Skip to main content

​
Health

Each node can serve GET /health (enable healthcheck in the configuration). It returns 200 when the node has connected to NATS and Consul and finished exchanging keys with its peers, and 503 until then, so use it for readiness, not liveness. Consul’s key-value store shows which nodes are registered (mpc_peers/) and which are ready (mpc_ready/). A node’s start-up sequence in the logs is: version banner → Connected to badger kv store → Loaded peers from consul → [READY] Node is ready → Starting consumers. Anything that stops short of [READY] is a configuration, identity or connectivity problem.

​
Logs

Logs go to standard output (containers) or the systemd journal (journalctl -u citadel-core). Search for keygen and signing with success to confirm operations, and for error to find failures. --debug at start-up adds protocol-level detail; do not leave it on in production.

​
Backups

With backup_enabled: true (the default) each node writes an encrypted backup of its share store to backup_dir every backup_period_seconds. The backups are encrypted with the same password as the store. Copy them off the host on a schedule, a CronJob to object storage on Kubernetes, AWS Backup on EFS for ECS, or your existing backup tooling for hosts. Three things must be backed up together, because each is useless without the others:
  1. the share-store backups;
  2. the node’s identity files (public and private);
  3. the share-store password and identity password, held in your secrets manager.
A lost share-store password means the node’s shares are gone. The remaining nodes can still sign if they meet the threshold, and the lost node can be replaced by resharing, but that only works while enough nodes remain. Losing more than n − t nodes’ shares or passwords is unrecoverable.

​
Restore a node’s share store

Stop the node, run recover with the backup directory and the store path from db_path, supply the password, and start the node again. It rejoins with the shares it had at the time of the backup.

​
Upgrades

Upgrades are coordinated across the cluster: a signing session needs t nodes on compatible versions. Blockops publishes release notes with each version that say whether the protocol changed.
  • Kubernetes: update the StatefulSet image; the rolling update restarts one pod at a time while the others keep signing.
  • ECS: update each service’s task definition in turn.
  • systemd: replace the binaries and systemctl restart citadel-core one host at a time.
Restart nodes one at a time and wait for [READY] before moving on, so the cluster never drops below threshold. For a release that changes the protocol, schedule a maintenance window and follow the release’s resharing instructions.

​
Add a node or change the threshold

  1. Add the new node to peers.json and distribute the file.
  2. Generate the new node’s identity; distribute its *_identity.json to every existing node and every node’s identity file to it.
  3. Register the updated peers in Consul (or start the new node with --peers).
  4. Deploy the new node.
  5. Run a reshare from citadel-api so every wallet’s shares are regenerated for the new set of nodes. Resharing keeps every wallet’s public key and addresses; only the shares change.
  6. If the threshold changes, update mpc_threshold on every node and restart them one at a time.
The same procedure retires a node: remove it from the registry, reshare to the smaller set, then decommission it and destroy its share store.

​
Recover a lost node

If a node’s storage is lost but its identity and passwords are intact, redeploy it with an empty store and reshare from citadel-api; the other nodes regenerate its shares. On Kubernetes, deleting the pod (and, if corrupted, its persistent volume claim) makes the StatefulSet recreate it. If the identity is lost as well, treat it as a new node.

​
Full cluster restart

Stop every node, then start them again; on Kubernetes scale the StatefulSet to 0 and back to 3. Nodes re-establish their peer sessions at start-up. Signing requests that arrive while the cluster is below threshold fail and are reported to citadel-api as failed, nothing is queued silently.

​
Threshold not met

insufficient participants in the logs means fewer than t nodes were ready when a request arrived. Check that every pod or service is running, that all nodes are registered in Consul, and that mpc_threshold matches the cluster size.