> ## Documentation Index
> Fetch the complete documentation index at: https://docs.blockops.network/llms.txt
> Use this file to discover all available pages before exploring further.

# Operate a cluster

> Health, logs, backups, upgrades, adding nodes and recovering a lost node in a self-hosted citadel-core cluster.

## Health

Each node can serve `GET /health` (enable `healthcheck` in the configuration). It returns `200` when the node has connected to NATS and Consul and finished exchanging keys with its peers, and `503` until then, so use it for readiness, not liveness. Consul's key-value store shows which nodes are registered (`mpc_peers/`) and which are ready (`mpc_ready/`).

A node's start-up sequence in the logs is: version banner → `Connected to badger kv store` → `Loaded peers from consul` → `[READY] Node is ready` → `Starting consumers`. Anything that stops short of `[READY]` is a configuration, identity or connectivity problem.

## Logs

Logs go to standard output (containers) or the systemd journal (`journalctl -u citadel-core`). Search for `keygen` and `signing` with `success` to confirm operations, and for `error` to find failures. `--debug` at start-up adds protocol-level detail; do not leave it on in production.

## Backups

With `backup_enabled: true` (the default) each node writes an encrypted backup of its share store to `backup_dir` every `backup_period_seconds`. The backups are encrypted with the same password as the store. Copy them off the host on a schedule, a CronJob to object storage on Kubernetes, AWS Backup on EFS for ECS, or your existing backup tooling for hosts.

Three things must be backed up together, because each is useless without the others:

1. the share-store backups;
2. the node's identity files (public and private);
3. the share-store password and identity password, held in your secrets manager.

<Warning>
  A lost share-store password means the node's shares are gone. The remaining nodes can still sign if they meet the threshold, and the lost node can be replaced by resharing, but that only works while enough nodes remain. Losing more than `n − t` nodes' shares or passwords is unrecoverable.
</Warning>

## Restore a node's share store

```bash theme={null}
citadel-core-cli recover --backup-dir <backups> --recovery-path <db path> [--force]
```

Stop the node, run `recover` with the backup directory and the store path from `db_path`, supply the password, and start the node again. It rejoins with the shares it had at the time of the backup.

## Upgrades

Upgrades are coordinated across the cluster: a signing session needs `t` nodes on compatible versions. Blockops publishes release notes with each version that say whether the protocol changed.

* **Kubernetes:** update the StatefulSet image; the rolling update restarts one pod at a time while the others keep signing.
* **ECS:** update each service's task definition in turn.
* **systemd:** replace the binaries and `systemctl restart citadel-core` one host at a time.

Restart nodes one at a time and wait for `[READY]` before moving on, so the cluster never drops below threshold. For a release that changes the protocol, schedule a maintenance window and follow the release's resharing instructions.

## Add a node or change the threshold

1. Add the new node to `peers.json` and distribute the file.
2. Generate the new node's identity; distribute its `*_identity.json` to every existing node and every node's identity file to it.
3. Register the updated peers in Consul (or start the new node with `--peers`).
4. Deploy the new node.
5. Run a **reshare** from citadel-api so every wallet's shares are regenerated for the new set of nodes. Resharing keeps every wallet's public key and addresses; only the shares change.
6. If the threshold changes, update `mpc_threshold` on every node and restart them one at a time.

The same procedure retires a node: remove it from the registry, reshare to the smaller set, then decommission it and destroy its share store.

## Recover a lost node

If a node's storage is lost but its identity and passwords are intact, redeploy it with an empty store and reshare from citadel-api; the other nodes regenerate its shares. On Kubernetes, deleting the pod (and, if corrupted, its persistent volume claim) makes the StatefulSet recreate it. If the identity is lost as well, treat it as a new node.

## Full cluster restart

Stop every node, then start them again; on Kubernetes scale the StatefulSet to `0` and back to `3`. Nodes re-establish their peer sessions at start-up. Signing requests that arrive while the cluster is below threshold fail and are reported to citadel-api as failed, nothing is queued silently.

## Threshold not met

`insufficient participants` in the logs means fewer than `t` nodes were ready when a request arrived. Check that every pod or service is running, that all nodes are registered in Consul, and that `mpc_threshold` matches the cluster size.