> ## Documentation Index
> Fetch the complete documentation index at: https://docs.blockops.network/llms.txt
> Use this file to discover all available pages before exploring further.
# Install on Kubernetes
> Deploy citadel-core signing nodes as a StatefulSet with secrets, persistent storage, probes and network policy.
This guide deploys a three-node citadel-core cluster on Kubernetes. Key material is generated offline, stored as Kubernetes Secrets, and mounted read-only into distroless containers that run as a non-root user.
## Prerequisites
* A Kubernetes cluster (v1.24 or later) with `kubectl` access, at least three worker nodes, and a persistent-storage provisioner.
* **NATS** with JetStream and **Consul**, either in the cluster (the official NATS and Consul Helm charts are the recommended route) or external and reachable from it.
* On a secure workstation: `citadel-core-cli`, `age`, and `kubectl`.
* The citadel-core container image from Blockops, available to your cluster's registry.
## Generate key material offline
```bash theme={null}
citadel-core-cli generate-peers -n 3
```
Pod names in a StatefulSet are `citadel-core-0`, `citadel-core-1`, `citadel-core-2`. Edit `peers.json` so its node names match those pod names.
```bash theme={null}
citadel-core-cli generate-initiator --encrypt
jq -r .public_key event_initiator.identity.json # goes into config.yaml
```
The encrypted private key goes to citadel-api; keep its password in your secrets manager.
```bash theme={null}
for n in citadel-core-0 citadel-core-1 citadel-core-2; do
citadel-core-cli generate-identity --node "$n" --peers ./peers.json \
--output-dir ./identity/"$n" --encrypt
done
```
Record each identity password.
```bash theme={null}
openssl rand -hex 32
```
One value, shared by every node, set as `chain_code` in the ConfigMap.
## Create the namespace, secrets and configuration
```bash theme={null}
kubectl create namespace citadel-core
# Passwords generated straight into a Secret; they never touch disk
kubectl create secret generic citadel-core-secrets -n citadel-core \
--from-literal=db-password.cred="$(< /dev/urandom tr -dc 'A-Za-z0-9!@#' | head -c 32)" \
--from-literal=identity-password.cred=""
# Identity files for every node
kubectl create secret generic citadel-core-identity -n citadel-core \
--from-file=identity/citadel-core-0/citadel-core-0_identity.json \
--from-file=identity/citadel-core-0/citadel-core-0_private.key.age \
--from-file=identity/citadel-core-1/citadel-core-1_identity.json \
--from-file=identity/citadel-core-1/citadel-core-1_private.key.age \
--from-file=identity/citadel-core-2/citadel-core-2_identity.json \
--from-file=identity/citadel-core-2/citadel-core-2_private.key.age
kubectl create configmap citadel-core-peers -n citadel-core --from-file=peers.json
```
Back up the share-store password immediately. BadgerDB encrypts the store under it at creation and it cannot be changed afterwards. In production, source both passwords from External Secrets Operator or Sealed Secrets rather than `--from-literal`.
Shared configuration:
```yaml theme={null}
apiVersion: v1
kind: ConfigMap
metadata:
name: citadel-core-config
namespace: citadel-core
data:
config.yaml: |
environment: production
db_path: /app/data/db
backup_dir: /app/data/backups
consul:
address: consul:8500
nats:
url: nats://nats:4222
mpc_threshold: 2
event_initiator_pubkey: ""
event_initiator_algorithm: "ed25519"
chain_code: "<64-character hex>"
backup_enabled: true
backup_period_seconds: 300
healthcheck:
enabled: true
address: "0.0.0.0:8080"
```
## Register peers in Consul
```bash theme={null}
kubectl port-forward -n citadel-core svc/consul 8500:8500 &
printf 'consul:\n address: localhost:8500\n' > register-config.yaml
citadel-core-cli register-peers --peers ./peers.json --config ./register-config.yaml --environment production
```
Alternatively, pass `--peers=/app/peers.json` to the node at start-up (as the manifest below does) and each node syncs the registry into Consul itself.
## Deploy the StatefulSet
```yaml theme={null}
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: citadel-core
namespace: citadel-core
spec:
replicas: 3
selector:
matchLabels: { app: citadel-core }
template:
metadata:
labels: { app: citadel-core }
spec:
securityContext:
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
fsGroup: 65532
containers:
- name: citadel-core
image:
env:
- name: POD_NAME
valueFrom: { fieldRef: { fieldPath: metadata.name } }
args:
- "start"
- "--name=$(POD_NAME)"
- "--config=/app/config.yaml"
- "--peers=/app/peers.json"
- "--password-file=/app/secrets/db-password.cred"
- "--identity-password-file=/app/secrets/identity-password.cred"
- "--decrypt-private-key"
volumeMounts:
- { name: config, mountPath: /app/config.yaml, subPath: config.yaml, readOnly: true }
- { name: peers, mountPath: /app/peers.json, subPath: peers.json, readOnly: true }
- { name: identity, mountPath: /app/identity, readOnly: true }
- { name: secrets, mountPath: /app/secrets, readOnly: true }
- { name: data, mountPath: /app/data }
- { name: tmp, mountPath: /tmp }
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
livenessProbe:
tcpSocket: { port: 8080 }
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet: { path: /health, port: 8080 }
initialDelaySeconds: 15
periodSeconds: 5
volumes:
- { name: config, configMap: { name: citadel-core-config } }
- { name: peers, configMap: { name: citadel-core-peers } }
- { name: identity, secret: { secretName: citadel-core-identity, defaultMode: 0400 } }
- { name: secrets, secret: { secretName: citadel-core-secrets, defaultMode: 0400 } }
- { name: tmp, emptyDir: {} }
volumeClaimTemplates:
- metadata: { name: data }
spec:
accessModes: ["ReadWriteOnce"]
resources: { requests: { storage: 5Gi } }
```
Use `/health` only as the **readiness** probe. It returns `503` while a node is still exchanging keys with its peers at start-up; a liveness probe on it would kill nodes that are initialising. Use a TCP probe for liveness, as above.
## Restrict network access
Nodes need to reach each other, NATS and Consul, and nothing else:
```yaml theme={null}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: citadel-core
namespace: citadel-core
spec:
podSelector:
matchLabels: { app: citadel-core }
policyTypes: [Ingress, Egress]
ingress:
- from: [{ podSelector: { matchLabels: { app: citadel-core } } }]
egress:
- to: [{ podSelector: { matchLabels: { app: nats } } }]
ports: [{ protocol: TCP, port: 4222 }]
- to: [{ podSelector: { matchLabels: { app: consul } } }]
ports: [{ protocol: TCP, port: 8500 }]
```
## Verify
```bash theme={null}
kubectl get pods -n citadel-core
kubectl logs -n citadel-core citadel-core-0 --tail=50
```
Expect the version banner, `Connected to badger kv store`, `Loaded peers from consul`, `[READY] Node is ready` and `Starting consumers`. In Consul's key-value store, `mpc_peers/` lists the three nodes. Then run a key generation from citadel-api to confirm the cluster signs end to end.
The image is distroless: there is no shell inside it. To inspect a running pod, attach an ephemeral debug container: `kubectl debug -n citadel-core -it --image=busybox --target=citadel-core`.
## Next
[Operate a cluster](/self-hosted/operate) covers backups, upgrades, adding nodes and recovering a lost pod; the [security checklist](/self-hosted/security-checklist) covers what to lock down before production.