Install on Kubernetes

This guide deploys a three-node citadel-core cluster on Kubernetes. Key material is generated offline, stored as Kubernetes Secrets, and mounted read-only into distroless containers that run as a non-root user.

Prerequisites

A Kubernetes cluster (v1.24 or later) with kubectl access, at least three worker nodes, and a persistent-storage provisioner.
NATS with JetStream and Consul, either in the cluster (the official NATS and Consul Helm charts are the recommended route) or external and reachable from it.
On a secure workstation: citadel-core-cli, age, and kubectl.
The citadel-core container image from Blockops, available to your cluster’s registry.

Generate key material offline

Peer registry

citadel-core-cli generate-peers -n 3

Pod names in a StatefulSet are citadel-core-0, citadel-core-1, citadel-core-2. Edit peers.json so its node names match those pod names.

Event initiator

citadel-core-cli generate-initiator --encrypt
jq -r .public_key event_initiator.identity.json   # goes into config.yaml

The encrypted private key goes to citadel-api; keep its password in your secrets manager.

Node identities

for n in citadel-core-0 citadel-core-1 citadel-core-2; do
  citadel-core-cli generate-identity --node "$n" --peers ./peers.json \
    --output-dir ./identity/"$n" --encrypt
done

Record each identity password.

Chain code

openssl rand -hex 32

One value, shared by every node, set as chain_code in the ConfigMap.

Create the namespace, secrets and configuration

kubectl create namespace citadel-core

# Passwords generated straight into a Secret; they never touch disk
kubectl create secret generic citadel-core-secrets -n citadel-core \
  --from-literal=db-password.cred="$(< /dev/urandom tr -dc 'A-Za-z0-9!@#' | head -c 32)" \
  --from-literal=identity-password.cred="<the identity password from generate-identity>"

# Identity files for every node
kubectl create secret generic citadel-core-identity -n citadel-core \
  --from-file=identity/citadel-core-0/citadel-core-0_identity.json \
  --from-file=identity/citadel-core-0/citadel-core-0_private.key.age \
  --from-file=identity/citadel-core-1/citadel-core-1_identity.json \
  --from-file=identity/citadel-core-1/citadel-core-1_private.key.age \
  --from-file=identity/citadel-core-2/citadel-core-2_identity.json \
  --from-file=identity/citadel-core-2/citadel-core-2_private.key.age

kubectl create configmap citadel-core-peers -n citadel-core --from-file=peers.json

Back up the share-store password immediately. BadgerDB encrypts the store under it at creation and it cannot be changed afterwards. In production, source both passwords from External Secrets Operator or Sealed Secrets rather than --from-literal.

Shared configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: citadel-core-config
  namespace: citadel-core
data:
  config.yaml: |
    environment: production
    db_path: /app/data/db
    backup_dir: /app/data/backups
    consul:
      address: consul:8500
    nats:
      url: nats://nats:4222
    mpc_threshold: 2
    event_initiator_pubkey: "<initiator public key>"
    event_initiator_algorithm: "ed25519"
    chain_code: "<64-character hex>"
    backup_enabled: true
    backup_period_seconds: 300
    healthcheck:
      enabled: true
      address: "0.0.0.0:8080"

Register peers in Consul

kubectl port-forward -n citadel-core svc/consul 8500:8500 &
printf 'consul:\n  address: localhost:8500\n' > register-config.yaml
citadel-core-cli register-peers --peers ./peers.json --config ./register-config.yaml --environment production

Alternatively, pass --peers=/app/peers.json to the node at start-up (as the manifest below does) and each node syncs the registry into Consul itself.

Deploy the StatefulSet

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: citadel-core
  namespace: citadel-core
spec:
  replicas: 3
  selector:
    matchLabels: { app: citadel-core }
  template:
    metadata:
      labels: { app: citadel-core }
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        runAsGroup: 65532
        fsGroup: 65532
      containers:
        - name: citadel-core
          image: <CITADEL_CORE_IMAGE>
          env:
            - name: POD_NAME
              valueFrom: { fieldRef: { fieldPath: metadata.name } }
          args:
            - "start"
            - "--name=$(POD_NAME)"
            - "--config=/app/config.yaml"
            - "--peers=/app/peers.json"
            - "--password-file=/app/secrets/db-password.cred"
            - "--identity-password-file=/app/secrets/identity-password.cred"
            - "--decrypt-private-key"
          volumeMounts:
            - { name: config,   mountPath: /app/config.yaml, subPath: config.yaml, readOnly: true }
            - { name: peers,    mountPath: /app/peers.json,  subPath: peers.json,  readOnly: true }
            - { name: identity, mountPath: /app/identity, readOnly: true }
            - { name: secrets,  mountPath: /app/secrets,  readOnly: true }
            - { name: data,     mountPath: /app/data }
            - { name: tmp,      mountPath: /tmp }
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
          livenessProbe:
            tcpSocket: { port: 8080 }
            initialDelaySeconds: 30
            periodSeconds: 10
          readinessProbe:
            httpGet: { path: /health, port: 8080 }
            initialDelaySeconds: 15
            periodSeconds: 5
      volumes:
        - { name: config,   configMap: { name: citadel-core-config } }
        - { name: peers,    configMap: { name: citadel-core-peers } }
        - { name: identity, secret: { secretName: citadel-core-identity, defaultMode: 0400 } }
        - { name: secrets,  secret: { secretName: citadel-core-secrets,  defaultMode: 0400 } }
        - { name: tmp,      emptyDir: {} }
  volumeClaimTemplates:
    - metadata: { name: data }
      spec:
        accessModes: ["ReadWriteOnce"]
        resources: { requests: { storage: 5Gi } }

Use /health only as the readiness probe. It returns 503 while a node is still exchanging keys with its peers at start-up; a liveness probe on it would kill nodes that are initialising. Use a TCP probe for liveness, as above.

Restrict network access

Nodes need to reach each other, NATS and Consul, and nothing else:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: citadel-core
  namespace: citadel-core
spec:
  podSelector:
    matchLabels: { app: citadel-core }
  policyTypes: [Ingress, Egress]
  ingress:
    - from: [{ podSelector: { matchLabels: { app: citadel-core } } }]
  egress:
    - to: [{ podSelector: { matchLabels: { app: nats } } }]
      ports: [{ protocol: TCP, port: 4222 }]
    - to: [{ podSelector: { matchLabels: { app: consul } } }]
      ports: [{ protocol: TCP, port: 8500 }]

Verify

kubectl get pods -n citadel-core
kubectl logs -n citadel-core citadel-core-0 --tail=50

Expect the version banner, Connected to badger kv store, Loaded peers from consul, [READY] Node is ready and Starting consumers. In Consul’s key-value store, mpc_peers/ lists the three nodes. Then run a key generation from citadel-api to confirm the cluster signs end to end. The image is distroless: there is no shell inside it. To inspect a running pod, attach an ephemeral debug container: kubectl debug -n citadel-core <pod> -it --image=busybox --target=citadel-core.

Operate a cluster covers backups, upgrades, adding nodes and recovering a lost pod; the security checklist covers what to lock down before production.

Introduction

Core concepts

Security & trust

Wallets & signing

Networks

Solutions

Self-hosted signing

Blockchain Infrastructure

Resources

Install on Kubernetes

Prerequisites

Generate key material offline

Create the namespace, secrets and configuration

Register peers in Consul

Deploy the StatefulSet

Restrict network access

Verify

Next

​Prerequisites

​Generate key material offline

​Create the namespace, secrets and configuration

​Register peers in Consul

​Deploy the StatefulSet

​Restrict network access

​Verify

​Next

Prerequisites

Generate key material offline

Create the namespace, secrets and configuration

Register peers in Consul

Deploy the StatefulSet

Restrict network access

Verify

Next