Install on AWS ECS Fargate

This guide deploys citadel-core on ECS Fargate: one service per node, secrets from AWS Secrets Manager, configuration and identity files from S3, and the encrypted share store on EFS. Infrastructure is expected to be provisioned with Terraform; this page covers what the application needs from it.

How a task is put together

Each task runs four containers in sequence:

init-secrets: reads the share-store and identity passwords from Secrets Manager into a shared /secrets volume.
init-config: downloads config.yaml, peers.json and the node’s identity files from S3.
init-data: copies the identity files and peers.json onto EFS so they survive restarts.
citadel-core: the node itself: distroless, non-root (UID 65532), read-only root filesystem, reading everything above from mounted volumes.

The main container has no shell, so every secret must already be a file by the time it starts.

Generate key material

Run these on a dedicated, ephemeral workstation (a temporary instance in a private subnet or an air-gapped machine) and wipe it after uploading the outputs. Never on a laptop or shared host.

citadel-core-cli generate-peers -n 3
citadel-core-cli generate-initiator --encrypt        # private key → citadel-api; public key → config.yaml
for node in node0 node1 node2; do
  citadel-core-cli generate-identity --node "$node" --peers ./peers.json \
    --output-dir ./identity/"$node" --encrypt
done
openssl rand -hex 32                                  # chain_code, shared by every node

Generate a share-store password per node and record it with each node’s identity password.

AWS resources

Resource	Requirement
ECR	A repository holding the citadel-core image.
EFS	Encrypted at rest; mount targets in every task subnet; one access point per node rooted at `/node0`, `/node1`, `/node2`, owned by `65532:65532`. Enable AWS Backup on the file system.
Secrets Manager	Per node: `citadel-core/<node>/db-password` and `citadel-core/<node>/identity-password`, stored as plain strings.
S3	A bucket with public access blocked: `citadel-core/peers.json` plus `citadel-core/<node>/config.yaml` and `citadel-core/<node>/identity/` for each node. TLS certificates under `citadel-core/certs/` if NATS uses mTLS.
CloudWatch Logs	Log group `/ecs/citadel-core`, retention set (30 days is a reasonable default).
ECS	A Fargate cluster; one service per node with `desired_count = 1`, `maximum_percent = 100` and `minimum_healthy_percent = 0`, so two instances of a node never hold the share store’s lock at once. Private subnets, no public IP.
Security group	Egress only: NATS `4222`, Consul `8500`, EFS `2049`, and `443` to S3, Secrets Manager and ECR (through VPC endpoints or NAT).

IAM

Execution role: AmazonECSTaskExecutionRolePolicy plus logs:CreateLogStream, logs:PutLogEvents and logs:CreateLogGroup on the log group.
Task role: secretsmanager:GetSecretValue on secret:citadel-core/*; s3:GetObject and s3:ListBucket on the bucket’s citadel-core/* prefix; elasticfilesystem:ClientMount, ClientWrite and ClientRootAccess on the file system.

Configuration for ECS

Leave the NATS and Consul addresses empty in config.yaml and inject them as the NATS_URL and CONSUL_ADDRESS environment variables on the citadel-core container; the same file then works in every environment.

environment: production
db_path: /app/data/db
backup_dir: /app/data/backups
consul:
  address: ""            # set by CONSUL_ADDRESS
nats:
  url: ""                # set by NATS_URL
  # username / password / tls block when NATS requires them
mpc_threshold: 2
event_initiator_pubkey: <initiator public key>
event_initiator_algorithm: ed25519
chain_code: <64-character hex>
max_concurrent_keygen: 3
max_concurrent_signing: 10
backup_enabled: true
backup_period_seconds: 300
healthcheck:
  enabled: true
  address: "0.0.0.0:8080"

Task definition

The main container’s command and mounts, with the init containers omitted for brevity:

{
  "name": "citadel-core",
  "image": "<ACCOUNT>.dkr.ecr.<REGION>.amazonaws.com/citadel-core:<TAG>",
  "essential": true,
  "command": [
    "start",
    "--name=<NODE_NAME>",
    "--config=/config/config.yaml",
    "--password-file=/app/secrets/db-password.cred",
    "--identity-password-file=/app/secrets/identity-password.cred",
    "--decrypt-private-key",
    "--peers=/config/peers.json"
  ],
  "environment": [
    { "name": "NATS_URL", "value": "<NATS_URL>" },
    { "name": "CONSUL_ADDRESS", "value": "<CONSUL_ADDRESS>" }
  ],
  "mountPoints": [
    { "sourceVolume": "secrets",  "containerPath": "/app/secrets",  "readOnly": true },
    { "sourceVolume": "config",   "containerPath": "/config",       "readOnly": true },
    { "sourceVolume": "identity", "containerPath": "/app/identity", "readOnly": true },
    { "sourceVolume": "data",     "containerPath": "/app/data" }
  ],
  "user": "65532:65532",
  "readonlyRootFilesystem": true,
  "linuxParameters": { "initProcessEnabled": true }
}

The data volume is the node’s EFS access point with transit encryption and IAM authorisation enabled. secrets, config and identity are ephemeral task volumes written by the init containers. Passing --peers makes each node sync the registry into Consul at start-up, so no separate register-peers step is needed.

Verify

CloudWatch should show, in order: the version banner, Connected to badger kv store, Loaded peers from consul, [READY] Node is ready, Starting consumers. GET /health on port 8080 reports readiness; add an ECS healthCheck on it if you want the service to track container health (allow a startPeriod of about 60 seconds).

Troubleshooting

Symptom	Likely cause
Init container exits non-zero	Task role lacks Secrets Manager or S3 permission; secret name or S3 path mismatch; identity files not uploaded.
Task stuck in `PROVISIONING`	No EFS mount target in the subnet; security group blocks `2049`; image pull failure.
Share-store error at start	Password in Secrets Manager differs from the one the store was created with; it cannot be changed after creation. Or the access point UID/GID is not `65532`.
`Failed to decrypt private key`	Identity password mismatch, or the `.age` file was corrupted in upload.
Health check failing	`healthcheck.enabled` is false, or `address` is `localhost` rather than `0.0.0.0:8080`.

Operate a cluster and the security checklist.

Introduction

Core concepts

Security & trust

Wallets & signing

Networks

Solutions

Self-hosted signing

Blockchain Infrastructure

Resources

Install on AWS ECS Fargate

How a task is put together

Generate key material

AWS resources

IAM

Configuration for ECS

Task definition

Verify

Troubleshooting

Next

​How a task is put together

​Generate key material

​AWS resources

​IAM

​Configuration for ECS

​Task definition

​Verify

​Troubleshooting

​Next

How a task is put together

Generate key material

AWS resources

IAM

Configuration for ECS

Task definition

Verify

Troubleshooting

Next