> ## Documentation Index
> Fetch the complete documentation index at: https://docs.blockops.network/llms.txt
> Use this file to discover all available pages before exploring further.

# Install on AWS ECS Fargate

> Run citadel-core signing nodes as Fargate tasks with Secrets Manager, S3 and EFS.

This guide deploys citadel-core on ECS Fargate: one service per node, secrets from AWS Secrets Manager, configuration and identity files from S3, and the encrypted share store on EFS. Infrastructure is expected to be provisioned with Terraform; this page covers what the application needs from it.

## How a task is put together

Each task runs four containers in sequence:

1. **init-secrets**: reads the share-store and identity passwords from Secrets Manager into a shared `/secrets` volume.
2. **init-config**: downloads `config.yaml`, `peers.json` and the node's identity files from S3.
3. **init-data**: copies the identity files and `peers.json` onto EFS so they survive restarts.
4. **citadel-core**: the node itself: distroless, non-root (UID 65532), read-only root filesystem, reading everything above from mounted volumes.

The main container has no shell, so every secret must already be a file by the time it starts.

## Generate key material

<Warning>
  Run these on a dedicated, ephemeral workstation (a temporary instance in a private subnet or an air-gapped machine) and wipe it after uploading the outputs. Never on a laptop or shared host.
</Warning>

```bash theme={null}
citadel-core-cli generate-peers -n 3
citadel-core-cli generate-initiator --encrypt        # private key → citadel-api; public key → config.yaml
for node in node0 node1 node2; do
  citadel-core-cli generate-identity --node "$node" --peers ./peers.json \
    --output-dir ./identity/"$node" --encrypt
done
openssl rand -hex 32                                  # chain_code, shared by every node
```

Generate a share-store password per node and record it with each node's identity password.

## AWS resources

| Resource            | Requirement                                                                                                                                                                                                                           |
| :------------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **ECR**             | A repository holding the citadel-core image.                                                                                                                                                                                          |
| **EFS**             | Encrypted at rest; mount targets in every task subnet; **one access point per node** rooted at `/node0`, `/node1`, `/node2`, owned by `65532:65532`. Enable AWS Backup on the file system.                                            |
| **Secrets Manager** | Per node: `citadel-core/<node>/db-password` and `citadel-core/<node>/identity-password`, stored as plain strings.                                                                                                                     |
| **S3**              | A bucket with public access blocked: `citadel-core/peers.json` plus `citadel-core/<node>/config.yaml` and `citadel-core/<node>/identity/` for each node. TLS certificates under `citadel-core/certs/` if NATS uses mTLS.              |
| **CloudWatch Logs** | Log group `/ecs/citadel-core`, retention set (30 days is a reasonable default).                                                                                                                                                       |
| **ECS**             | A Fargate cluster; **one service per node with `desired_count = 1`**, `maximum_percent = 100` and `minimum_healthy_percent = 0`, so two instances of a node never hold the share store's lock at once. Private subnets, no public IP. |
| **Security group**  | Egress only: NATS `4222`, Consul `8500`, EFS `2049`, and `443` to S3, Secrets Manager and ECR (through VPC endpoints or NAT).                                                                                                         |

### IAM

* **Execution role:** `AmazonECSTaskExecutionRolePolicy` plus `logs:CreateLogStream`, `logs:PutLogEvents` and `logs:CreateLogGroup` on the log group.
* **Task role:** `secretsmanager:GetSecretValue` on `secret:citadel-core/*`; `s3:GetObject` and `s3:ListBucket` on the bucket's `citadel-core/*` prefix; `elasticfilesystem:ClientMount`, `ClientWrite` and `ClientRootAccess` on the file system.

## Configuration for ECS

Leave the NATS and Consul addresses empty in `config.yaml` and inject them as the `NATS_URL` and `CONSUL_ADDRESS` environment variables on the citadel-core container; the same file then works in every environment.

```yaml theme={null}
environment: production
db_path: /app/data/db
backup_dir: /app/data/backups
consul:
  address: ""            # set by CONSUL_ADDRESS
nats:
  url: ""                # set by NATS_URL
  # username / password / tls block when NATS requires them
mpc_threshold: 2
event_initiator_pubkey: <initiator public key>
event_initiator_algorithm: ed25519
chain_code: <64-character hex>
max_concurrent_keygen: 3
max_concurrent_signing: 10
backup_enabled: true
backup_period_seconds: 300
healthcheck:
  enabled: true
  address: "0.0.0.0:8080"
```

## Task definition

The main container's command and mounts, with the init containers omitted for brevity:

```json theme={null}
{
  "name": "citadel-core",
  "image": "<ACCOUNT>.dkr.ecr.<REGION>.amazonaws.com/citadel-core:<TAG>",
  "essential": true,
  "command": [
    "start",
    "--name=<NODE_NAME>",
    "--config=/config/config.yaml",
    "--password-file=/app/secrets/db-password.cred",
    "--identity-password-file=/app/secrets/identity-password.cred",
    "--decrypt-private-key",
    "--peers=/config/peers.json"
  ],
  "environment": [
    { "name": "NATS_URL", "value": "<NATS_URL>" },
    { "name": "CONSUL_ADDRESS", "value": "<CONSUL_ADDRESS>" }
  ],
  "mountPoints": [
    { "sourceVolume": "secrets",  "containerPath": "/app/secrets",  "readOnly": true },
    { "sourceVolume": "config",   "containerPath": "/config",       "readOnly": true },
    { "sourceVolume": "identity", "containerPath": "/app/identity", "readOnly": true },
    { "sourceVolume": "data",     "containerPath": "/app/data" }
  ],
  "user": "65532:65532",
  "readonlyRootFilesystem": true,
  "linuxParameters": { "initProcessEnabled": true }
}
```

The `data` volume is the node's EFS access point with transit encryption and IAM authorisation enabled. `secrets`, `config` and `identity` are ephemeral task volumes written by the init containers. Passing `--peers` makes each node sync the registry into Consul at start-up, so no separate `register-peers` step is needed.

## Verify

CloudWatch should show, in order: the version banner, `Connected to badger kv store`, `Loaded peers from consul`, `[READY] Node is ready`, `Starting consumers`. `GET /health` on port 8080 reports readiness; add an ECS `healthCheck` on it if you want the service to track container health (allow a `startPeriod` of about 60 seconds).

## Troubleshooting

| Symptom                         | Likely cause                                                                                                                                                  |
| :------------------------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Init container exits non-zero   | Task role lacks Secrets Manager or S3 permission; secret name or S3 path mismatch; identity files not uploaded.                                               |
| Task stuck in `PROVISIONING`    | No EFS mount target in the subnet; security group blocks `2049`; image pull failure.                                                                          |
| Share-store error at start      | Password in Secrets Manager differs from the one the store was created with; it cannot be changed after creation. Or the access point UID/GID is not `65532`. |
| `Failed to decrypt private key` | Identity password mismatch, or the `.age` file was corrupted in upload.                                                                                       |
| Health check failing            | `healthcheck.enabled` is false, or `address` is `localhost` rather than `0.0.0.0:8080`.                                                                       |

## Next

[Operate a cluster](/self-hosted/operate) and the [security checklist](/self-hosted/security-checklist).