Operations & maintenance

Use this runbook after the production Compose stack is running. Run commands from the Cordy Gateway distribution directory. Every Compose command retains docker-compose.production-overrides.yml; omitting it can silently restore development-mode security behavior and disable the credit WAL on recreated containers. Compose uses --env-file .env for interpolation; the make scripts do not, so export the same DB_*, REDIS_URL, backup, and health-check values in the shell before running a make target.

Check the stack

Start with container state and recent logs:

docker compose -f docker-compose.prod.yml -f docker-compose.production-overrides.yml --env-file .env ps
docker compose -f docker-compose.prod.yml -f docker-compose.production-overrides.yml --env-file .env logs --tail=200 admin gateway worker nginx

Then check the public data-plane endpoints:

curl -sS http://<host>:8002/health
curl -sS http://<host>:8002/health/ready
curl -sS http://<host>:8002/health/providers
curl -sS http://<host>:8002/health/detailed

/health proves that the process is alive. Readiness and detailed health return 503 until at least one channel can serve traffic. /health/providers identifies the affected provider or channel; /health/detailed also reports database, Redis, capabilities, and license posture.

For a host-side check of PostgreSQL, Redis, detailed health, backup freshness, and writable paths, run:

make health-check

Set REQUIRE_PITR=1 in a production acceptance environment only after WAL archiving is configured; this changes missing PITR from a warning to a failure.

Investigate one request

1. Copy the response's request_id from an error body, or the successful response headers such as X-Channel-Id, X-Provider, and X-Failover-Count.
2. Open Admin → Monitoring → Request logs and search by request ID, user email, API-key name, or channel.
3. Check status, latency, token counts, selected channel, cost evidence, and trace links. Request logs are read-only evidence.
4. Use Usage stats for daily aggregates and the member portal for that member's recent 50 requests.
5. If tracing is enabled, follow the Tempo or Langfuse trace link. Langfuse links require ADMIN_LANGFUSE_TRACE_URL_TEMPLATE in the production Compose override.

Distinguish the two account limits before changing anything: 402 insufficient_credits means prepaid credit is empty; 429 quota_exceeded means that API key's token quota is exhausted. Rate-limit responses are a separate 429 rate_limit_exceeded and include Retry-After.

Monitor metrics and traces

For a Gateway behind public nginx, set GATEWAY_METRICS_TOKEN in .env, leave GATEWAY_METRICS_ALLOWED_CIDRS empty, and apply the production override:

docker compose \
  -f docker-compose.prod.yml \
  -f docker-compose.production-overrides.yml \
  --env-file .env up -d gateway nginx

Verify the token path before configuring your existing Prometheus-compatible scraper:

curl -fsS https://gateway.example.com/metrics/prometheus \
  -H "Authorization: Bearer $GATEWAY_METRICS_TOKEN"

The CIDR bypass checks the immediate socket peer before checking a token. Behind public nginx, allowing nginx or its network would admit all requests it proxies. Reserve CIDR access for a direct, isolated trusted scrape path. The production scraper example is in Deployment.

Watch request/error volume, tokens and USD cost, failover success, routing overhead, and provider circuit-breaker state.

For traces, configure OTEL_TRACING_ENABLED and all required OTEL_* values for your reachable operator-managed collector as shown in Configuration, then recreate the Gateway with the same production files:

docker compose \
  -f docker-compose.prod.yml \
  -f docker-compose.production-overrides.yml \
  --env-file .env up -d gateway

The bundled monitoring, tracing, and observability Compose files are development/reference profiles whose networks do not reach the production Gateway; the observability profile also carries CHANGEME defaults. Do not add them to production commands without a separately engineered and validated deployment.

Protect credit settlement evidence

Confirm the mounted WAL directory is present and writable after every image or UID change:

docker compose -f docker-compose.prod.yml -f docker-compose.production-overrides.yml --env-file .env \
  exec gateway sh -c 'test -d /var/lib/cordy && test -w /var/lib/cordy'

When a Redis error prevents a credit-consume buffer write, the request still succeeds and the Gateway appends {user_id, cost_usd, ts} JSON lines to /var/lib/cordy/credit-consume.jsonl. If this append also fails, logs report a counted consume_drop_total under-charge. Keep alerts on these conditions and preserve the named gateway_wal volume during container replacement.

There is no automated replay command. Before any adjustment, take a copy of the WAL as immutable evidence, match its user, cost, and timestamp records against Request logs, the credit ledger, and the corresponding upstream invoice, and document which rows were applied. Apply a reviewed manual credit adjustment only once; retaining this audit trail prevents a later duplicate charge.

Back up and test recovery

Install PostgreSQL client tools on the host running the commands, set the live DB_* variables, and choose a durable backup directory:

BACKUP_DIR=/var/backups/cordy BACKUP_RETENTION=7 make backup

The command writes a restorable SQL dump named for the database and UTC timestamp, prints its path, and retains the newest configured count. Copy backups off the application host according to your recovery policy.

Regularly prove that a dump restores into a throwaway database:

make dr-drill

The drill excludes pg_cron, restores application data into DR_DB_NAME (default cordy_link_drtest), checks VERIFY_TABLE (default django_migrations), reports restore time, and drops the throwaway database unless KEEP_DR_DB=1. It does not prove point-in-time recovery or high availability; test those separately when your deployment requires them.

Upgrade and roll back

Before an upgrade, record the current code reference and confirm the backup destination. Preview without changes:

make preflight
DRY_RUN=1 make upgrade

After deploying the intended source or image, run the real workflow:

BACKUP_DIR=/var/backups/cordy make upgrade
make health-check

The upgrade runs preflight, creates a database backup, applies Admin-owned migrations, and verifies health. Keep the printed backup and manifest paths.

If validation fails and the release procedure calls for rollback, redeploy the prior code/image and restore the matching database dump:

make upgrade-rollback BACKUP=/var/backups/cordy/cordy-<timestamp>.sql PREV_REF=<previous-ref>

Database restore is destructive. The command restores the dump and prints the code-ref step; it does not run git checkout for you.

Scale the data plane

Scale only the gateway replicas behind nginx:

docker compose \
  -f docker-compose.prod.yml \
  -f docker-compose.production-overrides.yml \
  --env-file .env up -d --scale gateway=4

All replicas share PostgreSQL and Redis, so rate limits, quota reservations, idempotency, and cooldown state remain cluster-wide. Recheck readiness, errors, latency, database pooling, and upstream limits after each change.

Common failures

Symptom	Check	Corrective action
Every key returns 401	Gateway DB_NAME, DB_USER, DB_PASSWORD; gateway logs	Match the Admin database values. The data plane does not use DATABASE_URL.
404 model_not_found	Public model, provider model, channel, and active binding	Complete or activate the five-record model chain.
503 readiness / no channel	/health/providers, channel status, Test connection	Repair the upstream credential/endpoint or enable a healthy binding.
402 insufficient_credits	Member balance and credit ledger	Top up through the Admin user action; do not edit the balance field.
429 quota_exceeded	API-key quota limit and used value	Increase/reset the key's quota according to policy or wait for the quota cycle.
403 scope_denied	Error's required_scope and the key's scopes	Add only the required scope.
Metrics return 403	Scraper bearer token	Match the scraper bearer credential to GATEWAY_METRICS_TOKEN. Do not allowlist a shared public nginx peer.
CORS unexpectedly allows every origin	Effective ENVIRONMENT and applied Compose files	Recreate with docker-compose.production-overrides.yml; the bare production file leaves the Gateway in development.
Redis consume writes fail	Gateway logs, gateway_wal, and directory permissions	Preserve and reconcile the JSONL evidence. If both Redis and WAL writes failed, investigate the counted under-charge; there is no automatic replay.
Costs are unknown	Effective Price history row at request time	Add a non-overlapping effective price window; do not edit old request evidence.

See Admin console for configuration and cost reconciliation, and API reference for the full error contract.