Concepts

How Cordy Gateway is built — control plane vs data plane, the model-resolution chain, channels and routing, and the request lifecycle.

This page explains the moving parts behind the single endpoint your clients call. Understanding them makes it clear where configuration lives, how a model name turns into a real upstream call, and what happens to a request from authentication to billing.

Two planes

Cordy Gateway is split into a control plane and a data plane that share one PostgreSQL database and one Redis instance.

Control plane — Admin

What it is: a Django application (with the django-unfold Admin UI).
Default port: 8001.
Owns: the database schema and all migrations. Every table — providers, channels, models, pricing, users, API keys, audit log — is defined and evolved here.
Used for: registering providers and channels, defining public models and pricing, managing users, issuing and revoking API keys, and reviewing usage and audit records.

Operators interact with the control plane. Clients never do.

Data plane — Gateway

What it is: a Starlette application served by the Granian ASGI server. It is the OpenAI-compatible API your clients call.
Default port: 8002.
Data access: raw psycopg against PostgreSQL through a connection pool, with no Django on the request path. This keeps the hot path lean and fast.
Used for: authentication, scope and quota enforcement, rate and concurrency limiting, model routing, failover, calling upstream providers, and settling usage.

The data plane treats the schema as read-mostly configuration. It reads providers, channels, models, keys, and pricing that the control plane wrote.

Shared state

PostgreSQL holds the durable configuration and records. The schema is owned by Admin migrations; the Gateway reads it.
Redis holds fast, ephemeral, cluster-shared state: rate-limit counters, quota reservations, response cache, idempotency keys, channel cooldown state, and the write buffers that batch usage back to Postgres. Money-related keys (accrued charges, usage buffers) are never evicted.

Because Redis is shared, running multiple gateway replicas is safe: rate limits, quotas, cooldown, and idempotency are all cluster-global. See Deployment for scaling.

The model-resolution chain

When a client sends "model": "gpt-4o-mini", that name is a public model slug — a stable identifier you choose. The gateway resolves it to a concrete upstream call through a chain of records:

PublicModel  ──<  ProviderModel  ──<  ChannelProviderModel  >──  Channel  >──  Provider
 (public slug)     (vendor model      (enables a model on       (routing      (upstream
                    name + pricing)    a channel)                target +       vendor)
                                                                 credentials)

PublicModel — the public identity clients request (for example gpt-4o-mini). Clients only ever see these slugs; they never see vendor model names or upstream keys.
ProviderModel — maps a PublicModel to a specific Provider's real model name (for example the vendor string gpt-4o-mini), and carries pricing (input_price, output_price) and capability flags (supports_chat, supports_embeddings, supports_streaming, context_window).
Provider — an upstream vendor integration (OpenAI, Anthropic, Azure, Bedrock, and so on), with a provider_type and status.
Channel — a concrete routing target on a Provider: an upstream credential (stored Fernet-encrypted), an optional base_url, and routing attributes such as weight, priority, region, rate_limit, concurrent_limit, timeout_seconds, and an optional cost_weight.
ChannelProviderModel — the binding that says "this channel can serve this provider model." A channel routes to exactly one provider, and the bound provider model must belong to that same provider.

One public model can be served by many channels — across the same provider or across different providers — which is what makes weighted balancing and failover possible.

Channels and routing

A channel is one way to reach a model. Because a single public model can map to several channels, the gateway must pick one per request, and fall back if it fails.

Selecting a channel

The gateway filters to the healthy, in-tier, active channels that can serve the requested model, then selects one according to a routing strategy:

Strategy	Behaviour
`weighted`	Priority-ordered, then weighted-random within the top priority band. The default.
`cheapest`	Lowest effective price first, using each channel's `cost_weight` multiplier over the provider-model price.
`fastest`	Lowest observed P95 latency first, from the in-process latency recorder; falls back to weighted when there is no data yet.
`region_aware`	Prefers channels whose region matches the caller's region, then weighted within the best bucket.

The default strategy is set by GATEWAY_DEFAULT_ROUTING_STRATEGY (default weighted). A client can override it per request with the X-Routing-Strategy header. See Providers and Client integration.

Failover, cooldown, and circuit breakers

Failover: if the selected channel fails with a retryable error, the gateway tries the next eligible channel for the same model. The number of failovers is reported in the X-Failover-Count response header.
Same-channel retries: transient errors are retried on the same channel with exponential backoff before failover kicks in.
Cooldown: a channel that repeatedly errors (or is rate-limited upstream) can be put into a cooldown window, during which it is skipped. Cooldown state lives in Redis and is shared across replicas.
Circuit breakers: each worker tracks provider health and trips a breaker for a failing provider, shedding load quickly. Breaker state is exposed in the metrics.

Request lifecycle

Every request flows through an ordered middleware chain before reaching the route handler, then through routing and settlement. The stages, in order:

Body size guard — requests larger than GATEWAY_MAX_BODY_BYTES (default 10 MiB) are rejected with 413 before anything is buffered.
Authentication — the wnx_ key is extracted from Authorization: Bearer or X-API-Key, hashed with SHA-256, and looked up (Redis-cached, Postgres-backed). Bad, revoked, inactive, or expired keys return 401. If the auth infrastructure itself is unavailable, the gateway returns 503 with Retry-After (a transient signal, not a bad-credential signal).
Scope and quota — the path is mapped to a required scope (chat, completions, embeddings, models, balance, admin). A key lacking the scope returns 403 scope_denied. A key over its token quota returns 429 quota_exceeded. A quota reservation is placed in Redis to admit the request.
Model permission — per-user model access rules are enforced, so a key can be restricted to a subset of public models.
Rate limiting — per-key requests-per-minute limits are enforced in Redis; over-limit requests return 429.
In-flight limit — concurrent requests per key are capped at GATEWAY_MAX_INFLIGHT_PER_KEY (default 50); the slot is held for the whole streaming lifecycle and released on completion or client disconnect.
Guardrails — optional inbound PII pattern scanning (log-only by default).
Region derivation — the caller's region is derived (honouring an explicit X-Region header first) to feed region-aware routing.
Idempotency — a POST with an Idempotency-Key is replay-protected: a repeat within the TTL returns the stored response instead of calling upstream again.
Routing and failover — the gateway selects a channel for the model and calls the upstream provider (via LiteLLM or the direct OpenAI-compatible path), failing over across channels as needed.
Billing and usage settlement — token usage and cost are computed, the quota reservation is settled, prepaid credit is debited, and a usage record is written. Settlement is buffered through Redis and drained to Postgres by the background worker.

Successful responses carry observability headers such as X-Channel-Id, X-Provider, X-Failover-Count, and X-Routing-Overhead-Ms.

Where things live

Concern	Home
Providers, channels, models, pricing	Control plane (Admin), stored in Postgres
Users, API keys, scopes, quotas	Control plane (Admin), stored in Postgres
Audit log and RBAC	Control plane (Admin)
Request routing, auth enforcement, metering	Data plane (Gateway)
Rate limits, quotas, cache, cooldown, idempotency	Redis (shared)
Upstream credentials (encrypted at rest)	Channel records in Postgres

Next steps

Providers — configure providers, channels, and routing.
API reference — the endpoints, scopes, and errors this lifecycle produces.
Security — how keys, credentials, and audit records are protected.