Cordy Gateway

Cordy Gateway is a self-hosted LLM gateway. It exposes a single OpenAI-compatible endpoint and routes each request to one of 100+ upstream providers, while adding authentication, per-key quotas, cost metering, and automatic failover on top. You run it on your own infrastructure: your API keys, your Postgres, your logs. Prompts pass through your Gateway deployment to the upstream provider you configured; no Cordy-operated service sits in that path.

Point any OpenAI SDK at the gateway by changing one line — the base_url — and keep using the chat completions, embeddings, and models APIs you already know.

What it is

• One endpoint, many providers. Clients call a single OpenAI-compatible URL. Behind it, the gateway routes through built-in integrations such as OpenAI, Anthropic, Azure OpenAI, Bedrock, Gemini, Groq, Together, and OpenRouter.
• Authenticated and metered. Every request carries a wnx_ API key. Keys have scopes, token quotas, rate limits, and concurrency caps.
• Cost-controlled. Per-model pricing, prepaid credit balances, and usage analytics let you cap and attribute spend.
• Self-hosted and single-tenant. Cordy Gateway is a privately deployed product. You operate one instance for your own organization; it is not a multi-tenant managed SaaS.

Who it's for

Cordy Gateway is for teams that want a single internal LLM endpoint they control:

• Platform and infrastructure teams consolidating many provider keys behind one auditable gateway.
• Product teams that need per-key quotas, cost attribution, and failover without building it themselves.
• Organizations with data-control requirements that need prompts and keys to stay on their own infrastructure.

Core capabilities

• OpenAI-compatible chat/completions, completions (legacy), embeddings, and models endpoints.
• 100+ provider integrations through LiteLLM. Direct OpenAI-compatible passthrough exists in the data plane, but this source snapshot does not connect the Admin's custom-endpoint field to its runtime input; see Providers.
• Per-key authentication with scopes, monthly token quotas, requests-per-minute rate limits, and per-key in-flight concurrency caps.
• Routing strategies (weighted, cheapest, fastest, region-aware) with automatic same-model failover across channels.
• Cost metering per provider and model, prepaid credit balances, and a self-service usage analytics endpoint.
• A /my/ member portal for sign-in, one-time key creation, balance/ledger review, recent usage, and password changes.
• Upstream credentials encrypted at rest (Fernet), append-only audit logging, and role-based access in the Admin control plane.
• Streaming (SSE), tool/function calling, and idempotent retries.

Honest limits

• Text only. Image and audio (multimodal) content is rejected today. Requests must be text.
• Not a managed SaaS. It is self-hosted and single-tenant by design.
• No compliance certifications. Cordy Gateway does not claim SOC 2, ISO 27001, HIPAA, FedRAMP, or any other certification. See Security for the honest trust model.

Version status

The highest source tag in the Gateway repository is v1.2.4; the repository has no published GitHub Release entries. These manuals are reconciled to current source commit d623472, which includes the v1.2.4 member/prepaid/Langfuse work and later source fixes. Obtain an approved distribution through your organization's delivery channel rather than assuming a public binary download exists.

Architecture at a glance

Cordy Gateway runs as two cooperating services over a shared PostgreSQL database and Redis:

                       ┌─────────────────────────────┐
   Admins ───────────► │  Control plane (Admin)       │  :8001
                       │  Django, owns schema +       │
                       │  migrations, providers,      │
                       │  channels, keys, pricing     │
                       └──────────────┬──────────────┘
                                      │ shared Postgres + Redis
                       ┌──────────────┴──────────────┐
   Clients (OpenAI ──► │  Data plane (Gateway)        │  :8002
   SDK) Bearer wnx_    │  Starlette + Granian,        │ ──► 100+ providers
                       │  raw psycopg, routing,       │     (LiteLLM / direct)
                       │  auth, metering, failover    │
                       └─────────────────────────────┘

• The control plane is a Django Admin where operators register providers, channels, models, pricing, users, and API keys. It owns the database schema.
• The data plane is the high-performance request path clients call. It reads configuration from the shared database and never bootstraps Django on the hot path.

See Concepts for the full request lifecycle and model-routing model.

A note on Cordy's client apps

Cordy also ships separate end-user client apps. Those are distinct products with their own documentation and can optionally be pointed at a Cordy Gateway deployment as their backend, but Cordy Gateway itself is a standalone, headless API — this documentation set covers only the gateway.

Next steps

• Quickstart — deploy the supported stack and make your first request.
• Admin console — configure providers, members, keys, pricing, and evidence.
• Member portal — use self-service keys, balance, and usage.
• Client integration — point your existing OpenAI SDK at the gateway.
• Concepts — how the control plane, data plane, and model routing fit together.