Local & on-device AI

Cordy can run prompts without a cloud AI provider. Chrome's built-in model and in-browser Gemma process prompts locally; a local-native endpoint receives prompts at the address you configure. Initial model setup may download weights from their hosting service, and a local-native server may require its own API key.

Local AI is genuinely useful for privacy-sensitive text, offline-ish workflows, and avoiding per-token costs. It's also experimental — quality and stability vary with your hardware and browser.

Enable Local AI

Local AI is off by default. Turn on the master switch (Settings → Local AI → Enable Local AI) and accept the confirmation dialog, titled "Local AI (Experimental)." It asks you to acknowledge the honest trade-offs:

• it can be unstable and may cause high memory use, slowdowns, or crashes;
• WebGPU support varies by device and browser, so some models may not load;
• models can require significant storage and are cached locally;
• results aren't guaranteed and may be lower quality than cloud models.

Until this switch is on, local models won't actually run, even if you've configured a runtime.

Check your hardware

The Local AI tab includes a Hardware Assessment card that checks for WebGPU — required for the in-browser Gemma models and Kokoro text-to-speech — and shows model compatibility. If WebGPU is unavailable, the in-browser models can't run; Chrome built-in AI and local-native runtimes may still be options.

The three kinds of local AI

1. Chrome built-in AI (Gemini Nano)

Chrome exposes its on-device model through the Prompt API. Availability and model download are managed by Chrome, but you must enable Chrome's AI flags first:

1. Open chrome://flags/#prompt-api-for-gemini-nano and set it to Enabled.
2. Open chrome://flags/#optimization-guide-on-device-model and set it to Enabled BypassPerfRequirement.
3. Relaunch Chrome.
4. In Settings → Local AI, use Initialize Model to download and warm up Gemini Nano.

Gemini Nano has a small context window (about 4,096 tokens) and is best for simple Q&A, short summaries, and translation — not complex reasoning or code. It does not support tool calling, so it won't use Cordy's browser tools.

2. In-browser models (WebGPU): Gemma

Cordy can download neural models that run in your browser on WebGPU, managed for you:

Model	Size	Requirement	Best for
Gemma 4 E2B (2B)	~3.1 GB	WebGPU	Everyday chat, summaries, quick Q&A
Gemma 4 E4B (4B)	~4.1 GB	WebGPU + ≥4 GB VRAM	Complex reasoning, longer analysis, code

Download a model from Settings → Local AI → Local LLM Models; it is cached locally for reuse. Both require WebGPU. Their catalog context limit is 128K, subject to available browser memory and GPU resources.

3. Local-native runtimes (Ollama, LM Studio & more)

If you already run a local model server, connect Cordy to it over loopback (localhost). Cordy talks to it using the OpenAI-compatible API, so the model runs in your server, not the browser.

Add one from Settings → Local AI → Local-Native Runtimes → Add Runtime. Templates are provided for:

Runtime	Default address
Ollama	http://localhost:11434
LM Studio	http://localhost:1234/v1
llama.cpp (server)	http://localhost:8080/v1
vLLM	http://localhost:8000/v1
LocalAI	http://localhost:8080/v1
Unsloth	http://localhost:8888/v1
OpenAI-compatible	(you fill in the URL)

Set the Base URL, add an API key if your runtime requires one, and click Detect. On first contact with localhost or 127.0.0.1, approve the matching optional Chrome origin permission. Cordy permits an unauthenticated connection only for loopback; authenticated local servers can store a key in the same encrypted key store used by cloud providers.

Still maturing. Local-native runtime support is newer and scoped to OpenAI-compatible model discovery (/v1/models). Cordy doesn't auto-detect or install the runtime software for you, and end-to-end reliability is still improving. Make sure your runtime is running and reachable before selecting it.

Using local models

Once Local AI is on and a model or runtime is ready, choose it under Settings → AI Config → Feature Models. Browser-local models can serve supported chat and text-action slots. Local-native runtimes can serve supported text slots, but the translation slot deliberately excludes local-native endpoints; its Local choice uses the browser-local translation engine. TTS has its own local engine settings.

Honest limits

• Experimental. Expect rougher edges than cloud models.
• No attachments. Local models don't accept image/PDF/file attachments — Cordy will ask you to switch to a cloud model.
• Tools depend on the path. Browser-local generation does not receive browser tools. A selected local-native chat model can use the normal Cordy tool set and follows the same approval rules as cloud chat.
• Hardware-bound. WebGPU models need a capable GPU; large models need VRAM and disk space.
• Quality varies. For hard reasoning or code, a cloud model is usually stronger.

Privacy

Prompts to Gemini Nano and in-browser Gemma stay in the browser. Prompts to a loopback runtime go to that local process. Model downloads can contact public model hosting, but do not include your prompt. The one-time cloud consent dialog does not appear for local routes.

Troubleshooting

• "WebGPU is required… but not available." Your browser/device doesn't expose WebGPU. Update Chrome and graphics drivers, or use Chrome built-in AI or a local-native runtime instead.
• Gemini Nano won't initialize. Re-check both Chrome flags and relaunch; the download can take a while on first use.
• A model download stalls. Cancel and retry; ensure you have enough free disk space (several GB for Gemma).
• Runtime shows "Not running" / "Wrong API." Start your local server, confirm the Base URL and port, and that it exposes an OpenAI-compatible /v1 API.
• A local model refuses attachments. Expected — switch that message to a cloud model.

Related

• Providers & models · Read aloud (TTS) · Translation workbench · Privacy & permissions