Local & on-device AI

Cordy can run AI without any cloud provider. You can use Chrome's built-in model, download a model that runs in your browser on WebGPU, or connect to a local server you already run — and in every case, nothing leaves your device. There's no API key, no consent dialog, and no network request to a third party.

Local AI is genuinely useful for privacy-sensitive text, offline-ish workflows, and avoiding per-token costs. It's also experimental — quality and stability vary with your hardware and browser.

Enable Local AI

Local AI is off by default. Turn on the master switch (Settings → Local AI → Enable Local AI) and accept the confirmation dialog, titled "Local AI (Experimental)." It asks you to acknowledge the honest trade-offs:

• it can be unstable and may cause high memory use, slowdowns, or crashes;
• WebGPU support varies by device and browser, so some models may not load;
• models can require significant storage and are cached locally;
• results aren't guaranteed and may be lower quality than cloud models.

Until this switch is on, local models won't actually run, even if you've configured a runtime.

Check your hardware

The Local AI tab includes a Hardware Assessment card that checks for WebGPU — required for the in-browser Gemma models and Kokoro text-to-speech — and shows model compatibility. If WebGPU is unavailable, the in-browser models can't run; Chrome built-in AI and local-native runtimes may still be options.

The three kinds of local AI

1. Chrome built-in AI (Gemini Nano)

Chrome ships a small on-device model (branded Gemini Nano; the weights are Gemma-family) accessible through Chrome's Prompt API. It needs no download of its own, but you must enable Chrome's AI flags first:

1. Open chrome://flags/#prompt-api-for-gemini-nano and set it to Enabled.
2. Open chrome://flags/#optimization-guide-on-device-model and set it to Enabled BypassPerfRequirement.
3. Relaunch Chrome.
4. In Settings → Local AI, use Initialize Model to download and warm up Gemini Nano.

Gemini Nano has a small context window (about 4,096 tokens) and is best for simple Q&A, short summaries, and translation — not complex reasoning or code. It does not support tool calling, so it won't use Cordy's browser tools.

2. In-browser models (WebGPU): Gemma

Cordy can download neural models that run in your browser on WebGPU, managed for you:

Model	Size	Requirement	Best for
Gemma 4 E2B (2B)	~3.1 GB	WebGPU	Everyday chat, summaries, quick Q&A
Gemma 4 E4B (4B)	~4.1 GB	WebGPU + ≥4 GB VRAM	Complex reasoning, longer analysis, code

Download a model from Settings → Local AI → Local LLM Models; it's cached locally for reuse. Both require WebGPU — Cordy fails clearly if it isn't available rather than silently degrading. These have an 8,192-token context and are stronger than Gemini Nano, at the cost of a large download and real GPU work.

3. Local-native runtimes (Ollama, LM Studio & more)

If you already run a local model server, connect Cordy to it over loopback (localhost). Cordy talks to it using the OpenAI-compatible API, so the model runs in your server, not the browser.

Add one from Settings → Local AI → Local-Native Runtimes → Add Runtime. Templates are provided for:

Runtime	Default address
Ollama	http://localhost:11434
LM Studio	http://localhost:1234/v1
llama.cpp (server)	http://localhost:8080/v1
vLLM	http://localhost:8000/v1
LocalAI	http://localhost:8080/v1
OpenAI-compatible	(you fill in the URL)

Set the Base URL and click Detect to discover the runtime's models. Because these are loopback addresses (localhost, 127.0.0.1, ::1), no API key is required — Cordy shows "Loopback address — no API key required" instead of a key field.

Still maturing. Local-native runtime support is newer and scoped to OpenAI-compatible model discovery (/v1/models). Cordy doesn't auto-detect or install the runtime software for you, and end-to-end reliability is still improving. Make sure your runtime is running and reachable before selecting it.

Using local models

Once Local AI is on and a model or runtime is set up, route any feature to it under Settings → AI Config → Scenario Model Routing — for example, chat on Gemma, translation on a local runtime. Local models power the side-panel chat, in-page translate/explain/summarize, the translation workbench's Local runtime, and local text-to-speech.

Honest limits

• Experimental. Expect rougher edges than cloud models.
• No attachments. Local models don't accept image/PDF/file attachments — Cordy will ask you to switch to a cloud model.
• Fewer tools. The local chat path is restricted to read-only public tools, and Gemini Nano has no tools at all — see Tools.
• Hardware-bound. WebGPU models need a capable GPU; large models need VRAM and disk space.
• Quality varies. For hard reasoning or code, a cloud model is usually stronger.

Privacy

This is the most private way to use Cordy: prompts to Gemini Nano, in-browser Gemma, or a loopback runtime never leave your device, and the one-time cloud consent dialog never appears for them. Only you and your own machine (or your own local server) see the text.

Troubleshooting

• "WebGPU is required… but not available." Your browser/device doesn't expose WebGPU. Update Chrome and graphics drivers, or use Chrome built-in AI or a local-native runtime instead.
• Gemini Nano won't initialize. Re-check both Chrome flags and relaunch; the download can take a while on first use.
• A model download stalls. Cancel and retry; ensure you have enough free disk space (several GB for Gemma).
• Runtime shows "Not running" / "Wrong API." Start your local server, confirm the Base URL and port, and that it exposes an OpenAI-compatible /v1 API.
• A local model refuses attachments. Expected — switch that message to a cloud model.

Related

• Providers & models · Read aloud (TTS) · Translation workbench · Privacy & permissions