--- title: On-device models description: Run the assistant's brain on the phone instead of the cloud — Automatic device-fit selection, the model ladder, the end-user download path, and on-device neural voices. Kotlin and Swift. type: concept platform: all vendor: meta related: - /docs/concepts/assistant - /docs/concepts/ai-gateway - /docs/sdk/android/install - /docs/sdk/ios/install - /docs/reference/errors --- Selecting a `local-*` model runs the assistant's brain **on the phone** instead of a cloud provider. Speech goes to on-device speech recognition, an open-weights model generates the reply, and the phone speaks it. No per-token cost, no conversation content leaving the device, and it keeps working in airplane mode. Everything else stays identical. The same `glasses.assistant.start(provider) { tool(...) { ... } }` block, the same tools, the same events. Only the brain changes. > **Both platforms, published.** Android: `com.extentos:glasses-local`. iOS: the `GlassesLocal` product from `swift-glasses`, which raises the package's deployment floor to **iOS 17** (see [install](/docs/sdk/ios/install)). Both have shipped since 1.11.0/1.11.1 — match the version to whatever `com.extentos:glasses` you're on and never mix majors. The code below is Kotlin; the Swift surface matches name-for-name, minus the `Context` parameter. ## Automatic Phones differ enormously in how much memory they can give one app, and a developer shipping to thousands of users cannot know which model each of them can run. **Automatic** answers that: pick it once, and the SDK resolves the best model each device can actually sustain. ```kotlin // From code — wins over the dashboard, and needs no account: val session = glasses.assistant.start( provider = AssistantProvider.Managed(model = "local-auto") ) { /* instructions, tools */ } // Or set it once in the dashboard: Agent → Model → "Automatic" (id: local-auto) ``` `Managed` is the provider type for every model — **the model id picks the vendor**, and a `local-*` id runs entirely on the phone with no network call. (It was called `OpenAi` before 2.0.0, which made exactly this line confusing; the old name still compiles, deprecated.) Precedence is code > dashboard > SDK default. At session start it reads the device, picks the largest model that device class supports, and **falls back to the cloud when no local model qualifies**. The developer picks one setting; every user gets the best brain their hardware allows. ### How it decides Two numbers, both read at runtime. **Automatic never looks up your phone model** — there is no device database to go stale. | Reading | Role | |---|---| | **Device class budget** — physical RAM × a sustainable share | Decides which model. Stable, so the same phone always gets the same answer. | | **Free memory right now** | A guard only. Can step the choice *down* for one session, never up. | Choosing on free memory alone would hand the same phone a 4B model on a fresh boot and a 1.7B with a browser open. The class budget keeps it deterministic. **The floor is quality, not fit.** Automatic never selects a model below Qwen 3 1.7B, even on a phone with room for one. Smaller models load fine but call tools unreliably, and an assistant that cannot act on "save a note" is a worse product than the cloud. The smaller rungs stay available if you pin them explicitly. ### It tells you what it chose Automatic is the one model id whose served model differs from the selected one, so it always reports the outcome: ```kotlin glasses.runtime.events.collect { ev -> val e = (ev as? RuntimeEvent.Assistant)?.event as? AssistantEvent.AutoModelResolved ?: return@collect println("serving ${e.servedModelId} local=${e.isLocal} reason=${e.cloudReason}") } ``` `servedModelId` is always a concrete model — never `local-auto`. It also appears in the simulator event log as `assistant.auto_model_resolved`. That report is what lets a privacy-sensitive app enforce its own policy: read the resolution, and if it is not local, refuse or warn in your own code. The SDK never decides that for you. ## The model ladder All rungs are Qwen open-weights models. Memory requirements differ per platform because Android runs GGUF quantisations and iOS runs 4-bit MLX checkpoints — **the same model needs different amounts of memory on each.** | Model | Android | iOS | Automatic may pick it | |---|---|---|---| | `local-qwen3-0.6b` | ~1.0 GB | ~0.5 GB | No — below the tool-calling floor | | `local-qwen25-1.5b` | ~1.6 GB | ~1.0 GB | No — below the tool-calling floor | | `local-qwen3-1.7b` | ~2.4 GB | ~1.2 GB | **Yes — the floor** | | `local-qwen3-4b` | ~3.4 GB | ~2.9 GB | Yes | | `local-qwen3-8b` | ~6.4 GB | ~5.8 GB | Yes — beyond current phones | Pinning a specific id serves exactly that model or refuses honestly. The SDK never silently substitutes a different one. ## The download path Weights are **gigabytes**, and the SDK will never download them behind your user's back — not at install, not at launch. Fetching is always the end user's deliberate act, through a surface in your app. Ask what this device needs, then offer it: ```kotlin val choice = ExtentosLocalTier.autoChoice(context) if (choice.isLocal) { // Already installed and running on-device — nothing to do. } else if (choice.downloadTarget != null) { // This phone CAN run a local model; it just isn't downloaded yet. ExtentosLocalTier.download(context, choice.downloadTarget!!) { done, total -> progress = done.toFloat() / total // byte-accurate } } else { // No local model fits this device — the cloud serves. Nothing to install. } ``` Swift is the same without `context`: ```swift let choice = ExtentosLocalTier.autoChoice() if let target = choice.downloadTarget { try await ExtentosLocalTier.download(modelId: target) { progress = $0 } } ``` Supporting calls on both platforms: `models()` (ids, display names, sizes, installed state), `delete(...)`, and a required-vs-available readout for building a manual picker — `deviceFit(context, modelId)` on Android, `deviceFit(for:)` on Swift. (Android takes a `Context` on these; Swift doesn't. That's the only shape difference.) **Until the weights land, the cloud serves.** A brand-new user on a perfectly capable phone still starts on the managed gateway and moves on-device once their download finishes. That transition is normal, not an error, and Automatic reports it each session. ## On-device voices The local tier speaks through the phone's built-in text-to-speech by default. Adding `glasses-local-voice` (Android) or `GlassesLocalVoice` (iOS) upgrades that to **Kokoro**, an on-device neural voice with eleven speakers: | | | |---|---| | **US female** | `kokoro`, `kokoro-bella`, `kokoro-nicole`, `kokoro-sarah`, `kokoro-sky` | | **US male** | `kokoro-adam`, `kokoro-michael` | | **UK female** | `kokoro-emma`, `kokoro-isabella` | | **UK male** | `kokoro-george`, `kokoro-lewis` | All eleven live in **one downloaded model**, so switching speaker costs no extra download, memory or load time. Every one has an audio preview in the dashboard's Agent section. The voice is independent of the brain — it pairs with any local model, and costs roughly 300 MB of memory plus some reply latency on older phones. Until its model is downloaded, the system voice serves. It isn't assistant-only, either. Any text your app wants read aloud can use it directly, with no assistant session anywhere: ```kotlin glasses.audio.speak(articleSummary, SpeakConfig(voice = "kokoro-emma")) ``` ```swift _ = await glasses.audio.speak(articleSummary, config: SpeakConfig(voice: "kokoro-emma")) ``` The text can come from anywhere — a cloud model's reply, app content, a notification. Routing follows the same per-call contract as the assistant's mouth: an unknown voice id or a not-yet-downloaded model serves the system voice for that call, `cancelSpeak()` interrupts a local synthesis just like platform TTS, and direct speak shares the assistant's loaded engine rather than holding a second copy in memory. ## On-device transcription **Android.** The same module has an ear as well as a mouth. `ExtentosSttModel` downloads a streaming recognition model (**73 MB**), after which `audio.transcriptions()` runs entirely on the phone: ```kotlin if (!ExtentosSttModel.isInstalled(context)) { ExtentosSttModel.download(context) { done, total -> showProgress(done, total) } } ``` It is the same contract as the voice in every respect — size known up front, byte progress, resumable, deletable, and never downloaded behind the user's back. Until it is present the system recogniser serves, so this is an upgrade rather than a requirement. Both engines come from the one library the module already vendors, so the voice and the ear cost a single dependency and a single `ExtentosLocalVoice.register(context)` line. What it buys over the system recogniser: no Android version floor, no network at recognition time, and no per-word cost. **There is no iOS equivalent, and that is not a gap.** Handing recorded audio to the system recogniser needs a trick on Android — `SpeechRecognizer` will accept a pipe instead of the microphone, but only from Android 13, and a given device's recogniser may still refuse. `SFSpeechAudioBufferRecognitionRequest` has always taken buffers, so iOS reaches the same destination with no model to download and no version floor. Both platforms end up in the same place; only Android needs a second route to get there. There is also a category of device where it matters more than that. Glasses that connect as a Bluetooth headset — Ray-Ban Meta, and ordinary earbuds — simply *become* the phone's microphone, so the system recogniser hears them with no help from the SDK. Glasses built around their own BLE link, like Brilliant Halo, never become a system audio device at all: their microphone audio arrives as bytes in your process, and anything built on "listen to the phone's mic" is deaf to them. You don't have to care which kind you're holding. `audio.transcriptions()` reads from the SDK's own audio stream rather than the phone's microphone, so the call is identical either way: ```kotlin glasses.audio.transcriptions().collect { transcript -> if (transcript is Transcript.Final) note(transcript.text) } ``` That is the whole point of the seam — [supported vendors](/docs/vendors) reach the audio path whether or not they speak Bluetooth audio, and your code does not branch on it. ## Setup **Android** — add the modules and register at app start: ```kotlin // build.gradle.kts implementation("com.extentos:glasses-local:2.1.2") // on-device models implementation("com.extentos:glasses-local-voice:2.1.2") // optional: Kokoro // Application.onCreate — BOTH must run before ExtentosGlasses.create(...) import com.extentos.glasses.local.ExtentosLocalTier import com.extentos.glasses.localvoice.ExtentosLocalVoice ExtentosLocalTier.register(applicationContext) ExtentosLocalVoice.register(applicationContext) ``` **iOS** — add the products and register: ```swift // Package.swift → .product(name: "GlassesLocal", package: "swift-glasses") // .product(name: "GlassesLocalVoice", package: "swift-glasses") import GlassesLocal import GlassesLocalVoice // Before Extentos.create(...) — in your App's init() ExtentosLocalTier.register() ExtentosLocalVoice.register() ``` **`register()` must run before you create the SDK handle** on both platforms. **On Swift this bites in a way that looks correct.** Stored-property initializers run *before* the enclosing `init()` body — so the common `let glasses = Extentos.create(...)` property form calls `create()` first and your `register()` second, no matter where in `init()` you put it. Declare the property without an initializer and assign it inside `init()`, after the `register()` calls: ```swift @main struct MyApp: App { let glasses: any ExtentosGlasses // no initializer here init() { ExtentosLocalTier.register() ExtentosLocalVoice.register() glasses = Extentos.create(config: ExtentosConfig( usedCapabilities: [.microphone, .speaker] )) } // … } ``` Registering afterwards leaves the assistant with no on-device brain for that process — which on iOS means it quietly serves from the cloud instead (see the callout below). Without `register()`, a `local-*` model has no on-device brain available, and **the two platforms diverge**. **Forgetting `register()` fails loudly on Android and quietly on iOS.** Android refuses the local model and says so. iOS serves the request from the cloud instead — so an app built for on-device privacy silently sends conversation audio to a provider, with nothing in the UI to show it. If privacy is the reason you chose the local tier, don't rely on the platform to tell you: subscribe to `AssistantEvent.AutoModelResolved` and check `isLocal` at the start of every session, refusing to proceed when it's `false`. That check is the same on both platforms. **`AutoModelResolved` fires for `local-auto`**, whose whole job is to report which model it picked. If you pin a concrete id (`local-qwen3-4b`) you don't get that report — so **use `local-auto` if you need to verify.** It reports every session and is the only path that does; pinning a concrete id buys determinism and gives up the receipt, and there is no session property that reports the served model. Don't assume silence means local. ## In the simulator The browser simulator serves local models **from Extentos infrastructure**, so every rung works immediately with no download. Because the model runs server-side there, the phone's memory does not constrain the choice — Automatic resolves to the best model the simulator hosts, which is how you exercise the local path on any device. That is the one deliberate difference between simulation and hardware, and it is contained at the substrate. Everything above it — resolution reporting, tools, events, voices — behaves identically. ## Cost and privacy On-device serving bills nothing per token and sends no conversation content to Extentos or any model vendor. Cloud fallback rides the [managed gateway](/docs/concepts/ai-gateway) like any other model and is billed under the model that actually served, never under `local-auto`. Because Automatic reports its resolution every session, "is this conversation staying on the device?" is a question your app can answer precisely rather than assume. ## Further reading [Realtime voice AI bills for every second it listens](/blog/local-models-zero-cost-voice) — why an always-on assistant's cost scales with engagement, the conversation-loop architecture that makes on-device turn-taking hold together, and the measurements behind the memory numbers on this page.