On-device models

Selecting a local-* model runs the assistant's brain on the phone instead of a cloud provider. Speech goes to on-device speech recognition, an open-weights model generates the reply, and the phone speaks it. No per-token cost, no conversation content leaving the device, and it keeps working in airplane mode.

Everything else stays identical. The same glasses.assistant.start(provider) { tool(...) { ... } } block, the same tools, the same events. Only the brain changes.

Both platforms, published. Android: com.extentos:glasses-local. iOS: the GlassesLocal product from swift-glasses, which raises the package's deployment floor to iOS 17 (see install). Both have shipped since 1.11.0/1.11.1 — match the version to whatever com.extentos:glasses you're on and never mix majors. The code below is Kotlin; the Swift surface matches name-for-name, minus the Context parameter.

Automatic

Phones differ enormously in how much memory they can give one app, and a developer shipping to thousands of users cannot know which model each of them can run. Automatic answers that: pick it once, and the SDK resolves the best model each device can actually sustain.

// From code — wins over the dashboard, and needs no account:
val session = glasses.assistant.start(
    provider = AssistantProvider.Managed(model = "local-auto")
) { /* instructions, tools */ }

// Or set it once in the dashboard: Agent → Model → "Automatic" (id: local-auto)

Managed is the provider type for every model — the model id picks the vendor, and a local-* id runs entirely on the phone with no network call. (It was called OpenAi before 2.0.0, which made exactly this line confusing; the old name still compiles, deprecated.) Precedence is code > dashboard > SDK default.

At session start it reads the device, picks the largest model that device class supports, and falls back to the cloud when no local model qualifies. The developer picks one setting; every user gets the best brain their hardware allows.

How it decides

Two numbers, both read at runtime. Automatic never looks up your phone model — there is no device database to go stale.

Choosing on free memory alone would hand the same phone a 4B model on a fresh boot and a 1.7B with a browser open. The class budget keeps it deterministic.

It tells you what it chose

Automatic is the one model id whose served model differs from the selected one, so it always reports the outcome:

glasses.runtime.events.collect { ev ->
    val e = (ev as? RuntimeEvent.Assistant)?.event as? AssistantEvent.AutoModelResolved ?: return@collect
    println("serving ${e.servedModelId}  local=${e.isLocal}  reason=${e.cloudReason}")
}

servedModelId is always a concrete model — never local-auto. It also appears in the simulator event log as assistant.auto_model_resolved.

That report is what lets a privacy-sensitive app enforce its own policy: read the resolution, and if it is not local, refuse or warn in your own code. The SDK never decides that for you.

The model ladder

All rungs are Qwen open-weights models. Memory requirements differ per platform because Android runs GGUF quantisations and iOS runs 4-bit MLX checkpoints — the same model needs different amounts of memory on each.

Pinning a specific id serves exactly that model or refuses honestly. The SDK never silently substitutes a different one.

The download path

Weights are gigabytes, and the SDK will never download them behind your user's back — not at install, not at launch. Fetching is always the end user's deliberate act, through a surface in your app.

Ask what this device needs, then offer it:

val choice = ExtentosLocalTier.autoChoice(context)

if (choice.isLocal) {
    // Already installed and running on-device — nothing to do.
} else if (choice.downloadTarget != null) {
    // This phone CAN run a local model; it just isn't downloaded yet.
    ExtentosLocalTier.download(context, choice.downloadTarget!!) { done, total ->
        progress = done.toFloat() / total          // byte-accurate
    }
} else {
    // No local model fits this device — the cloud serves. Nothing to install.
}

Swift is the same without context:

let choice = ExtentosLocalTier.autoChoice()
if let target = choice.downloadTarget {
    try await ExtentosLocalTier.download(modelId: target) { progress = $0 }
}

Supporting calls on both platforms: models() (ids, display names, sizes, installed state), delete(...), and a required-vs-available readout for building a manual picker — deviceFit(context, modelId) on Android, deviceFit(for:) on Swift. (Android takes a Context on these; Swift doesn't. That's the only shape difference.)

Until the weights land, the cloud serves. A brand-new user on a perfectly capable phone still starts on the managed gateway and moves on-device once their download finishes. That transition is normal, not an error, and Automatic reports it each session.

On-device voices

The local tier speaks through the phone's built-in text-to-speech by default. Adding glasses-local-voice (Android) or GlassesLocalVoice (iOS) upgrades that to Kokoro, an on-device neural voice with eleven speakers:

All eleven live in one downloaded model, so switching speaker costs no extra download, memory or load time. Every one has an audio preview in the dashboard's Agent section.

The voice is independent of the brain — it pairs with any local model, and costs roughly 300 MB of memory plus some reply latency on older phones. Until its model is downloaded, the system voice serves.

It isn't assistant-only, either. Any text your app wants read aloud can use it directly, with no assistant session anywhere:

glasses.audio.speak(articleSummary, SpeakConfig(voice = "kokoro-emma"))

_ = await glasses.audio.speak(articleSummary, config: SpeakConfig(voice: "kokoro-emma"))

The text can come from anywhere — a cloud model's reply, app content, a notification. Routing follows the same per-call contract as the assistant's mouth: an unknown voice id or a not-yet-downloaded model serves the system voice for that call, cancelSpeak() interrupts a local synthesis just like platform TTS, and direct speak shares the assistant's loaded engine rather than holding a second copy in memory.

On-device transcription

Android. The same module has an ear as well as a mouth. ExtentosSttModel downloads a streaming recognition model (73 MB), after which audio.transcriptions() runs entirely on the phone:

if (!ExtentosSttModel.isInstalled(context)) {
    ExtentosSttModel.download(context) { done, total -> showProgress(done, total) }
}

It is the same contract as the voice in every respect — size known up front, byte progress, resumable, deletable, and never downloaded behind the user's back. Until it is present the system recogniser serves, so this is an upgrade rather than a requirement. Both engines come from the one library the module already vendors, so the voice and the ear cost a single dependency and a single ExtentosLocalVoice.register(context) line.

What it buys over the system recogniser: no Android version floor, no network at recognition time, and no per-word cost.

There is no iOS equivalent, and that is not a gap. Handing recorded audio to the system recogniser needs a trick on Android — SpeechRecognizer will accept a pipe instead of the microphone, but only from Android 13, and a given device's recogniser may still refuse. SFSpeechAudioBufferRecognitionRequest has always taken buffers, so iOS reaches the same destination with no model to download and no version floor. Both platforms end up in the same place; only Android needs a second route to get there.

There is also a category of device where it matters more than that. Glasses that connect as a Bluetooth headset — Ray-Ban Meta, and ordinary earbuds — simply become the phone's microphone, so the system recogniser hears them with no help from the SDK. Glasses built around their own BLE link, like Brilliant Halo, never become a system audio device at all: their microphone audio arrives as bytes in your process, and anything built on "listen to the phone's mic" is deaf to them.

You don't have to care which kind you're holding. audio.transcriptions() reads from the SDK's own audio stream rather than the phone's microphone, so the call is identical either way:

glasses.audio.transcriptions().collect { transcript ->
    if (transcript is Transcript.Final) note(transcript.text)
}

That is the whole point of the seam — supported vendors reach the audio path whether or not they speak Bluetooth audio, and your code does not branch on it.

Setup

Android — add the modules and register at app start:

// build.gradle.kts
implementation("com.extentos:glasses-local:2.1.2")        // on-device models
implementation("com.extentos:glasses-local-voice:2.1.2")  // optional: Kokoro

// Application.onCreate — BOTH must run before ExtentosGlasses.create(...)
import com.extentos.glasses.local.ExtentosLocalTier
import com.extentos.glasses.localvoice.ExtentosLocalVoice

ExtentosLocalTier.register(applicationContext)
ExtentosLocalVoice.register(applicationContext)

iOS — add the products and register:

// Package.swift → .product(name: "GlassesLocal", package: "swift-glasses")
//                 .product(name: "GlassesLocalVoice", package: "swift-glasses")
import GlassesLocal
import GlassesLocalVoice

// Before Extentos.create(...) — in your App's init()
ExtentosLocalTier.register()
ExtentosLocalVoice.register()

register() must run before you create the SDK handle on both platforms.

@main
struct MyApp: App {
    let glasses: any ExtentosGlasses   // no initializer here

    init() {
        ExtentosLocalTier.register()
        ExtentosLocalVoice.register()
        glasses = Extentos.create(config: ExtentosConfig(
            usedCapabilities: [.microphone, .speaker]
        ))
    }
    // …
}

Registering afterwards leaves the assistant with no on-device brain for that process — which on iOS means it quietly serves from the cloud instead (see the callout below).

Without register(), a local-* model has no on-device brain available, and the two platforms diverge.

In the simulator

The browser simulator serves local models from Extentos infrastructure, so every rung works immediately with no download. Because the model runs server-side there, the phone's memory does not constrain the choice — Automatic resolves to the best model the simulator hosts, which is how you exercise the local path on any device.

That is the one deliberate difference between simulation and hardware, and it is contained at the substrate. Everything above it — resolution reporting, tools, events, voices — behaves identically.

Cost and privacy

On-device serving bills nothing per token and sends no conversation content to Extentos or any model vendor. Cloud fallback rides the managed gateway like any other model and is billed under the model that actually served, never under local-auto.

Because Automatic reports its resolution every session, "is this conversation staying on the device?" is a question your app can answer precisely rather than assume.

Further reading

Realtime voice AI bills for every second it listens — why an always-on assistant's cost scales with engagement, the conversation-loop architecture that makes on-device turn-taking hold together, and the measurements behind the memory numbers on this page.

Related