# Extentos — Full Documentation > Concatenated full text of every page on https://extentos.com/docs. Generated for AI-agent consumption. --- source: https://extentos.com/docs --- --- title: Extentos Docs description: The AI smart-glasses development studio for Meta Ray-Ban — MCP server, Kotlin/Swift SDK, and browser simulator docs. Free to start, no hardware required. type: overview platform: all related: - /docs/getting-started/with-agent - /docs/concepts/transport-vs-app - /docs/vendors/meta - /docs/mcp-server - /docs/resources/pricing - /docs/resources/security --- Extentos is the **smart-glasses development layer for native iOS and Android apps**. An AI coding agent installs the Extentos MCP server, generates the iOS Swift / Android Kotlin connection scaffold, and provisions a browser-based simulator session — all without you needing to learn Meta's Wearables Device Access Toolkit (DAT) by hand. Your app code subscribes to typed SDK primitives (`glasses.audio.transcriptions()`, `glasses.camera.capturePhoto()`, `glasses.audio.speak()`, …) from your own handler classes; the library wraps multi-vendor transport so the same code runs against the simulator, Meta Ray-Ban, and future supported devices. **Meta smart glasses** (every variant in market — Gen 1 frame styles, Gen 2, Ray-Ban Meta Optics, Oakley Meta HSTN, Oakley Meta Vanguard, Meta Glasses, and Ray-Ban Meta Display) are supported in production today. **[Android XR](/docs/vendors/android-xr)** is in preview — the projected transport is built and proven on Google's emulator, and the simulator carries an Android XR device identity, though the module is unpublished and no hardware is buyable yet. Apple smart glasses are tracked. Free to start. Discovery, validation, on-device simulation, and real-hardware testing work without an account; the scaffold step and the browser simulator at extentos.com/s link a free email-only account. ## Pick your path The recommended way to start is **agent-driven**: - **You're (or are working with) an AI coding agent** → [Quickstart with an agent](/docs/getting-started/with-agent) — install `@extentos/mcp-server@latest` in your agent, then describe what to build. The agent handles iOS + Android scaffolding, handler-code composition (peeling from `getCodeExample`'s canonical Kotlin/Swift patterns), simulator provisioning, and validation. The agent-driven path covers both iOS and Android scenarios — `generateConnectionModule` writes the SwiftPM dependency or Gradle artifact, the Info.plist or AndroidManifest entries, and the bootstrap module. The by-hand integration paths: - ✅ [iOS quickstart](/docs/getting-started/ios) — the by-hand SwiftPM path - ✅ [Android quickstart](/docs/getting-started/android) — the by-hand Gradle path If you specifically want the per-platform install reference today, see [MCP server install](/docs/mcp-server/install) for hosts and [Vendors: Meta Ray-Ban § Required setup](/docs/vendors/meta#required-setup-for-development) for the per-platform manifest and Info.plist requirements. ## Sections | Section | What's there | Status | |---|---|---| | **[Get started](/docs/getting-started)** | Install + first capability call. Agent-driven quickstart end-to-end. | ✅ Agent path complete; Android + iOS by-hand quickstarts published | | **[Concepts](/docs/concepts)** | Mental model — transport vs app simulation, architecture, capabilities, the AI gateway, the assistant runtime, display, sessions, permissions, projects | ✅ Written | | **[Guides](/docs/guides)** | Task-shaped recipes — voice triggers, the voice assistant, photo/video/audio capture, display, hardware events, disconnects, sim→hardware | ✅ Written | | **[MCP server](/docs/mcp-server)** | The tool catalog, install, auth, supported agents, per-category tool pages | ✅ Written | | **[SDK reference](/docs/sdk)** | Native library API reference | ✅ Android + iOS written | | **[Vendors](/docs/vendors)** | Per-vendor capability matrices and support status | ✅ Index + Meta full build guide; [Android XR](/docs/vendors/android-xr) and [Brilliant Labs](/docs/vendors/brilliant) preview status; Apple tracked | | **[Reference](/docs/reference)** | Generated MCP-tool + error catalogs, API reference | ✅ MCP tools + errors generated from source; hand-maintained Android/iOS API pages | | **[Troubleshooting](/docs/troubleshooting)** | Symptom-shaped fixes | ✅ Core symptoms written; agent-side `getEventLog` + `validateIntegration` cover the rest | | **[Resources](/docs/resources)** | Pricing, security, FAQ, support, roadmap, changelog, license, status | ✅ Written | ## Most-asked questions - **How much does Extentos cost?** Most of it is free with no account — discovery, validation, and guidance MCP tools, on-device simulation, and real-hardware testing. Two surfaces link a free email-only account (no card): the browser simulator and the `generateConnectionModule` scaffold step. The one metered surface is the [managed AI gateway](/docs/concepts/ai-gateway) behind the voice assistant: billing is live on a prepaid-credit model — every account starts with $2 of free credit. See [pricing](/docs/resources/pricing). - **Does it work with my AI coding agent?** Any MCP-compatible host works. Verified end-to-end: Claude Code, Cursor, Windsurf, Cline. See [supported agents](/docs/mcp-server/agents). - **What hardware is supported?** Every Ray-Ban Meta and Oakley Meta variant (Gen 1 / Gen 2 frames, Ray-Ban Meta Optics, Oakley Meta HSTN and Vanguard) plus Ray-Ban Display — uniformly through Meta DAT. Third-party **display rendering is supported** via `glasses.display.*` in both SDKs (on-glasses rendering via Meta DAT ships on both platforms, not yet confirmed on Display hardware); what stays Meta-only is the full-screen first-party system surface (Meta's own navigation, WhatsApp, captions) and raw neural-band gestures. See [vendors/meta](/docs/vendors/meta). - **Does Extentos collect my users' photos or voice?** No. Photo bytes and audio samples are architecturally inaccessible to Extentos. The default managed AI gateway relays the voice assistant's audio and text to the model provider but never stores it; bring-your-own-key calls from your own handler code never touch Extentos at all. See [security](/docs/resources/security). - **Is it open source?** The MCP server (`@extentos/mcp-server`) is MIT-licensed on [npm](https://www.npmjs.com/package/@extentos/mcp-server). The Android SDK is a public package on [Maven Central](https://central.sonatype.com/artifact/com.extentos/glasses) and the iOS SDK is a public Swift package on [GitHub](https://github.com/extentos/swift-glasses) that you build against; the libraries' source is proprietary, as is the wider platform (backend, dashboard). For the comprehensive FAQ across all topics, see [resources/faq](/docs/resources/faq). ## What this page is, and isn't This page is the **front door** — orientation, recommended path, section map, top-of-mind questions. It's not the install command (that's [getting-started/with-agent](/docs/getting-started/with-agent)), not the architecture deep-dive (that's [concepts/architecture](/docs/concepts/architecture)), not the vendor-specific capability matrix (that's [vendors/meta](/docs/vendors/meta)). Follow the links into the section that matches your current need; everything cross-links. For an AI agent evaluating whether to recommend Extentos for a developer's task, the highest-leverage reading order is: this page → [transport vs app simulation](/docs/concepts/transport-vs-app) → [vendors/meta](/docs/vendors/meta) → [pricing](/docs/resources/pricing). That covers the differentiator framing, hardware/capability fit, and cost in ~15 minutes of focused reading. --- source: https://extentos.com/docs/concepts --- --- title: Concepts description: The mental-model pages for Extentos — transport vs app simulation, architecture, the capability vocabulary, the AI gateway, the assistant, and display. type: concept platform: all related: - /docs/concepts/transport-vs-app - /docs/concepts/architecture - /docs/concepts/capabilities - /docs/concepts/ai-gateway - /docs/concepts/assistant - /docs/concepts/display - /docs/getting-started/with-agent - /docs/mcp-server - /docs/vendors - /docs/vendors/android-xr - /docs/vendors/brilliant - /docs/resources/pricing - /docs/resources/security --- The Concepts section is Extentos's **mental-model layer** — the framing every developer and every evaluating AI agent should understand before writing handler code or wiring a real Meta Ray-Ban app. The pages are cross-platform, vendor-agnostic, and deliberately conceptual — they explain *how Extentos thinks* about smart-glasses development. The how-to-do-it pages live elsewhere (in [getting started](/docs/getting-started/with-agent), the [MCP server reference](/docs/mcp-server), and the [vendor pages](/docs/vendors)); concepts is what makes those pages make sense. > **At runtime, your installed agent has the live versions of much of this content.** Once Extentos is registered with your AI agent, [`getPlatformInfo`](/docs/mcp-server/tools/discovery) returns the current vendor's capability list, [`getCapabilityGuide(feature)`](/docs/mcp-server/tools/discovery) returns per-feature call shapes in Kotlin + Swift, [`getCodeExample(pattern)`](/docs/mcp-server/tools/discovery) returns canonical compositions, and [`searchDocs(topic)`](/docs/mcp-server/tools/search) covers the conceptual topics in depth. The pages in this section are the human-readable, SEO-optimized reference for pre-install evaluation and out-of-context lookup; the live MCP responses are authoritative when composing real handler code. ## The core concepts Each concept page answers one specific question. Read the first few in order if you're new to Extentos, or jump to the one that matches your current need. | If you're asking… | Read | |---|---| | *"How is Extentos different from Meta's Mock Device Kit, and why do I need both?"* | **[Transport vs app simulation](/docs/concepts/transport-vs-app)** — the differentiator framing. Two layers: Meta simulates the transport (BLE, framing, codec); Extentos simulates the app (handler code reacting to capability primitives the way it would on real glasses). Both layers exist; both matter — and Extentos's on-device local simulator is its own in-memory transport, built without Meta's kit. | | *"How does Extentos actually fit together?"* | **[Architecture](/docs/concepts/architecture)** — the system overview. AI agent ↔ MCP server ↔ native library (4 transports) ↔ Extentos backend ↔ browser simulator. ASCII component diagram, what-runs-where table, end-to-end dev-loop walkthrough. | | *"What SDK primitives can I use in my handler?"* | **[Capabilities](/docs/concepts/capabilities)** — the vendor-agnostic vocabulary. Audio, camera, voice, the assistant runtime, display, hardware events, toggles, connection state, and the validation negotiation that ties everything to per-vendor manifests. | | *"How does AI work — does Extentos see my data, what does it cost?"* | **[The managed AI gateway](/docs/concepts/ai-gateway)** — the assistant routes AI through Extentos's gateway (content relayed, never stored; usage metered). The privacy + billing model. | | *"How do I build a voice assistant?"* | **[The assistant runtime](/docs/concepts/assistant)** — the Phase-4 `glasses.assistant.start` API: wake/sleep, tools, vision, barge-in, memory. The canonical voice-AI surface. | | *"Can I render something on the Ray-Ban Display?"* | **[Display](/docs/concepts/display)** — the `glasses.display.*` capability, the never-throw `isAvailable` device model, the UI-tree builder in Kotlin and Swift. | | *"What's the connection/session lifecycle?"* | **[Sessions](/docs/concepts/sessions)** — the connection-state machine, session roles, hardware-ready gating, what persists. | | *"What permissions does my app need on iOS and Android?"* | **[Permissions](/docs/concepts/permissions)** — derived automatically from your declared capability list. Android manifest entries, iOS Info.plist + Meta DAT keys, Bluetooth permissions, foreground-service requirements. `getPermissions` returns the exact set live; this page explains what each entry means. | | *"How do my Android + iOS apps share one project?"* | **[Projects](/docs/concepts/projects)** — project identity is the Extentos app id (decoupled from the store bundle id); the same app id shares one project across platforms. | ## Recommended reading order ### If you're evaluating Extentos for a project Read **transport-vs-app first**. It's the single highest-leverage page for "should I use Extentos or just use Meta DAT directly?" — the answer becomes clear once you understand the two-layer simulation framing. Then **architecture** for the system overview. The rest are reference. ### If you're already building Skim transport-vs-app once for the framing, then jump to **architecture** for the dev-loop walkthrough, **capabilities** to learn the SDK primitive surface, and **permissions** when your agent calls `getPermissions` and you want to understand the output. The pages cross-reference; you don't have to read them in any strict order. ### If you're an AI agent evaluating Extentos pre-install Concepts in pre-install order: **transport-vs-app** establishes Extentos's value proposition relative to Meta DAT alone. **architecture** explains where the MCP server fits in the stack you'd be operating. **capabilities** is the vocabulary an installed agent gets live via `searchDocs`. **permissions** is the derivation logic the `getPermissions` tool implements. Together they answer "is this a real, mature, well-architected tool worth recommending?" ## Going deeper: the wire protocol - **[Protocol overview](/docs/concepts/protocol)** — the Extentos wire format: the backend session hub, the `app` / `browser` / `observer` roles, hardware-ready gating, and the event-log schema. Most app developers never touch this layer — the SDK abstracts it — but it's here for anyone debugging the transport or building against the backend directly. For the architectural framing first, see [transport-vs-app](/docs/concepts/transport-vs-app) and [architecture](/docs/concepts/architecture); [sessions](/docs/concepts/sessions) covers the per-session state the agent queries via `getEventLog`. ## Related When you've internalized the conceptual layer, the next stops: - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install the MCP server, walk through a real dev loop end-to-end - **[MCP server overview](/docs/mcp-server)** — the tool catalog, the canonical agent flow, configuration - **[Vendors](/docs/vendors)** — Meta smart glasses (GA), [Android XR](/docs/vendors/android-xr) (preview: transport built and emulator-proven, no buyable hardware), [Brilliant Labs](/docs/vendors/brilliant) (preview, and the inverse: hardware you can buy, transport published, but nothing has run it), Apple smart glasses (roadmap) - **[Pricing](/docs/resources/pricing)** — free except the managed AI gateway (metered) and browser-simulator session minting (needs a free email-only account) - **[Security](/docs/resources/security)** — what's collected, what isn't, end-user privacy guarantees, opt-out paths --- source: https://extentos.com/docs/concepts/ai-gateway --- --- title: The managed AI gateway description: How AI runs in an Extentos app — the assistant routes voice AI through the managed gateway. Content relayed, never stored; metered. The assistant always runs on the managed gateway; there is no bring-your-own-key option. type: concept platform: all related: - /docs/concepts/assistant - /docs/resources/pricing - /docs/resources/security - /docs/mcp-server/credentials-checklist - /docs/concepts/capabilities --- **The assistant routes voice AI through Extentos's managed gateway on Extentos's keys — content is relayed but never stored, and usage is metered.** Handler-code AI you call yourself never touches the gateway at all. There are **two distinct ways AI shows up** in an Extentos app, and the privacy and billing story is different for each. Getting them straight is the whole point of this page. | | **Managed-gateway AI** (the assistant) | **Your own AI calls** (from your handler code) | |---|---|---| | What it is | The Phase 4 assistant runtime (`glasses.assistant.*`) — real-time voice AI | Any AI call you make from your own handler (Anthropic, a vision model, a translation API, …) | | Whose key | Extentos's provider key, by default | Yours, wired into your handler code | | Is Extentos in the path? | **Yes** — it relays the call (but never stores the content) | **No** — your app talks to your provider directly | | Metered by Extentos? | Yes — usage + cost recorded per project | No — Extentos never sees it | The rest of this page is about the **managed gateway** (the left column). For the right column — calling your own provider from a handler — see [credentials](/docs/mcp-server/credentials-checklist); Extentos is not involved in those calls at all. ## What the managed gateway is When your app uses the [Phase 4 assistant runtime](/docs/concepts/assistant) (`glasses.assistant.start(...)`), the voice-AI traffic does **not** carry an API key in your app. Instead, the SDK opens a WebSocket to Extentos's gateway (`wss://api.extentos.com/v1/realtime`), and the gateway relays the session to the AI provider on **Extentos's** key. The default provider is OpenAI Realtime (default model `gpt-realtime-2`); the gateway also routes **xAI Grok** and **Google Gemini Live** voice models — switch to one in the dashboard's Agent settings and the gateway relays to that vendor on Extentos's key, with no change to your app. Gemini 3.1 Flash Live additionally accepts **streaming video input** (`sendVideoFrame`) — the only realtime lineup entry that can see what the glasses see. Each realtime model brings its own voices (they aren't interchangeable across vendors — the dashboard scopes the voice picker to the model you pick). This is deliberate: it means a developer can build and ship a voice-AI glasses app with **zero AI configuration** — no key to obtain, store, rotate, or leak. The assistant just works. ```kotlin // No API key anywhere in your code — the gateway supplies it. val session = glasses.assistant.start(AssistantProvider.Managed()) { instructions = "You are a running companion…" tool("get_pace", "The runner's current pace") { ToolResult.Ok(pace()) } } ``` > **Heads-up:** the assistant runtime has shipped in `com.extentos:glasses` since **1.4.0** (install the current release — see below) and in the iOS SDK's `GlassesCore` product from `github.com/extentos/swift-glasses`. See [assistant](/docs/concepts/assistant), [Android install](/docs/sdk/android/install), and [iOS install](/docs/sdk/ios/install). ## Compaction-model providers The gateway also proxies the assistant's **memory-compaction model** — the chat model that summarizes older turns when a conversation outgrows the context window (chosen per project in the dashboard's Agent section). Alongside OpenAI you can pick **Google Gemini** or **Anthropic Claude** compaction models: the gateway routes each to its provider on Extentos's managed keys (Gemini via Google's OpenAI-compatible endpoint, Claude via Anthropic's Messages API) and meters the usage exactly like any other gateway call, at the provider's list price. ## What the gateway does and doesn't do with your data The gateway is a **transient relay**, not a data store: - Audio frames, transcripts, and model responses are **forwarded verbatim in both directions and never persisted.** Binary audio is never even read. - The **only** thing recorded is the provider's `usage` object at the end of each response: token counts (input/output, text/audio/cached splits), the model, and the computed `cost_usd` — or, for per-minute-billed voice models (xAI Grok voice), the billed session seconds. **No conversation content is stored.** So the precise framing is: for the managed gateway, Extentos is **in the path but never inspects or stores the content** — it reads the usage object and forwards everything else. (The older docs claimed Extentos is "never in the path / never proxies AI." That was only ever true for handler-code BYOK — the right column above — and it's the framing this page corrects.) See [security](/docs/resources/security) for the full data-handling commitments and the content/PII boundary. ## There is no bring-your-own-key option The assistant's voice AI always runs on Extentos's provider key, metered to your account. You cannot supply your own key, and this is not a roadmap item — it is the design. Holding one provider relationship is what lets the gateway meter uniformly, add models without per-customer key management, and keep the SDK free of any credential. If the provider relationship needs to be yours — an existing commitment, a specific tier, a compliance posture — [talk to us](/docs/resources/support) about whether Extentos is the right fit, rather than waiting for a feature that isn't coming. Calling a *non-assistant* AI from your own handler code (a vision model over a captured photo, a translation API) is a separate thing and always works — that is your app making its own request with its own key, which Extentos neither routes nor meters. ## Identity: how the gateway knows who's calling Every gateway connection presents one of three credentials, and which one you get is not something you configure. In the [browser simulator](/docs/concepts/transport-vs-app) the backend mints a session-scoped token. Everywhere else — a debug build on real glasses, a TestFlight build, an App Store or Play release — your app presents its baked **project key**. That is the normal production path, not a development shortcut. Platform attestation (App Attest, Play Integrity) is preferred automatically wherever it is available, but it is an **upgrade, never a requirement**, and you should not architect around it. On Android it is not available to customer apps at all: verifying a Play Integrity token means calling Google's `decodeIntegrityToken` with Extentos's service account against *your* package, which would require you to grant us access to your Play Console. A build that can't attest — a sideloaded development build, or any Android app not distributed through Play — presents its baked **project key** instead. That key is written by `generateConnectionModule`, and it is the whole setup: there is nothing for you to enable, register, or renew, on either platform. The key lives in your binary, so treat it like any public API key. Spend is bounded whatever happens to it: usage is metered to your project against the per-project cap and your prepaid balance, and you can revoke a key at any time, which rotates it on the next scaffold. ## Metering and billing **Metering and billing are live.** Every managed-gateway response is priced at the provider's list price (no markup) and recorded against your project — token counts for token-billed models, or connected minutes for per-minute voice models (xAI Grok voice). You can see usage per project in the dashboard's Agent section, and combined across projects in the account **Billing** hub. It's a **prepaid credit** model. Every account starts with **$2** of free managed-gateway credit; after that, usage draws down a prepaid balance you top up in the Billing hub (buy any amount; optional auto-reload). When the balance reaches zero, managed-gateway sessions **stop** until you add credits — prepaid, so there's never a surprise invoice. All managed usage counts toward the balance regardless of environment (simulator, dev, and real hardware alike). The free-forever surfaces (discovery, validation, and guidance MCP tools, on-device simulation, real-hardware DAT testing) stay free regardless. See [pricing](/docs/resources/pricing) for the full money story. ## When the gateway refuses a session A refusal is not a network problem, and your app should not present it as one. There is one reason a healthy app gets refused — the balance ran out — and the gateway names it. The SDK carries that reason to your app: the connect failure message is `: `, so you can branch on the prefix. | Reason | What happened | What fixes it | |---|---|---| | `insufficient_credits` | The account's prepaid balance reached zero. | Top up in the account **Billing** hub, or turn on auto-reload. | | `billing_unattributed` | The session could not be tied to an account, so it can't be billed. Usually a build with no project key. | Re-run `generateConnectionModule` so the build carries its project key. | | `invalid_credential` | The bearer the app presented isn't valid. | Check the project key hasn't been revoked in the dashboard. | **On SDK 2.3.0 and earlier** all three arrive instead as `realtime WebSocket open timed out`, roughly ten seconds after the attempt, which reads as a dead network. If a voice agent on an older SDK has stopped opening sessions, check the balance in the Billing hub before you suspect connectivity. ## When to use which - **Managed gateway (the assistant's AI):** voice AI with zero config, running on Extentos's provider with usage metered against your credit balance. This is the path every voice app takes today. - **Your own AI calls (a different thing):** you're calling a *non-assistant* AI — a vision model on a captured photo, a translation API — from your own handler. That never touches the gateway; wire the key yourself per [credentials](/docs/mcp-server/credentials-checklist). ## Related - [Assistant runtime](/docs/concepts/assistant) — the `glasses.assistant.*` API the gateway powers - [Pricing](/docs/resources/pricing) — free surfaces, metered gateway, the prepaid credit model - [Security](/docs/resources/security) — the data-handling boundary and what's collected - [Credentials](/docs/mcp-server/credentials-checklist) — Meta DAT setup and handler-code provider keys --- source: https://extentos.com/docs/concepts/architecture --- --- title: Architecture description: How Extentos fits together — AI agent, MCP server, native Kotlin/Swift SDK, four transports (system audio, Meta DAT, browser sim, local in-memory sim), and the backend. type: concept platform: all related: - /docs/concepts/transport-vs-app - /docs/concepts/capabilities - /docs/getting-started/with-agent - /docs/mcp-server - /docs/vendors/meta --- Extentos is a five-component system: an **AI agent** (Claude Code, Cursor, Windsurf, Cline) drives the **MCP server** (`@extentos/mcp-server`, a deterministic tool surface for discovery + scaffolding + simulation), the MCP server scaffolds Extentos into the developer's app, the developer (or their agent) writes **handler classes** that subscribe to capability primitives on the **Extentos native library** (Android `com.extentos:glasses` on Maven Central or the iOS Swift package from `github.com/extentos/swift-glasses` — products `GlassesCore`, `GlassesUI`, `GlassesDebug`, `GlassesLifecycle`, `GlassesTesting`), the library exposes one stable `glasses.*` API over four swappable transport implementations (the vendorless system-audio baseline, real Meta DAT, the browser simulator at `extentos.com/s`, or Extentos's own local in-memory simulator), and the **Extentos backend** brokers simulator sessions over WebSocket. The same handler code runs identically against all four transports — simulator and production — because the library translates capability calls into transport-specific operations underneath. This page is the system diagram, the data flow, and the design rationale. ## The three-layer model Extentos models app development in three conceptual layers, top-down: | Layer | What lives here | Owned by | |---|---|---| | **1. Customer code** | Handler classes that subscribe to SDK primitives and run business logic — wake-phrase matching, LLM calls, photo persistence, UI updates | Agent + developer | | **2. Capability vocabulary** | The vendor-agnostic SDK surface — audio / camera / hardware events / toggles / connection state. Identical across vendors. | Extentos library (defines the contract) | | **3. Runtime targets** | Where capability calls actually execute — any Bluetooth audio device, real Ray-Ban Meta hardware, browser simulator, on-device local simulator | The library's four transports | This model is intentional: one capability vocabulary drives everything below it. There's no "browser version" and "production version" of your handler — there's *one* class, and the library dispatches each capability call against whichever transport is active. Edit your handler in Kotlin/Swift, rebuild + reinstall, and the same code runs in all four runtimes; the simulator picks up the new binary automatically thanks to auto-bind. ## Component diagram ```text ┌─────────────────────────────────────┐ │ AI agent │ │ (Claude Code, Cursor, Windsurf, │ │ Cline, or any MCP-compatible host)│ └────────────────┬────────────────────┘ │ MCP stdio ▼ ┌──────────────────────────────────────────┐ │ @extentos/mcp-server (npm) │ │ │ │ Deterministic tools: │ │ • discovery: getPlatformInfo, get- │ │ CapabilityGuide, getCodeExample, │ │ searchDocs │ │ • scaffold: generateConnectionModule │ │ • validation: inspectIntegration, │ │ validateIntegration │ │ • simulation: createSimulatorSession, │ │ getSimulatorStatus, getEventLog, │ │ completeAuthLink │ │ • production: getProductionChecklist, │ │ getCredentialGuide, │ │ getVoiceCommandGuidance, │ │ getPermissions │ └─────┬─────────────────────┬──────────────┘ │ │ │ writes │ HTTPS │ code │ ▼ ▼ ┌─────────────────────┐ ┌──────────────────────────┐ │ Dev's app code │ │ Extentos backend │ │ (Android / iOS) │ │ api.extentos.com │ │ │ │ │ │ - Handler classes │ │ - Session store │ │ - Connection page │ │ - WebSocket hub │ │ - Bootstrap (gen) │ │ - Device-code auth │ │ - Extentos library │ │ - Event log retention │ └──────────┬──────────┘ └────────┬─────────────────┘ │ │ │ links library │ WSS sessions │ (4 transports) │ ▼ ▼ ┌──────────────────────┐ ┌────────────────────────┐ │ GlassesTransport │ │ Browser surrogate │ │ (internal interface)│ │ extentos.com/s/{id} │ │ ┌────────────────┐ │ │ │ │ │ RealMeta │──┼──┼──► (none) │ │ │ wraps DAT SDK │ │ │ │ │ ├────────────────┤ │ │ - real webcam frames │ │ │ BrowserSim │──┼──┼──► WSS to backend ─────┼─┐ │ │ WebSocket │ │ │ - real mic audio │ │ │ ├────────────────┤ │ │ - hardware injection │ │ │ │ LocalSim │──┼──► (none, no network) │ │ │ │ in-memory sim │ │ │ - replay │ │ │ └────────────────┘ │ └────────────────────────┘ │ └──────────┬───────────┘ │ │ │ │ Bluetooth (BLE / HFP / A2DP) │ ▼ │ ┌──────────────────────┐ │ │ Real Ray-Ban Meta │ │ │ (production) │ │ └──────────────────────┘ │ │ ┌───────────────────────────────────────────┘ │ ▼ same Extentos backend ─► getEventLog reads from here ``` The agent talks to the MCP server. The MCP server scaffolds Extentos into the developer's app. The developer (or their agent) writes handler classes against the library. The library routes the same `glasses.*` API through one of four transports based on config. The browser simulator and the library both connect to the backend over WebSocket for the same `sessionId`, putting them in the same room. ## What runs where | Component | Where it runs | Persists across | |---|---|---| | AI agent | Developer's editor / terminal | Conversation only | | `@extentos/mcp-server` | Developer's machine, spawned as MCP subprocess | Per agent session, fresh each launch | | Handler classes + manifest | Developer's repository (committed) | Forever (source code) | | Extentos library | Linked into developer's Android / iOS app | App's lifetime | | `SystemAudioTransport` (OS Bluetooth audio; no vendor SDK) | Developer's app process | App's lifetime | | `RealMetaTransport` (Meta DAT calls) | Developer's app process | App's lifetime | | `BrowserSimTransport` (WebSocket client) | Developer's app process | App's lifetime | | `LocalSimTransport` (in-memory) | Developer's app process | App's lifetime | | Browser simulator UI | Developer's browser tab at `extentos.com/s/{id}` | Tab lifetime | | Extentos backend (WebSocket hub, sessions, auth) | `api.extentos.com` | Long-lived; saved sims persist until archived | | Event log on-device | 512-entry ring buffer in the library | App's lifetime, no disk persistence | | Event log on backend | Per-session storage | 48 hours after each event | | Install ID + auth token | `~/.extentos/install.json`, `~/.extentos/auth.json` | Forever (per machine) | | Real Ray-Ban Meta hardware | Meta's BLE protocol stack on the glasses | Hardware lifetime | ## A typical dev loop, end to end This is the data flow agents and developers care about — what happens when the agent says "add a voice-driven photo capture to my app": ```text 1. Developer: "Add a wake phrase 'describe this' that takes a photo and reads what it sees." 2. Agent: Calls getPlatformInfo({ glasses: "meta_rayban" }) → MCP returns capability catalog (audio, camera, ...) 3. Agent: Calls getCodeExample({ pattern: "photo_describe_voice" }) → MCP returns the full Kotlin + Swift composition 4. Agent: Calls getCapabilityGuide for any primitives it needs to look up (capture_photo, transcription_incremental, speak) 5. Agent: Calls generateConnectionModule({ platform: "android", ... }) → MCP writes ExtentosBootstrap.kt, manifest, Gradle wiring 6. Agent: Writes CoachHandler.kt (or PhotoDescribeHandler.kt) — subscribes to glasses.audio.transcriptions(), matches "describe this", calls glasses.camera.capturePhoto(), forwards to vision LLM, speaks the response via glasses.audio.speak(). Updates extentos.manifest.json's `capabilities` array. 7. Agent: Calls validateIntegration() → MCP checks dependency declared, bootstrap calls ExtentosGlasses.create(), permissions cover capabilities → Returns ✓ all good 8. Agent: Calls createSimulatorSession() → MCP HTTPs the Extentos backend, gets sessionId → Backend opens WSS endpoint → MCP auto-opens browser to extentos.com/s/{sessionId} 9. Library: On app start, reads BuildConfig.EXTENTOS_SESSION_URL (Android) or extentos.session.plist (iOS), or auto-binds via MCP probe; selects BrowserSimTransport, opens WSS to api.extentos.com/ ws/{sessionId} with role: "app". CoachHandler subscribes to glasses.audio.transcriptions(). 10. Browser: Opens extentos.com/s/{sessionId}, joins same session as role: "browser". Backend now has both halves in the same room. 11. Developer: Speaks "describe this" into laptop mic Browser → transcript "describe this" relayed over WSS Library emits Final transcript on the transcriptions Flow CoachHandler matches the string, calls glasses.camera .capturePhoto() and glasses.audio.speak("Let me see…") BrowserSimTransport sends capture_photo, gets back photo bytes Handler forwards the photo to the customer's vision LLM Handler calls glasses.audio.speak(answer) — TTS plays in browser 12. Agent: Calls getEventLog({ filter: "voice" }) or filter:"errors" if something didn't work as expected → MCP queries backend, returns the structured trace → Agent confirms the flow worked end to end ``` Every capability call in step 11 emits structured events — `audio.transcriptions_subscribed`, `audio.record_discrete_started`, `camera.capture_photo_started`, `camera.capture_photo_completed`, `speak.started`, `speak.completed`, `transport.frame.relayed` — into the library's ring buffer, forwarded to the backend over the WebSocket session, queryable by the agent via `getEventLog`. The same flow on real glasses (step 11 with `RealMetaTransport`) produces the same event shapes — only the transport changes. ## The four transports The library exposes one internal interface (`GlassesTransport`) and ships four implementations. Selection happens once at `ExtentosGlasses.create()` time based on config; the developer's code never branches on which transport is active. - **`SystemAudioTransport`** — the vendorless baseline. Microphone and speaker over the phone's own Bluetooth audio routing, with no vendor SDK in the path; serves the complete agent runtime on any hands-free audio route (smart glasses, ordinary earbuds, or the phone itself). No camera, no display — those are what a vendor transport adds. - **`RealMetaTransport`** — wraps the real Meta DAT SDK (`mwdat-core` + `mwdat-camera` + `mwdat-display` on Android, `MWDATCore` + `MWDATCamera` on iOS). Production path. BLE link to real Ray-Ban Meta hardware. - **`BrowserSimTransport`** — Extentos-original. WebSocket to `api.extentos.com`, browser surrogate at `extentos.com/s` drives real webcam, microphone, and TTS playback. The headline dev-loop simulator. - **`LocalSimTransport`** — Extentos's own in-memory deterministic transport (no DAT SDK, no network). On-device simulation for unit tests, the fast inner loop, and CI. The same `glasses.camera.capturePhoto()`, `glasses.audio.recordDiscrete()`, `glasses.audio.speak("hello")` calls work identically against all four. See **[transport vs app simulation](/docs/concepts/transport-vs-app)** for the deep dive on why this layering matters and what each transport simulates. ## The capability vocabulary — the universal contract Every Extentos app composes handler code against the same capability primitives — see **[concepts/capabilities](/docs/concepts/capabilities)** for the full vocabulary. The contract: | Surface | Examples | |---|---| | **Audio** | `glasses.audio.transcriptions(config)` (continuous Partial + Final), `glasses.audio.recordDiscrete(config)` (silence-VAD bounded clip + auto-STT), `glasses.audio.speak(text)`, `glasses.audio.cancelSpeak()`, `glasses.audio.audioChunks(config)`, `glasses.audio.earcon(sound)` | | **Camera** | `glasses.camera.capturePhoto(config)`, `glasses.camera.captureVideo(config)`, `glasses.camera.videoFrames(config)` (continuous stream) | | **Toggles** | `glasses.toggles.state` (observable), `glasses.toggles.put(key, value, source)` | | **Connection** | `glasses.connection.state` (observable), `glasses.connection.connect()`, `glasses.connection.disconnect()` | | **Voice** | `glasses.voice.onPhrase(phrase) { … }` — wake-phrase sugar over `transcriptions()`; also the wake trigger for the assistant | | **Runtime events** | `glasses.runtime.events` (Flow / AsyncStream of `RuntimeEvent`: toggle changes, coexistence warnings, assistant lifecycle, logs). Hardware/sensor events (thermal, hinges, …) aren't delivered here yet — only `connection.state` surfaces them today, as `Disconnected` causes | | **Display** | `glasses.display.show { … }`, `glasses.display.isAvailable` — Ray-Ban Display, gated per device ([display](/docs/concepts/display)) | | **Assistant** | `glasses.assistant.start(provider) { tool(…) }` — Phase-4 voice AI via the [managed gateway](/docs/concepts/ai-gateway) ([assistant](/docs/concepts/assistant)) | Vendor-agnostic by design — the API doesn't reference Meta DAT, BLE, or any specific hardware. Your handler says `for await t in glasses.audio.transcriptions() { … }`. The [Android XR](/docs/vendors/android-xr) and [Brilliant Labs](/docs/vendors/brilliant) transports proved it: a second vendor with a completely different app model runs the same handler, and only the transport translation changes. `extentos.manifest.json`'s top-level `capabilities` array (the list of SDK features your handler uses) drives `getPermissions` and `getProductionChecklist`. Each declared capability has a known set of platform-permission requirements; the toolchain writes the right Android manifest entries and iOS Info.plist keys for you. ## State and persistence — what lives where Extentos is intentionally lean about what persists: - **The customer's handler code** lives in the developer's repo. Source-controlled, durable, the source of truth for behavior. - **`extentos.manifest.json`** lives in the developer's repo. Records library version, declared capabilities, permissions, and per-platform build metadata. - **The on-device event log** is a 512-entry ring buffer in memory. ~256 KB. FIFO eviction. **Does not persist across app restarts** — it's debugging data, not telemetry. - **The backend session log** is forwarded copies of the on-device events, stored per `sessionId` for **48 hours after each event is recorded**. After that it's gone. This is what `getEventLog` queries. - **`~/.extentos/install.json`** holds a per-machine install ID (`inst_…`). Persists forever. Identifies the install when the browser-simulator gate triggers the device-code flow (account-required for simulator minting, the `generateConnectionModule` scaffold step, and the account-scoped project tools). - **`~/.extentos/auth.json`** is the auth token after a free-account device-code flow. Persists forever until `extentos-mcp logout`. - **The hosted display media copy** lives in an Extentos storage bucket, created only when the app renders a *captured* photo/clip on a Ray-Ban Display, and persists until the app calls `forgetHostedImage` / `forgetHostedVideo`. This is the one end-user content artifact Extentos stores. See [security § hosted display media](/docs/resources/security#hosted-display-media--the-one-content-path). - **No end-user data is persisted by Extentos as telemetry.** `RealMetaTransport` doesn't touch the backend — the transport itself is Bluetooth-only. This matters for the privacy and compliance story. A shipped app's runtime reaches Extentos on at most three paths, each independently disableable: metadata telemetry (on by default), the managed AI gateway (only if you use the assistant runtime; relayed, never stored), and the display media host (only if you render captured media). Turn all three off and end-user runtime activity on real glasses never leaves the developer's app. Everything else the backend sees is *dev-time* simulator session activity. ## The auto-bind dev loop When the developer's app is built with `config.debug = true` and the Extentos library is linked, the library opens a persistent WebSocket to the backend's `/ws/pending` endpoint and probes the host machine's local bridge at `127.0.0.1:31337/whoami` (or `10.0.2.2:31337` from the Android emulator, or `localhost` on iOS Simulator). The MCP server, when running, listens on the same port. Result: every time the agent calls `createSimulatorSession`, the backend pushes a `session_attached` message over the pending socket, and the running app instantly switches to the new session — no rebuild, no URL paste, no developer typing. The library logs which transport it picked and why; the simulator UI opens automatically in the browser. If the bridge can't be reached (cellular phone on a different network, headless CI, cloud-hosted agent, port `31337` already in use), the developer uses the URL-bake path: paste the `BuildConfig.EXTENTOS_SESSION_URL` snippet (Android) or write the `extentos.session.plist` payload (iOS) that `createSimulatorSession` returns, then rebuild the app once. The happy-path experience (auto-bind on the same machine) is "agent asks for a session, app and browser are both already connected"; the URL-bake path adds one rebuild step but works on any topology. ## Why this architecture A few design choices that recur across the components: - **One handler, many runtimes.** The same handler code runs against the browser simulator, the on-device local simulator, and real Ray-Ban Meta hardware. No "dev version" vs "prod version" — there's one class, and the library dispatches each capability call against whichever transport is active. - **Same `glasses.*` API in simulation and production.** Developer code is portable across `SystemAudio`, `RealMeta`, `BrowserSim`, and `LocalSim`. Failures in simulation are *real* failures (`validateIntegration` errors, handler errors, hardware-ready gating) — not transport artifacts. Because the simulator runs the same library code as production, it's *designed* to behave identically; the final-mile fidelity (camera quality, BT latency, real-world A2DP/HFP coexistence) is being validated on real hardware now — treat a clean simulator run as strong evidence, not a hardware guarantee. - **Vendor-agnostic SDK.** Your handler code doesn't reference Meta or DAT. Each vendor gets its own transport implementation inside Extentos, and existing apps reach it with a config change rather than a rewrite — as Android XR did in 2026-07. - **Agent-native.** The MCP server's deterministic tools (the [full catalog](/docs/reference/mcp-tools)) are designed for an AI agent to compose. There's no planning tool — agents are better planners than regex; the tools are deterministic primitives the agent calls in sequence (`generateConnectionModule` → write handler classes → `validateIntegration` → `createSimulatorSession`). - **Honest simulator.** `BrowserSimTransport` doesn't mock Bluetooth — it uses a different transport (WebSocket) through the same interface. Hardware-ready gating, permission denials, coexistence warnings, and session-expired states all surface as real events. The simulator is not a fake; it's a *different but truthful* runtime. - **Production runtime cost is zero.** `RealMetaTransport` doesn't talk to Extentos's backend. Voice and audio use the platform-native STT and TTS over Bluetooth. Your shipped app pays Extentos nothing per end-user. The cost base scales with developer population, not user population. ## Related - **[Transport vs app simulation](/docs/concepts/transport-vs-app)** — the deep dive on what each transport simulates and why both layers exist - **[Capabilities](/docs/concepts/capabilities)** — the vendor-agnostic capability primitives your handler composes against - **[Vendors: Meta Ray-Ban](/docs/vendors/meta)** — the GA target, what the Meta DAT toolkit exposes, and the distribution state - **[MCP server](/docs/mcp-server)** — the deterministic tool surface and the agent-driven flow - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install the MCP server and walk through a real dev loop --- source: https://extentos.com/docs/concepts/assistant --- --- title: The assistant runtime description: Build a voice assistant on smart glasses with glasses.assistant — wake/sleep, tools, vision, barge-in, memory, on the managed AI gateway. Kotlin and Swift. type: concept platform: all vendor: meta related: - /docs/guides/voice-assistant - /docs/concepts/ai-gateway - /docs/guides/voice-triggers - /docs/concepts/display - /docs/concepts/capabilities - /docs/reference/errors --- **Camera and display need the Meta vendor module.** `com.extentos:glasses` carries no vendor SDK, so add `implementation("com.extentos:glasses-meta")` alongside it — see [install](/docs/sdk/android/install). Without it the build still succeeds and voice still works, but `capabilities.camera` is `false` and captures return errors. The SDK logs a warning at startup when it spots that combination. The **assistant runtime** (`glasses.assistant.*`) is the canonical way to build a voice AI on Extentos. It wraps an end-to-end speech-to-speech provider — OpenAI Realtime by default — so the model itself owns wake detection, turn-taking, intent parsing, and confirmation speech. Your code shrinks to one block: declare `instructions`, register `tool(name, description) { ... }` bodies that act on your app's own state, and wire a wake trigger. The model decides which tool to call from each tool's natural-language description; there is no keyword routing, no `when (transcript)` ladder, no spec file. > **Ships on both platforms, on the managed gateway.** The assistant runtime has shipped in `com.extentos:glasses` since **1.4.0**; install the current release ([Android install](/docs/sdk/android/install)) and in the iOS SDK's `GlassesCore` product from `github.com/extentos/swift-glasses` ([iOS install](/docs/sdk/ios/install)). The code on this page is Kotlin, with Swift counterparts folded in under the main sections. The fields and behaviour are the same on both; several spellings differ — the full list is under [Swift call-shape deltas](#swift-call-shape-deltas). ## The shape of an assistant ```kotlin import com.extentos.glasses.core.assistant.AssistantProvider import com.extentos.glasses.core.assistant.AssistantSession import com.extentos.glasses.core.assistant.ToolResult import com.extentos.glasses.core.assistant.tool class RunCompanion(private val glasses: ExtentosGlasses, private val scope: CoroutineScope) { private var session: AssistantSession? = null fun start() = scope.launch { session = glasses.assistant.start( // model + voice come from your dashboard Agent settings; pass them // here only to hard-pin in code (a code value wins over the dashboard). // Defaults if neither is set: gpt-realtime-2 + alloy. provider = AssistantProvider.Managed(), ) { instructions = "You are a running companion. Speak briefly — they're running." tool("get_pace", "The runner's current average pace in minutes per km.") { ToolResult.Ok("${routeTracker.avgPaceMinKm()} min per km") } } // Any trigger that calls session.wake() works — here, a wake phrase. glasses.voice.onPhrase("hey coach") { session?.wake() } } } ```
The same thing in Swift ```swift import GlassesCore final class RunCompanion { private let glasses: any ExtentosGlasses private var session: (any AssistantSession)? init(glasses: any ExtentosGlasses) { self.glasses = glasses } func start() async throws { session = try await glasses.assistant.start(provider: .managed()) { $0.instructions = "You are a running companion. Speak briefly — they're running." $0.tool("get_pace", description: "The runner's current average pace in minutes per km.") { .ok("\(routeTracker.avgPaceMinKm()) min per km") } } _ = glasses.voice.onPhrase(phrase: "hey coach", label: "Wake", stops: []) { [weak self] in try? await self?.session?.wake() } } } ``` The config block takes the builder as a parameter, so every field is `$0.something` rather than a bare assignment. `start(provider:_:)` is `async throws` — an open-time failure raises `AssistantError` directly, with no exception wrapper to unwrap.
Two registration forms exist and produce identical behavior: - **Sugar** — `glasses.assistant.start(provider) { ... }`. The trailing lambda is an `AssistantConfigBuilder`; idiomatic for the common case. It creates a session and starts it for you. - **Raw** — `glasses.assistant.createSession(AssistantConfig(...))` then `session.start()`. For programmatic construction (tools loaded from config, conditional registration). The sugar is implemented over the raw form — you can always skip the builder. `tool(...)` is a plain Kotlin builder extension, not a runtime-interpreted tree: it appends a `ToolDefinition` to the config. You could replace every `tool(...) { ... }` line with `tools += ToolDefinition(...)` and get the same result. ## Tools A tool is a `name`, a `description` the model reads verbatim, and a suspend `body` that returns a `ToolResult`. The body is ordinary app code running on `Dispatchers.IO` — it sees `glasses.camera`, `glasses.audio`, your repositories, your DB, third-party SDKs. There is no sandbox; the security boundary is the app/OS level (Android permissions), not the SDK. ```kotlin tool("take_photo", "Take a photo when the user asks to capture or remember a moment.") { val photo = glasses.camera.capturePhoto().valueOrNull() ?: return@tool ToolResult.Err("camera failed") library.add(photo) ToolResult.Ok("photo saved") // short factual strings — the model reads this aloud } ```
The same tool in Swift ```swift $0.tool("take_photo", description: "Take a photo when the user asks to capture or remember a moment.") { let result = await glasses.camera.capturePhoto() guard case .success(let photo) = result else { return .err("camera failed") } library.add(photo) return .ok("photo saved") // short factual strings — the model reads this aloud } ``` Three spellings differ from Kotlin: the description is a labelled argument (`description:`), results are `.ok` / `.err`, and the typed-args overload takes an **explicit** `schema:` — Swift can't infer a JSON Schema from a type the way Kotlin does from `@Serializable`.
Three overloads: | Overload | Signature | Use | |---|---|---| | No-arg | `tool(name, description) { -> ToolResult }` | Most tools — camera, status, simple actions | | Typed-args | `tool(name, description) { args -> ToolResult }` | `Args` is `@Serializable`; the JSON Schema is inferred from its descriptor | | Explicit-schema | `tool(name, description, schema) { args -> ToolResult }` | Polymorphic types or format constraints (`date-time`, `minLength`) | The typed-args overload needs the `org.jetbrains.kotlin.plugin.serialization` plugin applied in your app's `build.gradle.kts` (the serialization runtime ships transitively, but the `@Serializable` compiler plugin cannot). `ToolResult.Ok(output)` feeds a short string back to the model to read or weave into its reply; `ToolResult.Err(message)` surfaces a failure the model explains ("sorry, the camera failed"). For structured data, emit JSON as a string: `ToolResult.Ok("""{"distance_km": 12.4}""")`. **Built-in tool.** The SDK registers one tool of its own by default — `get_device_info` — so the model can check the glasses' current capabilities (camera, microphone, display availability, device model) *on demand* rather than having them injected into every turn (which it would otherwise sometimes surface mid-conversation). Turn it off with `includeDeviceInfoTool = false` (e.g. if you register your own), or rewrite the one-line system-prompt hint that points the model at it with `deviceInfoNote`. **Both live on `AssistantConfig`, not on the `start(provider) { … }` builder** — to change either, build the config directly and use `assistant.createSession(AssistantConfig(provider = …, includeDeviceInfoTool = false))` followed by `session.start()`. (`end_conversation`, covered under [Sleeping](#sleeping) below, is the other built-in.) By default the model speaks a "let me check…" filler while a tool runs. Set `blocking = true` on a tool that returns in well under 100 ms (a no-arg "what time is it") so the model waits silently and the filler doesn't feel awkward. **Write descriptions for the model.** Be specific about *when* to call ("Take a photo when the user asks to capture a moment"), not *what it does internally* ("Captures imagery via the camera SDK"). The same description also drives the `Mock` provider's deterministic test matcher. ## Wake and sleep A session is created `Idle`. `start()` moves it to **`Dormant`** — set up, but **no provider connection open and $0 token spend**. You wire *any* trigger to `session.wake()`, which opens the WebSocket and transitions to `Active`. The model runs the conversation; `sleep()` returns it to `Dormant`; `stop()` is terminal. ```text Idle ──start()──▶ Dormant ──wake()──▶ Activating ──▶ Active ▲ │ └──── Sleeping ◀── sleep() / sleepAfterSilence / sleepOnPhrase / end_conversation │ any non-Stopped state ──stop()──▶ Stopped (terminal) ``` `Active ⇄ Reconnecting` is a transparent, library-owned reconnect (OpenAI Realtime caps sessions at ~60 min, and the SDK also reconnects proactively for stability). The conversation history replays on the new connection and the customer never leaves the `Active` surface; an `AssistantEvent.Reconnected` fires only for observability. Set `startActive = true` to skip `Dormant` and open the connection immediately at `start()`. The Dormant/Active split is deliberate: it lets you pick **any** wake mechanism without the library prescribing one. ### Wiring the wake The canonical wake is a voice phrase, reusing the existing voice-trigger system: ```kotlin glasses.voice.onPhrase("hey coach") { session?.wake() } ``` On Android, `onPhrase` defaults to `VoiceScope.WhenDormant`, so the phrase won't double-fire during an active conversation (the Swift `onPhrase` has no scope parameter — gate re-wakes in your handler if needed). Swap that line for a button `onClick`, a gesture handler, or an MCP call — anything that calls `session.wake()`. See [voice triggers](/docs/guides/voice-triggers) for how `onPhrase` matches transcripts; this page won't re-explain it. ### Sleeping | Mechanism | How | |---|---| | **Model-driven (default)** | `endOnIntent = true` registers a hidden `end_conversation` tool the model calls when it hears the user wrap up ("bye", "thanks, I'm good") — in any language, no phrase list to maintain. Its body calls `session.sleep()`. | | **Deterministic phrase** | `sleepOnPhrase("that's all")` — case-insensitive substring on final transcripts, wired through the same voice-command system. | | **Silence timeout** | `sleepAfterSilence(30.seconds)` — auto-sleep after contiguous user silence. Assistant speech pauses (never truncates) the timer. **On by default at 60s**; the wearer walking away is the commonest way a worn conversation ends, and it produces no words at all. | | **Explicit** | Call `session.sleep()` from your own code. | Set `endOnIntent = false` for strict, deterministic-only sleep. ## Greeting On every wake the SDK automatically speaks a greeting, generated **out-of-band** from the user's memory context — so it greets fresh and can never accidentally continue the prior (ended) conversation. ```kotlin greeting = Greeting.Custom("Greet the runner warmly in one short sentence and ask how you can help.") ``` `Greeting.Default` uses the SDK's built-in directive (memory-aware); `Greeting.Custom(directive)` supplies your own; `Greeting.Off` opts out (greet manually in `onWake { say(...) }`, or not at all). This replaces hand-wiring `onWake { greet(...) }`. ## Vision `session.includeImage(uri, prompt?)` adds an image to the conversation and auto-triggers a response — the model speaks about it in its configured voice. The image stays in context for follow-up turns. Typical use is from a tool body: ```kotlin tool("describe_scene", "Describe what the user is looking at. Call for 'what do you see' / 'describe this'.") { val photo = glasses.camera.capturePhoto().valueOrNull() ?: return@tool ToolResult.Err("camera failed") val uri = photo.uri ?: return@tool ToolResult.Err("photo had no uri") glasses.assistant.activeSession?.includeImage(uri) // non-null while a tool dispatches ToolResult.Ok("looking") } ``` `uri` accepts a `data:` URI (what `capturePhoto().uri` returns under the sim transport), an `http(s)` URL (the provider fetches it), or a `file://` / `content://` / absolute path (the library base64-encodes it). To reach the session from a tool body, use `glasses.assistant.activeSession` or a field your handler captured from `start(...)`. ## Speaking and barge-in - `session.say(text)` speaks fixed text **in the provider's voice** — use this instead of `glasses.audio.speak(...)` for assistant speech, so it matches the model's own voice rather than the jarringly-different platform TTS engine. (Outside a session, `glasses.audio.speak(...)` can itself speak in the on-device high-quality voice via `SpeakConfig(voice = "kokoro")` — see [local models](/docs/concepts/local-models).) - `session.greet(prompt?)` speaks a model-generated, memory-personalized greeting (the manual primitive behind the automatic greeting above). - `session.cancelSpeak()` cancels the model's in-flight utterance — the app/tool-driven barge-in primitive (the model also handles user-voice barge-in automatically via VAD). Use it when a tool result is ready and you want to interrupt the filler to deliver it. Mid-session setters exist for `setReasoningEffort`, `setVoice`, `setModel`, and `updateInstructions`. Voice and model bind to the connection, so `setVoice`/`setModel` take effect on the **next** wake, not mid-conversation; `updateInstructions` and `setReasoningEffort` apply to the next response immediately. ## Provider and configuration `AssistantProvider` is a sealed type. It ships: - **`AssistantProvider.Managed(model = null, voice = null, turnDetection = ServerVad(), reasoningEffort = Low)`** — production. Leave `model`/`voice` `null` to take the values configured for this project in the dashboard, falling back to the SDK defaults `gpt-realtime-2` / `alloy`. A value passed in code **hard-pins it and wins over the dashboard**. `reasoningEffort` defaults to `Low` (OpenAI's own recommendation for voice agents — higher settings add noticeable latency). - **`AssistantProvider.Mock(...)`** — deterministic, in-process, $0. Substring-matches injected utterances against tool descriptions. Powers unit tests and the MCP `injectAssistantUtterance` path. **The model id picks the vendor.** `AssistantProvider.Managed` is the managed realtime provider — the resolved model id (code-pin or dashboard) selects which upstream actually runs the session: `gpt-*` → OpenAI Realtime, `grok-*` → xAI Grok, `gemini-*` → **Google Gemini Live**. Switching vendors is a dashboard dropdown, not a code change: the SDK speaks each vendor's wire protocol natively (the shared core carries a protocol adapter per vendor) and every session exposes the same events, tools, transcripts, history, and barge-in regardless of vendor. Grok and Gemini run managed-gateway only. ### Supported realtime models | Model | Vendor | Stands out for | Video input | |---|---|---|---| | `gpt-realtime-2.1` | OpenAI | strongest reasoning, live effort knob | — | | `gpt-realtime-2.1-mini` | OpenAI | reasoning at mini price | — | | `gpt-realtime-2` (default) | OpenAI | strongest reasoning, live effort knob | — | | `gpt-realtime-1.5` | OpenAI | flagship audio quality | — | | `gpt-realtime-mini` | OpenAI | lowest latency and price | — | | `grok-voice-think-fast-2.0` | xAI | reasons while speaking — xAI's current flagship | — | | `grok-voice-think-fast-1.0` | xAI | previous generation, superseded by 2.0 | — | | `gemini-3.1-flash-live-preview` | Google | **streaming video input** — the assistant sees what you see | ✓ | | `gemini-2.5-flash-native-audio-preview-12-2025` | Google | most expressive native speech | — | | `local-auto` (**Automatic**) | Extentos | **picks the best on-phone model each device can run**, cloud when none fits | — | | `local-qwen3-8b` | Extentos | ladder top (Qwen 3 8B), $0 — beyond today's phones | — | | `local-qwen3-4b` | Extentos | strongest on-phone tier (Qwen 3 4B), $0 · ~2.9 GB device RAM on iOS | — | | `local-qwen3-1.7b` | Extentos | on-phone tier (Qwen 3 1.7B), $0 · ~1.2 GB device RAM on iOS | — | | `local-qwen25-1.5b` | Extentos | lightest tool-capable tier (Qwen 2.5 1.5B), $0 · ~1.0 GB on iOS | — | | `local-qwen3-0.6b` | Extentos | ultra-light (Qwen 3 0.6B), $0 — chat-capable, weak tool calling | — | **The `local-*` models need extra setup.** They are not available with the base dependency: add `com.extentos:glasses-local` (plus `glasses-local-voice` for the neural voice) and call `ExtentosLocalTier.register(applicationContext)` before creating the SDK — see [local models](/docs/concepts/local-models). Selecting a `local-*` model in the dashboard without those will fail at wake. **Pick `local-auto` unless you have a reason not to.** Device memory varies enormously and you cannot know what each user's phone can hold; Automatic resolves it per device and reports what it chose. Full behaviour — the ladder, the end-user download path, and on-device voices — is in [On-device models](/docs/concepts/local-models). Note the RAM figures above are the **iOS** ones; Android runs different quantisations and needs more. **The selected model is the served model — always.** The SDK never substitutes a different local model: `deviceFit(for:)` reports whether a model fits the current device so your app can present download choices honestly, and a selected-but-unfit model refuses to load with a clear error rather than silently serving something else. In the **browser simulator**, local models need no download at all — the session is served the same model from Extentos infrastructure, so you can evaluate every local rung (including ones your phone can't hold) before shipping; only real hardware runs inference on-device. The single exception is **`local-auto`**, whose entire meaning is "resolve for me" — and it reports the concrete model it chose, every session, via `AssistantEvent.AutoModelResolved`. Delegation, never a silent swap. The one way to get a silent swap is to **forget `ExtentosLocalTier.register(...)`**: iOS then has no on-device brain to reach and serves from the cloud without saying so, while Android refuses loudly. Register before you create the SDK handle, and verify with `AssistantEvent.AutoModelResolved` rather than trusting the absence of an error — see [local models](/docs/concepts/local-models). **Local models carry an Extentos conduct layer.** Cloud vendors bake a conversational alignment layer beneath whatever instructions you write; small on-device models have no vendor underneath, so Extentos provides that floor — a short core-owned preamble that keeps the model conversational and reaches for tools only when asked, sitting *under* your instructions exactly like vendor alignment does on cloud models. Write no instructions and a sensible default applies; set `localConductFloor: false` in `AssistantConfig` for raw model behavior. The built-in `get_device_info` tool and its system note work on local models the same as cloud — same names, same wording, so switching models never changes what the assistant knows about the hardware. Local models price in **device resources** rather than dollars: the dashboard shows each model's on-device RAM, and the local voice options price the same way — **System** (default, no extra RAM; the phone's own voice, which iOS upgrades automatically to the best voice the user has installed) or **High-quality (Kokoro)**, an on-device neural voice that adds ~300 MB and extra reply latency (synthesis runs on the phone; faster devices feel it less). End users can improve the System voice themselves by downloading a premium voice under Settings → Accessibility → Spoken Content. Every model is selectable per-project in the dashboard's Agent tab; pass the id to `AssistantProvider.Managed(model = …)` only to hard-pin it in code. **Gemini Live models require SDK 1.8.0+** — earlier SDK versions predate the Gemini protocol adapter, and a session on a `gemini-*` model will fail at connect. Reasoning effort maps per vendor: a live knob on the OpenAI 2.x line (`gpt-realtime-2`, `gpt-realtime-2.1`, `gpt-realtime-2.1-mini`); on Gemini it sets the thinking level at the next wake (`Minimal` is Google's own speed-optimized default — recommended for snappy voice agents); Grok manages its own. Despite the name, **`gpt-realtime-2.1-mini` is a reasoning model** with the full 128k context window — it is a distilled 2.1, not a newer `gpt-realtime-mini`. The older `gpt-realtime-mini` has no effort knob and a 32k window. There is **no** `Claude` or `Cascaded` provider (Anthropic ships no Realtime API). Precedence for every model-side knob is **code > dashboard > SDK default** — leave a field `null` to defer to the dashboard. ### Vendor parity notes (Gemini Live) The event vocabulary and session lifecycle are identical across vendors. A few knobs behave differently on Gemini Live, where the session config is fixed at connect time: | Surface | OpenAI / Grok | Gemini Live | |---|---|---| | Wake→listen→answer, barge-in, tools, transcripts, history, sleep-on-silence | ✓ | ✓ identical | | `say(text)` | verbatim | near-verbatim (instruction-driven) | | `cancelSpeak()` | cancels at the source | stops playback locally; the turn's remaining audio is discarded | | `updateInstructions()` | applies immediately | applies at the next wake/reconnect | | `setVoice` / `setModel` / `setReasoningEffort` | next wake | next wake | | `withinSessionMemory` (dashboard) | smart = SDK summarization | handled natively by the model's sliding window (knob is a no-op; no summarizer cost) | | `includeImage(uri)` | ✓ | ✓ | | **`sendVideoFrame(jpegBytes)` — streaming video input** | ✗ (error event) | ✓ on `gemini-3.1-flash-live-preview` — the only realtime model that can *see what the user sees* | ### Streaming video input (Gemini 3.1 Flash Live) `session.sendVideoFrame(jpegBytes)` streams a camera frame into the live conversation — the assistant comments on what the glasses see, mid-dialogue. It is gated **per-model**: only `gemini-3.1-flash-live-preview` ingests video (other models emit an `Error(kind = "video_input_unsupported")` event instead of silently dropping the frame). `glasses.camera.videoFrames()` delivers JPEG frames by default — identical on simulator and hardware — so `frame.data` feeds it directly: ```kotlin val frames = glasses.camera.videoFrames(VideoFrameConfig(frameRate = 2, resolution = Resolution.LOW)) frames.sample(1_000).collect { frame -> // ~1 fps is plenty (and cheap) glasses.assistant.activeSession?.sendVideoFrame(frame.data) } ``` At the session's low media resolution a frame costs on the order of tens of input tokens, so a 1 fps loop is inexpensive; continuous camera + duplex audio is the fragile coexistence path on real glasses, so prefer on-demand bursts over always-on streaming. **Gate your video tools with `session.modelSupportsVideoInput`.** Only video-capable models ingest frames, and the honest failure belongs in your tool so the model *speaks* it: ```kotlin tool("start_watching", "Stream the user's view so you can see it live.") { if (!session.modelSupportsVideoInput) { return@tool ToolResult.Err( "this model can't see video — switch the project to Gemini 3.1 Flash Live " + "in the dashboard's Agent settings") } // ... start the frame loop ... } ``` Even without the check, the SDK never lets a blind session pretend: a `sendVideoFrame` on an unsupported model emits `AssistantEvent.Error(kind = "video_input_unsupported")` to your app **and injects a one-time note into the conversation telling the model it cannot see** — so the agent answers "I can't watch video on this model" instead of hallucinating a description. ### Resolution: recording quality vs. agent frames The glasses run **one** camera stream, and its resolution locks at first arm (it can be re-armed after the firmware retires the stream, ~5 min). Two facts make this easy to wire well: - **Gemini tokenizes frames at the session's low media resolution regardless of what you send** — feeding it high-resolution frames costs upload bandwidth, not extra tokens, and doesn't degrade its vision. The agent adapts to whatever the stream provides. - The **first camera call's config decides the armed stream** — you program this. If your app both records video and streams to the agent, arm HIGH (make the recording-quality call shape first, or request HIGH in your `videoFrames` config): the recording keeps full quality and the agent watches the same stream for free. Arming LOW first is the one trap — a later recording is stuck at LOW until the stream retires. `camera.activeStreamInfo()` tells you what the stream is currently armed at — it's a function, and it returns `null` when nothing is armed. - Want one quality for **every** stream instead of per-call control? Set the standing override — `glasses.camera.preferredStreamConfig = PreferredStreamConfig(Resolution.HIGH)` (or the same field on `ExtentosConfig` at startup): every arm then uses it and per-call values are ignored. And if a call ever requests better quality than the armed stream can give, the SDK emits one `stream_config_conflict` warning on `runtime.events` naming the armed config and the fix — it never re-arms behind your back. Key `AssistantConfig` / builder fields: | Field | Default | Meaning | |---|---|---| | `instructions` | `""` | The full system prompt — you own all of it; the library adds nothing | | `startActive` | `false` | `true` opens the connection at `start()` (skips Dormant) | | `onWake {}` / `onSleep {}` | — | Hooks run as coroutines with the session as receiver (`onWake { say("…") }`) | | `sleepAfterSilence(Duration)` | **60s** | Auto-sleep after contiguous user silence. Pass `null` / `nil` for a session that never sleeps on its own — but consider why: a session left listening holds the microphone, bills the gateway, and reacts to whatever the wearer says to somebody else. `AssistantConfig.defaultSilenceTimeout` (iOS) / `DEFAULT_SILENCE_TIMEOUT` (Android) name the value so you can scale it rather than retype it. | | `sleepOnPhrase(phrase)` | — | Deterministic sleep phrase | | `endOnIntent` | `true` | Register the hidden model-driven `end_conversation` tool | | `greeting` | `Greeting.Default` | Auto-greeting policy (see [Greeting](#greeting)) | | `historyCap` | `100` | Local replay-buffer cap, in turns | | `historyCompaction` | `Auto` | What happens as the buffer fills (see [Memory](#memory)) | ### What language does the assistant speak and hear? There is **no language setting on the assistant** — not on `AssistantProvider`, not in the `start(provider) { … }` block, not on `AssistantSession`. What happens instead depends on which brain is serving: **Cloud models (`gpt-*`, `grok-*`, `gemini-*`).** Speech recognition happens inside the provider, and Extentos sends no language field — so **input language is auto-detected**. Output language is whatever the model decides, which means you steer it through `instructions`: ```kotlin instructions = "You are an interpreter. Detect the language the user speaks " + "and reply in the other one. Speak only the translation." ``` That is prompt-level, not an API guarantee, but for multilingual conversation it's the working mechanism — and it's why an interpreter-style app is practical on the cloud tier today. **On-device models (`local-*`).** The ears are the SDK's own on-device recognizer, which the assistant path invokes with default settings — meaning **en-US**, with no way to change it from the assistant API. On real glasses and the audio baseline that recognizer is English-only regardless (see [audio streaming](/docs/guides/audio-streaming#language-and-where-it-is-and-isnt-honored)). The bundled on-device voices are US/UK English as well. **If your product is multilingual, the local tier is not ready for it.** On-device gives you privacy and $0 inference in English. Anything else needs a cloud model, where the language handling above applies. This is a real limitation, not a configuration you're missing — don't spend a day looking for the setting. ### Swift call-shape deltas The Swift surface carries the same fields and behaviour; the spellings differ more than "field-for-field" suggests. Assume these deltas rather than transliterating the Kotlin: - **The config block is an `inout` closure, not a receiver lambda.** Kotlin's `instructions = …` is Swift's `$0.instructions = …`. - **Names differ in case.** `AssistantProvider.Managed(...)` is `.managed(...)`; `ToolResult.Ok` / `.Err` are `.ok` / `.err`. - **Tool registration labels its description.** Kotlin `tool("name", "description") { … }` is Swift `$0.tool("name", description: "…") { … }`. - **Pinning a model** looks the same on both: `AssistantProvider.Managed(model = "local-auto")` / `.managed(model: "local-auto")`. - `glasses.assistant.start(provider:_:)` is `async throws` — open-time failures throw `AssistantError` directly (Swift enums conform to `Error`, so there's no `AssistantException` wrapper to unwrap). - `sleepAfterSilence(_:)` takes a `TimeInterval` in seconds instead of Kotlin's `Duration`. - Session state is observed via `session.state`, an `ObservableState` — a `current` snapshot plus a replaying `stream` (StateFlow semantics). - The typed tool overload takes an **explicit** `schema:` parameter — Swift can't infer a JSON Schema from a type the way Kotlin does from `@Serializable`. The no-arg `tool(name, description) { ... }` form is identical, and the schema sent to the provider is the same on both platforms. ## The managed gateway is the default The assistant carries **no API key in your app**. The SDK opens a WebSocket to Extentos's managed gateway, which relays the realtime session to OpenAI on Extentos's key and meters usage. There is no `setOpenaiApiKey` — it was removed when the assistant moved to gateway-only. (There is no option to run the assistant on your own provider account — see [the gateway](/docs/concepts/ai-gateway#there-is-no-bring-your-own-key-option); today it always runs on Extentos's managed key.) All gateway, identity/attestation, metering, and credit billing live in one place — see **[the managed AI gateway](/docs/concepts/ai-gateway)**. This page won't re-derive them. **PII note:** Phase 4 events carry **verbatim transcripts** in the dev event log (`user_spoke` / `assistant_spoke`) — fine for development, but the transcript is yours to govern in production; document retention in your app's privacy policy. (The gateway meters usage but does not persist conversation content.) ## Memory The assistant has two independent memory layers, both configured on `AssistantConfig` — neither is a separate capability. ### Within-session history and compaction The SDK keeps a local replay buffer of recent turns (capped at `historyCap`, default 100) and replays it to the provider on every reconnect — so a conversation survives wake/sleep and the transparent reconnects. The buffer lives in memory; it is **not persisted to disk** (persist to your own storage if you need it across launches, then restore with `replaceHistory`). `historyCompaction` controls what happens as the buffer fills (it fires in the background near ~80% of the cap): | Policy | Behavior | |---|---| | `Auto` (default) | Summarizes the oldest ~50% of turns via a cheap chat model (`compactionModel`, default `gpt-4o-mini`) into one summary turn — the conversation continues indefinitely without silent forgetting. ~$0.001 per compaction. | | `DropOldest` | Drop the oldest turn when full. Free, lossy. | | `Custom(compact)` | Your own `suspend (List) -> List` compactor — bring your own summarizer, model, or vector-DB recall. | | `None` | No compaction; the buffer holds at `historyCap` and you manage it manually via `clearHistory` / `appendHistory` / `replaceHistory`. | Session history methods — `conversationHistory(limit)`, `clearHistory()`, `appendHistory(turn)`, `replaceHistory(turns)` — let you snapshot, wipe (e.g. `onWake { session.clearHistory() }` for a fresh-each-wake notetaker), inject app-context hints, or restore persisted turns. ### Cross-session persistent memory (v0 preview) `persistentMemory = true` loads this end-user's stored profile at session start and merges durable signal back at session end — so the agent remembers the user *across* sessions (the automatic greeting personalizes from it). - **Opt-in and consent-gated.** It stores a person's data, so it requires the end-user's consent — which you can only obtain in your app. It is therefore a **code-side switch, never a dashboard toggle**. - **Keyed per-device by default.** `memoryUserId = null` keys the profile on the SDK's attested per-device id (memory follows the device). Set it to your app's stable id for the signed-in user to make memory follow the *person* across devices and reinstalls (isolated per user on a shared device). The profile is always scoped to your project by the attestation JWT, so one app can never reach another's memory. - **Gateway-backed** unless you supply a `MemoryStore`. By default the profile lives on the Extentos backend behind the gateway. Provide a `MemoryStore` to keep profiles entirely in your own infrastructure. ## Errors and events Open-time failures throw `AssistantException` wrapping an `AssistantError` (`NoApiKey`, `AlreadyActive`, `SessionEnded`, `NotReady`, `NetworkError`, `ProviderError`) — pattern-match on `.error`. Starting a second session while one is active throws `AlreadyActive` (the runtime is **singleton-active** per `ExtentosGlasses`). Once a session is `Active`, transient errors surface as events and the session rides them out through the reconnection state machine. Full table: [error reference](/docs/reference/errors#assistant-errors-phase-4). Lifecycle flows through the shared `glasses.runtime.events` stream as `RuntimeEvent.Assistant` wrapping an `AssistantEvent` (`SessionStarted`, `SessionEnded`, `UserSpoke`, `AssistantSpoke`, `AssistantAudioStarted`, `AssistantAudioFinished`, `ToolCalled`, `ToolResultEvent`, `Reconnected`, `Error`, `WentDormant`). In the simulator these land on the **`voice`** event-log chip — assistant lifecycle, STT and TTS all share it (an `Error` climbs to the **`errors`** chip automatically). The **`ai`** chip is reserved for customer-side BYOK calls wrapped in `glasses.observability.aiCall(...)`, not the assistant runtime. Capture transcripts off this stream: ```kotlin glasses.runtime.events .filterIsInstance() .onEach { (it.event as? AssistantEvent.UserSpoke)?.let { spoke -> notes.append(spoke.transcript) } } .launchIn(scope) ``` > `glasses.conversation.*` (the Phase 3 cascaded runtime) is **removed** on current Android — use `glasses.assistant.*`. ### Knowing when the assistant has stopped talking `AssistantSpoke` is the **transcript**, and it fires when generation finished — not when the audio finished. The model streams its reply faster than real time, so `AssistantSpoke` routinely lands **seconds before** the wearer stops hearing the sentence. Transitioning a screen or calling `cancelSpeak()` on it cuts the assistant off mid-word. Use the audio pair instead. `AssistantAudioStarted` fires on the first audible chunk of a stretch of speech; `AssistantAudioFinished` fires once the queued audio has played out **and** the model has finished the turn. They strictly alternate — exactly one finish per start. ```kotlin glasses.runtime.events .filterIsInstance() .onEach { when (it.event) { is AssistantEvent.AssistantAudioStarted -> speaking.value = true is AssistantEvent.AssistantAudioFinished -> speaking.value = false else -> Unit } } .launchIn(scope) ``` Three things worth knowing: - **A turn that calls a tool and keeps talking finishes after the last segment**, not at the tool call. The model often speaks, calls a tool, then speaks again with the tool's answer — all one turn. `AssistantAudioFinished` spans the whole round trip, so a transcript you close on it carries every segment. - **A barge-in fires it immediately.** If the wearer interrupts, or you call `cancelSpeak()`, the audio genuinely stopped — so the event fires rather than leaving you waiting. - **It is an estimate, accurate to roughly a quarter second.** The phone is the player and the glasses are a Bluetooth sink, so the last stretch of the audio hop is not observable from any layer of the SDK. That is plenty for a UI transition and not enough for sample-accurate timing. `glasses.audio.speak(...)` is unaffected — it has always had `waitForCompletion`, which suspends until that clip finishes. ### Reacting to how loudly the assistant is speaking For a live voice visualiser you want a number, not an edge. `session.audioLevels()` streams the RMS of the audio currently audible, normalised `0..1`, at about 25 Hz: ```kotlin session.audioLevels() .onEach { level -> ring.amplitude = level } .launchIn(scope) ``` The one property that makes it usable: **levels are placed on the playback clock, not on the moment audio was queued.** The model streams its reply faster than real time, so a chunk handed to the player stays audible for seconds afterwards — metering at enqueue would run the ring ahead of the voice by the whole queue depth, which reads as broken rather than as missing. Levels arrive only between `AssistantAudioStarted` and `AssistantAudioFinished`, with a final `0` when playback is cut short, so a visualiser lands at rest instead of freezing mid-swell at the moment the user interrupted. It is **opt-in and free when unused** — nothing is computed or emitted until the first subscriber, and metering stops when the last one goes away. Pair it with [`glasses.audio.audioChunks`](/docs/guides/audio-streaming#it-is-safe-to-subscribe-during-a-live-assistant-session) for the wearer's own voice, and you have both sides of the conversation. That one does *not* take the microphone away from the assistant — the SDK fans a single input tap out to every consumer. Two limits, stated rather than discovered: it is **RMS, not peak**, and like `AssistantAudioFinished` it is an estimate good to a fraction of a second, because the phone is the player and a wearable is a Bluetooth sink. Right for a visualiser, not for sample-accurate metering. It is also **not available on the on-device local tier** yet — that tier's speech pump runs outside the shared core, so `audioLevels()` there completes without emitting. ## Related - [Build a voice assistant](/docs/guides/voice-assistant) — the task guide: wake phrase → tools acting on app state → vision → sleep - [The managed AI gateway](/docs/concepts/ai-gateway) — gateway default, metering, and credit billing - [Voice triggers](/docs/guides/voice-triggers) — `glasses.voice.onPhrase`, the canonical wake mechanism - [The display capability](/docs/concepts/display) — assistant tools can render on the Ray-Ban Display via `glasses.display.*` - [Capabilities](/docs/concepts/capabilities) — the full vendor-agnostic SDK vocabulary - [Error reference](/docs/reference/errors) — `AssistantError` and the no-`DisplayError` model --- source: https://extentos.com/docs/concepts/capabilities --- --- title: Capabilities description: The Extentos capability vocabulary — the vendor-agnostic SDK primitives (audio, camera, voice, assistant, display, hardware events) your handler subscribes to. type: concept platform: all related: - /docs/concepts/assistant - /docs/concepts/display - /docs/concepts/ai-gateway - /docs/concepts/transport-vs-app - /docs/vendors/meta - /docs/getting-started/with-agent - /docs/concepts/architecture - /docs/vendors --- **Camera and display need the Meta vendor module.** `com.extentos:glasses` carries no vendor SDK, so add `implementation("com.extentos:glasses-meta")` alongside it — see [install](/docs/sdk/android/install). Without it the build still succeeds and voice still works, but `capabilities.camera` is `false` and captures return errors. The SDK logs a warning at startup when it spots that combination. **A capability is a vendor-agnostic SDK primitive your handler subscribes to** — `audio.transcriptions()`, `camera.capturePhoto()`, `audio.speak()`, the assistant, display, hardware events — and the same handler code runs across every supported vendor because the transport, not your code, adapts. A *capability* in Extentos is a vendor-agnostic SDK primitive your handler code subscribes to — `audio.transcriptions()` for continuous STT, `audio.recordDiscrete()` for a silence-VAD-bounded audio clip (raw audio — transcribe it via `audio.transcriptions()` or the assistant), `audio.speak()` for TTS, `camera.capturePhoto()` for a still, `camera.videoFrames()` for a frame stream, plus hardware-event flows for thermal / hinges / call-state / lifecycle. The capability vocabulary is the contract between your handler code (which calls capability primitives) and the underlying transport (which translates those calls into platform-specific operations — Meta DAT on Meta smart glasses, Google's Jetpack Projected on Android XR). This page is the technology behind that contract: the full vocabulary, how per-vendor manifests narrow it, how platform permissions derive automatically, and why a shared vocabulary plus a standard transport interface is the design that makes the same code run across every supported smart-glasses vendor. ## The capability layer Three coordinated layers turn an abstract capability into running code on real hardware: | Layer | What it does | Owned by | |---|---|---| | **1. Capability vocabulary** | The set of abstract SDK primitives — audio, camera, speak, hardware-event flows, the toggle surface, the connection-state machine. The same on every vendor. | Extentos (the language) | | **2. Per-vendor capability manifest** | Which capabilities a specific vendor exposes — e.g., Meta Ray-Ban supports `capture_photo` and `transcription_incremental` via the DAT public toolkit but not `custom_gesture`. Some capabilities are per-DEVICE within a vendor: `display` is available on Ray-Ban Display but not Ray-Ban Meta — apps branch on `glasses.display.isAvailable`, never the model name. | Each vendor (the subset) | | **3. Transport implementation** | The code that translates an abstract `glasses.camera.capturePhoto()` call into vendor-specific API calls (Meta DAT, Google's Jetpack Projected, etc.). One transport per vendor. | Each vendor's transport (the wiring) | Your handler is written in **layer 1** — pure capability calls, no vendor names. The MCP server's `validateIntegration` tool checks your `extentos.manifest.json`'s declared capability list against **layer 2** for the target vendor — flagging anything the vendor doesn't expose. At runtime, the library's selected transport (**layer 3**) translates your calls into actual platform operations. Same code, different transport — that's how a handler written for Meta Ray-Ban can later target a vendor with a different SDK shape without rewrites. > **At runtime, your installed agent has these live.** Once Extentos's MCP server is registered with your agent, the agent calls `getPlatformInfo` for the capability catalog scoped to the current vendor, `getCapabilityGuide(feature)` for per-feature call shapes, and `getCodeExample(pattern)` for canonical compositions in Kotlin + Swift. The static tables on this page are the human-readable reference for pre-install evaluation, SEO, and out-of-context lookup; the live MCP response is authoritative when composing real handler code. ## Audio primitives The audio surface is the most-used part of the SDK. The audio primitives: | Primitive | Shape | When to use | |---|---|---| | `glasses.audio.transcriptions(config)` | continuous `Flow` (Kotlin) / `AsyncStream` (Swift) — Partial + Final | Wake-phrase matching, live captions, continuous STT | | `glasses.audio.recordDiscrete(config)` | suspending one-shot, returns `AudioRecording` (raw audio + `audioDurationMs`) | Free-form question capture — silence-VAD turns the mic off when the user pauses | | `glasses.audio.speak(text)` | suspending TTS — phone's native engine by default, or the on-device [high-quality local voice](/docs/concepts/local-models) via `SpeakConfig(voice = "kokoro")` | Speaking responses through the glasses speaker (HFP) | | `glasses.audio.cancelSpeak()` | fire-and-forget interrupt | Barge-in — kill TTS the moment the user starts speaking | | `glasses.audio.audioChunks(config)` | continuous raw chunk stream | Custom on-device STT, passthrough, or non-text audio processing | | `glasses.audio.earcon(sound, volume)` | suspending one-shot canned tone | CONFIRMATION / ERROR / NOTIFICATION / START / STOP / COMPLETE confirmations | The canonical voice-Q&A pattern composes these: register a wake phrase with `glasses.voice.onPhrase(phrase, label, stops) { ... }` (sugar over `transcriptions()` that also surfaces the phrase on the connection page and the simulator's click-to-fire panel, with automatic handler cancellation when a `stops` phrase fires), then call `speak()` to acknowledge, `recordDiscrete()` to capture the user's question, and `speak()` again for the answer. Customers needing regex / stateful matching skip `onPhrase` and subscribe to `transcriptions()` directly, optionally calling `glasses.voice.registerHint(...)` to keep the UI affordance visible. `getCodeExample(pattern: "voice_qa_assistant")` returns the full ~100-line composition in Kotlin and Swift. For new voice-AI apps, prefer the [assistant runtime](/docs/concepts/assistant) over hand-composing this loop — `onPhrase` becomes the assistant's wake trigger. ## Camera primitives | Primitive | Shape | When to use | |---|---|---| | `glasses.camera.capturePhoto(config)` | suspending one-shot, returns `Photo` (URI + width / height / format) | Vision LLM input, save-to-gallery, single-frame analysis | | `glasses.camera.captureVideo(config)` | suspending one-shot, returns `VideoClip` | Bounded clip recording | | `glasses.camera.videoFrames(config)` | continuous `Flow` / `AsyncStream` | Live vision pipelines, frame-by-frame analysis (typically LOW resolution at 2 fps for cost) | Photo URI helpers (`Photos.loadBase64(uri)`, `Photos.loadBytes(uri)`, `Photos.loadBitmap(uri)`, `Photos.mediaTypeFromUri(uri)` on Android; `Photo.loadImage()` on iOS) bridge the data-URI / file-URI scheme variance across transports — write your handler against the helpers and the same code runs on BrowserSim, LocalSim, and real Meta Ray-Ban transports. Three `videoFrames` guarantees worth knowing for streaming pipelines: - **Timestamps are strictly increasing per stream** (`timestampMs` on Android, `presentationTimeUs` on iOS). The hardware's presentation clock can rebase during internal session recovery; the SDK normalizes so your pipeline can safely assume monotonic order. Don't treat frame timestamps as wall-clock. - **Requested resolution is a ceiling, not a guarantee.** On Meta hardware the tiers are LOW 360×640 / MEDIUM 504×896 / HIGH 720×1280, but Bluetooth bandwidth drives an automatic quality ladder (resolution steps down first, then frame rate, never below 15 fps) plus adaptive per-frame compression — effective quality can change mid-stream with no error. The stream is also armed once per app session and kept warm, so the first camera use fixes the quality tier. Read `frame.width` / `frame.height` for the effective resolution. - **Starting the camera never exits `Active`.** Arming the capture session is internal; the connection state stays `Active`, with its `camera` status refining through `starting` → `ready` (`broken` means a capture genuinely failed and a reconnect is needed). Treat only `Disconnected` — or a demotion out of `Active` — as a lost connection; never tear down app resources on `!Active`. ## Speech output `glasses.audio.speak()` routes through the phone's native TTS engine (`TextToSpeech` on Android, `AVSpeechSynthesizer` on iOS) and plays audio through the glasses speaker over Bluetooth HFP/SCO — the hands-free voice profile, not A2DP music streaming (a concurrent video stream can force an audio-route change; see [audio streaming](/docs/guides/audio-streaming)). The phone is the synthesizer; the glasses are the speaker. This is intentional — TTS quality is bound by the phone engine, which matters when comparing against premium voice providers like ElevenLabs (for which you'd fetch audio yourself from the customer's handler and play through the phone speaker until direct-audio-bytes routing to the glasses lands as a future SDK feature). `cancelSpeak()` interrupts the active utterance immediately for the barge-in flow. See `getCodeExample(pattern: "barge_in_speak")` for the canonical TaskGroup / structured-concurrency pattern that cancels speak the moment a Final transcript arrives. ## The assistant runtime — voice AI For voice-driven AI, the canonical surface is the **Phase-4 assistant runtime**: `glasses.assistant.start(provider) { tool(name, description) { … } }`. The model owns wake detection, turn-taking, and intent; you write tool bodies that act on app state. It runs through Extentos's [managed AI gateway](/docs/concepts/ai-gateway) by default — no API key in your app — and you wake it from any trigger (canonically `glasses.voice.onPhrase("hey …") { session?.wake() }`). It has shipped in `com.extentos:glasses` since 1.4.0 and in the iOS SDK's `GlassesCore` product from `github.com/extentos/swift-glasses`. Full detail: **[the assistant runtime](/docs/concepts/assistant)**. The lower-level audio primitives above remain for fine-grained, non-assistant control. ## Display On **Ray-Ban Display**, `glasses.display.show { … }` renders a small UI tree (text, buttons, images, full-surface video) the wearer sees, and `glasses.display.isAvailable` tells you whether the connected device has a display so you can branch. It never throws on non-display glasses — `show()` simply no-ops. **Display support is in beta**: the full surface ships in both SDKs and is fully tested in the simulator on both platforms; on-glasses rendering on Meta Ray-Ban Display hardware is in its verification phase. Full detail: **[the display capability](/docs/concepts/display)**. ## Hardware-event flows (mostly planned) Hardware events describe the world changing: temperature, hinge state, audio routing, call state, app lifecycle, notifications, location. This is a **forward-compatible vocabulary** — the event types are defined, but **most are not yet delivered to handler code on shipping hardware.** Today the only world-state your handler actually observes is connection state, via `glasses.connection.state` (a `Disconnected` value carries `ThermalCritical` or `HingesClosed` as its cause). The browser simulator can inject the rest so you can write handlers ahead of the runtime wiring, but they don't reach a consumer API yet. What `glasses.runtime.events` (`Flow` / `AsyncStream`) emits today is the SDK's own runtime stream — toggle changes, assistant lifecycle, and logs — **not** the hardware events in the table below. | Event | What it means | Status | |---|---|---| | `connection_state_changed` | The glasses connection transitioned states | **Live** — observe `glasses.connection.state` | | `thermal_warning` | The hardware is heating up (`light` … `critical`) | Planned — today only `Disconnected(ThermalCritical)` | | `hinges_closed` | The user folded the glasses | Planned — today only `Disconnected(HingesClosed)` | | `audio_route_changed` | The Bluetooth audio route changed (A2DP ↔ HFP/SCO) | Planned — sim-injectable, no consumer API yet | | `incoming_call_detected` | The phone has an incoming call | Planned — sim-injectable, no consumer API yet | | `app_lifecycle_changed` | The phone app moved foreground / background | Planned — sim-injectable, no consumer API yet | | `phone_notification_forwarded` | An OS notification was forwarded to the glasses | Planned — sim-injectable, no consumer API yet | | `location_updated` | A configured location threshold was crossed | Planned — sim-injectable, no consumer API yet | > **Write forward-compatible handlers, but gate on what's live.** Only `connection.state` surfaces world-state today; the simulator injects the rest for testing. See the **[Hardware events guide](/docs/guides/sensor-streams)** for exactly what's subscribable now versus planned. `searchDocs(topic: 'connection_state_model')` and `searchDocs(topic: 'event_log_schema')` cover the event types and the diagnostic surface in full. ## Toggles — runtime gates the user controls Eight runtime toggles gate capabilities at runtime; the user owns them via the connection page UI; your handler reads them via `glasses.toggles.state`. They're **session-scoped** — held in memory and re-seeded to defaults on each `ExtentosGlasses.create()`, not persisted across app restarts. | Toggle | What it gates | Default | |---|---|---| | `listening_mode` | STT recognizer (`off` disables transcriptions entirely) | `wake_word` | | `camera_streaming_enabled` | Every camera primitive | `true` | | `audio_capture_enabled` | Every audio-capture primitive | `true` | | `transcription_enabled` | Declared for the STT layer — **not yet enforced** (transcripts flow regardless today) | `false` | | `privacy_mode` | The super-toggle — kills every camera + mic capability | `false` | | `battery_save_mode` | Clamps `videoFrames` to LOW + 2 fps | `false` | | `voice_confirmations` | Auto-earcons around voice-initiated handler calls — **not yet enforced** | `true` | | `audio_video_coexistence_policy` | HFP / A2DP conflict policy — **cooperative, not yet enforced** | `prefer_video` | `searchDocs(topic: 'toggles')` covers each toggle's enforcement status and gotchas in depth. ## Permission derivation Each capability declares its platform-permission requirements once, centrally. The MCP server's `getPermissions(capabilities, platform)` returns the exact set: | Capability | Android (manifest) | iOS (Info.plist) | |---|---|---| | `capture_photo`, `capture_video`, `video_frames` | `CAMERA`, BT permissions | `NSCameraUsageDescription`, BT keys | | `record_audio`, `audio_chunks` | `RECORD_AUDIO`, BT permissions | `NSMicrophoneUsageDescription`, `NSSpeechRecognitionUsageDescription`, BT keys | | `transcription_incremental` | `RECORD_AUDIO`, `BLUETOOTH_*` | `NSMicrophoneUsageDescription`, `NSSpeechRecognitionUsageDescription`, BT keys | | `speak` | none of your own | none of your own | | `display` (Ray-Ban Display) | BT permissions only — rendering is outbound over the existing DAT connection | BT keys | | Any **vendor** connection (camera / display) | `BLUETOOTH_CONNECT`, `BLUETOOTH_SCAN` (the library declares these) | `NSBluetoothAlwaysUsageDescription`, MWDAT plist keys | The "BT permissions" column above means the library-declared Bluetooth pair, which is merged into your manifest automatically — it is never something you add. **A voice-only app needs no Bluetooth Info.plist key and no MWDAT keys on iOS**: it reaches the mic and speaker through the OS's own audio routing, not through the vendor SDK. `speak` needs nothing of your own on either platform — TTS runs through the platform engine. The authoritative per-capability derivation is [permissions](/docs/concepts/permissions), or `getPermissions` for your exact list. Update `extentos.manifest.json`'s `capabilities` array when you add a primitive to your handler — the next `getPermissions` call surfaces the new keys; `validateIntegration` confirms they land in the manifest and Info.plist. A developer who never had to think about iOS speech-recognition entitlements still ends up with a correct `NSSpeechRecognitionUsageDescription` because the capability said "transcription_incremental" and the toolchain knew what that meant on iOS. ## The declared footprint — one source, three surfaces Your `capabilities` array is the **single source of truth** for what your app uses. `generateConnectionModule` writes it once (into the manifest *and* the compiled `ExtentosConfig.usedCapabilities`) from a single input, so the three surfaces that need it can never disagree: - **OS permissions** — derived as above (`getPermissions`). - **The connection page** — one tile per declared capability, lit when the connected glasses provide it and dimmed when they don't (e.g. Display on a non-display Ray-Ban). Declare nothing and the page shows no capabilities section — it never guesses a default set ("declare nothing, show nothing"). - **Your dashboard** — the project Overview shows your declared capabilities as "what the app uses," with a per-capability *tested* marker once it's been exercised in the simulator. The footprint is **declared, not detected.** The SDK doesn't scan your code to infer it — static analysis can't reliably tell which capabilities a build actually uses (it would silently miss the camera and ship a runtime crash), which is why every platform's permission model is declare-then-use, not derive. Keep the list current by re-running `generateConnectionModule` with the capability when you add it. ## When a capability is absent One build ships to every device, and the hardware a user has connected changes minute to minute — glasses in a case, ordinary earbuds instead, nothing at all. (Those are two different states, and `capabilities` distinguishes them — see below.) Absent capabilities are therefore a normal production state, not an error path, and the SDK treats them that way. **Three rules:** 1. **Capability calls are always callable and always safe.** An absent capability never throws. Discrete calls (`camera.capturePhoto()`) return a typed error naming the cause; stream calls complete rather than hanging; display calls are a silent no-op. 2. **Your control flow is the same in development and production.** Only the *signal* changes — loud in dev (event-log warning, simulator banner), quiet in prod (a counter on your dashboard). 3. **Ask before you call.** `glasses.capabilities` is the flat, per-connection truth: `camera`, `microphone`, `speaker`, `display`. ```kotlin if (glasses.capabilities.camera) { val photo = glasses.camera.capturePhoto() } ``` That guard is the whole pattern. Branch on the capability, never on a model name — a new device or a new vendor then needs no code change from you. ### What this looks like with no glasses connected A camera app running on a phone whose glasses aren't there keeps working — but **"no glasses" is two different states**, and they report differently: | | Glasses on | Paired, in their case | Never paired | |---|---|---|---| | `transportChosen` | `REAL_META` | `REAL_META` | `SYSTEM_AUDIO` | | Voice agent (turn-taking, tools, barge-in) | ✅ | ✅ over earbuds or the phone | ✅ over earbuds or the phone | | `capabilities.microphone` / `.speaker` | `true` | `true` | `true` | | `capabilities.camera` | `true` | **`true`** | `false` | | `connection.state` | `Active` | not `Active` | `Active` | | `camera.capturePhoto()` | photo | `NotConnected` | `PlatformError("system_audio_no_camera")` | | Recovery | — | automatic when they're worn | needs an app restart | **`capabilities` reports what the resolved transport's vendor provides, not what is powered on right now.** A Bluetooth bond survives the glasses being switched off, so the vendor arm still claims and `capabilities.camera` stays `true` with them in the case. That's why it is not a presence check: for "can I capture *right now*," pair it with `connection.state`. ```kotlin val canCapture = glasses.capabilities.camera && glasses.connection.state.value is GlassesState.Active ``` You lose exactly the capabilities the hardware was providing, and nothing else. See [choose your path](/docs/getting-started/choose-your-path) for how this shapes an integration. ## Validation and capability negotiation Three MCP tools work together to keep a handler aligned with what the target vendor can actually do: - **`getPlatformInfo({ glasses: "" })`** — returns the vendor's capability manifest. Which audio / camera / hardware-event primitives it supports. Which it doesn't. Which are GA versus preview. - **`validateIntegration()`** — checks the project against the vendor's capability list for the configured target. Flags capabilities the manifest declares that the vendor doesn't expose, missing permissions, dependency drift. Returns structured errors the agent can act on. - **`getProductionChecklist()`** — late-stage gate. Verifies permissions are wired, credentials are set, foreground-service hints are present for continuous-capture flows, edge cases are handled. Run before shipping. In the typical agent flow, `getPlatformInfo` is the first call (discovery), `validateIntegration` runs after every structural change (correctness gate), and `getProductionChecklist` runs once the developer is preparing to ship. The capability layer is what lets these tools be *deterministic* — they answer yes/no against the manifest rather than guessing. ## The transport contract Each vendor provides a `GlassesTransport` implementation — the code that translates abstract capability calls into platform-specific API calls. The interface is identical across vendors: ```text GlassesTransport ├─ connect(deviceId) ├─ capturePhoto(config) ├─ captureVideo(config) ├─ recordAudio(config) ─► returns AudioRecording (transcript + bytes) ├─ videoFrames(config) ─► continuous stream ├─ audioChunks(config) ─► continuous stream ├─ transcriptions(config) ─► continuous stream (Partial + Final) ├─ speak(text, config) ├─ cancelSpeak() ├─ earcon(sound, volume) └─ events ─► transport state, hardware alerts, errors ``` A vendor that supports a capability implements the corresponding method against its SDK. A vendor that doesn't support a capability either fails fast at `validateIntegration` (preferred — caught before runtime) or surfaces a typed Result error at runtime (fallback for capabilities that depend on runtime state, like permissions). This is the engineering boundary that makes "add a vendor = implement the interface" a clean, bounded task — not a sprawling rewrite. For the deep dive on how transports work and what each implementation does, see **[transport vs app simulation](/docs/concepts/transport-vs-app)**. ## Why a shared vocabulary is the right design Five reasons the capability layer is shaped this way: 1. **Vendor portability is structural, not negotiated.** Because handler code is written against the capability primitives instead of vendor-specific calls, an app targeting Meta Ray-Ban today can target a future vendor by switching the transport — no code rewrite. The portability is a property of the architecture, not something an individual developer has to engineer per project. 2. **Validation is deterministic.** The capability vocabulary is finite and the per-vendor manifest is a known set. `validateIntegration` answers "does this app run on this vendor?" with a yes/no plus structured errors. That determinism is what lets an AI agent confidently mutate the integration — every change has a clear validation outcome. 3. **Permissions derive automatically.** Each capability declares its platform-permission requirements once, centrally. Add `transcription_incremental` to your handler and `extentos.manifest.json`'s capabilities array; the iOS Info.plist and the Android manifest get the right keys without the developer learning what `NSSpeechRecognitionUsageDescription` is. 4. **Simulators are honest.** The browser simulator and the on-device local simulator both implement the same capability vocabulary the production transports do. There's no "simulator-only" or "production-only" capability — anything you can run in simulation runs in production, and vice versa. 5. **New vendor onboarding is bounded work.** Adding a vendor is "implement the `GlassesTransport` interface against the new SDK and declare the capability manifest." No SDK shape changes, no migrations for existing developer handlers, no new MCP tools. Android XR is the worked example: a genuinely different app model (a projected phone activity, not a Bluetooth session) reached the same capability surface without moving it. Brilliant Labs then exercised the other half — **degradation** — with the first device that has a display and no speaker at all (Brilliant Frame), plus a round panel and hardware that cannot traverse focus. None of those needed a new concept; they are absent capabilities and panel geometry, handled by the rules already here. ## Targeting multiple vendors The capability layer is what makes this technically possible. The strategic story — what supported and roadmap vendors are, when each ships, how to think about portability when planning your app — lives on **[/docs/vendors](/docs/vendors)** as the section landing page, with the Extentos build guide at **[/docs/vendors/meta](/docs/vendors/meta)** and neutral platform pages for [Android XR](/docs/ecosystem/platforms/android-xr), [Brilliant Labs](/docs/ecosystem/platforms/brilliant-labs) and [Apple](/docs/ecosystem/platforms/apple) in the [ecosystem landscape](/docs/ecosystem). A future page will cover the **runtime semantics** of multi-vendor apps — graceful degradation when a target vendor doesn't expose a capability the handler uses, fallbacks, validation policies for "this app must run on at least N of these vendors." That's deferred until a second vendor is shipping, when the rules will be concrete enough to commit to. For now: target one vendor at a time, let `validateIntegration` confirm fit, and rely on the capability layer to keep your handler code portable when the time comes. ## Frequently asked questions ### Can I add a new capability that isn't in the vocabulary? Not directly — the capability vocabulary is a coordinated contract across the SDK, the validator, both simulators, and every vendor's transport. Extending it is an Extentos library change. If a capability you need doesn't exist, the path is to file an issue describing the use case; it's added when there's a cross-vendor primitive worth standardizing. For app-specific behavior that doesn't need a new SDK primitive — custom AI processing, business logic, network calls — that's just code in your handler class. The handler is your code; you can do anything in it. See `searchDocs(topic: 'custom_handlers')` for the canonical handler shape and `searchDocs(topic: 'custom_extensions')` for the framing of how to compose around missing primitives (e.g., custom on-device STT against `audio.audioChunks()`). ### How does the manifest know what permissions to derive on iOS vs Android? The capability vocabulary has a per-platform permission map baked in. `transcription_incremental` declares `NSSpeechRecognitionUsageDescription` on iOS and `RECORD_AUDIO` on Android, plus the Bluetooth keys both platforms need. The MCP server's `getPermissions` tool returns the current set given the capability list; `generateConnectionModule` writes them into the manifest and Info.plist on initial scaffold. ### Are streams metered the same as one-shot calls? The library emits `stream.started` and `stream.stopped` events into the structured event log regardless of one-shot vs continuous shape. Browser-simulator session minting requires a free email-only account (Google or email + password — see [pricing](/docs/resources/pricing)); once linked, sessions and the events they emit are unlimited. The discovery, validation, guidance, and search MCP tools work anonymously; besides simulator minting, only the `generateConnectionModule` scaffold step and the account-scoped project tools need the linked account. ### How is `glasses.audio.transcriptions()` different from "Hey Meta"? "Hey Meta" is Meta's system-level wake word — third-party apps can't hook it. `glasses.audio.transcriptions()` is the continuous-transcript primitive: the glasses microphone captures audio, streams it to the phone via Bluetooth HFP/SCO, on-device recognition on the phone (Vosk on real hardware; the platform recognizer on the dev path) emits Partial + Final transcripts, and your handler matches strings against them to detect wake phrases. No "Hey Meta" prefix; the wake word *is* whatever string your handler matches against. See [vendors/meta](/docs/vendors/meta) for the full audio-architecture story and `searchDocs(topic: 'voice_ux_guide')` for phrase-design rules. ### Does the capability layer add runtime overhead? Negligible. The library is a thin translation between abstract calls and the vendor's SDK. There's no extra serialization, no extra IPC, no proxy layer. Capability indirection is compile-time (the transport dispatch is a single method dispatch); runtime is direct SDK calls. ## Related - **[Architecture](/docs/concepts/architecture)** — how the capability layer fits into the broader system (MCP, library, backend, transports) - **[Transport vs app simulation](/docs/concepts/transport-vs-app)** — the deep dive on how each transport implements the capability vocabulary - **[Vendors](/docs/vendors)** — the strategic multi-vendor story; per-vendor capability manifests - **[Vendors: Meta Ray-Ban](/docs/vendors/meta)** — the GA target's full capability manifest - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install the MCP server and see capabilities in action --- source: https://extentos.com/docs/concepts/display --- --- title: The display capability description: Render UI on the Ray-Ban Display with the glasses.display builder DSL in Kotlin and Swift — text, images, buttons, media, and Neural Band input. Gated per device. Beta. type: concept platform: all vendor: meta related: - /docs/guides/display - /docs/concepts/capabilities - /docs/concepts/assistant - /docs/reference/errors - /docs/vendors/meta --- **Display needs the Meta vendor module.** `com.extentos:glasses` carries no vendor SDK, so add `implementation("com.extentos:glasses-meta")` alongside it — see [install](/docs/sdk/android/install). Without it your build still succeeds and voice still works, but `display.isAvailable` is `false` and display calls are silent no-ops. The SDK logs a warning at startup when it spots that combination. > **Display support is in beta.** The full display surface — the builder DSL, select/back input, per-device gating — ships in both SDKs and is fully tested in the simulator on both platforms. On-glasses rendering on Meta Ray-Ban Display hardware is in its verification phase; APIs are stable, with refinements expected as hardware verification completes. The **display capability** (`glasses.display.*`) renders UI on a glasses screen — today, the **Ray-Ban Display**. You compose a node tree with a small builder DSL (Kotlin or Swift) and hand it to `glasses.display.show { ... }`; the connected device renders it. On real hardware the tree is delivered over Meta DAT's `mwdat-display` path; in the [browser simulator](/docs/concepts/transport-vs-app) the same tree renders in the browser tab. There is no WebView, no canvas, no local-bitmap drawing — you declare a tree of containers and leaves, and the platform renders it. > **Ships in both SDKs; fully sim-testable on both.** `glasses.display.*` ships in the Kotlin and Swift SDKs, and the DSL, input routing, and event log are exercised end-to-end in the simulator on both platforms. On-glasses rendering via Meta DAT (`mwdat-display` on Android, `MWDATDisplay` on iOS, DAT 0.8.0) ships on BOTH platforms. Neither has been confirmed against real Meta Ray-Ban Display hardware yet — the tree, the translation and the input routing are exercised end to end in the simulator and in unit tests on both, and the on-glasses render is the step still outstanding. The local-media roots (`image(photo)` / `video(clip)`, `prepareVideo`, `forgetHostedVideo` / `forgetHostedImage`) ship on both platforms. See [the same tree in Swift](#the-same-tree-in-swift) for the Swift shapes. ## Per-device, never-throw Display is a capability of the specific **device model**, not the vendor. Within Meta Ray-Ban, the Ray-Ban Display has a screen and the Ray-Ban Meta / Oakley models do not. So you branch on a runtime signal, **never on a model name**: ```kotlin val isAvailable: Boolean = glasses.display.isAvailable // true only on a display-capable device if (glasses.display.isAvailable) { glasses.display.show { /* ... rich display UX ... */ } } else { glasses.audio.speak("Here's what I found …") // graceful voice fallback } ``` The model is **never-throw**: `show()` on a display-less device is a safe silent no-op — degradation never crashes and never diverges between the simulator and hardware. The gate is **live**: in the simulator, switching the device model pushes the new profile to the running app with no reconnect, so the same session can see `isAvailable` flip from `true` to `false` mid-run. Good UX guards rich display features with `isAvailable` and falls back to speech. ## The builder DSL `show(onBack) { ... }` runs against a `DisplayRootScope`. You nest **containers** and place **leaves**; `show {}` replaces the entire display each call (there are no incremental updates — own your view state app-side and re-render whole views on every transition). `show()` and `clear()` are suspend. **Containers:** - `column(...) { ... }` — a vertical stack. - `row(...) { ... }` — a horizontal stack. Both take `gap`, `padding`, `mainAlign`, `crossAlign`, `background`, and an optional `onClick` that makes the whole container tappable. A clickable container needs an explicit `id` for a stable, agent-targetable handle — otherwise it falls back to a positional id (`box-0`, `box-1`, …) that shifts when you reorder the tree. **Leaves:** - `text(text, style, color, align)` - `image(url, size, cornerRadius, align)` — a **remote** image fetched by URL. - `button(text, style, icon, align, id, onClick)` — pass an explicit `id`. - `icon(name, style, align)` — a vendor-specific icon by name. **Root-only, full-surface** (these render as the *entire* display and must be the sole top-level node — nesting them inside a `column` / `row` is a **compile error**, and the builder scope inside a container simply doesn't expose them): - `video(url)` — a remote MP4 by URL. - `video(clip)` — a locally-recorded `VideoClip` (the SDK hosts it; see below). - `image(photo)` — a locally-captured `Photo` rendered full-surface (the SDK hosts it). ```kotlin glasses.display.show(onBack = { scope.launch { glasses.display.clear() } }) { column(gap = 8, padding = 12) { text("My notes", style = TextStyle.HEADING) column( background = Background.CARD, onClick = { scope.launch { openNote(note) } }, id = "note-${note.id}", // stable identity-based id ) { text(note.title, style = TextStyle.BODY) text(note.preview, style = TextStyle.CAPTION, color = TextColor.SECONDARY) } } } ``` ### Enum vocabulary | Enum | Cases | |---|---| | `TextStyle` | `HEADING`, `BODY`, `CAPTION` | | `TextColor` | `PRIMARY`, `SECONDARY` | | `ButtonStyle` | `PRIMARY`, `SECONDARY`, `OUTLINE` | | `Background` | `NONE`, `CARD` | | `Alignment` | `START`, `CENTER`, `END`, `STRETCH` | | `ImageSize` | `ICON`, `FILL` | | `CornerRadius` | `NONE`, `SMALL`, `MEDIUM` | ### The same tree in Swift The Swift SDK builds the identical node tree — the tree vocabulary, wire JSON, and root-normalization rules are core-owned, so a tree composed in Swift renders exactly like its Kotlin twin. The builder differs in a few idioms: - `show(onBack:content:)` and `clear()` are `async` — `await glasses.display.show { ... }`. - **Containers:** one `flexBox(direction:)` method (`.column` / `.row`) instead of the `column` / `row` sugar — same `gap`, `padding` (an `EdgeInsets`), `mainAlign`, `crossAlign`, `background`, and optional `onClick`. - **Leaves:** `text(_:style:color:align:)`, `image(url:size:cornerRadius:align:)`, `button(_:style:icon:align:onClick:)`, `icon(_:style:align:)` — and `video(url:)` on the root scope. - **Ids auto-assign.** There is no explicit `id:` parameter — tappable nodes get stable-within-a-show ids automatically each `show`. ```swift await glasses.display.show(onBack: { Task { await glasses.display.clear() } }) { root in root.flexBox(direction: .column, gap: 8, padding: EdgeInsets(top: 12, right: 12, bottom: 12, left: 12)) { col in col.text("My notes", style: .heading) col.button("Open latest") { Task { await openLatest() } } } } ``` **Android-only:** the `column` / `row` container sugar (iOS uses `flexBox` directly). Everything else here works the same on both platforms, including the local-media roots `image(photo)` / `video(clip)`, `prepareVideo`, and `forgetHostedVideo` / `forgetHostedImage`. ## Additive-light design rule The Ray-Ban Display is a **waveguide that adds light** to the world — it can only brighten, never occlude. **Dark pixels render transparent.** So design **bright-on-dark**: bright text and strokes read well; dark fills disappear into the scene. The same applies double to video: bright-on-dark footage reads, dark scenes mostly vanish. The simulator composites the display additively (a screen blend) so previews match the optics. One consequence to know: a **dead or unfetchable media URL emits no light**, so the panel shows *nothing* even while the tree-level checks look green — the one failure additive optics can't surface visually. See [Errors and observability](#errors-and-observability). ## Content that does not fit Panels differ by more than size. Meta Ray-Ban Display is 600x600 and scrolls; Android XR display glasses are 450x394; Brilliant Halo is a 256px **circle** whose usable area is the inscribed square, and neither Brilliant device can scroll at all — Halo has one button and Frame has one tap gesture. So there is no row count that is right everywhere, and an app that picks one is wrong on every panel it was not tuned on. You do not pick one. Every container takes an `overflow` policy, and the default answers it per device: | Policy | What happens | | --- | --- | | `auto` (default) | Scrolls where the wearer can travel continuously, pages where they cannot. Nothing is lost either way. | | `scroll` | Insist on scrolling. Falls back to paging where scrolling is impossible — never to losing content. | | `paginate` | Insist on paging, even where scrolling is available. | | `clip` | Keep what fits, drop the rest. The only policy that loses content, and it applies only because you named it. | On a panel that scrolls, the SDK hands the whole tree over and the device's own view host handles travel. On a panel that cannot, the SDK splits the tree into pages that fit and appends a page button to each one — so **page travel rides selection**, the one action every display device can make. Your `onClick` map never sees it, and the wearer can see that there is more. Pages break at the root container's direct children and no deeper, so a card never loses its title to a page break. A child too tall for any page gets a page to itself rather than being skipped. The last page wraps to the first, because on a one-button device there may be no way back. The same split runs in the simulator and on glass, from the same panel table, so a page break you see in the browser is the page break the wearer gets. Switch the simulator's device to Brilliant Halo to see your own layout page. If you want more than the default gives you, branch on the connected device (`glasses.device.type`) and render a different tree — the default path is meant to be good, not to be the only option. ## Full-surface video and photos, with hosting `video(...)` is the one true-motion channel third-party content has — declarative tree updates are link-bound (a few Hz), but a root video plays on the glasses themselves. Playback auto-starts once the tree is sent and stops on the next `show()` / `clear()`. The glasses can only fetch an `http(s)` URL — a local `data:` / `file:` clip is rejected on hardware. So: - `video(url)` / `image(url)` — for media **already hosted** at an `http(s)` URL. - `video(clip)` / `image(photo)` — for a clip or photo you just **captured**. The SDK uploads it to a copy on Extentos's media host at `show()` time and renders the hosted URL; you never hand-roll hosting. A photo hosts in well under a second; a multi-MB clip takes a few seconds (you'll see a brief "Preparing…" only for video). ```kotlin // Show a just-captured photo full-surface — the SDK hosts the local capture. glasses.display.show(onBack = { scope.launch { showList() } }) { image(photo) } ``` The hosted copy intentionally **outlives** the on-screen display, so re-showing the same clip/photo reuses it instead of re-uploading. There is **no time-based expiry** — the copy lives at an unguessable public URL on Extentos's media host until your app removes it. That makes releasing it your responsibility, not a background job's: - `forgetHostedVideo(clip)` / `forgetHostedImage(photo)` — tear down the hosted copy. Idempotent and safe (a no-op if never hosted, already forgotten, or no display-video delivery is wired); does **not** touch the local file. - `prepareVideo(clip): Boolean` — speculatively pre-upload a clip (e.g. when you render a list of recordings) so a later `show { video(clip) }` plays instantly. Best-effort, idempotent, uploads at most once. There's no `prepareImage` sibling — photos host fast enough that there's no latency to move off the critical path. Build the `Photo` / `VideoClip` the **same way** for `show` and for `forget`, so both resolve to the same hosted copy. Use platform-hosted test clips for quick verification: `https://extentos.com/sample-videos/mountain-bike.mp4` and `https://extentos.com/sample-videos/run-in-city.mp4`. **Privacy: this is end-user media on our infrastructure.** `image(photo)` / `video(clip)` is the only SDK path that stores end-user content on Extentos. The copy is reachable by anyone holding the (unguessable, never-published) URL, because the glasses fetch it unauthenticated. Two consequences: wire `forgetHostedImage` / `forgetHostedVideo` into your app's delete path so a deleted photo really goes away, and disclose the upload in your own privacy policy. To avoid it entirely, host the media yourself and pass `image(url)` / `video(url)`. Full detail: [security § hosted display media](/docs/resources/security#hosted-display-media--the-one-content-path). ## Input model The display has two input rails, both driven the same way in the simulator (`injectInput`) as on hardware: - **Selection** — a Neural Band index-pinch on hardware (`injectInput(action: "select", targetId)` in the sim) fires the node's registered `onClick`, routed by node `id`. Give every clickable node a stable id. - **Back** — the back gesture (Neural Band mid-pinch on hardware; `injectInput(action: "back")` in the sim) fires the current show's `onBack`. Back is **view-contextual**, so register it **per `show`**: a detail view's `onBack` typically re-renders the browse view; a root view's `onBack` clears the display; a `show` with no `onBack` ignores the gesture. `clear()` drops the current handlers so a stale `onBack` can't fire after a wipe. `onClick` / `onBack` are plain `() -> Unit` — launch a coroutine inside them for the suspend `show()` / `clear()` calls. The slide (focus highlight) moves device/sim-side; your app doesn't see focus changes, but the focus target is observable in the event log. Keep an explicit Back **button** alongside `onBack` for discoverability. The back-gesture wiring is **sim-verified**; the real-hardware mid-pinch lands with the DAT display validation pass. Agent-driven verification reads the live tree with `getDisplayState` and drives input with `injectInput` — see the [MCP tools reference](/docs/reference/mcp-tools). ## Permissions The display capability contributes **zero** extra Android permissions and no iOS plist keys — rendering is outbound over the **existing** DAT connection (the standard Bluetooth set every glasses app already declares). If your app already connects to the glasses, you can render to the display — on real hardware the scaffold also flips the `DAM_ENABLED` manifest flag for you when Display is in your declared capability footprint (the Extentos SDK ships it off by default). ## Errors and observability There is **no `DisplayError` return type.** `show()` never throws; display delivery problems surface as **runtime log events** on `glasses.runtime.events` — `display.video_delivery_failed`, `display.image_delivery_failed`, `display.video_error` — and on the **`errors`** chip in the simulator event log (they are warn severity). A dead media URL additionally raises a visible "video failed to load" placeholder in the simulator viewport and a `display_error` entry on the **`errors`** chip — when a display flow looks green but shows nothing, check that chip first. App code can react to `display.video_error` (e.g. fall back to a text view). See the [error reference](/docs/reference/errors#there-is-no-displayerror). ## Related - [Render on the display](/docs/guides/display) — the task guide: capture → store → browse, plus the assistant-driven variant - [Capabilities](/docs/concepts/capabilities) — the full vendor-agnostic SDK vocabulary and the per-device capability model - [The assistant runtime](/docs/concepts/assistant) — assistant tools can drive `glasses.display.*` by voice - [Error reference](/docs/reference/errors) — the no-`DisplayError` model and the runtime log events - [Vendors: Meta Ray-Ban](/docs/vendors/meta) — the device family and the Ray-Ban Display --- source: https://extentos.com/docs/concepts/local-models --- --- title: On-device models description: Run the assistant's brain on the phone instead of the cloud — Automatic device-fit selection, the model ladder, the end-user download path, and on-device neural voices. Kotlin and Swift. type: concept platform: all vendor: meta related: - /docs/concepts/assistant - /docs/concepts/ai-gateway - /docs/sdk/android/install - /docs/sdk/ios/install - /docs/reference/errors --- Selecting a `local-*` model runs the assistant's brain **on the phone** instead of a cloud provider. Speech goes to on-device speech recognition, an open-weights model generates the reply, and the phone speaks it. No per-token cost, no conversation content leaving the device, and it keeps working in airplane mode. Everything else stays identical. The same `glasses.assistant.start(provider) { tool(...) { ... } }` block, the same tools, the same events. Only the brain changes. > **Both platforms, published.** Android: `com.extentos:glasses-local`. iOS: the `GlassesLocal` product from `swift-glasses`, which raises the package's deployment floor to **iOS 17** (see [install](/docs/sdk/ios/install)). Both have shipped since 1.11.0/1.11.1 — match the version to whatever `com.extentos:glasses` you're on and never mix majors. The code below is Kotlin; the Swift surface matches name-for-name, minus the `Context` parameter. ## Automatic Phones differ enormously in how much memory they can give one app, and a developer shipping to thousands of users cannot know which model each of them can run. **Automatic** answers that: pick it once, and the SDK resolves the best model each device can actually sustain. ```kotlin // From code — wins over the dashboard, and needs no account: val session = glasses.assistant.start( provider = AssistantProvider.Managed(model = "local-auto") ) { /* instructions, tools */ } // Or set it once in the dashboard: Agent → Model → "Automatic" (id: local-auto) ``` `Managed` is the provider type for every model — **the model id picks the vendor**, and a `local-*` id runs entirely on the phone with no network call. (It was called `OpenAi` before 2.0.0, which made exactly this line confusing; the old name still compiles, deprecated.) Precedence is code > dashboard > SDK default. At session start it reads the device, picks the largest model that device class supports, and **falls back to the cloud when no local model qualifies**. The developer picks one setting; every user gets the best brain their hardware allows. ### How it decides Two numbers, both read at runtime. **Automatic never looks up your phone model** — there is no device database to go stale. | Reading | Role | |---|---| | **Device class budget** — physical RAM × a sustainable share | Decides which model. Stable, so the same phone always gets the same answer. | | **Free memory right now** | A guard only. Can step the choice *down* for one session, never up. | Choosing on free memory alone would hand the same phone a 4B model on a fresh boot and a 1.7B with a browser open. The class budget keeps it deterministic. **The floor is quality, not fit.** Automatic never selects a model below Qwen 3 1.7B, even on a phone with room for one. Smaller models load fine but call tools unreliably, and an assistant that cannot act on "save a note" is a worse product than the cloud. The smaller rungs stay available if you pin them explicitly. ### It tells you what it chose Automatic is the one model id whose served model differs from the selected one, so it always reports the outcome: ```kotlin glasses.runtime.events.collect { ev -> val e = (ev as? RuntimeEvent.Assistant)?.event as? AssistantEvent.AutoModelResolved ?: return@collect println("serving ${e.servedModelId} local=${e.isLocal} reason=${e.cloudReason}") } ``` `servedModelId` is always a concrete model — never `local-auto`. It also appears in the simulator event log as `assistant.auto_model_resolved`. That report is what lets a privacy-sensitive app enforce its own policy: read the resolution, and if it is not local, refuse or warn in your own code. The SDK never decides that for you. ## The model ladder All rungs are Qwen open-weights models. Memory requirements differ per platform because Android runs GGUF quantisations and iOS runs 4-bit MLX checkpoints — **the same model needs different amounts of memory on each.** | Model | Android | iOS | Automatic may pick it | |---|---|---|---| | `local-qwen3-0.6b` | ~1.0 GB | ~0.5 GB | No — below the tool-calling floor | | `local-qwen25-1.5b` | ~1.6 GB | ~1.0 GB | No — below the tool-calling floor | | `local-qwen3-1.7b` | ~1.8 GB | ~1.2 GB | **Yes — the floor** | | `local-qwen3-4b` | ~3.4 GB | ~2.9 GB | Yes | | `local-qwen3-8b` | ~6.4 GB | ~5.8 GB | Yes — beyond current phones | Pinning a specific id serves exactly that model or refuses honestly. The SDK never silently substitutes a different one. ## The ladder keeps moving That table is a snapshot of one release, not a fixed specification. Three things underneath this tier are moving at once — the open-weights models, the runtimes that execute them, and the phones — and Extentos treats keeping current with all three as continuing work rather than a feature that shipped once. Most of what makes an on-device tier good is not the weights. Two changes give the shape of it. **The floor rung was re-quantised.** Qwen publishes only an 8-bit GGUF for the 1.7B while shipping 4-bit for the 4B and 8B, which left Automatic's floor carrying double the weights of every rung above it — a 5.4 GB Android phone fell back to the cloud while a 4 GB iPhone ran the same model happily. Re-sourcing the 4-bit build from ggml-org, the llama.cpp project's own org, cut peak memory from 2250 MB to 1584 MB on a Galaxy A25 with generation speed unchanged (20.1 s against 20.2 s). Mid-range Android phones moved from cloud fallback to running locally, under the same model id. **Ten voices came out of a file users already had.** Kokoro is multi-speaker and the downloaded model always contained all eleven speakers; the synthesiser was passing a hardcoded index and discarding the rest. One parameter reached them, with no extra download, memory or load time. Neither was new technology. Both were using what was already there properly, which is most of what improving an on-device tier looks like, and neither asked an app to change a line: | What changes underneath | What your app does about it | |---|---| | A rung is re-quantised or re-sourced | Nothing in code. The id keeps its meaning and the requirement usually drops, so more of your users' phones qualify. Each build is a distinct SHA-pinned file, so a device holding the old one fetches the new build on the same deliberate-download terms as any model. | | A rung is added, or becomes eligible for Automatic | Nothing on `local-auto` — it re-resolves at every session start, so devices climb as the ladder grows. A pinned id is never moved for you. | | Turn-taking, tool calling or barge-in improves | Nothing. The conversation state machine lives in the shared Rust core, so iOS and Android get the same behaviour in the same release. | | A memory figure moves | Nothing, unless you print it — read it from the SDK rather than from here. | **Take the numbers from the SDK, not from this page.** `models()` and `deviceFit(...)` report what the build you actually shipped requires; the table above is the human-readable copy of the same catalog and can lag a release. ## The download path Weights are **gigabytes**, and the SDK will never download them behind your user's back — not at install, not at launch. Fetching is always the end user's deliberate act, through a surface in your app. Ask what this device needs, then offer it: ```kotlin val choice = ExtentosLocalTier.autoChoice(context) if (choice.isLocal) { // Already installed and running on-device — nothing to do. } else if (choice.downloadTarget != null) { // This phone CAN run a local model; it just isn't downloaded yet. ExtentosLocalTier.download(context, choice.downloadTarget!!) { done, total -> progress = done.toFloat() / total // byte-accurate } } else { // No local model fits this device — the cloud serves. Nothing to install. } ``` Swift is the same without `context`: ```swift let choice = ExtentosLocalTier.autoChoice() if let target = choice.downloadTarget { try await ExtentosLocalTier.download(modelId: target) { progress = $0 } } ``` Supporting calls on both platforms: `models()` (ids, display names, sizes, installed state), `delete(...)`, and a required-vs-available readout for building a manual picker — `deviceFit(context, modelId)` on Android, `deviceFit(for:)` on Swift. (Android takes a `Context` on these; Swift doesn't. That's the only shape difference.) **Until the weights land, the cloud serves.** A brand-new user on a perfectly capable phone still starts on the managed gateway and moves on-device once their download finishes. That transition is normal, not an error, and Automatic reports it each session. ## On-device voices The local tier speaks through the phone's built-in text-to-speech by default. Adding `glasses-local-voice` (Android) or `GlassesLocalVoice` (iOS) upgrades that to **Kokoro**, an on-device neural voice with eleven speakers: | | | |---|---| | **US female** | `kokoro`, `kokoro-bella`, `kokoro-nicole`, `kokoro-sarah`, `kokoro-sky` | | **US male** | `kokoro-adam`, `kokoro-michael` | | **UK female** | `kokoro-emma`, `kokoro-isabella` | | **UK male** | `kokoro-george`, `kokoro-lewis` | All eleven live in **one downloaded model**, so switching speaker costs no extra download, memory or load time. Every one has an audio preview in the dashboard's Agent section. The voice is independent of the brain — it pairs with any local model, and costs roughly 300 MB of memory plus some reply latency on older phones. Until its model is downloaded, the system voice serves. It isn't assistant-only, either. Any text your app wants read aloud can use it directly, with no assistant session anywhere: ```kotlin glasses.audio.speak(articleSummary, SpeakConfig(voice = "kokoro-emma")) ``` ```swift _ = await glasses.audio.speak(articleSummary, config: SpeakConfig(voice: "kokoro-emma")) ``` The text can come from anywhere — a cloud model's reply, app content, a notification. Routing follows the same per-call contract as the assistant's mouth: an unknown voice id or a not-yet-downloaded model serves the system voice for that call, `cancelSpeak()` interrupts a local synthesis just like platform TTS, and direct speak shares the assistant's loaded engine rather than holding a second copy in memory. ## On-device transcription **Android.** The same module has an ear as well as a mouth. `ExtentosSttModel` downloads a streaming recognition model (**73 MB**), after which `audio.transcriptions()` runs entirely on the phone: ```kotlin if (!ExtentosSttModel.isInstalled(context)) { ExtentosSttModel.download(context) { done, total -> showProgress(done, total) } } ``` It is the same contract as the voice in every respect — size known up front, byte progress, resumable, deletable, and never downloaded behind the user's back. Until it is present the system recogniser serves, so this is an upgrade rather than a requirement. Both engines come from the one library the module already vendors, so the voice and the ear cost a single dependency and a single `ExtentosLocalVoice.register(context)` line. What it buys over the system recogniser: no Android version floor, no network at recognition time, and no per-word cost. **There is no iOS equivalent, and that is not a gap.** Handing recorded audio to the system recogniser needs a trick on Android — `SpeechRecognizer` will accept a pipe instead of the microphone, but only from Android 13, and a given device's recogniser may still refuse. `SFSpeechAudioBufferRecognitionRequest` has always taken buffers, so iOS reaches the same destination with no model to download and no version floor. Both platforms end up in the same place; only Android needs a second route to get there. There is also a category of device where it matters more than that. Glasses that connect as a Bluetooth headset — Ray-Ban Meta, and ordinary earbuds — simply *become* the phone's microphone, so the system recogniser hears them with no help from the SDK. Glasses built around their own BLE link, like Brilliant Halo, never become a system audio device at all: their microphone audio arrives as bytes in your process, and anything built on "listen to the phone's mic" is deaf to them. You don't have to care which kind you're holding. `audio.transcriptions()` reads from the SDK's own audio stream rather than the phone's microphone, so the call is identical either way: ```kotlin glasses.audio.transcriptions().collect { transcript -> if (transcript is Transcript.Final) note(transcript.text) } ``` That is the whole point of the seam — [supported vendors](/docs/vendors) reach the audio path whether or not they speak Bluetooth audio, and your code does not branch on it. ## Setup **Android** — add the modules and register at app start: ```kotlin // build.gradle.kts implementation("com.extentos:glasses-local:2.8.0") // on-device models implementation("com.extentos:glasses-local-voice:2.8.0") // optional: Kokoro // Application.onCreate — BOTH must run before ExtentosGlasses.create(...) import com.extentos.glasses.local.ExtentosLocalTier import com.extentos.glasses.localvoice.ExtentosLocalVoice ExtentosLocalTier.register(applicationContext) ExtentosLocalVoice.register(applicationContext) ``` **iOS** — add the products and register: ```swift // Package.swift → .product(name: "GlassesLocal", package: "swift-glasses") // .product(name: "GlassesLocalVoice", package: "swift-glasses") import GlassesLocal import GlassesLocalVoice // Before Extentos.create(...) — in your App's init() ExtentosLocalTier.register() ExtentosLocalVoice.register() ``` **`register()` must run before you create the SDK handle** on both platforms. **On Swift this bites in a way that looks correct.** Stored-property initializers run *before* the enclosing `init()` body — so the common `let glasses = Extentos.create(...)` property form calls `create()` first and your `register()` second, no matter where in `init()` you put it. Declare the property without an initializer and assign it inside `init()`, after the `register()` calls: ```swift @main struct MyApp: App { let glasses: any ExtentosGlasses // no initializer here init() { ExtentosLocalTier.register() ExtentosLocalVoice.register() glasses = Extentos.create(config: ExtentosConfig( usedCapabilities: [.microphone, .speaker] )) } // … } ``` Registering afterwards leaves the assistant with no on-device brain for that process — which on iOS means it quietly serves from the cloud instead (see the callout below). Without `register()`, a `local-*` model has no on-device brain available, and **the two platforms diverge**. **Forgetting `register()` fails loudly on Android and quietly on iOS.** Android refuses the local model and says so. iOS serves the request from the cloud instead — so an app built for on-device privacy silently sends conversation audio to a provider, with nothing in the UI to show it. If privacy is the reason you chose the local tier, don't rely on the platform to tell you: subscribe to `AssistantEvent.AutoModelResolved` and check `isLocal` at the start of every session, refusing to proceed when it's `false`. That check is the same on both platforms. **`AutoModelResolved` fires for `local-auto`**, whose whole job is to report which model it picked. If you pin a concrete id (`local-qwen3-4b`) you don't get that report — so **use `local-auto` if you need to verify.** It reports every session and is the only path that does; pinning a concrete id buys determinism and gives up the receipt, and there is no session property that reports the served model. Don't assume silence means local. ## In the simulator The browser simulator serves local models **from Extentos infrastructure**, so every rung works immediately with no download. Because the model runs server-side there, the phone's memory does not constrain the choice — Automatic resolves to the best model the simulator hosts, which is how you exercise the local path on any device. That is the one deliberate difference between simulation and hardware, and it is contained at the substrate. Everything above it — resolution reporting, tools, events, voices — behaves identically. ## Cost and privacy On-device serving bills nothing per token and sends no conversation content to Extentos or any model vendor. Cloud fallback rides the [managed gateway](/docs/concepts/ai-gateway) like any other model and is billed under the model that actually served, never under `local-auto`. Because Automatic reports its resolution every session, "is this conversation staying on the device?" is a question your app can answer precisely rather than assume. ## Further reading [Realtime voice AI bills for every second it listens](/blog/local-models-zero-cost-voice) — why an always-on assistant's cost scales with engagement, the conversation-loop architecture that makes on-device turn-taking hold together, and the measurements behind the memory numbers on this page. --- source: https://extentos.com/docs/concepts/permissions --- --- title: Permissions description: Extentos derives Android permissions and iOS Info.plist keys from the capabilities your handler uses, plus the always-required Bluetooth and Meta DAT keys. type: concept platform: all related: - /docs/concepts/architecture - /docs/concepts/capabilities - /docs/vendors/meta - /docs/getting-started/with-agent - /docs/mcp-server/tools/guidance --- Permissions are derived, not hand-maintained. Every SDK capability your handler subscribes to (`capture_photo`, `transcription_incremental`, `record_audio`, etc.) declares its own Android manifest permissions and iOS Info.plist keys, and the MCP server's `getPermissions` tool returns the exact set for your declared capability list. The agent calls `getPermissions`, gets the structured output, and writes the manifest and Info.plist for you — no developer needs to memorize that `transcription_incremental` requires `NSSpeechRecognitionUsageDescription` on iOS or that continuous-capture flows require `FOREGROUND_SERVICE` on Android. This page is the consolidated map: which capabilities require which keys, plus the always-required Bluetooth foundation and Meta DAT entitlements that every Ray-Ban Meta integration needs. ## How permission derivation works Three layers cooperate: 1. **`extentos.manifest.json`'s `capabilities` array declares what SDK primitives your handler uses** — `capture_photo`, `transcription_incremental`, `speak`, `video_frames`, etc. 2. **The derivation logic** (in `mcp-server/src/tools/util/permissions.ts`) maps each capability name to its required permissions per platform. 3. **`getPermissions`** returns the full set, plus structured details (Android manifest entries, iOS plist key/value/reason tuples, foreground-service declarations, NotificationListenerService stubs). The agent calls `getPermissions({ capabilities, platform })` whenever the declared capability list changes. The MCP server returns: ```json { "android": { "permissions": ["android.permission.CAMERA", "android.permission.BLUETOOTH_CONNECT", "..."], "manifestEntries": ["", "..."], "foregroundService": { "required": true, "types": ["camera"], "declaration": "", "devInstructions": "subclass GlassesForegroundService …" }, "notificationListener": { "required": false, "declaration": null, "devInstructions": null }, "minimumSdk": 31, "compileSdk": 35, "targetSdk": 34 }, "ios": { "plistKeys": [ { "key": "NSCameraUsageDescription", "value": "...", "reason": "capture_photo / capture_video block ..." }, { "key": "NSBluetoothAlwaysUsageDescription", "value": "...", "reason": "Required by Meta DAT SDK ..." } ] }, "metaDat": { "scopes": ["glasses.connection", "glasses.camera.photo", "..."], "registrationRequired": true, "registrationSteps": ["..."] }, "summary": "Android, 5 permissions (min SDK 31, target 34). Meta DAT registration required, 3 scopes." } ``` `generateConnectionModule` writes the manifest and Info.plist using this output during initial scaffolding. `validateIntegration` re-checks alignment after every change to your declared capabilities. > **At runtime, your installed agent has this live.** Once Extentos's MCP server is registered with your agent, the agent calls [`getPermissions({ capabilities, platform })`](/docs/mcp-server/tools/guidance) and gets the exact Android permissions, manifest entries, foreground-service declarations, and iOS Info.plist keys scoped to your declared capabilities — no manual lookup needed. The static derivation tables below are the human-readable reference for pre-install evaluation, SEO, and out-of-context lookup; the live `getPermissions` response is authoritative when wiring a real project. ## Android permissions ### Always required These are the foundation — every Extentos app on Android requests them, regardless of which capabilities you declare: | Permission | Why | |---|---| | `android.permission.BLUETOOTH_CONNECT` | Connect to paired vendor glasses over Bluetooth. `com.extentos:glasses` declares it in its own manifest for every app — you don't add it. A voice-only app on the system-audio transport never exercises it (the OS routes the audio). A **camera or display app must have it granted before `ExtentosGlasses.create(...)`**: the vendor arm of `TransportChoice.Auto` reads the bonded-device list, which this permission gates. Once the Meta transport is running the SDK requests it at runtime through your `activityProvider`, but that is too late to influence the transport already chosen. | | `android.permission.BLUETOOTH_SCAN` | Discover the paired device on the system bonded list. Declared by the library. | | `android.permission.INTERNET` | The managed gateway, the browser simulator, and telemetry. **You declare this one** — unlike the Bluetooth pair above, it is not in the library's manifest. An app that runs entirely on local models and sends no telemetry can omit it **after** the weights are on the device — the download itself is a network call, so an app that ships without `INTERNET` can never fetch a model in the first place. | ### Derived from camera primitives | Capability | Android permission(s) | Note | |---|---|---| | `capture_photo` | `android.permission.CAMERA` | | | `capture_video` | `android.permission.CAMERA` | + `RECORD_AUDIO` only if your handler's `VideoConfig` requests audio. | | `video_frames` | `android.permission.CAMERA` + `FOREGROUND_SERVICE` + `FOREGROUND_SERVICE_CAMERA` | Continuous stream; survives backgrounding only with a foreground service. `FOREGROUND_SERVICE_CAMERA` is required on Android 14+. | ### Derived from audio primitives | Capability | Android permission(s) | Notes | |---|---|---| | `record_audio` | `android.permission.RECORD_AUDIO` | One-shot capture | | `audio_chunks` | `android.permission.RECORD_AUDIO` + `FOREGROUND_SERVICE` + `FOREGROUND_SERVICE_MICROPHONE` | Continuous raw audio stream. `FOREGROUND_SERVICE_MICROPHONE` is required on Android 14+. | | `transcription_incremental` | `android.permission.RECORD_AUDIO` + `FOREGROUND_SERVICE` + `FOREGROUND_SERVICE_MICROPHONE` | Continuous STT stream. `FOREGROUND_SERVICE_MICROPHONE` is required on Android 14+. | | `assistant_runtime` | `android.permission.RECORD_AUDIO` + `FOREGROUND_SERVICE` + `FOREGROUND_SERVICE_MICROPHONE` | The Phase-4 assistant holds the mic for the life of a session — same set as continuous transcription. | | `speak` | (none) | TTS doesn't need a manifest permission — mediated through the platform's `TextToSpeech` engine | ### Derived from hardware events Hardware events your handler subscribes to via `glasses.runtime.events` and pattern-matches the variants from: | Event | Android permission(s) | Notes | |---|---|---| | `thermal_warning`, `hinges_closed`, `audio_route_changed`, `app_lifecycle_changed`, `connection_state_changed` | (none) | Platform / library mediated, no app-side permission | | `location_updated` | `android.permission.ACCESS_FINE_LOCATION` | Geofence-style flows | | `phone_notification_forwarded` | `android.permission.BIND_NOTIFICATION_LISTENER_SERVICE` | System-managed; **cannot be requested programmatically** — see below | | `incoming_call_detected` | `android.permission.READ_PHONE_STATE` | | ### Foreground-service requirement for continuous capture **Microphone-type and camera-type services differ, and `getPermissions` reflects that.** The library ships `GlassesForegroundService` with `foregroundServiceType="microphone"` and declares it in its own manifest — for a mic-only footprint you don't declare anything, you just call `GlassesForegroundService.start(context)`. A **camera**-type service is yours: subclass `GlassesForegroundService` (or write your own) and declare it, because the library can't declare a camera service on behalf of apps that don't use the camera. That's why `getPermissions` returns a `declaration` stub for camera footprints and none for mic-only ones. Capabilities that capture continuously (`video_frames`, `audio_chunks`, `transcription_incremental`) require `FOREGROUND_SERVICE` because the capture continues across app backgrounding — Android 14+ enforces foreground-service types per category (camera, microphone). `getPermissions` returns the required `foregroundServiceType` set in the response so your agent can configure the service correctly. See `searchDocs(topic: 'concurrency_modes')` for the background-survival model. ### `NotificationListenerService` for phone notifications If your handler subscribes to `phone_notification_forwarded` events, the library's `ExtentosNotificationListenerService` must be declared in your manifest: ```xml ``` `getPermissions` returns this declaration verbatim under `notificationListener.declaration` when it's needed. Critically, `BIND_NOTIFICATION_LISTENER_SERVICE` is a system-managed permission — your app cannot request it via the runtime permission API. The user must enable it manually: > *Settings → Notifications → Notification access → toggle your app on* Use `NotificationAccessHelper.openSettings(context)` from the library to deep-link the user to the right screen. ### SDK version requirements | Field | Default | Source | |---|---|---| | `minimumSdk` | 31 (Android 12) | `VERSION_INFO.android.minimumSdk` | | `compileSdk` | 35 | `VERSION_INFO.android.compileSdk` | | `targetSdk` | 34 | `VERSION_INFO.android.targetSdk` | These come from a single source of truth (`mcp-server/src/tools/data/version.ts`) so library bumps don't drift. `minSdk` and `compileSdk` are real floors — take them from `getPermissions` rather than hardcoding a guess. **`targetSdk` is not a requirement**: the library declares none, so the 34 above is the value this advice was derived against, not a value to copy into your build. Yours is your own choice — see [install](/docs/sdk/android/install). ## iOS Info.plist keys ### Always required | Key | Privacy string default | Why | |---|---|---| | `NSBluetoothAlwaysUsageDescription` | "Required to connect to your glasses over Bluetooth." | Required by Meta DAT SDK for **all** glasses connections | ### Derived from camera primitives | Capability | iOS plist key | Privacy string | |---|---|---| | `capture_photo` / `capture_video` | `NSCameraUsageDescription` | "Used to capture photos and video from your glasses." | | `video_frames` | `NSCameraUsageDescription` | (same) | | `capture_video` requesting audio | also `NSMicrophoneUsageDescription` | (same as record_audio) | ### Derived from audio primitives | Capability | iOS plist key(s) | Notes | |---|---|---| | `record_audio` | `NSMicrophoneUsageDescription` + `NSSpeechRecognitionUsageDescription` | One-shot silence-VAD capture; returns raw audio (no auto-STT — the speech key is derived for the whole audio-capture family) | | `audio_chunks` | `NSMicrophoneUsageDescription` + `NSSpeechRecognitionUsageDescription` | Continuous raw audio (the speech key is derived for the whole audio-capture family) | | `transcription_incremental` | `NSMicrophoneUsageDescription` + `NSSpeechRecognitionUsageDescription` | Phone-side recognition via `SFSpeechRecognizer` | | `assistant_runtime` | `NSMicrophoneUsageDescription`, `NSSpeechRecognitionUsageDescription` | The Phase-4 assistant. Add `UIBackgroundModes: ["audio"]` if it must keep listening while backgrounded. | | `speak` | (none) | TTS via `AVSpeechSynthesizer` — no entitlement needed | ### Derived from hardware events | Event | iOS plist key(s) | Notes | |---|---|---| | `location_updated` | `NSLocationWhenInUseUsageDescription` | Geofence-style flows | | All others (thermal, hinges, audio-route, lifecycle, call, notifications) | (none) | Platform / library mediated | ### Privacy strings `getPermissions` returns each plist key with three fields: `key`, `value` (the default privacy string), `reason` (which SDK capability triggered it). Defaults from `mcp-server/src/tools/util/permissions.ts`: | Key | Default privacy string | |---|---| | `NSCameraUsageDescription` | "Used to capture photos and video from your glasses." | | `NSMicrophoneUsageDescription` | "Used to capture audio from your glasses for voice commands." | | `NSSpeechRecognitionUsageDescription` | "Used to transcribe voice commands from your glasses." | | `NSLocationWhenInUseUsageDescription` | "Used to deliver location-aware glasses experiences." | | `NSBluetoothAlwaysUsageDescription` | "Required to connect to your glasses over Bluetooth." | You can override these strings in your Info.plist — the App Store review process strongly prefers app-specific reasons over generic ones. ## Meta DAT-specific iOS Info.plist keys These are **not** derived from your declared capabilities — they're required by every app that uses a **vendor transport**, regardless of which camera/display capabilities it declares. `generateConnectionModule` writes them once during initial scaffolding. A **voice-only** app needs none of them: it has no Meta DAT dependency at all, and its microphone and speaker are reached through the phone's own Bluetooth audio routing. Skip this whole section if you're on the [voice path](/docs/getting-started/choose-your-path). ### `MWDAT` dictionary ```xml MWDAT MetaAppIDYOUR_META_APP_ID ClientTokenYOUR_CLIENT_TOKEN TeamIDYOUR_APPLE_TEAM_ID AppLinkURLSchemeyourapp DAMEnabled ``` `AppLinkURLScheme` is a custom URL scheme (e.g. `yourapp://`), **not a universal link**. Meta DAT uses it for the registration callback from the Meta companion app. **Set `DAMEnabled` explicitly — leaving it out breaks camera and audio apps on real hardware.** Meta's default when the key is *missing* is `true`, and DAM runs an on-glasses readiness probe that **reaps the device session about ten seconds after it starts** on any model without the display web-app runtime — which is every current non-display model. Set it `true` only if you use the display; `false` otherwise. `generateConnectionModule` emits the right value from your declared capabilities. Full detail: [Info.plist reference](/docs/sdk/ios/info-plist). ### `CFBundleURLTypes` Must include the same scheme: ```xml CFBundleURLTypes CFBundleURLSchemes yourapp ``` ### `LSApplicationQueriesSchemes` ```xml LSApplicationQueriesSchemes fb-viewapp ``` Required to query whether the Meta AI / Meta View companion app is installed. ### `UISupportedExternalAccessoryProtocols` ```xml UISupportedExternalAccessoryProtocols com.meta.ar.wearable ``` Declares Meta's external-accessory protocol — required for the BLE link. ### `UIBackgroundModes` ```xml UIBackgroundModes bluetooth-central bluetooth-peripheral external-accessory ``` Those three are the **vendor-transport** set — a camera or display app needs them. A **voice-only** app needs none of them, but if the assistant must keep running while the app is backgrounded it needs `audio` instead: ```xml UIBackgroundModes audio ``` Add `audio` to the array as well if your app is both. Required to maintain the glasses connection across app backgrounding. ## Worked example — a wake-phrase photo-capture handler A handler that subscribes to `audio.transcriptions()` for the wake phrase, calls `camera.capturePhoto()`, forwards to a vision LLM, and `audio.speak()`s the result. Capability declarations in the manifest: ```text manifest.capabilities = [ "transcription_incremental", // subscribe to audio.transcriptions() "capture_photo", // camera.capturePhoto() "speak" // audio.speak() ] ``` `getPermissions({ capabilities, platform: "android" })` returns: | Permission | Source | |---|---| | `android.permission.BLUETOOTH_CONNECT` | always | | `android.permission.BLUETOOTH_SCAN` | always | | `android.permission.INTERNET` | always | | `android.permission.CAMERA` | `capture_photo` | | `android.permission.RECORD_AUDIO` | `transcription_incremental` | | `android.permission.FOREGROUND_SERVICE` | `transcription_incremental` (continuous) | | `android.permission.FOREGROUND_SERVICE_MICROPHONE` | `transcription_incremental` (Android 14+) | → 7 permissions. `foregroundService.required: true` (continuous transcription). `notificationListener.required: false`. `getPermissions({ capabilities, platform: "ios" })` returns these plist keys: | Key | Source | |---|---| | `NSBluetoothAlwaysUsageDescription` | always | | `NSCameraUsageDescription` | `capture_photo` | | `NSMicrophoneUsageDescription` | `transcription_incremental` | | `NSSpeechRecognitionUsageDescription` | `transcription_incremental` | → 4 plist keys. Plus the always-required Meta DAT-specific keys (`MWDAT` dict, `CFBundleURLTypes`, `LSApplicationQueriesSchemes`, `UISupportedExternalAccessoryProtocols`, `UIBackgroundModes`). ## Runtime permission requests vs manifest declarations Declaring permissions in the manifest / Info.plist isn't the same as having them granted at runtime: - **Android:** `CAMERA`, `RECORD_AUDIO`, `ACCESS_FINE_LOCATION`, `READ_PHONE_STATE`, and `BLUETOOTH_CONNECT` (on Android 12+) are **dangerous permissions** — declared in the manifest, but also requested via `ActivityCompat.requestPermissions` at runtime. The library's `ExtentosConnectionPage` handles the runtime flow during pairing, and once a vendor transport is running the SDK requests `BLUETOOTH_CONNECT` and `CAMERA` itself through your `activityProvider`. **Neither covers an app with no connection page** — a voice app owns its `RECORD_AUDIO` request outright, and a camera app must grant `BLUETOOTH_CONNECT` before `ExtentosGlasses.create(...)` for the vendor arm to resolve at all. See the [voice-assistant bootstrap](/docs/guides/voice-assistant#1-bootstrap). - **iOS:** plist keys *declare* the permission and provide the privacy string the OS shows to the user. The OS prompts on first use (e.g., the first time the camera is accessed). The library handles this; you don't need a separate request flow. - **`BIND_NOTIFICATION_LISTENER_SERVICE` on Android** is special: declared in the manifest, but **cannot be requested programmatically**. The user must toggle it on manually in system Settings. Use `NotificationAccessHelper.openSettings(context)` to deep-link them there. ## Common gotchas ### Android 12 (API 31) Bluetooth split Pre-Android 12, `BLUETOOTH` and `BLUETOOTH_ADMIN` were normal permissions auto-granted at install. Android 12 split them into runtime-requested `BLUETOOTH_CONNECT` and `BLUETOOTH_SCAN`. Extentos targets API 31+ — the new permissions are required (no legacy fallback); the library handles the user prompt during pairing. ### iOS `NSBluetoothAlwaysUsageDescription` even if your app doesn't expose Bluetooth in the UI Meta DAT's BLE connection requires this key. Even if your app appears purely visual to the user, the absence of this key causes the DAT registration flow to crash silently on iOS 13+. ### Audio on `capture_video` `VideoConfig` lets you request an audio track on the captured video; if you do, `RECORD_AUDIO` (Android) / `NSMicrophoneUsageDescription` (iOS) is required too. The default is video-only — only request audio if your handler actually needs it. ### Foreground service types on Android 14+ Android 14 enforces `foregroundServiceType="camera"` or `="microphone"` per continuous-capture flow. `getPermissions` returns the right type set; if you write the manifest by hand, make sure the service declaration includes the matching type attribute. ## Frequently asked questions ### Why does Extentos derive permissions instead of letting me write them? Three reasons. (1) Permissions drift the moment a handler changes which capability it subscribes to — adding `transcription_incremental` requires `NSSpeechRecognitionUsageDescription`, and forgetting to add it manifests as a confusing runtime crash. (2) Platform requirements evolve (Android 12 BT split, Android 14 foreground-service types) — centralized derivation lets the library track them. (3) AI agents can't reliably memorize platform permission rules; `getPermissions` is a deterministic primitive they can call. ### Can I add permissions Extentos doesn't derive? Yes. The `getPermissions` output is the *minimum* set Extentos needs for the declared capabilities. Your app can add more permissions for features outside the SDK (analytics, push notifications, etc.) by editing the manifest / Info.plist directly. `validateIntegration` flags missing required permissions but doesn't complain about extras. ### What's the difference between `BLUETOOTH_CONNECT` and `BLUETOOTH_SCAN` on Android? `BLUETOOTH_CONNECT` is for connecting to an already-paired device. `BLUETOOTH_SCAN` is for discovering nearby devices. Extentos requests both because the pairing flow scans for the glasses on first run, then connects to the bonded device on subsequent runs. ### Why is `NSBluetoothAlwaysUsageDescription` required and not `NSBluetoothPeripheralUsageDescription`? Apple deprecated `NSBluetoothPeripheralUsageDescription` in iOS 13 in favor of `NSBluetoothAlwaysUsageDescription`. Meta DAT v0.6+ requires the modern key. ### Does my app need to declare every key Meta's documentation lists? It depends on the path. The Meta DAT-specific keys (`MWDAT` dict, `LSApplicationQueriesSchemes`, `UISupportedExternalAccessoryProtocols`, `UIBackgroundModes`) are mandatory for any app that uses a **vendor transport** — that is, one declaring camera or display. A **voice-only** app has no DAT dependency at all and needs none of them; its microphone and speaker are reached through the phone's own Bluetooth audio routing. ### What happens if I miss a permission? - **Android dangerous permissions** (CAMERA, RECORD_AUDIO, etc.) — the runtime request fails silently; the library logs a `permission.denied` event into the structured event log; the affected capability's result is a typed failure. **On the capture path that failure is `CaptureError.PlatformError`, not `PermissionDenied`** — `CaptureError.PermissionDenied` is declared but not currently emitted, so don't hang your "ask for the camera again" recovery on it (see [photo capture fails](/docs/troubleshooting/photo-capture-fails)). Audio calls do return `AudioError.PermissionDenied`. - **iOS plist keys** — the OS terminates your app the moment it tries to access the gated API. There's no graceful fallback — fix the plist. - **`BIND_NOTIFICATION_LISTENER_SERVICE`** — declared but not granted means `phone_notification_forwarded` events never reach your handler. The library logs the gap; the user has to enable it in system Settings. `validateIntegration` checks alignment between your declared capabilities and your manifest / Info.plist, and flags missing keys before testing. ## Related - **[Architecture](/docs/concepts/architecture)** — how the SDK, library, and platform fit together; permissions are part of the boundary - **[Capabilities](/docs/concepts/capabilities)** — the vocabulary that drives permission derivation - **[Vendors: Meta Ray-Ban](/docs/vendors/meta)** — the Meta DAT-specific Info.plist keys, registration flow, audio architecture - **[`getPermissions` tool](/docs/mcp-server/tools/guidance)** — the MCP tool the agent calls to retrieve the current permission set - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — how the agent wires permissions during initial scaffolding --- source: https://extentos.com/docs/concepts/projects --- --- title: Projects description: How Extentos identifies a project across Android and iOS — the Extentos app id (config.appId) decoupled from the store bundle id, and how the same app id shares one project across platforms. type: concept platform: all related: - /docs/concepts/sessions - /docs/getting-started/with-agent - /docs/concepts/architecture --- A **project** in Extentos is one app — your Android version and your iOS version live under the same project. The dashboard at [extentos.com/projects](https://extentos.com/projects) groups them automatically, and the agent's `createSimulatorSession` calls flow into the right project without you doing anything special — *if* you follow one convention. This page explains exactly how project identity works, what makes two platforms join under one project, what to do when they don't, and what each piece of stored data means. ## TL;DR - Each project has a **Project ID** (a reverse-DNS identifier, e.g. `com.example.myapp`). - The Project ID is the **Extentos app id** you scaffold with (`config.appId`) — it is **decoupled from your store bundle id**. It defaults to your bundle id but does not have to equal it. - Give both platforms the **same Extentos app id** and they share one project — two platform chips, one card, zero manual setup. This works **even when the store bundle ids differ** (Android `applicationId` ≠ iOS bundle identifier), which is common on apps already shipped to both stores. - The id each shipped app actually broadcasts is preserved per row, so the Settings page always shows the truth. ## How project identity works Project identity is the **`appPackage` you pass to `generateConnectionModule`** — recorded in `extentos.manifest.json` and baked into the app as `config.appId` (iOS: the `EXTENTOS_APP_ID` Info.plist key; Android: the `appId` field of `ExtentosConfig`). The SDK reports this id for agent config, telemetry, and sounds. **It is independent of the store bundle id** — you can set it equal to your bundle id (the default expectation) or to any shared value you choose. This value is sent to the backend as `projectInstallId`. The backend's get-or-create logic looks for an existing saved sim under `(your account, this projectInstallId, this platform)`; if one exists, it's returned; otherwise a new one is minted. **The dashboard groups all of an account's sessions by `projectInstallId` — one card per unique value.** If the project doesn't have a manifest yet (you've called `createSimulatorSession` before `generateConnectionModule`), the MCP server falls back to: 1. Android: `rootProject.name` from `settings.gradle.kts`, prefixed as `android:` (e.g. `android:my-app`) 2. iOS: no fallback — anonymous mint, no project grouping Once `generateConnectionModule` writes the manifest, future mints use the manifest's `appPackage`, so the project identity becomes stable. **Two different things are easy to confuse here.** Your **project identity** (`projectInstallId`, from the `appPackage` you scaffold with) is a grouping label — it is what puts Android and iOS under one dashboard card, and it is not secret. Your **project key** (`EXTENTOSProjectKey` / `EXTENTOS_PROJECT_KEY`, a `pk_…` value) is a credential — it authenticates your app to the managed AI gateway and bills usage to your account. This page is about the first one. ## Sharing one project across Android + iOS Give both platforms the **same Extentos app id** (the `appPackage` you scaffold with): - Scaffold Android with `appPackage = com.example.myapp`. - Scaffold iOS with `appPackage = com.example.myapp`. Both apps then broadcast that id to Extentos, so `createSimulatorSession` from each platform produces the **same `projectInstallId`**: - First mint (e.g. Android): a new project row is created with `projectInstallId = com.example.myapp`. - Second mint (e.g. iOS): the get-or-create lookup finds the existing project, and the iOS row joins under the same `projectInstallId`. - Dashboard: ONE project card with two platform chips (Android · iOS). **This holds even when your store bundle ids differ.** Because the Extentos id is `config.appId` and not the store identifier, an app whose Android `applicationId` is `com.co.app` and iOS bundle id is `com.co.App` (different, and locked by the stores) still shares one project — you just give both the same Extentos `appPackage`. Integrating Extentos is itself a store release, so you pick the shared id at integration time at no extra cost. Matching the `appPackage` is the mechanism. ## Linking a project (recovery path) The same-id approach above is what you want in almost every case. But if you've **already shipped Extentos** on both platforms under different, store-locked ids — and can't re-release to unify them — you'd otherwise be stuck with two cards. For that case there's **Link**, on a project's Settings tab: 1. On the project you want to be canonical, open Settings → Project identifier → **Link project…** 2. Pick the other project. Its install-id is registered as an **alias** of this project, and its simulators move here. Unlike a cosmetic relabel, the alias is resolved **server-side**: the linked platform's shipped app keeps broadcasting its original id, and Extentos maps that id to the canonical project for agent config, sounds, and analytics. The real per-platform id is preserved and shown in Settings. This is the recovery path — matching the `appPackage` up front is simpler and needs no linking. ## What's stored on each session row There are three identifier fields on every session row, each with a distinct role: - **`projectInstallId`** — the **dashboard grouping key** = the `appPackage` (`config.appId`) at mint time. Controls which card a session appears under. - **`platformInstallId`** — the id the shipped app actually broadcasts, set on insert and never modified. Equals `projectInstallId` in the normal case; differs only when a separate install-id has been linked to this project. Surfaces in Settings → Project info → per-platform identifiers when it diverges. - **`appLabel`** — the **friendly display name** shown on the dashboard card and the project header. Cosmetic, editable from Settings → Project info → Display name. ## Per-platform truth in Settings Because `platformInstallId` is preserved per row, a project stays honest about the id each platform actually broadcasts. On the project's Settings page: - **Project ID** displays the canonical `projectInstallId` (the grouping value). - **Per-platform identifiers** appears as a separate section *only* when at least one platform's `platformInstallId` differs from the Project ID. It lists each diverging platform with its actual identifier, e.g.: - `Android: com.example.android-original` - `iOS: com.example.ios-original` If your identifiers match (the happy path), this section is hidden — the project ID alone is the complete picture. ## Editing the project name The Project ID is **read-only on the dashboard** — your app's source is the source of truth. Editing the dashboard's stored `projectInstallId` would just create a mismatch with the next mint (which derives the value from the manifest again). The **display name** (`appLabel`) is editable on the Settings tab. Saving it propagates atomically across every platform row of the project, so Android and iOS rows display the same label. ## What the agent does `createSimulatorSession` is **get-or-create** under the hood. The first call from a project mints; subsequent calls return the existing saved sim with `status: "resumed"`. Rotating the session ID (URL hijack recovery, a deliberate fresh identity) is a two-step act: `deleteSimulatorSession({ sessionId })`, then mint again. There is no one-call force-fresh — a silent replace left device registrations pointing at apps that were already gone. When the agent mints from a second platform with a matching identifier, the second-platform row simply joins the existing project — no special tool call, no flag, no agent prompt required. The agent just calls `createSimulatorSession` for the iOS or Android project as normal. ## Reference - Settings UI lives at [extentos.com/projects/[id]/settings](https://extentos.com/projects). - Backend endpoints: - `PATCH /api/projects/[projectId]` updates the display name across all rows. - `POST /api/sessions/[id]/delete` soft-deletes a session — the only per-sim lifecycle action. Rotating an ID is delete, then mint. - Schema columns on `simulator_sessions`: `project_install_id`, `platform_install_id`, `app_label` (plus the rest of the row). --- source: https://extentos.com/docs/concepts/protocol --- --- title: Protocol overview description: The Extentos simulator wire protocol — WebSocket roles, hardware-ready gating, the inject envelope, event log, and voice proxy. The SDK speaks it for you. type: concept platform: all related: - /docs/concepts/transport-vs-app - /docs/concepts/sessions - /docs/concepts/architecture - /docs/mcp-server/tools/simulation --- You almost never need this page. The protocol here is the **WebSocket wire format between the SDK's `BrowserSimTransport` and the Extentos backend** during simulator mode — the SDK speaks it for you. It matters only when you're debugging a simulator session (via [`getEventLog`](/docs/mcp-server/tools/simulation)) or reasoning about *why* the simulator behaves exactly like hardware. On real glasses (`RealMetaTransport`) and the in-process mock (`LocalSimTransport`) no backend protocol is involved at all. For the bigger picture of why the simulator runs the same app code as production, read [transport vs app simulation](/docs/concepts/transport-vs-app). ## The backend hub The backend is a custom Node server (`backend/server.ts`) that wraps Next.js and a `ws.WebSocketServer`. Every simulator session is a room on that hub, addressed by `sessionId`. Three kinds of client connect to a room: | Role | Who it is | Singleton? | |---|---|---| | `app` | The consumer's running app (the SDK's `BrowserSimTransport`) | One per session | | `browser` | The hardware surrogate — the simulator tab at `extentos.com/s/` that renders the wearer view, captures webcam/mic, and exposes the click-to-fire panels | One per session | | `observer` | A read-only spectator (a dashboard, a second tab) | Up to 8 | The hub enforces the `app` and `browser` singletons — a second live client of either role is rejected with a role conflict — and caps observers at 8. ## Hardware-ready gating Capability traffic doesn't flow until the surrogate is ready. The hub flips `hardware_ready` true once both the app and browser roles are attached, and the SDK gates capability calls on that signal client-side — `capturePhoto()` before the surrogate is up returns a clean "not ready" locally instead of putting a frame on the wire. This mirrors a real device that isn't connected yet. ## The inject envelope The simulator's click-to-fire panels (and the agent-driven [`injectTranscript`](/docs/mcp-server/tools/simulation) / `injectAssistantUtterance` / `injectInput` / `injectHardwareButton` tools) deliver synthetic input wrapped in an **inject envelope**. The hub unwraps it and forwards it **browser → app only**, so an injected transcript reaches the app exactly as a real STT result would and handler dispatch is identical to hardware. Injected input never travels app → browser. ## Camera stream state The surrogate mirrors the glasses' capture-button model: the first shared-stream use (a frame-grab photo, a video recording, or `videoFrames`) arms the camera stream, and the capture button — the sim page's Hardware-buttons panel or `injectHardwareButton` — drives it exactly like the right temple on real glasses: **tap = pause/resume, hold = stop**. The SDK core owns this state machine and announces transitions as `camera_stream_opened` / `camera_stream_paused` / `camera_stream_resumed` / `camera_stream_closed` events (camera chip); the sim page's capture LED tracks them (lit = streaming, dark = paused or closed, mirroring the privacy LED on the physical glasses). Every camera consumer follows the same state: `videoFrames` delivery halts while paused (the app's picture freezes) and dies on close, and an in-flight `captureVideo` recording captures no footage while paused (the clip splices across the gap; a close ends it with the footage so far). While paused, stream-needing captures fail with `CaptureError.StreamPaused` and a `capture_denied` event (errors chip) records why — the same shared gate that fires on hardware. One surfaced substrate delta: on real glasses a hold stops the whole device session (the connection drops and the SDK auto-recovers over a few seconds); the sim closes the stream without the connection blip. After a stop, the next camera use re-arms the stream on both substrates. ## The event log Everything in a session lands in a structured event log. The SDK keeps a 512-entry ring buffer on-device and mirrors it to the backend during simulator mode, where it's queryable per `sessionId` with [`getEventLog`](/docs/mcp-server/tools/simulation) for 48 hours after each event is recorded. Each event carries exactly one **chip** — there are seven: `errors` · `voice` · `camera` · `display` · `ai` · `lifecycle` · `custom` `errors` is special: any event at severity ≥ warn lands there regardless of modality, so a failed photo capture lands in `errors` (not `camera` — chips are mutually exclusive); read both chips to see the request-in-`camera` plus failure-in-`errors` flow. `voice` covers mic/STT, TTS, voice triggers, and Phase-4 assistant lifecycle; `ai` is reserved for customer-side BYOK calls wrapped in `glasses.observability.aiCall(...)`. Filter with `getEventLog({ sessionId, filter: "voice" })` and union chips client-side when you need more than one. ## The voice proxy Simulator STT and TTS run through the backend at `/api/voice/{stt, tts}` so the browser surrogate doesn't need its own model keys — the dev STT/TTS path is one of the deliberately-contained [substrate deltas](/docs/concepts/transport-vs-app) between simulator and hardware. App-level behavior is identical; only the provider differs. ## Versioning The protocol is internal and versioned with the backend; the SDK and hub negotiate a compatible version on connect. There's no third-party-implementable spec today — if you're building an Extentos app you consume capability primitives, never this wire format. A public protocol spec would only ship if a third party needed to implement an alternate hub. ## Related - [Transport vs app simulation](/docs/concepts/transport-vs-app) — what the simulator swaps and what it keeps identical - [Sessions](/docs/concepts/sessions) — roles, lifetime, and the account gate on session minting - [getEventLog](/docs/mcp-server/tools/simulation) — reading the event log while debugging --- source: https://extentos.com/docs/concepts/sessions --- --- title: Sessions description: How an Extentos session works — the glasses connection-state machine, what persists across backgrounding and reconnects, and the three browser-simulator roles. type: concept platform: all related: - /docs/guides/handle-disconnects - /docs/guides/background-sessions - /docs/concepts/transport-vs-app - /docs/reference/errors --- "Session" means two related things in Extentos, depending on the transport. On real glasses it's the **connection** between your app and the hardware, modeled by a six-state machine you observe through `glasses.connection.state`. In the [browser simulator](/docs/concepts/transport-vs-app) it's additionally a **backend session** that multiple parties join with distinct roles. This page covers both, plus what survives backgrounding and reconnects. ## The connection-state machine `glasses.connection.state` is a `StateFlow` (Android) / `ObservableState` (iOS). `GlassesState` is a sealed type with six buckets, advancing roughly in order from "not set up" to "connected", with `Disconnected` reachable from anywhere: ```text NotRegistered ─▶ Registered ─▶ DeviceDiscovered ─▶ Connecting ─▶ Active ▲ │ └──────────────────── Disconnected(cause) ◀────────────────────┘ (reachable from any state) ``` | State | Payload | Meaning | |---|---|---| | `NotRegistered` | — | The install hasn't registered/authenticated. Show a pairing prompt. | | `Registered` | — | Registered; scanning for a known device. | | `DeviceDiscovered` | `deviceId: String` | A device was found; connection is about to begin. | | `Connecting` | `deviceId: String` | Handshake in progress. | | `Active` | `state: ActiveState` | Connected and usable — capability calls (camera, audio, …) work here. | | `Disconnected` | `cause: DisconnectCause` | The connection ended. `cause` carries *why* (user intent, thermal, hinges, transport failure, or a simulator-session event). | `Active` is the "transport is up" state — necessary for using the glasses, but not sufficient, and **not** a signal that glasses are present. On the vendorless audio baseline it is reported whenever audio is routable — which the phone's own mic and speaker always satisfy — so it does not by itself tell you a camera exists. Gate on `state == Active` **and** the capability: `glasses.capabilities.camera`. See [choose your path](/docs/getting-started/choose-your-path#what-a-device-app-does-when-there-are-no-glasses). The transitions are driven by the transport — your code observes; it doesn't drive the machine by hand (you nudge it only via `connect()` / `disconnect()` / `reconnect()`). `Disconnected.cause` is the actionable part — it distinguishes "the user took the glasses off" from "Bluetooth dropped" from "the simulator session expired," so you can choose the right recovery. The full branch-on-cause pattern, the retry policy, and the reconnect call live in **[Handle disconnects](/docs/guides/handle-disconnects)**; the complete `DisconnectCause` and `ConnectError` payload shapes are in the [error reference](/docs/reference/errors). > You rarely render this machine yourself. `ExtentosConnectionPage` renders the canonical pairing → connecting → connected → disconnected flow from this exact state. Observe `glasses.connection.state` directly only for custom in-app affordances (e.g. disabling a button while not `Active`). ## What persists The `ExtentosGlasses` instance is the durable object — construct **one** at app init (in your `Application` subclass / `App`) and hold it for the app's lifetime. It owns the transport, the auto-bind probe, the connection state, registration, and telemetry hooks. Constructing more than one is a bug: there's no caching, and each fires its own auto-bind probe and telemetry registration. - **Across reconnects:** `reconnect()` re-runs registration and re-fetches device metadata (name, firmware, link state), then the state flips back to `Active`. Toggle state (`glasses.toggles`) and your registered voice phrases / assistant configuration are held by the library across the drop — you don't re-register them. - **Across app backgrounding:** whether the *connection* survives backgrounding depends on platform plumbing — an Android foreground service or iOS background mode. That setup is its own topic: see **[Background sessions](/docs/guides/background-sessions)**. Without it, the OS may tear the connection down when your app leaves the foreground, surfacing as a `Disconnected`. - **What does NOT persist on the device:** the event log is a 512-entry ring buffer — it's mirrored to the backend during simulator mode but is not durable storage on the glasses. Assistant conversation history is an in-memory replay buffer (replayed to the provider on reconnect), not disk-persisted by the SDK; if you need it across app restarts, you persist it yourself. ## Simulator sessions and roles The browser simulator adds a second sense of "session": a **backend session**, identified by a `sessionId`, that more than one party joins. Where a real-glasses session is a 1:1 link between your app and hardware, a simulator session is a small hub with three roles ([transport vs app simulation](/docs/concepts/transport-vs-app)): | Role | Who | What it does | |---|---|---| | `app` | Your running app (via `BrowserSimTransport`) | The code under test — exactly what runs in production, only the transport swapped. | | `browser` | The hardware surrogate at `extentos.com/s/{sessionId}` | Drives real webcam frames, microphone audio, speech-to-text, and TTS playback; injects hardware alerts. Stands in for the glasses. | | `observer` | Read-only spectators | Watch the session both directions without participating. Capped at 8 per session. | The `app` and `browser` roles are singletons per session (one each); observers are the only multi-party role. ### `hardware_ready` gating Because the surrogate (the `browser` role) *is* the simulated hardware, a simulator session isn't usable until that surrogate has joined and acknowledged. The backend tracks a **`hardware_ready`** flag and only flips it true once the browser surrogate is present and ack'd; it then fans `hardware_ready` out to the app and any observers. Until then, the app side behaves as "hardware not present" — the analogue of `Connecting` on real glasses. This is what makes the simulator faithful: there's no app-layer behavior available before the (simulated) hardware is ready, exactly as on glasses. Simulator sessions also carry per-device facts the app reacts to — for example `display_capable` is true only on display-capable device models (Ray-Ban Display), so a [display](/docs/concepts/capabilities) feature lights up in the simulator only against the right device. When a simulator session ends (idle timeout, lifetime cap, user-ended, backend restart), the app sees `Disconnected(SimulatorSessionExpired(reason, …))` — a simulator-only `DisconnectCause` you won't encounter on hardware. ## The same code on both The point of the connection-state machine is that it's *transport-independent*: your handler code observes `glasses.connection.state` and calls `glasses.camera.capturePhoto()` identically whether the transport underneath is `BrowserSimTransport`, `LocalSimTransport`, or `RealMetaTransport`. The simulator is designed to behave like the glasses, and that fidelity is under active validation on hardware — so a session that's green in the simulator is strong evidence — the final gate is a real-hardware run. The substrate deltas (transport, audio format, STT/TTS provider) are contained at the transport layer; the session model your app sees is the same on both. See [Transport vs app simulation](/docs/concepts/transport-vs-app) for exactly what differs. ## Related - [Handle disconnects](/docs/guides/handle-disconnects) — branching on `Disconnected.cause`, reconnect, and retry policy. - [Background sessions](/docs/guides/background-sessions) — keeping the connection alive when backgrounded. - [Transport vs app simulation](/docs/concepts/transport-vs-app) — the transports and what the simulator simulates. - [Error reference](/docs/reference/errors) — `GlassesState`, `DisconnectCause`, and `ConnectError` payloads. --- source: https://extentos.com/docs/concepts/transport-vs-app --- --- title: Transport vs app simulation description: Meta's Mock Device Kit simulates the transport layer; Extentos simulates the app layer — voice, photo capture, and the wearing experience. Both matter. type: concept platform: all vendor: meta related: - /docs/concepts/architecture - /docs/concepts/capabilities - /docs/concepts/vs-meta-dat - /docs/vendors/meta - /docs/getting-started/with-agent - /docs/guides/simulator-to-real-glasses --- Building a smart-glasses app for Meta Ray-Ban means working across two simulation layers, not one. **Meta's Mock Device Kit (`mwdat-mockdevice` on Android, `MWDATMockDevice` on iOS) simulates the transport layer** — the Bluetooth Low Energy link, the protocol framing, the video/audio codec, the SDK call surface — so your mobile app *thinks* it's talking to glasses. **Extentos simulates the app layer** — voice triggers being recognized, photos being captured, the wearer's audio output, hardware alerts (thermal, hinges, audio-route changes), permission flows, and the moment-to-moment user experience as it would feel on real Ray-Ban Meta glasses. These two layers are complementary, not competing. Meta's Mock Device Kit is Meta's own transport-layer tool — Extentos doesn't use it internally (the local sim is Extentos's own in-memory transport). Extentos's headline browser-based simulator (`BrowserSimTransport` at [extentos.com/s](https://extentos.com/s)) is a separate architecture — a WebSocket protocol with a browser-side hardware surrogate that drives real webcam and microphone input through the same `glasses.*` API your production code uses. This page explains the distinction, the implications for development speed, and why an AI coding agent building a Meta Ray-Ban app should reach for Extentos rather than wiring Mock Device Kit by hand. ## TL;DR | Layer | What gets simulated | Tool | Where it runs | |---|---|---|---| | **Transport** | BLE link, MWDAT SDK calls, framing, codec | Meta Mock Device Kit (`mwdat-mockdevice` / `MWDATMockDevice`) | Inside your mobile app | | **App** | Voice triggers, photo capture, TTS playback, hardware alerts, wearer UX | Extentos browser simulator (`BrowserSimTransport`) | Browser tab — no mobile app build needed in the inner loop | You can think of Mock Device Kit as testing the *wire* between your app and the glasses. Extentos's browser simulator tests the *experience* of using your app *while* wearing glasses. A Meta Ray-Ban app can benefit from both — Meta's kit used directly for wire-level checks, Extentos's simulators for everything app-level. ## The two layers, separately ### What Meta's Mock Device Kit does Meta's Device Access Toolkit (DAT) ships a mock layer alongside the real SDK: - **Android:** the `mwdat-mockdevice` Maven artifact, with `MockDeviceKit` and `MockCameraKit` classes - **iOS:** the `MWDATMockDevice` Swift Package Manager module, with `MockDeviceKit.pairRaybanMeta()` and `MockCameraKit.setCameraFeed(...)` APIs When you wire it up, your app's `Wearables.configure()` succeeds without scanning Bluetooth, your `createSession(...)` returns a `MockDeviceSession` whose publishers emit events as if a paired Ray-Ban Meta were responding, and your camera-stream subscription returns frames from a bundled video file or the phone's own camera. The DAT API surface — `Wearables`, `DeviceSession`, `StreamSession`, `videoFramePublisher`, `photoDataPublisher` — behaves as it would in production. This is the **transport layer**. The mock answers the SDK; the SDK shape is what's being exercised. You write the rest of your app — the wake-phrase matching, the user flow, the UI, the AI calls, the audio routing decisions — yourself, in code, against that mocked SDK. **What Mock Device Kit is excellent at:** - Verifying your DAT integration compiles, links, and calls the SDK correctly - Local testing on the developer's phone or emulator with no network - Pulling synthetic camera frames from a controlled source (bundled file or phone camera) - Exercising connection / session / stream lifecycle without bonding real glasses **What Mock Device Kit doesn't do:** - Recognize voice triggers — there's no speech-recognition simulation - Play TTS audio out so you can hear what your wearer hears - Inject hardware alerts (thermal warning, hinges closed, incoming call) through a UI — you'd write code to fire them programmatically - Show you what the wearer would see and hear *as a holistic experience* - Surface a structured, queryable event log for an AI agent to debug - Persist logs after the app process dies - Hot-reload your app's behavior without rebuilding ### What Extentos's browser simulator does `BrowserSimTransport` is the simulator developers spend most of their time in. It's not a fork of Mock Device Kit — it's a different architecture entirely: - The Extentos library opens a WebSocket to `wss://api.extentos.com/ws/{sessionId}` - A browser tab at `extentos.com/s/{sessionId}` joins the same session as the *hardware surrogate* - The browser drives **real webcam frames, real microphone audio, and real speech-to-text recognition** through the developer's machine - TTS plays back through the developer's speakers so they can *hear* what the wearer would hear - Voice transcripts can be injected from the browser UI and via MCP tools (hardware-alert injection buttons are roadmap — the SDK has no consumer API for those alerts yet) - Every event flows through Extentos's structured 7-chip event log (`errors` / `voice` / `camera` / `display` / `ai` / `lifecycle` / `custom`) and is queryable via the `getEventLog` MCP tool The same `glasses.camera.capturePhoto()` call that runs against `BrowserSimTransport` in dev runs against `RealMetaTransport` (Meta DAT) in production. Identical API, different wire underneath. This is the **app layer**. Your handler code defines what your glasses app does — wake-phrase matching, photo capture, AI calls, TTS response. The simulator runs that code exactly as production would, against a hardware surrogate the developer drives through a familiar browser UI. ## Where Meta's Mock Device Kit fits next to Extentos's transports Extentos has four transport implementations behind its `GlassesTransport` interface. None of them wraps Meta's Mock Device Kit — the local sim is Extentos's own in-memory transport: | Extentos transport | Wraps | When it's used | |---|---|---| | `SystemAudioTransport` | Nothing from Meta — the phone's own Bluetooth audio routing, no vendor SDK at all | Voice apps, and any app on a phone with no vendor glasses claimed | | `RealMetaTransport` | Meta DAT real SDK (`mwdat-core`, `mwdat-camera`, `mwdat-display` on Android / `MWDATCore`, `MWDATCamera` on iOS) | Production — real Ray-Ban Meta glasses paired | | `BrowserSimTransport` | Nothing from Meta. WebSocket to Extentos backend, browser-side hardware surrogate | The headline dev experience at `extentos.com/s` | | `LocalSimTransport` | Nothing from Meta — Extentos's own in-memory deterministic transport (no DAT SDK) | Fast on-device testing, no network, no browser, headless CI | So the precise answer: Extentos uses Mock Device Kit *nowhere* — `LocalSimTransport` is Extentos's own in-memory fast path for inner-loop testing without a network round-trip, and `BrowserSimTransport` is the higher-level simulator where the developer spends the bulk of their time. Meta's kit remains available to you directly when you want wire-level simulation. This is intentional. Mock Device Kit is the right tool for "does my SDK integration compile and run" — Extentos defers to it. It's the wrong tool for "does my voice-trigger app actually feel right when worn" — Extentos builds a separate simulator for that. ## Why testing requires both layers Consider a typical Meta Ray-Ban app: the user says *"summarize this"*, the handler matches the wake phrase, the camera captures a frame, the handler calls a vision model, and the glasses speak the answer. To validate that flow, you need to test: 1. **Does my app speak the DAT SDK correctly?** (transport layer — Mock Device Kit's job) 2. **Does my wake phrase match against real STT output with reasonable variance tolerance?** (app layer — needs real STT) 3. **Does the captured frame look right and route to my handler?** (app layer — needs real frames) 4. **Does the spoken response sound right at the right moment?** (app layer — needs TTS playback the developer can hear) 5. **What happens when the user removes the glasses mid-flow?** (app layer — needs hardware-alert injection) 6. **What happens when the BLE link drops?** (transport layer — Mock Device Kit-style mocking) 7. **Does my AI handler return cleanly under timeout?** (app layer — needs the full execution to play out) Mock Device Kit answers (1) and (6). Extentos's browser simulator answers (2) through (7). Doing all seven with raw Mock Device Kit means writing your own speech-recognition simulator, your own TTS playback harness, your own hardware-alert injection UI, your own event correlator, and your own replay tool. That's a lot of infrastructure. Extentos is that infrastructure — built once, shared across every Meta Ray-Ban app developer. ## The complete advantage list The case for `BrowserSimTransport` over a hand-rolled Mock Device Kit setup, grouped by what an AI agent and a human developer each gain: ### Setup and iteration speed 1. **Zero-config simulation.** Extentos resolves the right transport at `ExtentosGlasses.create()` based on `BuildConfig.EXTENTOS_SESSION_URL` (Android) or `extentos.session.plist` (iOS). With raw Mock Device Kit you write the mock initialization, device pairing, state-machine walks, and camera-feed wiring by hand for every app. 2. **Auto-bind dev loop.** Once the Extentos library is in your app, the MCP server and the library coordinate via a `127.0.0.1:31337/whoami` localhost bridge. Every new `createSimulatorSession` call auto-attaches the running app to the new session — no rebuild, no URL paste, no typed code in the happy path. Mock Device Kit has no MCP integration at all. 3. **Higher-level SDK on top of the SDK.** Your handler code subscribes to capability primitives (`glasses.audio.transcriptions()`, `glasses.camera.capturePhoto()`, `glasses.audio.speak()`, …) instead of the raw DAT call surface. Same primitives across simulator and production — you write the handler once, it runs on the laptop simulator and on real Ray-Ban Meta glasses without changes. Mock Device Kit gives you a mocked DAT SDK; you write the rest of the abstraction layer yourself. 4. **Reattach without remint.** Rebuild + reinstall your app and it auto-reattaches to the same persistent simulator session — the URL is stable, no fresh-mint required. Mock Device Kit re-runs the mock pairing from scratch on every app launch. 5. **No browser/mobile context switch for the inner loop.** Browser simulation runs without an Android emulator or iOS Simulator window even open (your device or emulator runs your real app code, the *simulated hardware* is in the browser). Mock Device Kit needs the emulator/device app actually running. ### What you can actually see and do 6. **Phone, glasses, and agent in one view.** The browser shows the wearer-perspective alongside your phone-app UI alongside the live event stream. Mock Device Kit shows you camera frames inside your app — that's it. 7. **Real webcam and microphone in the loop.** Browser simulation uses the developer's actual webcam and microphone for input. Mock Device Kit can play back a bundled video file or the phone camera, but no real microphone-driven speech recognition. 8. **Input injection (transcripts today).** The browser UI and `injectTranscript` fire voice input; hardware-event buttons (`thermal_warning`, `hinges_closed`, `audio_route_changed`, `incoming_call_detected`, `app_lifecycle_changed`) are roadmap — the SDK exposes no consumer API for those alerts yet. With Mock Device Kit you'd write code to programmatically trigger these — and you'd write that code separately for each scenario. 9. **Permission flow modeled.** Browser sim respects user-denied webcam or mic permission and surfaces `TransportError.HardwareUnavailable` exactly as production would. Mock Device Kit assumes permissions granted. 10. **A2DP/HFP coexistence.** Routing conflicts surface as `CoexistenceBlocked` errors on the affected call (the `coexistence.warning` runtime event is declared but not yet emitted). Mock Device Kit doesn't simulate this Bluetooth audio-profile interaction either. ### Debugging the AI agent can use 11. **7-chip structured event log.** Every event is tagged with one of `errors`, `voice`, `camera`, `display`, `ai`, `lifecycle`, or `custom` — one label per event, with `errors` absorbing severity≥warn regardless of modality so failures find a single home. Mock Device Kit logs are raw DAT-level `Logcat` / `os_log` output — unstructured, unfiltered, untagged. 12. **Capability-call correlation.** Each capability invocation (record_discrete, capture_photo, speak, …) emits a request + result/completed event with a stable id, plus any failures (which route to the `errors` chip). The agent calls `getEventLog(filter="voice")` and gets every audio call's lifecycle — request, result, completed — in order. Mock Device Kit has no per-call event model. 13. **Voice transcripts as first-class events.** The library emits `audio.transcriptions` events with the transcript text + confidence score + final/partial flag. Your handler matches them; the agent debugs them via `getEventLog`. Mock Device Kit doesn't simulate voice transcription at all, so there's nothing to log. 14. **Backend-side log retention.** Logs persist for 48 hours after each event, queryable by the agent from anywhere. Mock Device Kit logs die when the app process dies — gone forever after a crash. 15. **Replay.** Scroll back through everything that happened in the session, including streams, hardware events, and the full event log. Mock Device Kit has no replay. 16. **`getEventLog` MCP tool.** The agent can query the event log directly — `filter: "errors"`, `filter: "voice"`, `filter: "camera"`, plus an opaque `cursor` for incremental reads — and ingest the structured response. Mock Device Kit has no agent-facing query API. ### Things Mock Device Kit literally cannot do 17. **Voice-recognition simulation.** Real speech recognition through the dev's microphone, with confidence scores and partial-match diagnostics. Mock Device Kit gives you camera and sensor mocks; voice is your problem. 18. **TTS playback simulation.** Browser sim plays the synthesized speech out loud through the dev's speakers so they can *hear* what the wearer would hear. Mock Device Kit gives you no audio output simulation. 19. **End-to-end wake → record → LLM → speak rehearsal.** Browser sim runs the entire handler code path. Mock Device Kit only mocks the SDK; the rest of your flow is your own code without an environment to exercise it in. ### Architecture and ship-readiness 20. **Same API, simulator and production.** `glasses.camera.capturePhoto()` works identically against `LocalSimTransport`, `BrowserSimTransport`, and `RealMetaTransport`. With Mock Device Kit alone, you're writing against the DAT API directly — Extentos gives you a higher-level surface that's stable across simulator and hardware. 21. **Pre-test correctness gate.** `validateIntegration` catches missing dependencies, bootstrap wiring gaps, permission/capability mismatches *before* you run anything. Mock Device Kit fails at runtime. 22. **Ship-readiness checklist.** `getProductionChecklist` is a tool the agent invokes to verify the app is ready to ship — credential setup, permissions, capability declarations, edge-case coverage. No equivalent for Mock Device Kit. 23. **Multi-vendor, demonstrated.** A second and third vendor mean second and third transport implementations inside Extentos, not a rewrite of your app: Android XR landed in 2026-07 without changing a single app-facing API, and the same handler code runs under an Android XR device identity in the simulator today ([status](/docs/vendors/android-xr)). Mock Device Kit is Meta-only by definition — every additional vendor is a fresh integration project. 24. **Designed for AI agents to operate.** A tight, deterministic MCP tool surface (capability discovery, code-example reference, scaffolding, validation, simulator orchestration, debug log query) the agent calls. Mock Device Kit was designed for human developers writing Android or iOS code by hand against the DAT SDK. ## How they stack — example A developer's day on Meta Ray-Ban with Extentos looks like this: ```text Morning — design and validate Agent: getPlatformInfo → getCodeExample(pattern) → getCapabilityGuide(...) Agent: generateConnectionModule Agent: Agent: validateIntegration ✓ wired correctly Inner loop — most of the day, browser-driven Agent: createSimulatorSession → extentos.com/s/ opens Dev: speaks "summarize this" into laptop mic sees photo capture in browser, hears TTS response clicks "thermal warning" button to test edge case Agent: getEventLog(filter: "errors") → zero results, all good Dev: edits handler — changes wake-phrase string from "summarize" to "describe" Dev: rebuild + reinstall ✓ app auto-attaches to same session → BrowserSimTransport (no Mock Device Kit involved) Headless CI — fast smoke check Test runner spins up LocalSimTransport (Extentos's in-memory local sim) Verifies the bootstrap connects and the handler subscribes cleanly → LocalSimTransport (in-memory, no DAT SDK) Real-hardware verification — late afternoon, periodically Dev pairs Ray-Ban Meta over Bluetooth, runs app Same handler code, real glasses → RealMetaTransport (Meta DAT real SDK) Evening — ship Agent: getProductionChecklist ✓ ready Dev: builds release, submits. ``` Mock Device Kit's role: Meta's own wire-level building block, available to you directly when you want BLE/SDK-layer simulation. Extentos's network-free fast path for CI and headless tests is `LocalSimTransport` — its own in-memory transport. The browser simulator is the primary surface, because the things developers actually need to validate (does my voice trigger feel right, is the photo timing correct, does the TTS land at the right moment, what does the wearer experience) all live at the app layer. ## When to reach for which | Question | Right tool | |---|---| | Does my Android or iOS app compile against the Meta DAT SDK? | A real build against DAT (Meta's Mock Device Kit for wire-level checks) | | Does my wake phrase match against real STT output? | Extentos browser simulator | | Does my TTS response land at the right moment? | Extentos browser simulator | | What happens when the wearer takes off the glasses mid-flow? | Extentos browser simulator (end the session from the sim; `hinges_closed` injection is roadmap) | | Does my background app come back cleanly after a phone call? | Extentos browser simulator (`incoming_call_detected` injection is roadmap) | | Does my AI handler return within the timeout budget? | Extentos browser simulator (full handler execution) | | Does the BLE link drop and recover correctly? | `LocalSimTransport` (in-memory link-state machine) | | Do I have a complete trace of one capability call? | Extentos `getEventLog(filter="voice" \| "camera" \| ...)` | | Is my app ready to ship? | Extentos `getProductionChecklist` | | Does the photo *actually* look right on the wearer's view? | Real Ray-Ban Meta hardware (`RealMetaTransport`) | ## Frequently asked questions ### Is Extentos a wrapper around Meta's Mock Device Kit? No. Extentos has four transport implementations: one is the vendorless system-audio baseline (`SystemAudioTransport`), one wraps Meta's real SDK (`RealMetaTransport`), one is Extentos's own in-memory local sim (`LocalSimTransport` — no DAT SDK, no network), and one is an Extentos-original architecture using a WebSocket protocol and a browser-side hardware surrogate (`BrowserSimTransport`). Meta's Mock Device Kit remains Meta's own separate tool for transport-layer testing. ### Can I use Extentos without using Meta's Mock Device Kit at all? Yes. The `BrowserSimTransport` does not import or depend on `mwdat-mockdevice` — it talks to the Extentos backend over WebSocket and uses the developer's webcam/mic via the browser. If you only ever use the browser simulator, Mock Device Kit is never loaded. ### Can I use Meta's Mock Device Kit without Extentos? Yes — Mock Device Kit is shipped by Meta as part of the Device Access Toolkit. You can integrate it directly. You'll write the simulation harness, the trigger logic, the audio routing, the event correlation, the agent integration, and the multi-vendor abstraction yourself. ### Does Extentos work with real Ray-Ban Meta glasses? Yes. `RealMetaTransport` wraps Meta DAT — `mwdat-core` + `mwdat-camera` + `mwdat-display` `v0.8.0` on Android (group `com.meta.wearable`, singular), `MWDATCore` + `MWDATCamera` `v0.8.0` on iOS via SPM. The same code that runs against the simulator runs against real hardware — only the transport changes. ### Does Extentos support other smart-glasses vendors? Meta smart glasses (vendor token `meta`) are the GA target. [Android XR](/docs/vendors/android-xr) is in preview: the projected transport is built and proven on Google's emulator, and the simulator carries an Android XR device identity, but the module is unpublished and no hardware is buyable. [Brilliant Labs](/docs/vendors/brilliant) (`brilliant`) is preview the other way round: the BLE transport, on-device Lua bundle and display translation are built and published, and Halo is on sale — but no device has run any of it, and there is no emulator standing in either. Apple smart glasses await a third-party SDK. The SDK surface and your handler code are vendor-independent — adding a vendor doesn't break existing code. ### Why doesn't Mock Device Kit simulate voice triggers? Meta's Device Access Toolkit operates at the BLE/SDK layer. Voice triggers in production are recognized on the phone — on-device Vosk on Android hardware (the platform recognizer on the dev path), `SFSpeechRecognizer` on iOS — running on audio streamed from the glasses microphone over Bluetooth HFP/SCO. Mock Device Kit replaces the BLE side of that path; it doesn't try to replace the phone's speech recognizer because the phone is real. Extentos's browser simulator replaces the *whole* path — using the developer's laptop microphone and a browser-side recognizer — so voice triggers can be exercised end-to-end without glasses or a phone. ### What's the cost difference? Meta's Mock Device Kit is included with Meta DAT and free. Extentos is free for the whole dev loop — the discovery, validation, guidance, and search MCP tools work without an account, and so do `LocalSimTransport` and real-hardware testing via your own Meta DAT credentials. A free email-only account (Google sign-in, or email + password — no payment, no card) covers browser-simulator session minting at `extentos.com/s`, the `generateConnectionModule` scaffold step, and the account-scoped project tools. Once linked, sessions and the events they emit are unlimited. The one metered surface is the [managed AI gateway](/docs/concepts/ai-gateway) — prepaid credits, with $2 free to start. See [pricing](/docs/resources/pricing) for details. ## Related - **[Architecture](/docs/concepts/architecture)** — how the MCP server, the SDK, the simulator backend, and your app fit together - **[Capabilities](/docs/concepts/capabilities)** — the abstract capabilities (camera, mic, voice, sensors, alerts) that Extentos exposes across vendors - **[Vendors: Meta Ray-Ban](/docs/vendors/meta)** — the GA capability matrix, what Meta DAT supports today, and what's preview-only - **[From simulator to real glasses](/docs/guides/simulator-to-real-glasses)** — when you're ready for hardware: the transport switch, Meta credentials, and the verification pass - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install the MCP server and see the simulator in action --- source: https://extentos.com/docs/concepts/vs-meta-dat --- --- title: Extentos vs Meta DAT directly description: Extentos wraps Meta's Device Access Toolkit — it doesn't replace it. What you write against raw DAT vs what Extentos gives you, and when to go direct. type: concept platform: all vendor: meta related: - /docs/concepts/transport-vs-app - /docs/concepts/capabilities - /docs/concepts/architecture - /docs/vendors/meta - /docs/getting-started/with-agent - /docs/guides/migrating-from-meta-dat --- **Extentos isn't an alternative to Meta's Device Access Toolkit — it's a layer on top of it.** Extentos's production transport (`RealMetaTransport`) calls Meta DAT under the hood; the question isn't "Extentos *or* DAT," it's "compose against Extentos's typed primitives and tooling, or wire the raw DAT API yourself." This page is the honest comparison, including when going direct to DAT is the better choice. ## What Meta DAT gives you, and what you still have to build Meta's [Device Access Toolkit](/docs/vendors/meta) is the official SDK for connecting a mobile app to Ray-Ban Meta — it handles Bluetooth pairing, the session/stream lifecycle, camera frames, photo capture, and the audio path. It's solid plumbing. But it stops at the SDK surface. Building a real voice-glasses app on raw DAT means you also write, yourself: - **The wake-phrase path** — DAT doesn't recognize phrases. You stream the glasses mic to the phone's `SpeechRecognizer` / `SFSpeechRecognizer` over Bluetooth HFP, normalize transcripts, and match strings. - **A capability abstraction** — DAT exposes raw publishers (`videoFramePublisher`, `photoDataPublisher`, …). You build the typed surface your handlers call. - **A test harness** — DAT ships a Mock Device Kit for the *transport*, but you write the speech-recognition simulation, TTS playback, hardware-alert injection, and an event correlator to actually exercise your app's behavior. (See [transport vs app simulation](/docs/concepts/transport-vs-app).) - **Permission derivation** — you map each capability to the right Android manifest entries and iOS Info.plist keys (including the Meta DAT-specific ones) by hand. - **The audio-coexistence decisions** — A2DP playback racing an HFP capture is fragile; you write the routing policy. - **Multi-vendor portability** — DAT is Meta-only. Supporting another vendor later is a fresh integration. ## What Extentos adds Extentos builds that layer once, shared across every Meta Ray-Ban app: - **Typed capability primitives** — your handler subscribes to `glasses.audio.transcriptions()`, `glasses.camera.capturePhoto()`, `glasses.audio.speak()`, `glasses.assistant.start()`, `glasses.display.show()` — the same API across the [browser simulator](/docs/concepts/transport-vs-app), the on-device local simulator, and real hardware. - **A real app-level simulator + agent-driven testing** — drive voice, camera, and display from a browser (or from MCP tools, no human), and read a structured [7-chip event log](/docs/concepts/sessions). The bulk of the dev loop happens without pairing glasses. - **MCP scaffolding for AI agents** — `generateConnectionModule` writes the wiring, `getPermissions` derives the entitlements, `validateIntegration` gates correctness. See the [MCP tools reference](/docs/reference/mcp-tools). - **A typed error model** — lifecycle ops return [`ExtentosResult`](/docs/reference/errors) with concrete failure variants instead of raw exceptions. - **A managed AI gateway** (optional) — the [assistant runtime](/docs/concepts/assistant) routes voice AI on Extentos's keys by default; no key in your app. See [the gateway](/docs/concepts/ai-gateway). - **A multi-vendor seam** — handler code is vendor-agnostic; adding a vendor is a new transport, not a rewrite. ## Side by side | | Raw Meta DAT | Extentos (on top of DAT) | |---|---|---| | Bluetooth pairing, session/stream lifecycle | ✅ DAT | ✅ (wraps DAT) | | Camera / photo / audio primitives | ✅ raw publishers | ✅ typed `glasses.*` | | Wake-phrase recognition | ❌ you build it | ✅ `glasses.voice.onPhrase` / `transcriptions()` | | App-level simulator (voice, TTS, alerts) | ❌ you build it | ✅ browser simulator | | Agent-driven E2E testing | ❌ | ✅ inject + assert + event log | | Permission derivation | ❌ by hand | ✅ `getPermissions` | | Typed error model | ❌ raw exceptions | ✅ `ExtentosResult` | | Voice AI / assistant runtime | ❌ you wire a provider | ✅ managed gateway + `glasses.assistant` | | Ray-Ban Display rendering | DAT 0.8.0 primitive | ✅ `glasses.display.*` DSL | | Multi-vendor portability | ❌ Meta-only | ✅ vendor-agnostic handlers | | Dependencies | DAT only | DAT + Extentos SDK | ## When to use Meta DAT directly Extentos is the faster path for almost any voice/camera/display glasses app — but going direct to DAT is the right call when: - **You need a DAT capability Extentos doesn't surface yet.** Extentos exposes a curated primitive set; if you need a raw DAT API that isn't wrapped, use DAT directly (and consider [emailing us](mailto:hello@extentos.com) so it gets surfaced). - **You want zero added dependencies.** A minimal app that only does, say, one photo capture may not need the abstraction. - **You're doing something very low-level** with the transport that the capability model would get in the way of. Adopting Extentos on an existing DAT app is a **full cutover today, not a side-by-side blend**: the Meta `DeviceSession` is a single-owner handle, so raw DAT and Extentos can't both own the hardware in one process. The cutover is a call-site swap, not a rewrite (Extentos calls the same DAT underneath) — see **[Migrating from raw Meta DAT](/docs/guides/migrating-from-meta-dat)** for the symbol-by-symbol map. ### Escape hatches: lower altitude, same ecosystem When you need something the packaged surface doesn't give you, Extentos's answer is an **escape hatch that stays inside the ecosystem** — an explicitly opt-in API at a lower altitude (Android `@ExtentosEscapeHatch`, iOS `@_spi(ExtentosEscapeHatch)`), never a handout of the raw vendor session. That's a deliberate hard rule, for the same single-owner reason as above: two owners of one Bluetooth transport is split-brain by construction. Everything you do through a hatch still rides the Extentos session — same connection state machine, same event log, same toggles and permission gates. What that looks like today: the shipped hatch lets you replace the packaged connection page with fully custom UI built on `glasses.connection.state`, `glasses.toggles.state`, and `glasses.voice.hints`. And a lot of "low-level" visibility needs no hatch at all — `connection.state` is a plain observable stream, video frames carry per-frame effective `width`/`height`, and `glasses.runtime.events` streams the SDK's runtime activity. Vendor-specific extension hatches (Meta-only knobs the neutral vocabulary doesn't cover) follow the same in-ecosystem rule as they land; if you're blocked on a specific DAT capability today, [tell us](mailto:hello@extentos.com) and it gets prioritized. ## The bottom line Extentos doesn't hide Meta DAT — it does the repetitive, fragile, per-app work (wake-phrase plumbing, the app-level simulator, permission derivation, the test harness, the error model) once, so you write your app's logic instead of its infrastructure. If you'd end up rebuilding that layer anyway, Extentos is the layer. If you genuinely only need raw transport, DAT alone is fine. ## Related - **[Migrating from raw Meta DAT](/docs/guides/migrating-from-meta-dat)** — the symbol-by-symbol cutover map for an existing DAT app - **[Transport vs app simulation](/docs/concepts/transport-vs-app)** — Mock Device Kit (transport) vs the Extentos app-level simulator - **[Capabilities](/docs/concepts/capabilities)** — the vendor-agnostic primitive surface - **[Vendors: Meta Ray-Ban](/docs/vendors/meta)** — the DAT capability matrix and setup - **[Quickstart with an agent](/docs/getting-started/with-agent)** — install and build the first flow --- source: https://extentos.com/docs/ecosystem --- --- title: The third-party smart-glasses landscape seoTitle: Third-party smart glasses in 2026 — the developer landscape description: "Can you build third-party apps for smart glasses today? A platform-by-platform comparison — Meta, Snap Spectacles, Brilliant Labs, Rokid, RayNeo, Even Realities, Vuzix, Android XR, Apple and more: which have an official SDK, how apps are built and distributed, and whether you can publish publicly." type: overview verified: "2026-08-24" platform: all related: - /docs/ecosystem/distribution - /docs/ecosystem/openness - /docs/ecosystem/ai - /docs/vendors - /docs/vendors/meta --- "Can I build a third-party app for smart glasses?" has a different answer on every platform, and the answers move quarter to quarter. This is the neutral, dated map: which platforms expose an official developer SDK, what shape a third-party app takes on each, how it reaches real users, whether public publishing is open yet, and how AI works. It's the landscape view — for the per-platform deep dives, follow through to each platform page below. ## The active platforms These are the platforms with a real (or credibly imminent) third-party developer story — a shipping SDK and a path, however gated, to real users. Rows marked ⚠ are thinly sourced — verified, but resting on weak or contested sources (docs behind a sign-in, a dead or archived page, or sources that contradict each other). Meta is the platform we build on daily, so its row is the most battle-tested; see [Meta smart glasses](/docs/vendors/meta) for the full Extentos-focused picture. Every other row was primary-source-verified but moves fast — follow the source links and the platform pages. ## On the horizon [Apple](/docs/ecosystem/platforms/apple) is the one entry on this map with **no device, no runtime and no SDK** — there is nothing to build against today, so it doesn't belong in the table above. It's on the map because the accessory frameworks a tethered design would need already ship (AccessorySetupKit, ExternalAccessory, and the MFi program, which is a real gating surface), and because if Apple moves, the market moves. Treat it as a watch item, not a platform. ## The wider landscape Real developer platforms that sit outside the consumer third-party race: enterprise/industrial glasses, sensing devices, and tethered "birdbath" viewers used mainly as a wearable screen. They belong on the map, but most developers building a consumer smart-glasses app won't start here. ## The one distinction that explains most of it The biggest source of confusion isn't capabilities — it's **app model**: *does your code run on the phone (with the glasses as a peripheral), or on the glasses themselves?* (A VR-style "headset app" — Android XR *headsets* like Project Moohan, or Magic Leap / HoloLens — is a separate device category, not smart glasses.) - **Companion / extension app** (runs on the phone) — you extend an app that ships through an existing store; the glasses are the peripheral — camera, mic, speaker, or display — over Bluetooth. **Meta's DAT, Android XR, Brilliant Labs, Solos, and Vuzix** take this approach — which is *why* distribution is often a two-part story (ship the app *and* get the user's glasses linked to it). - **An app that runs on the glasses** — either a **native standalone** app (**Snap Spectacles, Rokid, RayNeo, INMO**) or a **web app you deploy as a URL** (**Meta Ray-Ban Display Web Apps**, or Rokid's AIUI/JSAR runtime). And a single platform can offer *both*: **Meta ships two entirely separate systems** — DAT (native companion) and Display Web Apps (a web app on the Ray-Ban Display) — and **Rokid** exposes four dev paths at once. Most "why can't I just publish my glasses app to a store" questions dissolve once you see which system you're in. Full breakdown: [how third-party apps are distributed](/docs/ecosystem/distribution). ## What this means if you're deciding today - **Building now, shipping to real users:** **Meta** is the most mature — shipping consumer hardware + a public SDK — but publishing is invite/partner-gated during preview ([start here](/docs/vendors/meta)). **Snap Spectacles** has arguably the deepest on-glasses developer ecosystem (Lens Studio, public Lens Explorer) but hardware access narrowed — Snap has withdrawn the published sign-up for its subscription developer program, leaving a $2,195 Specs pre-order as the only listed purchase route. **Brilliant Labs** is the most *open* (permissive OSS SDKs). - **Openness varies wildly, and on two axes.** A platform can hand you a free SDK and still refuse to let you ship — see [how open is smart-glasses development](/docs/ecosystem/openness) for the mental model and where each platform lands. - **AI is mostly closed at the assistant layer.** Meta AI, Gemini, and most first-party assistants are reserved; you bring your own AI on nearly every platform. [How AI works on smart glasses →](/docs/ecosystem/ai) - **Forward-looking:** **Android XR** ([tracking](/docs/ecosystem/platforms/android-xr)) is the other big-platform ecosystem play, and **Apple** ([no SDK yet](/docs/ecosystem/platforms/apple)) is worth watching, not yet building on. ## Platform deep-dives Neutral, per-platform developer references — SDK, app model, distribution, capabilities & AI: [Snap Spectacles](/docs/ecosystem/platforms/snap) · [Brilliant Labs](/docs/ecosystem/platforms/brilliant-labs) · [Rokid](/docs/ecosystem/platforms/rokid) · [RayNeo](/docs/ecosystem/platforms/rayneo) · [Even Realities](/docs/ecosystem/platforms/even-realities) · [Vuzix](/docs/ecosystem/platforms/vuzix) · [INMO](/docs/ecosystem/platforms/inmo) · [Solos](/docs/ecosystem/platforms/solos) · [HTC VIVE Eagle](/docs/ecosystem/platforms/htc-vive-eagle) · [Alibaba Qianwen](/docs/ecosystem/platforms/alibaba-qianwen) Extentos ships a production transport for **Meta**, which gets a deeper, build-focused guide under [vendors](/docs/vendors): the [Meta build guide](/docs/vendors/meta). **Android XR** has a preview transport — built and proven on Google's emulator, module unpublished, no buyable hardware ([status](/docs/vendors/android-xr)). Apple is tracked. The neutral platform pages above stay vendor-neutral regardless of what Extentos supports. Extentos's bet is that the *capability surface* (camera, voice, audio, display) generalizes across all of these even though the app models don't — so one app can target the platform that's open today and the platform that opens next with a config change, not a rewrite. That's the [vendor model](/docs/vendors). ## Related - [How third-party smart-glasses apps are distributed](/docs/ecosystem/distribution) — the app-model taxonomy in depth - [How open is smart-glasses development?](/docs/ecosystem/openness) — the openness spectrum, both axes - [AI on smart glasses](/docs/ecosystem/ai) — reserved assistants vs. bring-your-own-AI - [Vendors overview](/docs/vendors) — what Extentos supports, and the vendor-portability model --- source: https://extentos.com/docs/ecosystem/ai --- --- title: "AI on smart glasses" seoTitle: "AI on smart glasses: reserved assistants vs. bring-your-own-AI across platforms" description: "How third-party AI works across smart-glasses platforms — which first-party assistants are reserved (Meta AI, Gemini, Ari), where you bring your own AI (phone-side, cloud, on-device), wake-word limits, and the voice-in → AI → voice-out surface that generalizes even where the assistant is closed." type: guide verified: "2026-08-24" platform: all related: - /docs/ecosystem - /docs/ecosystem/distribution - /docs/ecosystem/openness - /docs/concepts/assistant --- "AI on smart glasses" sounds like one feature, but for a developer it's three separable primitives wired in a line: **voice in** (capture the wearer's speech), **the AI** (an LLM or agent that decides what to do), and **voice out** (speak the answer back). Almost every platform in this category exposes those three primitives — even the ones whose *branded* assistant is sealed shut. The reserved assistant and the buildable pipeline are different questions, and conflating them is the most common way developers mis-scope a glasses AI project. This page teaches that distinction: where the first-party assistant is closed, where you bring your own model, how the AI can sit phone-side / in the cloud / on the device, and what the wake-word constraint actually blocks. It places each platform on that map and links to the [per-vendor pages](/docs/vendors) for the specifics. ## The surface that generalizes Strip away branding and a glasses "AI assistant" is a pipeline: ```text microphone ──▶ speech-to-text ──▶ your AI ──▶ text-to-speech ──▶ speaker (voice in) (STT) (LLM / agent / tool loop) (TTS) (voice out) ``` The pipeline is what generalizes. A platform can reserve its *own* assistant (the thing that answers to a branded wake word) while still handing you the microphone, an STT stream, and a way to speak audio back — which is everything you need to run **your** model in the middle. So the useful question is never "does it have AI?" but two sharper ones: 1. **Is the first-party assistant open to third parties?** Almost always **no** (see the next section). 2. **Can I get voice in and voice out to run my own model?** Almost always **yes** — that's the surface Extentos and every serious glasses app build on. Where the two diverge is *placement* — whether the AI runs on the phone, in the cloud, or on the glasses themselves — and *invocation* — whether you can trigger it hands-free without colliding with a reserved wake word. ## First-party assistants are usually reserved The branded assistant — the one wired to the hardware button and the vendor's wake word — is closed to third-party code on nearly every platform. It's a product surface the vendor owns, not an API. Treat first-party assistant access as the exception, not the default: | Platform | First-party assistant | Open to your code? | |---|---|---| | [Meta](/docs/vendors/meta) | Meta AI ("Hey Meta") | No — curated partners only, no public route | | [Android XR](/docs/vendors) | Gemini | Assistant surface reserved, but apps are **Gemini-invocable** and can call the **Gemini Live API** | | DigiLens | Google Gemini voice (via Google Cloud) | No — wired to the OS | | RealWear | Ari | End-user assistant; you build apps *around* it | | [RayNeo](/docs/ecosystem/platforms/rayneo) | Gemini (West) / Qwen (China) | No — closed **and** region-split by firmware | | [INMO](/docs/ecosystem/platforms/inmo) | "OK Amu" | No — reserved wake word; managed agent layer is separate | | [Rokid](/docs/ecosystem/platforms/rokid) | Platform assistant | Wake word reserved, but **open at the model layer** | | Mentra | Mentra AI | Exposed as **actions + transcription events**, not a sealed box | Two nuances worth internalizing. First, "reserved" usually means *the wake word and the button*, not the microphone — Meta AI is closed, but Meta's DAT still routes glasses audio to your phone app. Second, a handful of platforms deliberately don't seal the assistant: [Rokid](/docs/ecosystem/platforms/rokid) reserves the wake phrase but lets you swap the model behind it, and Mentra surfaces its Mentra AI assistant as callable actions — which it states map 1:1 onto MCP tools — plus a transcription-event stream rather than an opaque service. Those are the outliers; design for the closed case and treat openness as a bonus. Several platforms ship **no** first-party assistant at all — [Vuzix](/docs/ecosystem/platforms/vuzix), Epson Moverio, XREAL, VITURE, Iristick (on-device voice control only, no AI), Lenovo ThinkReality, and Jorjin's J-Reality J7EF line (its industrial J7M markets an "AI-powered Assistant", but no assistant API is published on any model). On those, there's nothing to be locked out of — the AI is entirely yours to bring. ## Bringing your own AI Because the assistant is usually closed but the pipeline is usually open, **bring-your-own-AI is the mainstream path**, not a fallback. What differs across platforms is *where the model runs*. Three placements, each with real trade-offs: ### Phone-side The AI runs in a companion app on the paired phone; the glasses are a microphone-and-speaker (and maybe display) peripheral. This is the lowest-friction path — you ship an ordinary phone app, use any model SDK, and the glasses never need to run inference. It's the model for display-less or thin-client hardware: [Vuzix](/docs/ecosystem/platforms/vuzix) (BYO-AI fully open; the Z100 is a Bluetooth display peripheral, so the app *is* a phone app), [Solos](/docs/ecosystem/platforms/solos) (audio-first, no HUD — voice in, your AI, voice out), MICROOLED ActiveLook (open BLE SDK, no first-party AI), and [Even Realities](/docs/ecosystem/platforms/even-realities) (BYO-AI via `fetch` from the companion, no assistant API). It is also **Extentos's model** — see [where Extentos fits](#where-extentos-fits) below. ### Cloud / gateway The glasses (or the phone) reach a hosted model, often through a vendor-provided gateway that handles auth and network egress. [Snap Spectacles](/docs/ecosystem/platforms/snap) is the clearest example: its **Remote Service Gateway** lets a Lens call out to your own AI backend, and — importantly — such bring-your-own-AI Lenses are *publicly publishable* through Lens Explorer, not stuck in an experimental sandbox. [Rokid](/docs/ecosystem/platforms/rokid) exposes a comparably open cloud/BYO-model agent path, and [INMO](/docs/ecosystem/platforms/inmo) is building a managed agent layer (an n8n-style workflow platform, currently beta) alongside its open BYO-AI story. ### On-device The model runs on the glasses themselves. This is the frontier, and it's advancing fastest on the open platforms. [Brilliant Labs](/docs/ecosystem/platforms/brilliant-labs) is the reference point: an on-device Lua VM over plain BLE plus an **open-source, self-hostable, BYO-key assistant** — you own the whole stack, model included. Omi (OmiGlass) offers full AI access with BYO keys *or* local models. And the standalone-Android devices — [INMO Air3](/docs/ecosystem/platforms/inmo) (Android 14 with Play Store) and [RayNeo](/docs/ecosystem/platforms/rayneo) X-series (standalone APKs, not phone-companion) — can host inference on-glasses because they're full computers, not peripherals. [Rokid](/docs/ecosystem/platforms/rokid)'s on-glasses AIUI runtime (an open-source JavaScript/JSAR web layer) is a related trend: app logic, and increasingly AI glue, running *on the lens*. On-device buys you latency and privacy at the cost of the compute/thermal budget of a wearable. Today it's most practical for small models, wake/keyword spotting, and routing — with the heavy generation still delegated to the phone or cloud. Expect the split point to keep moving on-device as the hardware improves. ## Voice invocation and wake words Hands-free invocation is where "bring your own AI" hits its sharpest limit. The pipeline is open, but the *trigger* often isn't: the vendor's branded wake word is reserved, and on some platforms it's the only truly hands-free path. - **Reserved wake words** you cannot bind: "Hey Meta" ([Meta](/docs/vendors/meta)), "OK Amu" ([INMO](/docs/ecosystem/platforms/inmo)), and the platform wake phrase on [Rokid](/docs/ecosystem/platforms/rokid). Firing your AI on those phrases is not available. - **Your own wake path** generally means phone-side keyword spotting on a continuous transcript — you run STT and match your own trigger phrase, rather than getting a hardware "wake" event. That works wherever you get a live mic/STT stream (most phone-companion platforms), at the cost of running detection yourself. - **Button and gesture invocation** is the reliable fallback where a wake word is reserved: bind a tap, a button, or a captouch gesture to open your session, sidestepping the wake-word wall entirely. The practical rule: don't design an AI feature around a hands-free wake word until you've confirmed the platform gives you a bindable trigger. If it doesn't, build on a continuous-transcript keyword spotter or an explicit button/gesture — both generalize across the whole category. ## Where each platform sits Putting the three questions together — first-party assistant, BYO-AI placement, invocation: | Platform | First-party assistant | Your AI runs | Notable | |---|---|---|---| | [Snap](/docs/ecosystem/platforms/snap) | None open | Cloud (Remote Service Gateway) | BYO-AI Lenses are *publicly publishable* | | [Brilliant Labs](/docs/ecosystem/platforms/brilliant-labs) | OSS, BYO-key | On-device + self-host | Most open; you own the whole stack | | [Rokid](/docs/ecosystem/platforms/rokid) | Wake reserved, model open | Cloud / on-glasses (AIUI) | Open at the model layer | | [RayNeo](/docs/ecosystem/platforms/rayneo) | Closed, region-split | In your standalone APK | Gemini (West) / Qwen (China) | | [Vuzix](/docs/ecosystem/platforms/vuzix) | None | Phone-side | Display-only peripheral; BYO-AI fully open | | [Solos](/docs/ecosystem/platforms/solos) | Closed | Phone-side | Audio-first, no HUD | | [HTC VIVE Eagle](/docs/ecosystem/platforms/htc-vive-eagle) | VIVE AI, closed | Phone-side | Displayless; SDK gives camera, audio and transcription | | [INMO](/docs/ecosystem/platforms/inmo) | "OK Amu" reserved | On-glasses + managed layer | Standalone Android 14; agent platform (beta) | | [Even Realities](/docs/ecosystem/platforms/even-realities) | Even AI | Phone-side (`fetch`) | Agent Configuration routes Even AI to your own endpoint | | Omi | None | On-device / BYO keys | Full AI access, local models | | Mentra | Mentra AI (as actions) | Phone-side | Transcription events + MCP-shaped actions | | [Meta](/docs/vendors/meta) | Meta AI, closed | Phone-side (DAT companion) | Production target for Extentos | | [Android XR](/docs/vendors) | Wake reserved, API open | System / your app | GA-gated | Platforms with no first-party AI surface reachable by your code — Epson Moverio, XREAL, VITURE, Iristick, Lenovo ThinkReality, and Jorjin (whose "AI-XR Solution" and J7M "AI-powered Assistant" publish no API) — are pure phone-side BYO-AI: whatever you build is the only AI on the device. Emteq is a special case: its ML is sensing (affect/expression), not a conversational assistant, and it's not open to third-party models. ## Where Extentos fits Extentos targets [Meta](/docs/vendors/meta) today, where Meta AI is closed — so Extentos is a concrete instance of the pattern this page describes: **the branded assistant is reserved, but the voice-in → AI → voice-out surface is open, and Extentos builds on that surface.** Its [assistant runtime](/docs/concepts/assistant) wraps the full pipeline (wake, STT, tool-calling model, TTS, barge-in) behind one API, running the model **phone-side / cloud** through a [managed gateway](/docs/concepts/ai-gateway) — the same placement as Vuzix, Solos, and Even Realities in the table above, just packaged as a capability instead of a hand-wired pipeline. The reason this generalizes is structural: Extentos's [capability vocabulary](/docs/concepts/capabilities) — microphone, STT, `speak`, voice triggers — is the same voice-in/voice-out surface every vendor exposes in some form. A closed first-party assistant on the next vendor doesn't block an Extentos app, because the app was never depending on it. That's the whole bet: build on the pipeline that's open everywhere, not the assistant that's closed almost everywhere. (Extentos doesn't and can't expose a vendor's reserved wake word or first-party assistant — those stay the vendor's, as they do for any third party.) ## Related - [The assistant runtime](/docs/concepts/assistant) — Extentos's voice-in → AI → voice-out pipeline as one API - [The managed AI gateway](/docs/concepts/ai-gateway) — where the cloud-placement model actually runs and meters - [Capabilities](/docs/concepts/capabilities) — the vendor-agnostic microphone/STT/speak vocabulary this all builds on - [Vendors](/docs/vendors) — the full per-platform reference this page draws from --- source: https://extentos.com/docs/ecosystem/distribution --- --- title: How third-party smart-glasses apps are distributed seoTitle: How are third-party smart glasses apps distributed? description: "Smart glasses don't have one app store. The app-model taxonomy — companion mobile app vs. an app that runs on the glasses (native or web) — and why a single platform (Meta) offers two entirely separate developer systems: DAT and Ray-Ban Display Web Apps." type: guide verified: "2026-08-24" platform: all related: - /docs/ecosystem - /docs/vendors/meta - /docs/concepts/architecture --- There is no single "smart glasses app store," and that surprises almost everyone coming from phones. Distribution on smart glasses is downstream of one architectural choice — **where your app runs, and whether it's native or web** — and that choice differs not just by platform, but sometimes *within* one platform. Meta alone ships two entirely separate third-party systems. This page is the taxonomy; per-platform SDK detail lives on the [vendor pages](/docs/vendors). ## Two app models — plus a wrinkle The dividing question is: does your code run on the **phone** (with the glasses as a peripheral), or on the **glasses** themselves? Everything about distribution follows from that. (A VR-style "headset app" — Android XR *headsets* like Project Moohan — is a separate device category, not smart glasses, and is out of scope here.) ### 1. Companion / extension app (runs on the phone) You extend a mobile app that ships through an existing store; the glasses are camera, microphone, and speaker, reached over Bluetooth. The platform is expanding a broad third-party ecosystem around a large install base. **Meta's DAT and Android XR** are here. On **Meta** (the Wearables **Device Access Toolkit**), this makes distribution a **two-part** problem, which is the crux of the confusion: - **The phone app** ships through the channel you already know — Google Play or sideload/internal tracks on Android; TestFlight / enterprise / Ad Hoc on iOS. - **The glasses link** is separate: the user must be authorized to register your integration with their glasses (on Meta, an invited tester in a release channel — see below). Installing the phone app is not enough on its own. There is no standalone on-glasses app in *this* model — which is exactly why there's no store you submit a "glasses app" to. **Android XR** takes the same broad-ecosystem approach on the Android app + Play foundation (developer program still gating on Google's GA). Coverage here expands as that program firms up. ### 2. An app that runs on the glasses Here the app lives on the device, and distribution is its own path — not tied to a phone-app store. Two flavors: - **Web app (deploy a URL).** **Meta Ray-Ban Display Web Apps** — you host a standard web app over HTTPS and the glasses load it. See the dedicated section below; this is the "separate Meta system" most developers don't know exists. - **Native standalone.** **Snap Spectacles, Rokid, RayNeo and INMO** — dedicated glasses apps that run on the device's own OS, built and distributed through the platform's own (often open) ecosystem, aimed at a different, more hardware-forward audience. ## Meta's two systems: DAT vs. Display Web Apps This is the part that trips people up: on Meta smart glasses, "build a third-party app" has **two different answers**, and they barely overlap. | | **Device Access Toolkit (DAT)** | **Display Web Apps** | |---|---|---| | What you build | A native **iOS/Android app** | A **web app** (HTML/JS, standard web APIs) | | Where your code runs | On the **phone**; glasses are the peripheral | **On the glasses** (a WebView on Ray-Ban Display) | | Which glasses | Every Meta model (camera + audio); native display on Ray-Ban Display | **Ray-Ban Display only** | | Capability focus | Camera, microphone, audio, plus a native display tree (`glasses.display.*`) | Display, input, sensors, location, local storage (web APIs) — no camera/audio path | | How you deploy | Publish/sideload the mobile app | **Host it over HTTPS at a public URL** | | How a user adds it | Register the integration via the Meta AI app (release-channel tester) | **Meta AI app → App Connections → Web Apps → Add → your URL** | | Distribution today | Release channels (invited testers); public publishing partner-gated | **Share a link** (recipient needs Developer Mode); appears in the glasses app grid | | Docs / repo | `meta-wearables-dat-*` | `meta-wearables-webapp` (separate) | They're complementary, not competing: DAT is the path for camera/voice/audio apps across the whole lineup; Web Apps is a fast, hostless-of-app-stores way to put an interactive **UI on the Ray-Ban Display**. **Extentos builds on DAT** — the native, cross-model, camera/voice/display path — not the Web Apps system. ### Deploying a Ray-Ban Display Web App The Web Apps flow is refreshingly light — no store submission, no review, no native build: 1. **Host** your web app over HTTPS at a publicly reachable URL (any host works; the glasses just load the page). 2. In the **Meta AI app**, enable **Developer Mode**, then go **App Settings → App Connections → Web Apps → Add a Web App**, give it a name and your URL, and tap **Connect**. 3. It appears immediately in your **Ray-Ban Display app grid** — pin it, launch it, drive it with swipes + pinch (a middle-pinch opens a Restart / Resume / Permissions menu). 4. **Share it** with the *Share link* button; anyone with Developer Mode is one tap from adding it. Requirements today: Ray-Ban Display glasses on software `v125+`, Meta AI app `v272+`. Meta ships **no first-party store** for Web Apps, so official "distribution" is link-sharing — and unlike the DAT path there is no invite list, tester roster or release channel at all; the only friction is that a recipient needs Developer Mode enabled. Discovery isn't zero, though: **community catalogs** — [MetaGlassApps](https://metaglassapps.com/) and [GlassApps](https://glassapps.io/) — index Display Web Apps and accept submissions. They're unofficial, but they're where these apps actually get found today. ## Meta DAT: the two distribution paths in practice For the native (DAT) path — the one Extentos targets — distribution has two lanes: - **Release-channel distribution (works today).** In the Meta Wearables Developer Center you create versions of your integration and assign them to release channels, then invite testers by email. Channel membership is what lets a user register your app with their glasses through the Meta AI app — there's no Meta content review. The phone app ships separately through your normal channels. **iOS App Store publishing isn't supported yet** (Meta's SDK triggers App Store rejection under Apple's MFi + privacy-manifest rules); TestFlight + release channels is the iOS path. - **Public publishing (partner-gated).** Publishing so *any* user can register your integration without an invitation is limited to select Meta partners during the developer preview. Meta has said broader publishing is expected in 2026. There's no store-style storefront for glasses integrations today — users manage them in the Meta AI app. The full mechanics, the launch-partner picture, and the required setup are on the [Meta smart glasses](/docs/vendors/meta#distribution-state-2026-07) page. ## Why this matters for how you build 1. **Pick the system before the framework.** On Meta, "camera/voice app for any model" means DAT (native); "an interactive panel on the Ray-Ban Display" can mean either DAT's `glasses.display.*` or a Web App — different toolchains, different distribution. Choose deliberately. 2. **Plan for tester enrollment, not a store launch.** Across every platform's preview, go-to-market today is release channels / link-sharing behind Developer Mode — not "submit and wait for approval." 3. **Keep app logic vendor-agnostic.** The capability surface (camera, voice, audio, display) is stable across platforms even where the app models and distribution stories differ, so let the *transport* differ and keep your handler code portable. That's the [vendor-portability model](/docs/vendors). ## Related - [The third-party smart-glasses landscape](/docs/ecosystem) — the platform comparison matrix - [Meta smart glasses → distribution state](/docs/vendors/meta#distribution-state-2026-07) — the concrete DAT mechanics - [Architecture](/docs/concepts/architecture) — how the phone-app-plus-glasses model fits together --- source: https://extentos.com/docs/ecosystem/openness --- --- title: "How open is smart-glasses development?" seoTitle: "Smart-Glasses Development Openness: The Full Spectrum" description: "A mental model for the openness spectrum in smart-glasses development — SDK availability, license, build-time gating, and publishing gating — with today's platforms placed on it." type: guide verified: "2026-08-24" platform: all related: - /docs/ecosystem - /docs/ecosystem/distribution - /docs/ecosystem/ai --- "Is this platform open?" is the wrong question. It compresses at least four independent decisions into one word, and every smart-glasses vendor answers them differently. A platform can hand you a free, permissively-licensed SDK and still refuse to let you ship to real users. Another can gate the SDK behind a paid program yet let you publish to a public store in a weekend. If you pick where to build on a single "openness" vibe, you will be surprised at exactly the wrong moment — usually the day you try to put your app in someone else's hands. This page gives you the model we use to reason about it, then places today's platforms on it. ## Openness has two phases, not one The single most useful distinction: **build-time openness and ship-time openness are separate, and they don't correlate.** - **Build-time** — Can you get the SDK, write code, and run it on the hardware? This is where the SDK license, price, and the "who is allowed to build" gate live. - **Ship-time** — Can you get that code into an end user's hands, and how wide can that audience be? This is a completely different gate: a store, a review pipeline, a partner agreement, or nothing at all. [Meta](/docs/vendors/meta) is the canonical example of the split: the SDK is free and public, and Developer Mode runs your own build on your own glasses with no invite and no review — but getting that build onto anyone else's glasses means inviting them by email to a release channel, and public publishing is partner-gated. [Vuzix](/docs/ecosystem/platforms/vuzix) is the opposite corner — open SDK, and because the Z100 is a display-only Bluetooth peripheral, you ship an ordinary phone app through Google Play or the App Store with no glasses-specific gate at all. [Even Realities](/docs/ecosystem/platforms/even-realities) splits the other way: a genuine public web/TypeScript SDK, but its first-party store is manually reviewed. Hold those two phases apart and the rest of the model falls out cleanly. ## The four dimensions Score any platform on these four and you have its real posture: 1. **SDK availability & cost** — Is there a public SDK you can download *today* without an invite? Free, paid, or application-gated? 2. **License** — Permissive open source (BSD / MIT / Apache-2.0), or a proprietary closed binary (an AAR, an xcframework, a signed runtime)? This governs whether you can fork, self-host, audit, and survive the vendor changing direction. 3. **Who-can-build gating** — Anyone; a dev-mode toggle; a paid developer program; a light application form; or invite/partner-only. 4. **Publishing gating** — Self-distribute with no gatekeeper; an open store with light review; a curated/manually-reviewed store; phone-store-only (ordinary app review); partner-gated; enterprise-MDM-only; or no path at all. A platform that is open on all four is a place you can move fast and keep control. A platform that is open on (1) and (2) but closed on (4) is fine for prototypes and demos and a trap for a product. ## The spectrum From most open to most closed. The tiers are about *net* freedom to build and ship to real users — a permissive license pulls a platform up, a publishing wall pushes it down. ### 1. Fully open — OSS SDK, self-distribute, no gatekeeper You get open-source SDKs, you keep the keys, and you distribute your own host app, so there is no one to ask permission of at either phase. - [Brilliant Labs](/docs/ecosystem/platforms/brilliant-labs) (Frame & Halo) is the most open in the category: OSS multi-language SDKs (BSD-3/MIT), an on-device Lua VM over plain BLE, a first-party IDE, an open-source reference host app, open hardware/firmware, and an OSS BYO-key/self-host assistant. You ship your own app; nobody gatekeeps either phase. - MICROOLED ActiveLook — an open BLE SDK (Android + iOS + a generic API) you embed in your own app, no gatekeeper. - Omi (OmiGlass) — open SDKs (Python, Swift, RN, Expo), full AI access with BYO keys or local models, an app marketplace *and* open GitHub self-distribution. - Mentra (MentraOS) — an MIT TypeScript SDK (`@mentra/miniapp`, in beta) with a store and LAN QR install. Notably it is *cross-vendor*, which is why it also shows up as the ungated path on hardware whose first-party channel is narrower (see Even Realities and Vuzix below). ### 2. Open to build and ship — permissive-enough SDK, open publishing, proprietary core Publishing is genuinely open, but the SDK itself is proprietary or carries a cost/hardware gate. Net freedom is still high. - [Snap Spectacles / Specs](/docs/ecosystem/platforms/snap) — a free public SDK (Lens Studio 5) plus an independent WebXR path, and public Lens Explorer distribution with a narrow ~2–3 day review. Camera/mic/location + internet *is* publicly publishable via Transparent Permissions, including BYO-AI Lenses through the Remote Service Gateway. The only real wall is hardware access — and it got higher: Snap has withdrawn the published sign-up for its subscription developer program (announced at $99/month US, €110/month in six European countries) without an end-of-life notice, leaving a $2,195 Specs pre-order as the only listed purchase route to hardware. - [Vuzix](/docs/ecosystem/platforms/vuzix) — a free, publicly downloadable native SDK (Android + Apple) plus MIT-licensed MentraOS; a Z100 app is an ordinary phone app in the normal stores. - [HTC VIVE Eagle](/docs/ecosystem/platforms/htc-vive-eagle) — a free, anonymously downloadable iOS + Android SDK (0.6.0 beta) and an ordinary phone app that ships through the App Store and Google Play, so ship-time is the normal store review. The qualification is real, though: the app-registration step that lets your app talk to the glasses is undocumented, and the only published route to it closed in July 2026. - [Solos](/docs/ecosystem/platforms/solos) — a paid but public SDK (standard iOS/Android, commercial use allowed) with no application step; ship-time is wide open through the normal app stores. Audio-first, no HUD. - Iristick — a native Android + iOS SDK behind a paid, NDA-bound developer program (€1,137.50 one-time including the first three months, then €1,000/yr); distribution through Google Play and the App Store requires the Certified Software Partner tier, while the SDK Licence path is internal-use only. - [Even Realities](/docs/ecosystem/platforms/even-realities) *(first-party track)* — a public web/TypeScript SDK with **self-serve** developer entry (signing in to the web hub flips your account to developer); the remaining gate is ship-time, since the Even Hub store is manually reviewed. Private and email-invited Beta builds install unreviewed, and MentraOS + raw-BLE community tooling reaches both G1 and G2 with no Even gate at all. - [INMO (Air3)](/docs/ecosystem/platforms/inmo) — a standalone Android 14 device that ships internationally with the Google Play Store, plus a Unity SDK; publishing mixes an open Play path with a curated first-party Air3 store. Qualified openness — the prose API reference sits behind a login-walled Feishu doc, though the SDK and the XMLDoc reference shipped inside it are login-free. ### 3. Dev-mode / sideload — open to build, no consumer publishing surface You can build freely and run on-device via ADB/sideload, but there is **no public store or review pipeline** to reach a wide audience yet. Great for internal tools and demos; you own the distribution problem. - [RayNeo](/docs/ecosystem/platforms/rayneo) — an open, standard-based build path (MIT OpenXR Unity ARDK + native Android ARDK, no invite gate), but a real developer-mode unlock and everyday delivery via ADB or community sideload tooling. No self-serve store, though RayNeo's X-series docs publish a manually brokered submission and review route. - [Rokid](/docs/ecosystem/platforms/rokid) — four parallel dev axes, and the build path needs no account at all: the Maven repo and the AIUI tooling both resolve anonymously. The on-glasses AIUI runtime (JSAR) is Apache-2.0 while the native SDKs are closed AARs. The exception is the cloud OpenAPI behind the BYO-model agents, which requires an enterprise account arranged through sales. Sideload works today (phone app, WebUSB, community stores); a first-party native store is still emerging. - XREAL — a Unity SDK; sideload / OTA via ControlGlasses. - VITURE — a cross-platform + Unity SDK; distribution through host-platform stores. - Epson Moverio — a native Android AAR SDK; sideload APK, no store. - Lenovo ThinkReality A3 — Snapdragon Spaces (OpenXR, Unity/Unreal); sideload, Google Play, or Lenovo-managed. ### 4. Application- or partner-gated build The SDK exists but you must be approved to get it or to onboard through the first-party channel. - Jorjin J-Reality — the J7 SDK is sales-gated; distribution is partner/ODM. ### 5. Invite / partner preview — open to build, gated on who can receive it - [Meta](/docs/vendors/meta) — a DAT companion SDK and Display Web Apps, free to build against: Developer Mode runs your unpublished app on your own glasses with no invite and no publishing review, though Meta's FAQ limits the full toolkit — including the Wearables Developer Center — to developers in AI-glasses supported countries. The gate is ship-time: every DAT tester must be invited by email to an invite-only release channel, and public publishing is partner-gated; Meta AI is closed to developers. The most-watched platform, and among the most gated at ship time. - [Android XR](/docs/vendors) — the Jetpack XR SDK is public, but the platform is GA-gating with Gemini as the assistant; treat it as staged-rollout rather than freely shippable today. ### 6. Enterprise-managed Open enough to build, but distribution assumes a managed fleet — the "public store" is an MDM console, not a consumer channel. - RealWear — a standard Android SDK plus WearHF/WearML; RealWear Cloud + APK sideload; the Ari assistant is end-user-facing, BYO-AI for your own apps. - DigiLens (ARGO) — a DigiOS SDK (AOSP 12) + Unity/Spaces; sideload + enterprise MDM; Gemini voice via Google Cloud. - ThirdEye Gen — Android (Java) + Unity + a VisionEye SLAM SDK; you upload APKs to a ThirdEye App Store dashboard. - Emteq Labs (OCOsense) — a Unity SDK + Python LSL client shipped as a developer kit via GitHub/UPM; a specialized affect/expression sensing platform rather than a general app target. ### 7. Closed / none - [Apple](/docs/vendors) — no third-party glasses SDK yet. There is nothing to build against. ## The openness map at a glance | Platform | SDK license | Who-can-build gate | Publishing gate | |---|---|---|---| | [Brilliant Labs](/docs/ecosystem/platforms/brilliant-labs) | OSS (BSD-3/MIT) | None | Self-distribute, none | | ActiveLook | Open BLE SDK | None | Embed in your app, none | | Omi | OSS | None | Marketplace + GitHub | | Mentra | MIT | None | Open store + web-link | | [Snap](/docs/ecosystem/platforms/snap) | Free, proprietary | Subscription sign-up withdrawn; buy Specs ($2,195) | Open Lens Explorer, ~2–3 day review | | [Vuzix](/docs/ecosystem/platforms/vuzix) | Free native + MIT MentraOS | None | Ordinary phone stores | | [HTC VIVE Eagle](/docs/ecosystem/platforms/htc-vive-eagle) | Free, proprietary | None to build; undocumented to register | Ordinary phone stores; glasses link needs HTC registration | | [Solos](/docs/ecosystem/platforms/solos) | Paid, public | Buy the $1,999 kit; no application | Ordinary phone stores | | Iristick | Native, credential-gated | Paid program: €1,137.50 one-time, then €1,000/yr + NDA | Certified Software Partner: Play + App Store; SDK Licence internal-only | | [INMO](/docs/ecosystem/platforms/inmo) | Unity + native | Buy device; no application | Play + curated Air3 store | | [RayNeo](/docs/ecosystem/platforms/rayneo) | MIT OpenXR + native | Dev-mode unlock | Sideload; brokered submission | | [Rokid](/docs/ecosystem/platforms/rokid) | Mixed (AIUI Apache-2.0; native closed) | None | Sideload; native store emerging | | XREAL | Unity SDK | None | Sideload / OTA | | VITURE | Cross-platform + Unity | None | Host stores | | Epson Moverio | Native AAR | None | Sideload, no store | | Lenovo A3 | Snapdragon Spaces | None | Sideload / Play / managed | | Jorjin | J7 SDK | Sales request | Partner/ODM | | [Even Realities](/docs/ecosystem/platforms/even-realities) | Public web/TS (+ MentraOS) | Self-serve (web-hub sign-in) | Curated store (or self-host BLE) | | [Meta](/docs/vendors/meta) | DAT companion, closed AI | Developer Mode; no invite (supported countries) | Invite-only release channels; partner-gated | | [Android XR](/docs/vendors) | Jetpack XR | GA-gating | Staged | | RealWear | Standard Android | None | Cloud/MDM + sideload | | DigiLens | DigiOS (AOSP 12) | None | Sideload + MDM | | ThirdEye | Android + Unity | None | ThirdEye dashboard store | | Emteq | Unity + Python | Dev kit | GitHub/UPM | | [Apple](/docs/vendors) | None yet | — | — | ## Where the phone-companion model sits A cross-cutting factor the table hints at: **who runs your code changes the gate.** When the glasses are a display-or-audio peripheral and your app runs on the phone, publishing collapses to ordinary phone-store review — which is why display-only peripherals like Vuzix and audio-only Solos land in the "open to ship" tier despite proprietary SDKs. When the app runs *on the glasses*, you inherit that device's store, sideload story, or partner gate. Extentos targets the phone-companion side of that line: the app lives on the phone, the glasses are the transport, and distribution rides the phone stores you already ship through. That inherits the ship-time openness of the peripheral model rather than a per-vendor on-glasses store — one reason the model ports across several of the vendors above. Which posture is right is a function of your product, not a universal ranking; the point of this page is to make the trade legible, not to sell one answer. ## What this means for choosing where to build Read the two phases against your actual goal: - **Prototype or internal tool, wide reach not required?** Tier 3 (sideload/dev-mode) is fine — [RayNeo](/docs/ecosystem/platforms/rayneo), XREAL, [Rokid](/docs/ecosystem/platforms/rokid). Don't over-index on the missing store; you don't need it yet. - **Consumer product you must ship broadly?** Weight **ship-time** heavily. Open publishing (Tiers 1–2) beats a great SDK trapped behind a partner gate. A brilliant SDK you can't distribute is a demo. - **You need to fork, audit, self-host, or outlive the vendor?** Weight **license.** Only Tier 1's permissive OSS (Brilliant Labs, ActiveLook, Omi, Mentra) gives you that guarantee; a proprietary AAR can change under you. - **You want to own the AI layer?** Openness of the *SDK* and openness of the *assistant* are yet another independent axis — Snap and Brilliant Labs invite BYO-AI, while Meta's assistant is closed and RayNeo's is closed and region-split. If AI is your product, check that axis explicitly, not the general openness label. - **Betting on a not-yet-open platform?** [Meta](/docs/vendors/meta), [Android XR](/docs/vendors), and [Apple](/docs/vendors) may open further; build with an eye on portability so a gate that opens late doesn't strand you. The honest summary: there is no single "most open" platform for every job. Rank the four dimensions by what your product actually needs to survive, score your candidates on each, and let the two-phase split — build vs ship — do the deciding. Each vendor page linked above carries the specifics for that scoring. --- source: https://extentos.com/docs/ecosystem/platforms/alibaba-qianwen --- --- title: "Alibaba Qianwen AI Glasses" seoTitle: "Alibaba Qianwen AI glasses for developers: Skill platform, MCP & distribution (2026)" description: "Alibaba Qianwen AI Glasses for third-party developers — the 千问AI硬件开放平台 Skill platform, MCP tool integration, app model, distribution, capabilities & AI, and where it sits in the 2026 smart-glasses landscape." type: reference verified: "2026-08-25" platform: all vendor: alibaba related: - /docs/ecosystem - /docs/ecosystem/openness - /docs/ecosystem/ai - /docs/ecosystem/distribution --- **Openness verdict.** Read the entire platform and its documentation set without an account, and wire your own MCP server into a pair of glasses in an afternoon — but you cannot register, bind a device or publish anything without a mainland-China identity document. **Covered here:** Alibaba's 千问AI眼镜 (Qianwen AI Glasses) S1 and G1, and the 千问AI硬件开放平台 developer platform at device.qianwen.com — both its self-serve Skill platform and its brokered industry platform. **Not covered here:** Tmall Genie speakers and the 哇哦闺蜜机 portable screen (same organisational lineage, not eyewear) · Alibaba Cloud Bailian (百炼) and Coze (扣子), which this platform *consumes* as Agent plugins rather than competing with · the Qwen model family and the 千问 assistant app, which are software, not wearables. ## Overview Alibaba sells two pairs of AI smart glasses and runs a developer platform for them that looks like nothing else in this landscape: on the self-serve path there is no SDK to download, no app to compile, and no code that runs on the device. A third-party integration is a **Skill** — a `SKILL.md` file with YAML frontmatter, executed by Alibaba's cloud agent, which calls tools you register. Those tools can be your existing REST API, an **MCP server**, or an agent you already built on Bailian or Coze. The platform describes itself as the official ecosystem platform for the glasses, "为企业和个人开发者提供AI眼镜定制和眼镜技能一站式搭建能力" ([千问AI硬件开放平台](https://device.qianwen.com/)), and it splits in two: a **技能平台** (Skill platform) open to individual and enterprise developers, and a **行业开放平台** (industry platform) for enterprises building custom units. **Two models, and the difference decides what you can build.** The industry platform names them as the two base products: **千问AI眼镜S1（带屏显）或者G1（不带屏显）** — S1 with a display, G1 without ([platform introduction](https://device.qianwen.com/docs/qwen/6674064)). Alibaba's own portal product cards describe S1 as a 全感知旗舰 AI 终端 whose 显示方案 reads **双显双光机** (dual display, dual optical engines), aimed at industrial inspection, retail, remote collaboration and announcements; and G1 as a 轻量化智能 AI 终端 whose display field simply reads **无屏幕**, aimed at in-car use, voice announcements and lightweight capture ([device.qianwen.com](https://device.qianwen.com/), product cards in the portal's own shipped front-end). No optical stack, resolution or brightness figure appears on any surface readable without a login, so none is claimed here. The one display property Alibaba does state sits on the UI-tool pages: the on-glasses display is **单原色**, single primary colour, with a small display area. **Shipping status and price need stating carefully.** Both SKUs have live Tmall retail listings, linked by item id from the developer portal's own 产品中心 footer — S1 as item `1035065479567`, G1 as item `1020553105651`. Both of those pages bounce an unauthenticated request to `login.taobao.com`, from desktop, mobile and crawler user agents alike, and on the one shell that does render the price element is served empty. So **no price is quoted on this page**, and purchasability and stock could not be confirmed for either SKU. The dated evidence that the hardware ships is indirect but solid: the 千问AI穿戴 companion app has been live on the China App Store since **21 October 2025** and was last updated **10 August 2026** ([App Store listing](https://apps.apple.com/cn/app/id6744239632)), and the developer docs treat real hardware as the normal test target. Regions: no purchase path outside mainland China was found. There is no English developer surface — `/en` and `/en/docs` both 404, `global.qianwen.com`, `developer.qianwen.com` and `device.qwen.com` do not resolve, and all 49 documentation entries are `lang="zh-Hans"`. One nuance cuts against a flat "China-only": the companion app itself *is* listed in the US and Hong Kong App Stores as well as China, and absent from GB, DE, JP and SG. That is an app listing, not a way to buy the glasses. **On the name.** Our earlier watch entry called these Quark AI Glasses. Alibaba has moved them onto the 千问 brand: the old portal host `wearables.aligenie.com` now 302s to `device.qianwen.com` (while the new portal's JSON-LD still carries `"url":"https://wearables.aligenie.com/"`), the portal's own careers link points at `talent.quark.cn`, the account-upgrade guide walks developers off their 原淘宝账号 onto a 千问账号, and **zero** occurrences of 夸克 remain anywhere in the current portal front-end or its documentation. The old name survives on a sibling surface Alibaba forgot: its Tmall Genie hardware site still lists `天猫精灵,夸克ai眼镜,夸克智能眼镜,哇哦闺蜜机` in its meta keywords ([tmallgenie.com](https://tmallgenie.com/pages/pcHome.html), re-read 25 August 2026). Use **千问AI眼镜 / Qianwen AI Glasses**. ### What you can buy ## Access | | | | --- | --- | | **What you actually get** | On the self-serve path, not an SDK: a **web console** where you register tools, author a Skill, debug it in a sandbox, bind a real device by QR, and publish. Nothing is downloaded and nothing is compiled. A real native SDK exists for enterprises — see below | | **Languages** | None, in the usual sense. A Skill is Markdown; your tools are whatever your own server is written in. The one place a language appears is **Groovy** — "脚本语言仅支持 Groovy，不支持其他脚本语言" — and only for OpenAPI-tool request signing ([tool access](https://device.qianwen.com/docs/qwen/6679330)) | | **Maturity** | Brand new and moving fast. All **49 entries** in the published documentation tree carry a `gmtModify` stamp, and every one falls between **14 July 2026** and **18 August 2026**. The RTC audio/video path is explicitly labelled **功能公测中** (in public beta). No version number is published for the platform itself | | **Who can build** | Anyone who can pass Chinese identity verification. Sign-in is 手机号 / 千问App扫码 / 淘宝 / 支付宝, and then "登录成功后，请根据页面引导完成个人实名认证或企业认证" ([access guide](https://device.qianwen.com/docs/qwen/6668302)). The portal's own registration form validates an **18-digit mainland-China ID number** and tells Hong Kong, Macau and Taiwan applicants to email `AliGenie@list.alibaba-inc.com` instead ([device.qianwen.com](https://device.qianwen.com/), validator in the shipped front-end). Enterprise verification needs a **corporate Alipay account**; applicants who cannot use one — the portal names large foreign enterprises, "如大型外企", as its example — are routed to an offline email approval process | | **Cost** | **No fee is published, and no page states that there is none.** The words 免费, 收费, 费用, 付费 and 价格 appear on none of the 49 documentation entries, and no billing console is documented. What *is* published are quotas: official demo tools are capped at **30 calls per API per user per day** ([usage limits](https://device.qianwen.com/docs/qwen/6683675)), and the platform model tools carry an unpublished daily quota you raise by asking support in a DingTalk group ([model service](https://device.qianwen.com/docs/qwen/6817147)) | | **Licence** | No developer licence or terms of use are published in the documentation. The only published legal text is the Professional Edition SDK's compliance guide and privacy notice, both carrying 更新日期/生效日期 **15 July 2026** | | **Docs** | [device.qianwen.com/docs](https://device.qianwen.com/docs) — Chinese only, and genuinely public: pages return HTTP 200 with `window.loginStatus = false` and no cookie, while the console at `/guide` 302s to `/account/login` | **There is a native SDK, and it is the part you cannot have.** The **千问AI硬件开放平台专业版 SDK** "面向接入方 App 提供双端一致的设备接入能力" — consistent device access across both mobile platforms — and its capability page reads like a real SDK reference: Bluetooth scan and dedupe, connect and disconnect, battery and brightness state, camera capture returning bytes versus on-device save, video record, live preview, unsynced-media badges, mic pickup direction, and Android process revival for background stability ([APP capability opening](https://device.qianwen.com/docs/qwen/6668299)). Alibaba's own privacy notice for it states "本SDK支持的平台：iOS版、Android版" and lists its core functions as 账号授权、蓝牙连接、设备管理、拍照、录像、实时视频流、媒体同步、麦克风录音, with 上海智信普惠科技有限公司 as provider ([SDK privacy notice](https://terms.alicdn.com/legal-agreement/terms/privacy_permission_description/20260709101902905/20260709101902905.html)). And there is no download link on any of the 49 public documentation entries — no Maven, Gradle, CocoaPods, `.aar` or `.framework` instruction of any kind. Access begins with "需要业务侧与官方客服获取 accessKeyId、accessKeySecret 等必要信息", and the industry platform states outright that **当前暂无完全自助式配置后台，接入采取线下对接的方式** — no fully self-service console; integration happens offline ([access guide](https://device.qianwen.com/docs/qwen/6668302)). A GitHub search turns up no official repository — nothing under the `alibaba`, `QwenLM` or `alibaba-inc` orgs, and the `aligenie` and `qianwen` orgs do not exist — only a handful of zero-star third-party repos. The logged-in console behind the identity gate was not inspected. ## App model For a self-serve Skill, code runs **in Alibaba's cloud and on your own servers** — never on the glasses, and never in a phone app you ship. The unit of work is a Skill directory, and its shape will look familiar: ``` skill-name/ ├── SKILL.md # required, main file ├── tool_schema.json # required, generated by the platform ├── references/ # optional, loaded on demand └── assets/ # optional, loaded on demand ``` `SKILL.md` carries YAML frontmatter with exactly two required fields — `name` (2–64 chars) and `description` (up to 1000 chars, "Agent 凭此字段决定是否激活 Skill") — and the platform is strict about it: "文件名必须是 SKILL.md，大小写敏感，不接受 skill.md 或 SKILL.MD" ([Skill specification](https://device.qianwen.com/docs/qwen/6679386)). Only `name` and `description` load at startup; the full instructions load when a request matches. Body content is recommended under 500 lines and hard-capped at 2000 lines or 80,000 characters, with overflow pushed into `references/` for on-demand loading. Tools are referenced from the body with a `^^{pluginCode}.{toolCode}^^` syntax, inserted with `@` in the editor, and the platform writes `tool_schema.json` for you. The layout mirrors Anthropic's Agent Skills convention closely, though the vendor never says so — Anthropic is not named anywhere in the documentation set. **Three enterprise-only routes escape the cloud-only model:** the Professional Edition SDK inside your own phone app; your own **H5 pages hosted inside Alibaba's companion app** (see Distribution); and an RTC stream your own server consumes. The 千问AI穿戴 companion app is Alibaba's and is not optional. | | | | --- | --- | | **How your service is reached** | Three plugin types, one model. **OpenAPI** for an existing REST API (manual or document import, per-parameter position and source mapping, Groovy for signing). **MCP** for a Model Context Protocol server. **Agent** for an existing Bailian or Coze app, configured with `appId` + `apiKey`. A plugin is the connection layer, tools are the capability layer, and `${pluginCode}.${toolCode}` is what the model calls ([tool access](https://device.qianwen.com/docs/qwen/6679330)) | | **MCP specifics** | The platform connects on save, calls `tools/list` itself and registers everything it finds — "**平台自动从 MCP Server 拉取工具列表**". Three transports: **SSE / Streamable HTTP / 无状态 HTTP**. Auth is none, Bearer, or a custom header or query parameter, never a path parameter. Import is a Claude-style `mcpServers` JSON blob, and a 刷新工具 button re-pulls when your server changes. Parameter descriptions come from your server and cannot be edited on the platform, and MCP tools do not support streaming responses (暂不支持) where OpenAPI tools do | | **Transport to the device** | Bluetooth from glasses to the 千问AI穿戴 companion app, then cloud. Photos take the documented route 下发拍照指令 → 眼镜拍照 → 蓝牙回传App → App上传云端 → 云端临时存储 → 返回照片可访问地址. Real-time media takes a separate path, pushed to an RTC service the protocol pins to `"provider": "ALIYUN_RTC"` ([interaction protocol](https://device.qianwen.com/docs/qwen/6817146)). The Professional Edition SDK adds a local direct network channel for low-latency video | | **Multiple dev systems?** | Yes, and they are unequal. The Skill platform is self-serve. The industry platform — on-glasses desktop apps, custom wake words, an outbound OpenAPI, two-way authorisation, third-party H5 inside the companion app, the native SDK — is enterprise-only, and every one of its pages ends by inviting you to 提交合作意向 | | **Authoring** | Five ways to start a Skill: 一句话创建 (one-sentence AI generation), 工具创建 (AI generation with tools pre-selected), 模板创建 (from a template), 空白创建 (blank), or 上传技能 — uploading a `.zip`/`.tar`/`.tar.gz` up to **20 MB**, one archive at a time, including a Skill someone else wrote ([create](https://device.qianwen.com/docs/qwen/6679381), [usage limits](https://device.qianwen.com/docs/qwen/6683675)) | The inner loop is short. Debug in a sandbox that is isolated from production, watching each tool call's parameters and return values; then bind real hardware: "进入 Skill 编辑页，点击「真机」菜单，页面将生成设备绑定二维码", scanned from 我的 - 扫码 in the 千问 AI穿戴 app, after which you talk to the glasses ([real-device testing](https://device.qianwen.com/docs/qwen/6681505)). Binding is sticky — once on, the device stays in that mode until every Skill is removed in device management. ## Distribution For a consumer, an app reaches a wearer exactly one way: the **skill market inside Alibaba's own 千问AI穿戴 companion app**. There is no sideload and no APK. Publishing offers three scopes ([publish](https://device.qianwen.com/docs/qwen/6679382)): | Scope | Who gets it | | --- | --- | | **全局可见** | 上架到千问AI穿戴APP 中的技能市场 — listed in the skill market in the companion app; any consumer can install it | | **个人私有** | 发布到自己的私有设备上 — your own bound devices only | | **企业专属** | Bulk-mounted onto an enterprise's custom units, after platform staff mount it for you | **One documented route runs inside the companion app rather than through the market.** An enterprise that takes the 双向授权 (two-way authorisation) path can host its own H5 pages and have users open them from within the companion app with no second login — "用户在千问AI穿戴APP 中访问企业开发的 H5 页面无需再次登录", governed by a one-time, ≤5-minute, HTTPS-bound ticket ([third-party H5](https://device.qianwen.com/docs/qwen/6668316)) — and can have a custom entry placed on that app's home screen, either replacing an existing 推荐服务 entry or adding one under 更多服务 ([custom app entry](https://device.qianwen.com/docs/qwen/6668317)). Enterprise on-glasses desktop apps reach devices through an offline process instead: contact platform staff for a `productKey`, agree a `desktopAppCode`, prepare five icon assets, and have the platform enter the configuration for you ([desktop menu](https://device.qianwen.com/docs/qwen/6668306), [icon specification](https://device.qianwen.com/docs/qwen/6668315)). **Public publishing is open in principle and gated twice in practice.** There is no partner list and no invite: any registered developer can submit to the public market. But every version 自动进入小二审核环节 — automatic entry into platform-staff review, with a rejection returning the Skill to 待上架 for resubmission — and **no review criteria are published anywhere**. The page that should carry them, 应用开发与上架规范, exists in the navigation and is published with nothing but its own title. The harder gate is upstream, at the account: without mainland-China real-name or enterprise verification you never reach the console at all. That, not a partner programme, is what closes this platform to a developer outside China. Publishing has one structural constraint worth designing around: "SKILL发布上线时，其引用的所有工具都必须处于已发布", and a published plugin or tool enters a locked state and cannot be edited, which the platform explains as avoiding disruption to live Skills. Plan tool versions before you ship, not after. ## Capabilities, limits & AI The published tool catalogue is short, explicit, and sorted into three categories — 镜端工具 (glasses-side), 云服务工具 (cloud), and 屏显工具 (screen display) ([overview](https://device.qianwen.com/docs/qwen/6820009)). | Capability | Third-party access | | --- | --- | | **Camera (stills)** | `tQ8fCF.capture_photo` — "控制眼镜拍摄一张照片，返回照片的公网可访问地址", returning `{success, errorCode, errorMsg, model:{url, format, mimeType, requestId}}`. Two hard limits: **800 × 600** resolution and a **1-hour** URL expiry. Capture is always perceptible to the wearer — "拍照时眼镜端会有提示音以及提示灯闪烁" ([capture_photo](https://device.qianwen.com/docs/qwen/6817086)) | | **Camera (live) and microphone** | `tQ8fCF.rtc_start_audio_stream` and `tQ8fCF.rtc_start_video_stream` push the wearer's audio, or audio and video, to an RTC service for your own server to receive by joining the room. Published specifications: audio **16-bit / 1 channel / 16 kHz**, video **720p at 2 fps, about 320 kbps**. Marked **功能公测中** (public beta). Alibaba is explicit that you must bring your own ASR and image recognition to close the loop ([RTC streaming](https://device.qianwen.com/docs/qwen/6820210)) | | **Audio out** | Through the agent, not through a device API. A Skill speaks with `Speak` in its body, and during an RTC session your server drives the glasses over the return channel — the open messages are "SSEoutput 事件中的 SPEAK、STOP_OUTPUT", which is a genuine barge-in primitive. `SPEAK` takes plain UTF-8 text only, no SSML and no Markdown ([interaction protocol](https://device.qianwen.com/docs/qwen/6817146)) | | **On-lens display (S1)** | Three fixed templates, no canvas: `UiTool.all_text` renders markdown 到带屏眼镜上 ([text UI](https://device.qianwen.com/docs/qwen/6817149)), `UiTool.pic_and_text` takes title/subtitle/image/content ([image-text UI](https://device.qianwen.com/docs/qwen/6817150)), `UiTool.list` takes a title plus items of icon/type/title/subtitle ([list UI](https://device.qianwen.com/docs/qwen/6817151)). You supply content, the platform owns layout; none of the three takes a callback or input parameter. Some emoji do not render. The vendor's own limits describe the display as **单原色** (single primary colour) with a small display area, ask for simply-styled images — ≤400 × 400 px for image-text, ≤120 × 120 px for list icons — and require every image URL to be served over HTTPS with a fully valid certificate or it will not render. Nothing draws on G1 | | **Location** | `tQ8fCF.get_geo_location` returns longitude, latitude and a report timestamp, but read the fine print: the value is "千问 AI 穿戴 App 上报的原始数据（App 数据源自手机系统定位）" — the *phone's* fix passed through, accurate to the handset, no more than 5 seconds stale, and conditional on the wearer having granted location permission to the companion app ([get_geo_location](https://device.qianwen.com/docs/qwen/6817145)) | | **Sensors and input** | **None exposed in the published tool list.** It is exactly 眼镜拍照、开启音频推流、开启音视频推流、获取经纬度 — no IMU, no head pose, no gesture, no touchpad and no button event appears in it, nor in the enterprise 镜端能力开放 section, nor in the Professional Edition SDK capability table, which tops out at 拾音方向 (mic direction) | | **Reserved surfaces** | The wake word, the launcher and desktop, firmware, the companion app, and the review queue. The wake word is the interesting exception: enterprises can add a **second** one, and "生效后设备将同时支持千问系列唤醒词和客户定制唤醒词唤醒（双唤醒词）", constrained to Mandarin, 3–5 characters, no everyday words and no reduplication ([custom wake word](https://device.qianwen.com/docs/qwen/6668310)) | | **Hard runtime limits** | API and MCP tools must respond within **3000 ms**; Agent tools must return a first packet within **6000 ms**. Skill upload is one archive, **≤20 MB**. Official demo tools are capped at **30 calls/day/user** ([usage limits](https://device.qianwen.com/docs/qwen/6683675)) | | **Enterprise-only additions** | On-glasses desktop apps whose icons launch a tool service that pushes "受控 UI 页面下发至设备端渲染", desktop menu and top-bar customisation, a 自定义画面 render path, RTC streaming triggered from a desktop icon, an outbound OpenAPI, two-way authorisation, and single-sign-on H5 pages inside the companion app. All configured by Alibaba staff, not by you. The icon specification is itself a hardware tell: each 3D desktop icon needs 28 PNG frames **per eye**, 56 in total | ### AI & the assistant This is where Alibaba diverges hardest from everyone else on this map. The platform states plainly that it is built on Alibaba's own model — "平台基于阿里千问模型" ([platform introduction](https://device.qianwen.com/docs/qwen/6679322)) — and rather than reserving that model behind a wake word, it hands a general-purpose model to third parties as two ordinary callable tools. `4F2df8.general_qa_stream` is documented as a "通用大模型问答工具，基于用户 query 生成回答", streaming its answer back in chunks ([model service](https://device.qianwen.com/docs/qwen/6817147)). `4F2df8.image_qa_stream` takes a publicly reachable `imageUrl` plus a query and streams back an understanding of the image ([image recognition](https://device.qianwen.com/docs/qwen/6817148)). Neither tool page names a model, a version or a family, so treat "Qwen" here as the platform's brand rather than a documented model identity. Compose them with `capture_photo` and you have wearable visual Q&A in three tool calls and no AI vendor of your own — Alibaba's own worked example is a museum-artefact identifier that does exactly that. **Your Skill is the agent.** Intent routing, prompting, refusal wording and tool selection are all yours, written in Markdown; the platform supplies the model and the runtime. And bring-your-own is a first-class alternative rather than a workaround: the RTC path streams the wearer's microphone and camera to your infrastructure and lets your server speak back through `SPEAK`, so a fully self-hosted ASR → LLM → TTS loop is a documented shape, not a hack. Third-party agents are welcome from the other direction too — the Agent plugin type imports a Bailian or Coze app with nothing but an `appId` and `apiKey`, and the platform's own quick-start builds its first Skill on a Bailian MCP service ([quick start](https://device.qianwen.com/docs/qwen/6679323)). The one thing that is not open is the front door: Skills are reached by the agent matching your `description` against what the wearer said, and the wake phrase itself stays first-party unless you are an enterprise buying a second one. ## Roadmap **21 October 2025 — the companion app appears.** 千问AI穿戴 lands on the China App Store, published by Shanghai Zhixin Puhui Technology Co., Ltd. (上海智信普惠科技有限公司), and has been updated steadily since; version 1.11.5 shipped 10 August 2026, and the app reached the US and Hong Kong storefronts in November 2025 ([App Store](https://apps.apple.com/cn/app/id6744239632)). This is the earliest hard date available for the product line. **14 July to 18 August 2026 — the entire developer platform is written.** Every one of the 49 documentation entries was last modified inside this five-week window: the industry platform's pages cluster on 16 July, the Skill platform's on 14–17 July, and the whole 开放能力介绍 capability catalogue — photo, RTC streaming, geolocation, model service, image recognition and all three UI tools — on 17–18 August. The Professional Edition SDK's compliance guide and privacy notice both took effect **15 July 2026**. A platform this young with documentation this specific is a real investment, not a placeholder. **In flight now.** The RTC audio and video path is labelled 功能公测中 (public beta) and is the only capability carrying a status marker. The account system is mid-migration, moving developers off Taobao accounts onto 千问 accounts. Six of the 49 entries are section landing pages carrying nothing but their own title, and a seventh — 应用开发与上架规范, the page that should carry the review criteria — is published with nothing but its title as well. **Announced but unshipped, and the honest gaps.** No international surface of any kind has been announced: no English documentation, no non-China storefront found, and no route for a foreign individual developer beyond an email address. No self-service console exists for the industry platform. No SDK download, no versioning scheme for the platform itself, and no published developer licence. And no vendor page we could read states a price, a fee for the platform, a launch date, or the S1's optical specification — the storefront that would answer the first and third sits behind a Taobao login. **What to watch.** Three signals would each change this row materially: an English or international developer surface appearing; the Professional Edition SDK becoming self-serve rather than brokered; and any widening of the three display templates into something a developer can lay out. The first would open the platform to everyone reading this page. ## In the landscape Alibaba's Qianwen platform is one of the platforms tracked in the third-party [smart-glasses landscape](/docs/ecosystem). It is the sharpest illustration of why that map separates build-time from ship-time openness — see [how open it is](/docs/ecosystem/openness) relative to platforms whose gate is a partner list rather than a passport, and [how AI works](/docs/ecosystem/ai) across them, where it stands nearly alone in handing a general-purpose model to third parties instead of reserving it behind a wake word. --- source: https://extentos.com/docs/ecosystem/platforms/android-xr --- --- title: "Android XR" seoTitle: "Android XR for developers: SDK, app model & distribution (2026)" description: "Android XR for third-party developers — SDK access, app model, distribution, capabilities & AI, and where it sits in the 2026 smart-glasses landscape." type: reference verified: "2026-08-24" platform: all vendor: android-xr related: - /docs/ecosystem - /docs/ecosystem/openness - /docs/ecosystem/ai - /docs/ecosystem/distribution - /docs/vendors/meta - /docs/concepts/capabilities --- **Openness verdict.** Open, well-documented SDK available to every developer (Jetpack XR SDK, Developer Preview 4), and Google has an on-the-record launch window — audio glasses "coming this fall" (2026), display glasses to follow. But as of July 2026 the glasses class is still pre-production: no consumer Android XR smart glasses have shipped, and there is no public app-distribution channel for the audio/display glasses class yet (Play distribution today covers only XR headsets and wired XR glasses). **Covered here:** Android XR audio glasses · Android XR display glasses (in-lens optical see-through display class; Samsung's display model is codenamed 'Haean', reportedly targeted at 2027). **Not covered here:** Samsung Galaxy XR (Project Moohan) · Wired XR glasses / tethered viewers (e.g. XREAL Project Aura) · Android XR headsets generally. ## Overview Android XR is Google's Android-based operating system for extended-reality devices, spanning immersive headsets and, in the class this page covers, lightweight **smart glasses**. Google's SDK formally distinguishes two glasses device classes: **audio glasses** (screen-free, driven by camera, speakers, microphones and Gemini) and **display glasses** (an in-lens optical see-through display), both treated as "augmented experience" devices distinct from headsets ([developer.android.com](https://developer.android.com/develop/xr/jetpack-xr-sdk/glasses/build)). Google has confirmed hardware partnerships with Samsung and eyewear makers **Gentle Monster** and **Warby Parker** ([Android Developers Blog, Dec 2025](https://android-developers.googleblog.com/2025/12/build-for-ai-glasses-with-android-xr.html)), and **Gucci (Kering)** is a confirmed in-class partner running Android XR + Gemini and targeting 2027 ([reporting on Kering CEO Luca de Meo, Apr 2026](https://vr.org/articles/gucci-warby-parker-google-fashion-ar-glasses-race)). Unlike the draft framing that treated the launch as purely rumor, Google made an on-the-record commitment at **Google I/O 2026** that Android XR audio glasses are **"coming this fall" (2026)** — audio-first, with display glasses to follow ([blog.google, I/O 2026](https://blog.google/products-and-platforms/platforms/android/android-xr-io-2026/)). As of July 2026 no consumer unit has shipped, but Samsung formally unveiled its Android XR **intelligent eyewear** at Galaxy Unpacked on **22 July 2026** — audio-first with no display, a built-in camera, the Snapdragon AR1 Gen1 platform and up to nine hours per charge, in Gentle Monster and Warby Parker designs, "scheduled to launch this fall in select markets" ([Samsung Newsroom, Galaxy Unpacked](https://news.samsung.com/global/samsung-brings-galaxy-ecosystem-into-everyday-eyewear); the fall / select-markets window comes from the [20 May Google I/O first look](https://news.samsung.com/global/samsung-and-google-give-first-look-at-new-intelligent-eyewear)). Samsung disclosed neither price nor camera resolution, so the ~$379–$499 / 12MP figures circulating in secondary reporting remain unconfirmed. The only Android XR hardware actually on sale today is the Samsung Galaxy XR headset, which is out of scope here. Net: the platform, SDK, and an official launch window are all real and public; the glasses hardware and a public publishing path are the two gates still closed. ### What you can buy ## Access The SDK is the **Jetpack XR SDK**, a Kotlin-first stack (Java usable via the Android framework) built around Jetpack Compose. | | | | --- | --- | | **SDK** | Jetpack XR SDK. For glasses the load-bearing libraries are **Jetpack Projected** (bridges a phone app to glasses hardware — sensors, speakers, camera, display) and **Jetpack Compose Glimmer**, a Compose UI toolkit optimized for display glasses; ARCore for Jetpack XR, Compose for XR and SceneCore round out the perception/3D surface ([build guide](https://developer.android.com/develop/xr/jetpack-xr-sdk/glasses/build)) | | **Languages** | Kotlin-first; Java usable via the Android framework | | **Maturity** | **Developer Preview 4**, released **19 May 2026**, with the XR Runtime, SceneCore and ARCore perception libraries since reaching **1.0.0-beta02** on **12 August 2026** (Jetpack Projected and Compose Glimmer remain alpha) ([DP4 blog](https://android-developers.googleblog.com/2026/05/android-xr-sdk-developer-preview-4-updates.html)) | | **Who can build** | **All developers**, since Developer Preview 3 in December 2025 — no allowlist to write and emulate apps | | **Cost** | Free | | **Without hardware** | Android Studio's bundled **AI Glasses emulator** simulates touchpad and voice input, so apps can be built without hardware | | **License** | Jetpack libraries are open-source (Apache 2.0); documentation is under Google's content license | | **Physical hardware** | Pre-release hardware is gated behind the **Android XR Developer Catalyst Program** — application-based; the 2026 cycle closed to applications on 30 June 2026 with selections notified 15 July, and development kits ship only to developers in the US, Canada, Japan, the UK and the EU | | **Docs** | [developer.android.com/develop/xr](https://developer.android.com/develop/xr/jetpack-xr-sdk) | ## App model Glasses apps use a distinctive **projected** model, not a standalone on-device app. Per Google: "Unlike XR headsets that run a full APK on the device, audio glasses and display glasses use a dedicated activity that runs within your phone's existing app. This activity is projected from the host device to the glasses" ([first-activity guide](https://developer.android.com/develop/xr/jetpack-xr-sdk/ai-glasses/first-activity)). | | | | --- | --- | | **Where your code runs** | **Natively on the paired phone** (Android/Kotlin, compute and the app process stay phone-side) while UI and I/O are projected to the glasses over the phone link — the glasses are a tethered display/sensor surface, not a compute target | | **UI** | Display UI is authored in Jetpack Compose via Glimmer; audio-only glasses render no visual UI and lean on ASR/TTS and Gemini | | **Entry point** | Apps declare a projected launcher activity (`XR_PROJECTED_LAUNCHER` intent category) as their glasses entry point, and that activity must also declare `android:requiredDisplayCategory="android.hardware.display.category.XR_PROJECTED"` | | **The API surface** | Apps obtain projected context via `ProjectedContext.createProjectedActivityOptions(context)`, request hardware permissions through `ProjectedActivityCompat.requestPermissions()` (which superseded the deprecated `ProjectedPermissionsResultContract` in Jetpack Projected 1.0.0-alpha10, 15 July 2026, and was removed outright in 1.0.0-alpha11 on 12 August 2026 — the narrative guides still show the old contract), observe connection with `ProjectedContext.isProjectedDeviceConnected()` (returns a `Flow<Boolean>`), and can query `ProjectedDeviceController.capabilities` (e.g. `CAPABILITY_VISUAL_UI`) to branch between display and audio-only hardware | | **Known Preview limitations** | Launching a projected activity does **not** auto-turn-on the glasses display, and the display briefly flashes white on activity start ([glasses first-activity guide](https://developer.android.com/develop/xr/jetpack-xr-sdk/glasses/first-activity)) | | **Open standards** | There is a **single first-party development system** (Jetpack XR) for this class. **WebXR and OpenXR are supported dev paths but Google's device-types page lists them exclusively for XR headsets and wired XR glasses, never for the audio/display glasses class** ([Android XR devices](https://developer.android.com/develop/xr/devices)) — so on the projected glasses model treat both as undocumented and unavailable today; Google gives no rationale for the omission and has committed to nothing either way | | **Game engines** | Unity, and newly Unreal and Godot, likewise target the immersive/headset side rather than the projected glasses model | One nuance is easy to overreach on, so state it precisely: Google's I/O 2026 post says the **glasses pair with both Android and iOS phones** ([blog.google](https://blog.google/products-and-platforms/platforms/android/android-xr-io-2026/)). That is a statement about **hardware pairing and the first-party Gemini experience** — it is *not* a documented third-party app path on iOS. A projected app is an Android `Activity`, declared in an Android manifest, and Google documents no iOS host SDK for it; so a **third-party** projected app today implies an **Android** host. Treat iPhone pairing as a consumer-hardware fact, not a developer-distribution one. ## Distribution Distribution is Google Play–based in principle but **not yet open for glasses**. Google's guidance is explicit: Play Store distribution is currently available only for immersive apps on **XR headsets and wired XR glasses**; for augmented experiences on **AI (audio/display) glasses**, developers should "get started developing and run and debug apps on the Android XR emulator," with "more updates on distribution coming in the future" ([package-and-distribute](https://developer.android.com/develop/xr/package-and-distribute)). | | | | --- | --- | | **Glasses apps today** | The publishing channel is unstated and unavailable. Treat glasses publishing as not-yet-open and gate any go-to-market on Google's forthcoming announcement | | **Headsets and wired XR glasses** | A standard **Play Console** account, choosing a mobile release track or a dedicated Android XR track. The XR track requires an `android.software.xr.api.spatial` or `...openxr` manifest feature | | **Review** | Normal Google Play policies plus the [Android XR app quality guidelines](https://developer.android.com/docs/quality-guidelines/android-xr) | | **The one documented reach path** | Not distribution: an existing phone app's notifications are bridged to audio and display glasses through the standard Android notification framework "with little to no additional configuration", with optional glasses-side behaviour via `NotificationCompat.ProjectedExtender` ([notification behavior](https://developer.android.com/develop/xr/jetpack-xr-sdk/glasses/notifications/behavior)). It is opt-in per app — "By default, app notifications are toggled off in the Glasses app" — and it surfaces an already-installed phone app on the glasses rather than publishing a projected glasses app | Google says the glasses also pair with iOS phones, but that is a hardware-pairing statement about the first-party experience; no third-party projected-app path on an iOS host is documented, so don't plan distribution around one. ## Capabilities, limits & AI | Capability | Third-party access | | --- | --- | | **Camera** | Accessible through Jetpack Projected with runtime permission prompts (requested via `ProjectedActivityCompat.requestPermissions()`, which replaced the deprecated `ProjectedPermissionsResultContract` in Jetpack Projected 1.0.0-alpha10 and fully removed it in 1.0.0-alpha11); the glasses camera is a headline capability (Samsung's rumored unit cites 12MP, unconfirmed). | | **Mic/audio in** | on-device microphones with ASR/speech input via the projected stack. | | **Audio out / TTS** | Speakers with text-to-speech synthesis. | | **On-lens display** | Display glasses only, driven by Jetpack Compose Glimmer over an optical see-through panel — small, transparent, legibility-first (Google Sans Flex font added in DP4); audio glasses have no display. | | **Sensors / perception** | ARCore for Jetpack XR brings motion tracking to the platform, but note a device-class caveat — the **Geospatial/VPS API early preview announced in DP4 is for _wired_ XR glasses** (the tethered, out-of-scope class), *not* the in-scope projected audio/display glasses; Google's DP4 wording is "an early preview of the Geospatial API for wired XR Glasses… high-precision anchoring… in over 87 countries" ([DP4 blog](https://android-developers.googleblog.com/2026/05/android-xr-sdk-developer-preview-4-updates.html)). Do not assume Geospatial anchoring is available on the projected glasses this page covers. Richer perception (depth, hand/eye tracking, spatial anchors) is documented primarily for headsets and may be a subset on glasses (verify per device). | | **Input** | Touchpad **tap and swipe** plus **voice**; custom gestures beyond these are not documented. | | **Reserved assistant / wake path** | Gemini is the platform's built-in assistant, and the reserved invocation is concretely the wake phrase **"Hey Google"** plus a **frame-side tap** ([blog.google, I/O 2026](https://blog.google/products-and-platforms/platforms/android/android-xr-io-2026/)) — analogous to how competing platforms reserve "Hey Meta." | | **AI & the assistant** | Third-party apps are **not** locked out of Gemini — a projected activity tagged `XR_PROJECTED_LAUNCHER` becomes discoverable by **Gemini voice commands** ("Open/Launch/Start [app name]"), and apps can integrate the **Gemini Live API** for real-time audio interactions ([first-activity](https://developer.android.com/develop/xr/jetpack-xr-sdk/ai-glasses/first-activity), [build guide](https://developer.android.com/develop/xr/jetpack-xr-sdk/glasses/build)). Whether a third party can technically *replace* Gemini as the system assistant (versus only being invoked by it and building on the Gemini Live API) is not spelled out in a primary source — the documented model is invoke-and-integrate, so replaceability should be treated as unconfirmed rather than asserted. Likewise, the platform-wide on-device-vs-cloud AI split for glasses is not documented, though the third-party path is explicitly cloud — the Gemini Live API "does require a persistent internet connection" and is integrated and billed through Firebase AI Logic; given phone-hosted compute, treat the rest as an open question rather than a stated guarantee. | ## Roadmap Momentum is steady and, importantly, has an **on-the-record launch window**: at **Google I/O 2026** Google committed that Android XR audio glasses are **"coming this fall" (2026)** — audio-first, display glasses to follow, delivered via Samsung / Gentle Monster / Warby Parker frames and with **iPhone host support** ([blog.google, I/O 2026](https://blog.google/products-and-platforms/platforms/android/android-xr-io-2026/), [9to5Google](https://9to5google.com/2026/05/19/android-xr-audio-glasses/)). Samsung then formally unveiled that hardware at Galaxy Unpacked on **22 July 2026** — audio-first intelligent eyewear with a built-in camera, Snapdragon AR1 Gen1 and up to nine hours per charge, launching this fall in select markets ([Samsung Newsroom, Galaxy Unpacked](https://news.samsung.com/global/samsung-brings-galaxy-ecosystem-into-everyday-eyewear); the fall / select-markets window is from the [20 May Google I/O first look](https://news.samsung.com/global/samsung-and-google-give-first-look-at-new-intelligent-eyewear)); price and camera resolution stay undisclosed, so the ~$379–499 / 12MP specs from secondary reporting remain unverified. Dated SDK milestones: **Dec 2025** — Developer Preview 3 opens AI-glasses development to all developers, ships Jetpack Projected + Glimmer + the AI Glasses emulator, and confirms Samsung / Gentle Monster / Warby Parker frame partners. **19 May 2026** — Developer Preview 4 adds a Device Availability API, ProjectedTestRule testing, new Glimmer components (Stacks, Title Chips) and Google Sans Flex, and signals XR Runtime/SceneCore/ARCore perception moving to Beta (the DP4 Geospatial preview is for *wired* XR glasses, not the projected class). **15 June 2026** — ecosystem update adds Unreal and Godot engine support (headset side) and the Android XR Engine Hub. **15 July 2026** — the Beta DP4 signalled lands: XR Runtime, SceneCore and ARCore for Jetpack XR all ship 1.0.0-beta01, while Jetpack Projected and Compose Glimmer stay alpha. **12 August 2026** — all three perception libraries move to 1.0.0-beta02 and Jetpack Projected 1.0.0-alpha11 removes the deprecated projected-permissions APIs outright; Google formalised the beta milestone in an **18 August 2026** post, with Jetpack Compose for XR still alpha. Longer-horizon in-scope hardware: **Samsung** is pursuing two SKUs — the audio device this fall 2026 and a separate **display model codenamed 'Haean' targeted at 2027** ([AndroidHeadlines](https://www.androidheadlines.com/samsung-galaxy-glasses)) — and **Gucci/Kering** targets 2027. Still-open gate: a public glasses app-distribution channel ("coming in the future"). The trajectory points toward glasses GA on a named timeline; production hardware and public publishing are the two things not yet delivered. ## In the landscape Android XR is one platform in the third-party [smart-glasses landscape](/docs/ecosystem). See [how open it is](/docs/ecosystem/openness) relative to other platforms, [how AI works](/docs/ecosystem/ai) across them, and the full [platform comparison](/docs/ecosystem). --- source: https://extentos.com/docs/ecosystem/platforms/apple --- --- title: "Apple smart glasses" seoTitle: "Apple smart glasses for developers: SDK, app model & distribution (2026)" description: "Apple smart glasses for third-party developers — SDK access, app model, distribution, capabilities & AI, and where it sits in the 2026 smart-glasses landscape." type: reference verified: "2026-08-24" platform: all vendor: apple related: - /docs/ecosystem - /docs/ecosystem/openness - /docs/ecosystem/ai - /docs/ecosystem/distribution - /docs/vendors/meta - /docs/concepts/capabilities --- **Openness verdict.** Closed by absence: as of July 2026 there is no shipping Apple glasses product and no glasses developer SDK. The only concrete developer surface is phone-side on-device AI — and that surface is real and increasingly open (Foundation Models, the new Core AI framework, MLX, and a first-party LoRA-adapter toolkit on iOS/iPadOS/macOS/visionOS), just not scoped to any glasses device. **Covered here:** Apple smart glasses. **Not covered here:** Apple Vision Pro · Rumored "Vision Air" · Camera-equipped AirPods (rumored, reported ~late-2027). ## Overview Apple has **not** shipped a smart-glasses product, and as of July 2026 has made **no official announcement** of one — nothing on the [Apple Newsroom](https://www.apple.com/newsroom/) or [Apple Developer](https://developer.apple.com/) site names a glasses device. Multiple press reports (Bloomberg's Mark Gurman, relayed via outlets such as [9to5Mac](https://9to5mac.com/2026/05/31/apple-glasses-launching-late-2027-with-vision-air-to-follow-by-2029/)) describe an internal program ("N50") aimed at an audio-and-camera AI accessory — cameras, mics, speakers, Siri/Apple Intelligence for interaction, **no on-lens display** in the first generation, iPhone-tethered — reportedly slipped to a **late-2027** launch. Treat all of that as unverified signal, not product. The closest in-class precedent is not Vision Pro but Apple's own audio accessories: reporting points to **camera-equipped AirPods** on a similar ~late-2027 window and the same audio-first + camera + Siri-visual-intelligence design ([MacRumors](https://www.macrumors.com/2026/06/16/airpods-camera-late-2027/)), and **AirPods Live Translation** (iOS 26, processed on-device on the paired iPhone) is the actual shipping pattern for an Apple audio accessory driving on-device Apple Intelligence ([Apple Support](https://support.apple.com/en-us/123185)) — the concrete template a glasses accessory would most plausibly inherit. The only Apple hardware in the adjacent MR space that actually ships is **Apple Vision Pro** (out of scope here — it is not glasses). Pricing, hardware spec, and ship date for any Apple glasses are unknown; the figures circulating point to a reported **~$200–$500 target**, which is analyst/rumor, not Apple. ### What you can buy ## Access There is **no Apple glasses SDK** — no framework, package, or documentation for building glasses apps, because there is no device. What Apple does ship is a deep and increasingly open **phone-side on-device AI stack**, none of it glasses-scoped. | | | | --- | --- | | **Foundation Models** | The [Foundation Models framework](https://developer.apple.com/documentation/FoundationModels) — a native **Swift** API giving direct access to the same on-device model that powers Apple Intelligence, **introduced at WWDC in June 2025** ([session 286](https://developer.apple.com/videos/play/wwdc2025/286/)) and made **available with iOS 26 / iPadOS 26 / macOS 26 / visionOS 26 in September 2025** ([Newsroom](https://www.apple.com/newsroom/2025/09/apples-foundation-models-framework-unlocks-new-intelligent-app-experiences/)), backed by **Private Cloud Compute** for heavier inference | | **Core AI** | Announced at **WWDC26 (June 2026)** and currently **developer-beta only** — Apple's docs mark every platform `27.0+ Beta`, so it is gated on the unreleased OS 27 line. A memory-safe Swift API to load, specialize, and run **arbitrary** on-device models, 'everything from compact vision models to large-scale generative AI,' with ahead-of-time compilation and fine-grained inference/memory control, paired with a curated **Core AI Models** repo (Qwen, Mistral, SAM3) each convertible with one command via standard Python/PyTorch workflows ([What's new in ML](https://developer.apple.com/machine-learning/whats-new/), [session 324](https://developer.apple.com/videos/play/wwdc2026/324/)) | | **MLX** | Apple's open-source array framework for Apple silicon, which lets developers run and **fine-tune** arbitrary LLMs on-device via the MLX / MLX-LM Python packages (the mlx-community org hosts thousands of quantized models) ([opensource.apple.com](https://opensource.apple.com/projects/mlx/)) | | **Who can build** | All of it is buildable in **Xcode** by anyone in the paid **Apple Developer Program** under the standard license agreement; models run on-device and offline | This is the substrate an eventual glasses accessory would most plausibly build on — but it is phone/Mac software, not a glasses platform. ## App model Undefined for glasses — no app model exists because no device or SDK ships. | | | | --- | --- | | **Where phone-side code runs** | **Natively in Swift** inside a standard iOS/iPadOS/macOS/visionOS app: Foundation Models runs the on-device model with optional escalation to **Private Cloud Compute**, while **Core AI** and **MLX** let an app load and run arbitrary on-device models with no server dependency and no token cost | | **What does not exist** | No glasses runtime, no web-app layer, and no game-engine target | | **The accessory frameworks that do exist** | Apple has published no glasses-specific transport, but it already ships the frameworks a phone-tethered accessory uses: **AccessorySetupKit** (iOS 18+, proximity-triggered secure pairing without broad Bluetooth scanning), **AccessoryTransportExtension** (iOS 26.2+, secure data transfer to accessories you develop), the **ExternalAccessory** framework, and the **MFi Program** (Apple-licensed accessory protocol strings via `UISupportedExternalAccessoryProtocols`) ([AccessorySetupKit](https://developer.apple.com/documentation/accessorysetupkit/), [Accessories](https://developer.apple.com/accessories/)) | A reported audio-first iPhone-companion accessory would almost certainly surface through these — MFi in particular is a genuine gating/reserved surface — though nothing glasses-specific is documented. The plausible shape is an iPhone-companion accessory (phone does the AI) rather than an on-device third-party app platform, but that is inference from reporting, not documentation. ## Distribution Not applicable to glasses — with no device and no SDK there is no distribution channel, no publishing path, and no review process for glasses apps. For context, Apple's general model is the **App Store** with mandatory App Review, and a licensed hardware accessory additionally routes through the **MFi Program** ([Accessories](https://developer.apple.com/accessories/)). A companion glasses accessory would most likely surface as features inside existing iPhone apps distributed through the App Store rather than as a separate storefront, but Apple has published nothing on this. Any developer/publishing timeline is unknown and gated on Apple first announcing hardware. ## Capabilities, limits & AI | Capability | Third-party access | | --- | --- | | **All glasses hardware** | **Unknown/unshipped.** Reporting (unverified) describes cameras, microphones, and speakers with **no on-lens display** in the first generation and Siri/audio as the primary interaction — but Apple has documented none of it, so camera access, mic/audio streaming, TTS/audio-out APIs, sensors, gesture input, and any reserved system surfaces are all undefined for third parties | | **AI & the assistant** | There is no glasses assistant to open up. On the phone side, **Siri/Apple Intelligence is not a general third-party programmable assistant** — developers integrate via App Intents/Shortcuts and, for on-device generation, the [Foundation Models framework](https://developer.apple.com/documentation/FoundationModels), which supports **tool calling**, **guided (structured) generation**, and **streaming**, runs **on-device** and offline, and — as of WWDC26 — is materially more capable in glasses-relevant ways: it now accepts **multimodal prompts** (images alongside text), can call on-device **Vision tools** (OCR, barcode), supports **Dynamic Profiles** (swap models/tools/instructions at runtime), ships an **Evaluations** framework, and gives free **Private Cloud Compute** access to the latest server model for apps under 2M first-time downloads ([What's new in ML](https://developer.apple.com/machine-learning/whats-new/)) | | **Bring your own model** | The BYO-model story is genuinely rich and mostly **first-party**: the **Foundation Models adapter training toolkit** lets you fine-tune Apple's own on-device model with your data via rank-32 **LoRA adapters** (~160MB each, loadable at runtime, with an optional speculative-decoding draft model) — a small model specialized for an accessory's task, though the toolkit is frozen: **version 26.0.0 is its last release** and is not compatible with iOS/iPadOS/macOS/visionOS 27 and later ([adapter toolkit](https://developer.apple.com/apple-intelligence/foundation-models-adapter/)); the new **Core AI** framework (OS 27 beta) and **MLX** let you load, run, and fine-tune arbitrary open models on-device; and a third-party provider layer lets other LLM vendors plug in, with **Anthropic (Claude)** and **Google (Gemini)** packages announced/upcoming ([WWDC26 session 339](https://developer.apple.com/videos/play/wwdc2026/339/)) | | **On a wearable** | Apple's docs now list **watchOS 27.0+** availability for the core Foundation Models types, extending on-device generation to a constrained wearable — announced for the OS 27 line rather than shipping, but the closest analogue to what a glasses accessory would inherit | | **Voice invocation** | Apple ships no custom wake-word API, so cold invocation while your app isn't running goes through Siri/App Intents. An app that is already running can detect its own wake phrase in-process — the Speech framework does continuous live-audio recognition, and third-party on-device engines (Picovoice Porcupine, VoxRT) ship iOS packages | Apple's own [next-generation Apple Intelligence](https://www.apple.com/newsroom/2026/06/apple-intelligence-brings-powerful-ai-capabilities-into-everyday-experiences/) is powered by Apple Foundation Models custom-built in collaboration with Google and its Gemini models, which run on device and on servers using Private Cloud Compute. None of these AI hooks are exposed on or scoped to a glasses device today. ## Roadmap **Shipped/dated (primary):** Foundation Models framework introduced at **WWDC June 2025** ([session 286](https://developer.apple.com/videos/play/wwdc2025/286/)), available with iOS/iPadOS/macOS 26 in **September 2025**; at **WWDC26 (June 2026)** Apple announced the **Core AI** framework for running arbitrary on-device models, expanded Foundation Models to multimodal + Dynamic Profiles + Evaluations + free PCC under 2M downloads, and surfaced a third-party provider layer (Claude/Gemini packages announced/upcoming, [session 339](https://developer.apple.com/videos/play/wwdc2026/339/)); next-generation Apple Intelligence + a more capable Siri, powered by next-generation Apple Foundation Models custom-built with Google and its Gemini models and running on device and on Private Cloud Compute servers, detailed [June 2026](https://www.apple.com/newsroom/2026/06/apple-intelligence-brings-powerful-ai-capabilities-into-everyday-experiences/). **Glasses (reported, not announced):** press reporting through 2026 (Gurman/Bloomberg via outlets like [AppleInsider](https://appleinsider.com/articles/26/05/31/longtime-leaker-joins-others-in-saying-apple-glasses-wont-arrive-until-late-2027) and [9to5Mac](https://9to5mac.com/2026/05/31/apple-glasses-launching-late-2027-with-vision-air-to-follow-by-2029/)) points to an unveil at **WWDC in June 2027** and a **late-2027** launch for an audio-first "N50" accessory, delayed to let Apple's visual AI mature; a lighter "Vision Air" headset has since been reported cancelled, with the head-mounted roadmap pruned to two glasses projects. **Momentum:** heavy, accelerating investment in the on-device AI substrate (Foundation Models → Core AI → MLX → the adapter toolkit — the likely foundation for any future glasses), but **no official glasses hardware or SDK yet** — no announced device, no developer preview, no dated SDK, though the reported WWDC 2027 unveil would be the natural developer-preview moment. Extentos will add an Apple glasses transport if and when Apple ships an actual SDK; until then there is nothing to target. ## In the landscape Apple smart glasses is one platform in the third-party [smart-glasses landscape](/docs/ecosystem). See [how open it is](/docs/ecosystem/openness) relative to other platforms, [how AI works](/docs/ecosystem/ai) across them, and the full [platform comparison](/docs/ecosystem). --- source: https://extentos.com/docs/ecosystem/platforms/brilliant-labs --- --- title: "Brilliant Labs" seoTitle: "Brilliant Labs for developers: SDK, app model & distribution (2026)" description: "Brilliant Labs for third-party developers — SDK access, app model, distribution, capabilities & AI, and where it sits in the 2026 smart-glasses landscape." type: reference verified: "2026-08-24" platform: all vendor: brilliant related: - /docs/ecosystem - /docs/ecosystem/openness - /docs/ecosystem/ai - /docs/ecosystem/distribution - /docs/vendors/meta - /docs/concepts/capabilities --- **Openness verdict.** The most open platform in the category: OSS multi-language SDKs (BSD-3/MIT) + an on-device Lua VM over plain BLE, a first-party IDE (AR Studio) and a complete open-source reference host app, open firmware on both Frame and Halo, plus Frame's design files (Halo's design files are not yet published), and an open-source self-hostable BYO-key assistant — with no gatekeeper on either building or shipping, since you distribute your own host app. **Covered here:** Halo (the only product on sale — $399, orderable; first units ship early August 2026 per the vendor, checked 2026-07-29) · Frame (earlier model, NO LONGER SOLD; still documented by the vendor and still owned in the wild). **Not covered here:** Monocle. ## Overview Brilliant Labs is a Singapore/Hong-Kong open-source hardware startup that builds developer-first AI smart glasses. Two models are in scope. **Frame** (no longer sold — its product page 404s and it is absent from the store, verified 2026-07-29; the vendor still documents it and existing units still work) is a lightweight (~40g class) glasses with a monocular **640×400 color micro-OLED** (0.23", 20° FOV), an [OV09734-based 720p color camera](https://docs.brilliant.xyz/frame/hardware/) cropped to 720×720, a single TDK ICS-41351 MEMS mic, a 6-axis MC6470 accel/e-compass, an nRF52840 Cortex-M4F MCU + FPGA for display acceleration, a 210mAh battery, and **Bluetooth 5.3** — no speaker. **Halo** (orderable; the vendor states shipments begin early August 2026, verified 2026-07-29) is a redesign: a **640×480 full-RGB VGA020 OLEDoS micro-display** (0.2" panel, up to 5000 nit, 10000:1), a PixArt PAG7982J1 global-shutter VGA camera (81.2° H-FOV), **dual T5838 MEMS mics** with audio-activity detection, **stereo bone-conduction speakers**, a Bosch BMA580 accelerometer + QST QMC6308 magnetometer with tap detection, and notably an **Alif Balletto B1 (Cortex-M55 + Ethos-U55 NPU)** for on-device AI, a 300mAh battery, BLE 5.3, [just over 40g](https://brilliant.xyz/products/halo). Halo is $399, sold direct via brilliant.xyz, with prescription lenses via a partner (adjustable +2 to −6 dioptres) and a single black colourway — one hardware model, not a product line. Both are open-source in intent, though the artifacts are uneven: Frame's firmware and FPGA RTL are published as [`frame-codebase`](https://github.com/brilliantlabsAR/frame-codebase) and the SDKs are BSD-3/MIT, while Halo runs Zephyr OS and its firmware went public on 2026-08-19 as [`halo-firmware`](https://github.com/brilliantlabsAR/halo-firmware) (Brilliant's own code Apache-2.0 inside a multi-licence Alif Zephyr fork, so the repo is not uniformly Apache-2.0), with owner-flashable signed OTA images from release 0.8.8; Halo's design files are still unpublished beyond a downloadable STL, and some vendor doc pages still carry stale "coming soon" firmware links. Neither is standalone: both are phone/host peripherals over BLE. (n/a: neither is a tethered viewer, controller-driven AR, or MR headset.) ### What you can buy ## Access The [Brilliant SDK](https://github.com/brilliantlabsAR/brilliant_sdk) is a single open-source monorepo covering **both Halo and Frame** (device type auto-detected on connect). | | | | --- | --- | | **Host bindings** | Three — **Python** (`brilliant-ble` low-level BLE via Bleak + `brilliant-msg` for rich objects), **Flutter/Dart** for iOS+Android, and **Web Bluetooth/TypeScript** | | **On-device** | An **on-device Lua 5.4 VM** with a full [Lua hardware API](https://docs.brilliant.xyz/halo/halo-sdk-lua/) reachable directly on the glasses | | **License** | Genuinely permissive: the unified `brilliant_sdk` is **BSD-3-Clause**, the older [`frame-sdk-python`](https://github.com/brilliantlabsAR/frame-sdk-python) is **MIT**, and the Noa assistant backend is ISC | | **IDE workflow** | **[AR Studio](https://marketplace.visualstudio.com/items?itemName=brilliantlabs.brilliant-ar-studio)**, Brilliant's VSCode extension, is a full IDE-integrated on-device workflow — device-file explorer, an interactive on-glasses REPL, auto-run/build that transfers files and soft-resets over BLE, FPGA-binary/firmware updates, and community-project browse/publish. The [Frame SDK docs](https://docs.brilliant.xyz/frame/frame-sdk/) present it as the officially-documented way to "directly write and execute Lua code on Frame" (its lineage is the Monocle MicroPython tool, now extended to the glasses) | | **Reference app** | The complete **[Noa companion app](https://github.com/brilliantlabsAR/noa-flutter)** is open-source Flutter and is promoted as the canonical reference — "a great example of how to build your own Frame apps" — so you can start from a shipping production host app rather than raw snippets | | **Maturity** | Actively-developed and past 1.0. The bindings version independently rather than in lockstep (PyPI `brilliant-sdk` 1.0.0, npm `brilliant-sdk` 2.0.0, pub.dev `brilliant_sdk` 2.0.0, checked 2026-08-24), and the docs still note each platform SDK is under active development. Frame's SDK is battle-tested; Halo's shipped with the hardware | | **Who can build** | **Anyone** — open signup, no dev-mode toggle, invite, partner tier, enterprise contract, or fee | | **Docs** | [docs.brilliant.xyz](https://docs.brilliant.xyz/); source at [github.com/brilliantlabsAR](https://github.com/brilliantlabsAR) | ## App model Explicitly phone/host-companion. Per the [Frame SDK docs](https://docs.brilliant.xyz/frame/frame-sdk/): "Rather than installing apps directly on Frame, it typically functions as a peripheral accessory for 'host' apps running on computers or mobile devices," with the glasses running a simple event-handler loop while the host drives logic. | | | | --- | --- | | **Transport** | **Bluetooth LE** in every binding. `brilliant-msg` streams rich objects (images, audio, IMU data, rasterized text) between host and glasses | | **A second dev path** | Developers can also write Lua scripts that execute on Frame/Halo directly on the on-device Lua 5.4 VM (on Frame via AR Studio; on Halo via the BLE Lua REPL plus a `main.lua` that runs at wakeup), supplementing — not replacing — the host app, so you can move logic on-glasses when latency or offline operation matters, and Halo's Ethos-U55 NPU makes some on-device inference viable | This is a near-exact match for Extentos's phone-companion model: a mobile app owning the primitives over BLE. The divergence is that Brilliant's transport/messaging is its own BLE protocol (not Meta's DAT), and Brilliant additionally exposes the on-device Lua path that Extentos's Meta-oriented model does not. ## Distribution There is **no gatekeeper and no first-party developer app store** for compiled SDK apps. | | | | --- | --- | | **What you ship** | An SDK app is just your own host program — a Python/desktop app, or an iOS/Android app built on the Flutter SDK — that you distribute through ordinary channels (the OS app stores for a mobile companion, or run it yourself) | | **Review** | Brilliant reviews and gates nothing about shipping, and because the SDK, hardware, and firmware are open source you can fork firmware or flash custom Lua freely | | **AR Studio sharing** | A lightweight community browse-and-publish channel ("Browse community projects, or publish your own") for on-device scripts/projects — Monocle-rooted, but a real first-party discovery path | | **Noa Miniapps** | The Noa companion app ships a consumer **Miniapps** system — which the Halo page calls "our App Store" — where an end user builds new experiences in natural language and shares them with others, included free out of the box with daily usage caps (see the [Halo product page](https://brilliant.xyz/products/halo)) | | **Gating** | No preview→GA gating, no developer invite waitlist, no enterprise-MDM requirement | These sharing surfaces are authoring/sharing surfaces, not compiled-SDK third-party publishing channels comparable to a reviewed native store. Publishing status: effectively **open/unrestricted** for SDK apps, with the caveat that "publishing" means shipping your own host app rather than submitting to a Brilliant-run store. ## Capabilities, limits & AI | Capability | Third-party access | | --- | --- | | **Camera** | Frame captures 720×720 color stills to the host via `brilliant-msg`. Halo's PAG7982J1 is a global-shutter VGA color module (81.2° H-FOV); `brilliant-msg` lists image transfer, but Brilliant markets Halo's camera primarily as "AI vision input" and the [product page](https://brilliant.xyz/products/halo) frames it as a low-power optical sensor for AI inference — photo capture to host is nonetheless a documented first-class primitive on Halo: the Lua API exposes `frame.camera.capture()` / `image_ready()` / `read()` and states "Camera images are read in Lua and sent to the host over the regular data channel using `frame.bluetooth.send()`", and the SDK's device-support table marks "Camera / photo capture" supported on both Halo and Frame. No documented capture-gating/permission-broker layer — access is direct over BLE. | | **Microphone/audio capture** | Yes — Frame single MEMS mic; Halo dual T5838 mics with audio-activity detection, streamable to host. | | **Audio output/TTS** | Frame has no speaker (host-side audio only); Halo has stereo bone-conduction speakers you can drive. | | **On-lens display/UI** | Yes, a real first-class surface — you render directly to the color OLED via Lua graphics APIs or host-pushed rasterized text/images (Frame 640×400, 20° FOV; Halo's 640×480 RGB OLEDoS panel exposes a 256×256 circular drawable area). | | **Sensors** | IMU/accelerometer + magnetometer with tap detection on both; no GPS on-glasses (location comes from the phone host). | | **Input** | Accelerometer-based tap detection on the frame; Halo adds a physical button + status LED. No gesture-camera, controller, or neural-band input. | | **Reserved surfaces** | Essentially none — because the assistant, wake handling, gestures, and IMU are all exposed to developer code, there is no locked first-party layer the way Meta reserves "Hey Meta." | | **AI & the assistant** | Noa is the bundled multimodal assistant, and its backend is genuinely open: [`noa-assistant`](https://github.com/brilliantlabsAR/noa-assistant) is open-source (ISC), self-hostable, and bring-your-own-key — you supply your own provider credentials in a `.env` and point the glasses at a model you host. Two caveats: it is a Frame-era repo with no commits since October 2024, and Brilliant's own [Halo guide](https://docs.brilliant.xyz/halo/halo/) still lists the Noa app itself as coming soon on both app stores (checked 2026-07-30). Provider selection is first-class and multi-vendor: `--assistant` accepts `gpt`, `claude` or `groq`, `--search-api` accepts `perplexity` (the default), `serp` or `dataforseo`, and `--vision` accepts GPT-4o/GPT-4-Vision or three Claude models — each routed to a dedicated client class, with OpenAI, Anthropic and Groq keys read from the environment. Voice invocation and vision queries are developer-controllable; Halo's Ethos-U55 NPU enables some on-device inference, though the shipped Noa experience is cloud-backed. | ## Roadmap **Frame** has been shipping and its Python/Flutter SDKs are mature. **Halo** was announced in 2026 as the successor at $399 (an earlier $349 introductory price is no longer listed); shipping was stated to start the week of July 20, 2026 and the product page now says shipments begin in early August — the unified `brilliant_sdk` monorepo (Python/Flutter/Web Bluetooth, Lua 5.4, auto device detection) lands with it. Momentum signals: consolidation of Frame+Halo into one open SDK, a move to a modern NPU-class SoC (Alif Balletto B1) for on-device AI, and continued full open-sourcing of hardware/firmware/design files. Announced-but-still-settling at time of writing: Noa's paid "Plus" tier pricing and the exact scope of Halo's first-party on-device (vs cloud) AI — Brilliant's own spec sheet lists a "cloud-based AI agent" under software while the silicon carries an NPU. What *is* settled is the third-party surface: the [Halo Lua API](https://docs.brilliant.xyz/halo/halo-sdk-lua/) exposes **no inference, model, tensor or NPU namespace** (verified 2026-08-24), and Brilliant's own MobileNet classification example runs the model **on the phone** rather than on the glasses. So the Ethos-U55 is not reachable from the Lua or host-SDK surface an app calls into — but since Halo's firmware went public on 2026-08-19 the NPU is no longer vendor-reserved either: the Ethos-U node is enabled on the halo board and the west manifest pulls `hal_ethos_u` and `tflite-micro`, so targeting it means building custom firmware and flashing it over BLE OTA, which Brilliant supports outright ("No cable, no dev kit, no vendor tooling", per the [firmware README](https://github.com/brilliantlabsAR/halo-firmware)). That is a firmware fork rather than an app calling an API, and the wiring is still yours to do: the shipped Halo build leaves the Ethos-U driver off, the in-tree TFLite-Micro/Ethos-U samples carry board overlays for Alif's dev kits and none for `halo`, and Brilliant's own in-tree note still lists "driver wiring — confirm Ethos is enabled in this build" as open work; what the NPU-class silicon does reach developers as is narrow sensor-level work — audio-activity detection on the mics and tap detection on the IMU. No developer-publishing gate is announced or planned — the platform's trajectory is more-open, not gated. The predecessor **Monocle** (MicroPython clip-on) is legacy and out of scope. ## In the landscape Brilliant Labs is one platform in the third-party [smart-glasses landscape](/docs/ecosystem). See [how open it is](/docs/ecosystem/openness) relative to other platforms, [how AI works](/docs/ecosystem/ai) across them, and the full [platform comparison](/docs/ecosystem). --- source: https://extentos.com/docs/ecosystem/platforms/even-realities --- --- title: "Even Realities" seoTitle: "Even Realities for developers: SDK, app model & distribution (2026)" description: "Even Realities for third-party developers — SDK access, app model, distribution, capabilities & AI, and where it sits in the 2026 smart-glasses landscape." type: reference verified: "2026-08-24" platform: all vendor: even-realities related: - /docs/ecosystem - /docs/ecosystem/openness - /docs/ecosystem/ai - /docs/ecosystem/distribution - /docs/vendors/meta - /docs/concepts/capabilities --- **Openness verdict.** Build-time openness is real but multi-track. For the G2, Even ships a genuine public web/TypeScript SDK (Even Hub) — and first-party developer onboarding is now self-serve (signing in to the web hub flips the account to developer), though ship-time publishing still goes through an Even-run, manually-reviewed app store. Separately, a mature open-source third-party OS/SDK (MentraOS) officially supports BOTH G1 and G2 with no Even pilot gate and no Even store review, and an active raw-BLE community drives both models directly — so the ungated developer path is broader than the official channel alone implies. The older G1 has no first-party SDK or store, but it is reachable via MentraOS, an official BSD-licensed protocol demo, and mainstream tools like Gadgetbridge. **Covered here:** Even G1 (2024 HUD glasses) · Even G2 (Nov 2025 HUD glasses). **Not covered here:** Even R1 smart ring. ## Overview Even Realities is a Shenzhen/US eyewear startup making discreet, prescription-friendly **HUD glasses** — monochrome green heads-up displays in a normal-looking frame, deliberately **no camera and no speaker** (privacy positioning against Ray-Ban Meta). Two in-scope models: the **G1** (2024; ~44 g; 640×200 green, reportedly ~20 Hz refresh, ~25° FOV, mic array, no camera) and the **G2** (announced/shipped Nov 12 2025; ~36 g titanium/magnesium; dual micro-LED, ~576×288 px per eye per Even's developer docs — Even's own consumer spec sheet cites 640×350 — 60 Hz, up to 1,200 nits, ~27.5° FOV, four-mic array, no camera, no speaker). Both are **peripheral displays paired to a phone over Bluetooth LE (5.4 on G2, up from 5.2 on the G1)**; on-board compute is display/firmware only (LVGL rendering), not standalone. The G1 retails **$499**, the G2 **$599**; the optional **R1 control ring** is **$249** (out of scope as an accessory). Audio output routes through the phone, not the glasses. Sources: [G2 overview (Even Hub docs)](https://hub.evenrealities.com/docs/get-started/overview), [G2 launch (9to5Google)](https://9to5google.com/2025/11/12/even-realities-launches-g2-smart-glasses-r1-control-ring/), [G2 engineering blog (Bluetooth 5.4)](https://www.evenrealities.com/blogs/even-insider/how-we-rebuilt-g2-from-the-inside-out), [G1 review (Engadget)](https://www.engadget.com/wearables/even-realities-g1-review-limited-but-effective-smart-glasses-140059586.html), [evenrealities.com](https://www.evenrealities.com/). ### What you can buy ## Access **G2 has a real, official developer platform — Even Hub.** | | | | --- | --- | | **SDK** | A web/TypeScript kit published to npm (`@evenrealities/even_hub_sdk`) with a CLI (`@evenrealities/evenhub-cli`) and a local **simulator** (`@evenrealities/evenhub-simulator`) | | **Languages** | **Plain HTML/CSS/JS or TypeScript, any stack (Vite/React)** — there is no proprietary language or engine | | **Starter code** | The bundle is documented in the official [`everything-evenhub`](https://github.com/even-realities/everything-evenhub) (MIT) repo with [`evenhub-templates`](https://github.com/even-realities/evenhub-templates) (MIT, TypeScript) starter scaffolds | | **Maturity** | The store and platform publicly launched **April 3 2026**. Docs currently say you can *"Build plugins for the Even G2. Widgets, dashboards, and AI skills are next"* — i.e. plugins are GA-ish, other surfaces are roadmap | | **Who can build** | **Self-serve** — you register an Even Realities account in the phone app and signing in at hub.evenrealities.com flips it to developer ("There is no toggle"), with no application, invite or allowlist step in the documented Quickstart. A separate [Even Hub Pilot / Early-Access program](https://hub.evenrealities.com/) opened applications in early 2026 for closer collaboration with Even's team ([Android Authority](https://www.androidauthority.com/even-realities-hub-smart-glasses-developers-3629083/)) | | **Cost** | No developer fee is documented | | **License** | SDK tooling and templates are MIT/BSD-2-Clause open source | | **Docs** | [hub.evenrealities.com/docs](https://hub.evenrealities.com/docs) | Even Hub is not the only way in. Three further tracks reach these glasses, two of them with no Even gate at all. | Track | What it gives you | | --- | --- | | **MentraOS** | [MentraOS](https://github.com/Mentra-Community/MentraOS) (by Mentra, the platform formerly known as AugmentOS) is a full open-source, cross-vendor smart-glasses OS whose TypeScript SDK moved off the cloud to phone-local miniapps in MentraOS 3.0 whose supported-glasses list officially includes **both Even G1 and G2** (alongside Mentra Live, Vuzix Z100, and NIMO). You write one app that runs phone-side in the Mentra Runtime and drives display/mic over the glasses' BLE link — with **no Even pilot gate and no Even store review** | | **Raw BLE (G2)** | An active community bypasses Even Hub entirely — [i-soxi/even-g2-protocol](https://github.com/i-soxi/even-g2-protocol) documents the G2 service UUIDs, 7-packet auth handshake, CRC-16/CCITT framing and protobuf payloads (with working teleprompter/calendar/notification demos), and MentraOS's own `G2.kt` is the widely-ported reference BLE implementation | | **G1** | **No first-party SDK or program**, but far from closed: Even ships an official [`EvenDemoApp`](https://github.com/even-realities/EvenDemoApp) (BSD-2-Clause) reference for the raw dual-BLE protocol (LC3 audio, 1-bit 576×136 BMP image packets), MentraOS supports G1, and the mainstream open-source Android companion app [Gadgetbridge](https://gadgetbridge.org/gadgets/others/even_realities/) has explicit Even Realities (G1) support — so third parties reach the G1 despite the missing first-party SDK | ## App model **Phone-companion web model.** | | | | --- | --- | | **What an app is** | A G2 app ("plugin") is a web app whose code runs off-glasses — loaded in a WebView hosted by the Even Realities companion app on the paired phone (community docs describe the frontend loading in the iPhone Even App's WebView, with the app free to fetch from its own server backend) | | **What the glasses do** | The **glasses are a thin display+input client**: the SDK sends layout/UI commands that the glasses render via **LVGL v9 firmware**, and relays mic audio, touchpad/ring input, IMU and device status back | | **Transport** | **Bluetooth LE** | | **What you can call** | Standard browser APIs (`fetch`, `Canvas`) are available because the runtime is a WebView — but every outbound origin must be listed in the `app.json` network whitelist (full origins, no wildcards), and anything else is blocked before the request leaves the WebView | | **What does not exist** | **No native on-glasses app runtime** and no Unity/engine path; Even Hub's app model is web-only (the MentraOS and raw-BLE tracks in [Access](#access) are not, though neither runs code on the glasses either) | This maps cleanly onto Extentos's **phone-companion** framing (phone mediates a peripheral display), with the key difference that Even's contract is a web/WebView + LVGL rendering pipeline rather than a native Android/iOS SDK, and Even ships no camera/speaker to drive. Sources: [Even Hub overview](https://hub.evenrealities.com/docs/get-started/overview), [awesome-even-realities-g2](https://github.com/pangoleen/awesome-even-realities-g2), [SDK feature notes (zenn)](https://zenn.dev/bigdra/articles/eveng2-sdk-features?locale=en). ## Distribution **G2 ships through Even Hub, Even Realities' own on-device app store.** | | | | --- | --- | | **How users install** | **Over-the-air** from an "Apps"/Even Hub tab inside the Even Realities mobile companion app (also surfaced via a tab in the Even online shop); no cable or sideload needed for end users | | **How you publish** | You package the app (`.ehpk` code package) with the `evenhub` CLI (`evenhub pack app.json dist -o myapp.ehpk`) and upload it through the **Even Hub Developer Portal**. For local testing you scan a QR code in the app's Developer Center to sync a build to the glasses | | **The two gates** | (1) Developer access is no longer one of them — onboarding is self-serve, since signing in to the web hub flips the account to developer; (2) apps reach the public store only via Even's review — though Private builds and email-invited Beta groups install unreviewed on real hardware — so Even is the gatekeeper/curator of public listing (the April 3 2026 launch shipped a curated set of ~50 apps) | | **Review policy** | Even **does publish a detailed content-review policy**: the [App Submission & QA Guidelines](https://hub.evenrealities.com/docs/ship/app-submission) mandate a manual review against a rejection-note checklist — package/manifest rules (reverse-domain ID, edition string, name ≤20 chars and must not contain "Even"), branding/screenshot-accuracy rules, functionality requirements (end-to-end on glasses+ring, 2-minute idle liveness, double-tap shutdown), and content restrictions (no medical diagnosis, financial advice, emergency-routing, or NSFW/hateful content) | | **Promotion flow** | A documented four-stage state machine — Draft → Test → Submitted → Released, with automated reviewer assignment, emailed decisions, and immutable Released versions (fix-forward only) — though a review-turnaround SLA is still not published | | **Without the store** | Because the G2 speaks an open BLE protocol, a developer can drive it directly from their own host/agent — [gpsnmeajp/men-g2-ble-gateway](https://github.com/gpsnmeajp/men-g2-ble-gateway/) is a Python gateway (its BLE layer ported from MentraOS's `G2.kt`) that bridges the glasses to HTTP, WebSocket, MCP and a local browser UI, i.e. a concrete self-hosted, no-store way to ship G2 functionality | | **G1** | **Even runs no store or official distribution path** — but a G1 owner reaches third-party apps through the Mentra app store via MentraOS, and anything outside that is a private/community app talking to the raw BLE protocol | Sources: [Even Hub launch (AndroidGuys)](https://androidguys.com/news/even-realities-even-hub-app-store-for-g2-smart-glasses-launches/), [App Submission & QA Guidelines](https://hub.evenrealities.com/docs/ship/app-submission), [Even Hub](https://hub.evenrealities.com/). ## Capabilities, limits & AI | Capability | Third-party access | | --- | --- | | **Camera** | None on either device — Even ships **no camera by design**; a G2 plugin can request access to the **phone's** camera/album via SDK plugins with user permission, but there is no glasses-camera photo/video/frame API. | | **Microphone/audio capture** | Yes — G2 exposes the four-mic array as **PCM 16 kHz mono 16-bit** streams, and `AudioInputSource` lets a plugin switch between the glasses mics and the phone mic at runtime (G1 streams LC3 over BLE). | | **Audio output/TTS** | The glasses have **no speaker**; any voice output must route through the paired phone. | | **On-lens display/UI** | Yes — monochrome green HUD; you render via SDK layout primitives (text containers up to ~1–2k chars, lists, images max 288×144 px, ~4 per page) compiled to **LVGL** on-glasses, with full-rebuild and flicker-free partial-update paths. Limits: **no font-size API in the Even Hub SDK** (the G1's raw BLE text protocol does carry a settable font size), **no official animation API**. | | **Sensors** | IMU/accelerometer at ten selectable pacing codes (`ImuReportPace.P100`–`P1000`, which the docs note are "protocol pacing codes, not literal Hz values"); no dedicated on-glasses GPS — location comes from the phone bridge, and Private and Beta builds fire real permission prompts exactly as an end user sees them while QR sideload skips some during dev. Key-value string storage persists across reboot. | | **Input** | Left/right temple touchpads (press/swipe/double-press) plus the optional **R1 ring**. | | **Reserved surfaces** | Even's own first-party apps are the reserved surfaces on **both** models — **Even AI** (reached by tap or the reserved "Hey, Even" wake phrase), real-time **Translation** (~35 languages), **Teleprompter** (AI/Manual/Automatic scroll modes), **QuickList**, **Conversate**, **Navigation**, and the **Dashboard**/notification HUD ([Even Realities smart glasses](https://www.evenrealities.com/smart-glasses)); the SDK exposes no wake-word/gesture-system hook, so third-party plugins cannot hook the assistant/wake path. | | **AI & the assistant** | The Even Hub plugin SDK exposes **no assistant or wake-word API**, so a plugin cannot hook the Even AI path — and "AI skills" is an announced-but-unshipped roadmap surface. The Even App itself ships an **"Agent Configuration"** setting (Settings → Even AI, visible only to enrolled Even Hub developers) that routes Even AI voice input to a developer-supplied OpenAI-compatible chat-completions endpoint and renders the reply on the HUD ([Even's own tutorial](https://support.evenrealities.com/hc/en-us/articles/16280926743695-Tutorial-Bridging-G2-to-OpenClaw-Bring-Your-Own-Agent), [agent PoC](https://github.com/fuschini/eveng2-agent-poc)) — a third-party assistant path Even documents in its support centre rather than in the developer docs. The practical path is **bring-your-own-AI**: because plugins are WebView apps with `fetch`, developers wire their own STT (Deepgram/Soniox/browser ASR) and LLM (Claude/OpenAI/Gemini) directly; there is no on-device AI. The BYO-AI framing is a direct consequence of that reserved assistant surface. Sources: [SDK feature verification (zenn)](https://zenn.dev/bigdra/articles/eveng2-sdk-features?locale=en), [Even Hub overview](https://hub.evenrealities.com/docs/get-started/overview), [Even Realities smart glasses](https://www.evenrealities.com/smart-glasses), [awesome-even-realities-g2](https://github.com/pangoleen/awesome-even-realities-g2). | ## Roadmap - **Recent, dated:** G2 hardware announced and on sale **Nov 12 2025** ([9to5Google](https://9to5google.com/2025/11/12/even-realities-launches-g2-smart-glasses-r1-control-ring/)); Even Hub **Pilot Program** applications opened in early 2026 ([Android Authority](https://www.androidauthority.com/even-realities-hub-smart-glasses-developers-3629083/)); Even Hub store + developer platform **publicly launched April 3 2026** with ~50 launch apps and a claimed network of 2,000+ independent developers ([AndroidGuys](https://androidguys.com/news/even-realities-even-hub-app-store-for-g2-smart-glasses-launches/)). - **Announced but unshipped:** the docs explicitly flag **widgets, dashboards, and AI skills as "next"** (plugins are the only live first-party surface today). - **Momentum:** high and improving — a real npm-published SDK/CLI/simulator, active first-party templates and demos, a documented app-QA policy, and an established store, all within ~months, plus an independent open-source track (MentraOS) that already ships to both G1 and G2. - **Gating that remains:** ship-time store review, not developer entry — onboarding became self-serve, and the Draft → Test → Submitted → Released promotion pipeline is now fully documented, though a review-turnaround SLA is still not public. G1 is effectively legacy for Even's own tooling (no roadmap toward an official SDK), but stays developer-reachable through MentraOS, the BSD protocol demo, and Gadgetbridge. ## In the landscape Even Realities is one platform in the third-party [smart-glasses landscape](/docs/ecosystem). See [how open it is](/docs/ecosystem/openness) relative to other platforms, [how AI works](/docs/ecosystem/ai) across them, and the full [platform comparison](/docs/ecosystem). --- source: https://extentos.com/docs/ecosystem/platforms/htc-vive-eagle --- --- title: "HTC VIVE Eagle" seoTitle: "HTC VIVE Eagle for developers: SDK, app model & distribution (2026)" description: "HTC VIVE Eagle for third-party developers — SDK access, app model, distribution, capabilities & AI, and where it sits in the 2026 smart-glasses landscape." type: reference verified: "2026-08-24" platform: all vendor: htc related: - /docs/ecosystem - /docs/ecosystem/openness - /docs/ecosystem/ai - /docs/ecosystem/distribution - /docs/vendors/meta - /docs/concepts/capabilities --- **Openness verdict.** Genuinely open at build time — undocumented at ship time, and sold across four Asian markets while HTC's own Western pages still say Taiwan-only. **Covered here:** VIVE Eagle (HTC's AI smart glasses) and the VIVE AI Glasses SDK for iOS and Android. **Not covered here:** VIVE XR Elite, the VIVE Focus series and VIVE Flow (VR/MR headsets on the VIVE Wave / OpenXR stack) · VIVERSE (HTC's web metaverse platform, not a wearable). ## Overview HTC's **VIVE Eagle** is a displayless pair of AI smart glasses: a 12MP ultra-wide camera with AI auto-correction, a four-mic beamforming array, and bass-enhanced open-ear speakers on a Qualcomm **Snapdragon AR1 Gen 1** with 4 GB RAM and 32 GB storage, driven by a phone ([spec sheet](https://www.vive.com/tw/product/vive-eagle/spec/)). There is no display and no HUD — HTC's own structured data for the product says so outright, describing the lenses as "鏡片為一般鏡片，無 AR 或 HUD 顯示" (ordinary lenses, no AR or HUD display) ([VIVE Eagle, Taiwan](https://www.vive.com/tw/product/vive-eagle/overview/)). Everything the device tells the wearer, it says out loud. Video is capped at ten minutes per clip in a vertical, social-native format, and two privacy behaviours are enforced in hardware: camera functions auto-disable when the glasses are not worn or when the capture LED is covered. A smartphone is mandatory (iOS 17.6+ or Android 10+), along with an HTC account and the VIVE Connect companion app, with use gated at 13+. **Regional availability is the sharpest thing to get right about this platform, because HTC's own pages disagree with each other.** Eagle is **buyable now in at least four markets**, each publishing its own price: Taiwan from **TWD$15,600** (TWD$19,888 for the limited photochromic UV lens), with a first-party configurator and structured data marking it `InStock` ([Taiwan](https://www.vive.com/tw/product/vive-eagle/overview/)); Hong Kong at **HK$3,988** ([Hong Kong](https://www.vive.com/hk/product/vive-eagle/overview/)); Singapore from **S$730** (S$880 for AdaptiveSun lenses) ([Singapore](https://www.vive.com/sea/product/vive-eagle/overview-sg/)); and Japan from **税込82,500円** (98,000円 for the photochromic lens) ([Japan](https://www.vive.com/jp/product/vive-eagle/overview/)). HTC's newsroom dates each expansion — Hong Kong 11 Dec 2025, Singapore 9 Mar 2026, Japan 22 Apr 2026 ([newsroom](https://www.vive.com/tw/newsroom/2026/)). Meanwhile **vive.com/us, /uk, /de, /eu and /fr still state "VIVE Eagle is currently only available in Taiwan"** — a claim HTC's own newsroom and its Asian storefronts have overtaken. Those pages are stale marketing surfaces, not evidence of unavailability; they are also why no US-dollar price for Eagle should be trusted, since no US purchase path exists at all. The device is not sold in the US, UK or EU. The openness verdict then splits along the build/ship line: the SDK, its full API reference and a working simulator are free to anyone on earth, while running on real hardware means importing from one of four Asian markets and getting your app registered by HTC through a route HTC does not document. ### What you can buy ## Access The **VIVE AI Glasses SDK** covers **Android (Kotlin/Java)** and **iOS (Swift)** from a single documented API surface, and it is **not gated**. | | | | --- | --- | | **Who can download** | Anyone. The download page lists both archives with direct links — no sign-in wall, no approval step, no NDA. An unauthenticated request for `Android_ViveGlassSDK_0.6.0_beta.7z` returns HTTP 200 and a 12.4 MB 7z archive; the iOS archive returns 16.4 MB the same way ([0.6.0 release](https://developer.vive.com/resources/vive-ai-glasses-sdk/download/latest-release/060/)). This corrects a common assumption that HTC's glasses SDK sits behind a partner gate: documentation and binaries are both public today | | **Maturity** | Early but genuinely active. **Six versions are published**, and the cadence is tight: 0.1.0 (6 Mar 2026), 0.2.0 (30 Mar 2026), 0.5.0 (1 Jun 2026), 0.5.1 (5 Jun 2026), 0.5.2 (9 Jun 2026) and **0.6.0 (3 Jul 2026)**, which added advanced video streaming configurations ([archive](https://developer.vive.com/resources/vive-ai-glasses-sdk/download/archive/)). Versions 0.3.0 and 0.4.0 were never posted. The whole SDK is under six months old, every artifact is still labelled `beta` in its filename, and none has reached 1.0 | | **How you install it** | **By archive only.** On Android the download does contain a real Gradle coordinate — `com.htc.viveglass.sdk:viveglass_client:0.6.0`, packaging `aar` — but served from a file-based Maven repository bundled inside the 7z and consumed via `url = uri("$rootDir/repository")`, so there is no remote repository to resolve against and no automated version bumping | | **iOS install** | A manual drop-in of `ViveGlassKit.xcframework` and `ViveGlassSimulator.xcframework` at **Embed & Sign**, with **Xcode 15.0+ and iOS 16.0+** as the floor per the archive's own readme; no SPM, no CocoaPods | | **Android floor** | No `minSdk` is documented on the site, though the shipped AAR declares `minSdkVersion 28` | | **Cost** | **Zero** to download | | **License** | HTC publishes **no licence or EULA** — not on the download page, and not inside either archive, which contain no licence, EULA, terms or notice file at all. Commercial terms of use are simply unstated | | **Docs** | [developer.vive.com/resources/vive-ai-glasses-sdk](https://developer.vive.com/resources/vive-ai-glasses-sdk/) | The real access wall sits one step later, at runtime. The Android setup page opens with **"Step 1: Register the application — You must register your app to enable communication between the software and hardware"** ([Android setup](https://developer.vive.com/resources/vive-ai-glasses-sdk/documentation/android/setup/)), and the SDK enforces it as `UnregisteredAppException`, "thrown when the app has not yet been officially registered" ([exceptions](https://developer.vive.com/resources/vive-ai-glasses-sdk/documentation/android/exceptions/unregisteredappexception/)). HTC documents no way to become registered — no registration form, portal, email address or contact route for it appears in the developer resources, in either shipped archive, or in any public repository. The generic channels the SDK pages do link — a developer support page, a "Contact us" form and the VIVE Developer Forums — cover no part of registration. Anyone may build and simulate; who may ship against hardware is decided off-document. ## App model Code runs **on the phone, not on the glasses**. There is no documented way to deploy an application, applet or web view to the device; Eagle is a peripheral and the developer's own iOS or Android app is the application. The root object is `ViveGlass`, which "provides a unified interface for managing connections and interactions with VIVE AI Glasses" across four documented jobs — device connection, voice interaction (TTS/STT), audio input, and image input ([ViveGlass reference](https://developer.vive.com/resources/vive-ai-glasses-sdk/documentation/android/viveglass/)). On Android you instantiate `ViveGlass`, assign a `ViveGlassKit` as the adapter, and call `connect()` with a `ViveGlassClientCallback` receiving `onConnectionStateChanged`, `onImageCaptured`, `onSpeechTranscribed`, `onTextSpoken` and `onKeyEvent`. | | | | --- | --- | | **Transport** | Documented most explicitly on iOS, and there are **two paths**: direct Bluetooth (`NSBluetoothAlwaysUsageDescription`, with `UIBackgroundModes: bluetooth-central` for background operation) and a local link to the **VIVE Connect** companion app via Bonjour service `_viveglass_connect._tcp` plus the `viveglass-connect` queried URL scheme, with your app exposing its own `CFBundleURLTypes` scheme for the callback ([iOS setup](https://developer.vive.com/resources/vive-ai-glasses-sdk/documentation/ios/setup/)) | | **VIVE Connect** | Not optional infrastructure on either platform — it owns user state, `getUserPreferredLocale()` "retrieves the language settings from VIVE Connect", and the shipped Android manifest declares a `` intent for `com.htc.viveglass.action.AUTH`, commented "Required to query the host app for the auth handshake" | | **Authoring systems** | **One**: no Unity path, no web or WebXR path, no on-device runtime | What HTC does provide is a **first-party simulator**, which matters given the hardware sells only in Asia. You import `com.htc.viveglass.sdk.simulator.ViveGlassSimulator`, call `create(appContext)`, set it as the `ViveGlass.adapter` in place of the real kit, and call `showUI(supportFragmentManager)`. Sample media is swapped by pushing files into the app's files directory with Device Explorer at documented specs — HEIC 1440×1920 for stills, MP4 480×856 30 FPS with AAC 44.1 kHz 32 kbps stereo audio — and the page is explicit that **"Your app must be a debug version"** ([simulator setup](https://developer.vive.com/resources/vive-ai-glasses-sdk/documentation/android/simulator-setup/)). The iOS simulator works differently: it borrows the iPhone's own microphone and Speech framework to stand in for the glasses rather than replaying files. ## Distribution An Eagle app is an **ordinary phone app**, so it reaches users through the App Store and Google Play like any other — there is no HTC store, no Eagle storefront and no on-device catalogue, because there is no on-device runtime to host one. HTC's own developer programme confirmed the shape by asking finalists for exactly what a normal mobile release requires: the **live App Store listing URL** plus presentation assets including icon and UI screenshots for a consumer app, or a **genuinely working connected backend** for a business app, with TestFlight builds and APKs accepted as supporting material ([2026 VIVE Eagle developer programme, as the page stood on 15 May 2026](https://web.archive.org/web/20260515234049/https://www.vive.com/tw/aiglassesdevprogram/) — HTC has since removed it). Content review is therefore Apple's and Google's; HTC documents no review or certification process of its own for third-party Eagle apps. The gating that does exist is upstream of publishing. Your app must be **officially registered with HTC** before it can communicate with the glasses at all, enforced both as `UnregisteredAppException` and — importantly for anyone building a connection flow — as a connection state, `ConnectionState.ERROR_UNREGISTER_APP`, which HTC's own bundled sample handles alongside `ConnectionState.ERROR_UNSUPPORTED_ROM_VERSION`. An unregistered app does not crash; it silently fails to connect. Publishing one to a store is possible but pointless. Because HTC publishes no self-serve registration route, the only documented path to registered status ran through its developer programme — and **that programme was geographically restricted and has now ended**. Entrants had to be ROC (Taiwan) nationals, foreign nationals holding valid Taiwan residency and actually resident there, or organisations legally registered in Taiwan. Its published schedule ran proposals 8 Jan – 9 Feb 2026, 15 teams selected 12 Feb (each receiving a pair of glasses), POC submissions 9 Mar – 19 Apr, 10 teams advancing 22 Apr, commercial development 30 Apr – 14 Jul, live demos 15–16 July, and a final round awarding **four winning entries NT$100,000 each**. The second round was defined as teams who passed screening and "取得 HTC VIVE Eagle AI SDK" — HTC treating hardware-capable SDK entitlement as something it grants, even though the archives download publicly. Selected teams were also bound by a confidentiality and non-compete clause covering the SDK and technical documentation, a notable contrast with binaries that download with no NDA at all. **No successor programme or replacement registration route is currently published.** For a developer outside Taiwan the practical position today is: build and simulate freely, ship to a real wearer only by arrangement with HTC. The public third-party footprint reflects that — a GitHub search for `viveglass` or "vive eagle" returns two repositories in total, both with zero stars. ## Capabilities, limits & AI | Capability | Third-party access | | --- | --- | | **Camera** | `captureImage(ImageQuality)` for stills, and three `startVideoStreaming(...)` overloads of increasing configurability — basic `VideoQuality`, then with `AudioStreamingFormat`, then with a full `VideoStreamingFormat` — plus `stopVideoStreaming()` and `isVideoStreaming()`, with `Framerate` as a separate enumeration ([ViveGlass reference](https://developer.vive.com/resources/vive-ai-glasses-sdk/documentation/android/viveglass/)). Frames arrive through `StreamingBufferCallback` with lifecycle on `StreamingEventCallback`, so this is a genuine live-frame pipeline, not just a shutter button. The sensor is 12MP ultra-wide, producing 3024×4032 stills and 1512×2016 @ 30 fps video, capped at ten minutes per clip ([spec sheet](https://www.vive.com/tw/product/vive-eagle/spec/)). | | **Microphone and audio** | `startAudioStreaming(Microphone, bitrate, sampleRate, AudioChannel, ...)` gives unusually fine control for this class of device, including a `Microphone` enumeration documented as defining "the directional pickup pattern of the microphone" ([enumerations](https://developer.vive.com/resources/vive-ai-glasses-sdk/documentation/android/enumerations/)) — meaningful because the hardware is a four-mic beamforming array (1 directional + 3 omnidirectional), not a single capsule. | | **Speech** | `startTranscription(boolean bContinuous)` covers both continuous dictation and single-utterance capture, with results on `onSpeechTranscribed`; `speakText(String, Locale)` performs TTS out of the glasses' speakers, with progress on `SynthesisEvent`. STT and TTS are supplied by the platform, so a voice loop needs no third-party speech vendor. **Display/UI: none.** The device has no display and the API has no render, overlay or display surface anywhere in it — the only channel back to the wearer is synthesized speech through 2x stereo open-ear speakers. Any UI belongs on the phone. **Sensors and input: minimal, and this is the sharpest limit.** `KeyEvent` has exactly **one** case, `AIBUTTON` — "The AI button on the VIVE AI Glasses was pressed" ([KeyEvent](https://developer.vive.com/resources/vive-ai-glasses-sdk/documentation/android/enumerations/keyevent/)). The gap is quantifiable: HTC's spec sheet lists four hardware inputs — voice control, touchpad, capture button and AI button — and the SDK exposes one of them. There is no documented IMU, head tracking, gesture or location surface. | | **Wear state and permissions** | `GlassesNotWornException` is thrown during connection if the user is not wearing the glasses, and a `requestPermission()` / `checkPermission()` pair with `Permission` and `PermissionResult` governs access, backed by `GlassesPermissionDeniedException` and `NoPairedGlassesException`. Pairing happens in VIVE Connect, not in your app. | | **Reserved surfaces** | The "Hey VIVE" wake phrase and HTC's assistant, the touchpad and capture button, and app registration. | | **AI & the assistant** | HTC's platform AI is **closed to third parties, but the useful primitives are not**. Eagle ships with HTC's own VIVE AI, notable for letting the *end user* switch the underlying model — the spec sheet lists "VIVE AI 可存取以下模型：• Google Gemini • OpenAI GPT（目前為 Beta 版本）" — and Eagle can additionally wake the phone's native assistant, Siri or Google Assistant, from the glasses ([VIVE Eagle, Taiwan](https://www.vive.com/tw/product/vive-eagle/overview/)). None of that is reachable from the SDK: there is **no assistant, chat, prompt or LLM entry point** in the API, and the model switch is a user setting rather than a developer choice. HTC's own legal footnote describes VIVE AI responses as "由第三方語言模型生成" (generated by a third-party language model), and full functionality requires **VIVE AI Plus**, a paid tier carrying a 24-month free trial on signing into VIVE Connect, with post-trial pricing undocumented beyond a note that it "可能需額外註冊、同意條款並支付相關費用". The third-party path is therefore **bring-your-own-model**, and it is workable: the AI button press arrives in your app as a `KeyEvent`, platform STT turns speech into text, you call whatever model you like from the phone, and `speakText` returns the answer through the glasses' speakers. HTC brokers no AI gateway and imposes no model choice on developers. | ## Roadmap **3 July 2026:** SDK **0.6.0** shipped, adding advanced video streaming configurations — the sixth release in four months, after 0.1.0 (6 Mar), 0.2.0 (30 Mar), 0.5.0 (1 Jun), 0.5.1 (5 Jun) and 0.5.2 (9 Jun), all beta-labelled and pre-1.0 ([archive](https://developer.vive.com/resources/vive-ai-glasses-sdk/download/archive/)). That cadence, and the steady growth of the video API, mark active development rather than a parked SDK. **8 January to July 2026:** HTC and the Institute for Information Industry ran the **2026 VIVE Eagle AI Glasses Developer Programme** ([HTC, 13 January 2026](https://www.vive.com/tw/newsroom/2026-01-13/)) with published dates — proposals 8 Jan – 9 Feb, 15 teams selected 12 Feb (each receiving hardware), POC submissions 9 Mar – 19 Apr, 10 teams advancing 22 Apr, commercial development 30 Apr – 14 Jul, live demos online 15 July and at HTC headquarters 16 July, and a final round awarding **four winning entries NT$100,000 each** ([programme page, as it stood on 15 May 2026](https://web.archive.org/web/20260515234049/https://www.vive.com/tw/aiglassesdevprogram/); that capture schedules the final announcement for 17 July). HTC has since deleted the programme page — it returns HTTP 404 as of 24 August 2026 — and has published no results. HTC has not announced a successor, and whether it reopens is the single most useful signal to watch, because it was the only documented route to a registered app — and with the page deleted, HTC now documents no route at all. **Market expansion is the live storyline.** HTC's newsroom dates a steady march outward from Taiwan — Hong Kong 11 Dec 2025, Singapore with Singtel 9 Mar 2026, Japan 22 Apr 2026 (alongside a Red Dot award) — plus recurring feature drops: AI continuous-conversation mode (13 Nov 2025), Cantonese support (9 Mar 2026) and VIVE AI Notes with 12-language transcripts and summaries (31 Mar 2026) ([newsroom](https://www.vive.com/tw/newsroom/2026/)). The programme materials also dangled international promotion for selected teams "搭配HTC海外上市計畫" (in step with HTC's overseas launch plan), pointing at further expansion. But **no Western market, date or price has been announced**, and HTC's US, UK, German, EU and French pages have not been updated since the Asian launches — they still assert Taiwan-only availability, which is now inaccurate rather than informative. Treat US/UK/EU availability as unannounced, while treating a US launch as signposted: six VIVE Eagle SKUs appeared on Amazon US in July 2026 at $499 with a 1 September 2026 ship date, were pulled within days and have never been confirmed by HTC, and HTC's own US store has run a [VIVE Eagle "Notify Me" page](https://shop-us.vive.com/pages/vive-eagle-notify-me) since 7 August 2026 with no product, price or date attached. **Also outstanding:** a 1.0 SDK, any published licence terms, a remote dependency-manager distribution, a documented `minSdk` and ROM-version compatibility matrix, and a self-serve app-registration route. ## In the landscape HTC VIVE Eagle is one platform in the third-party [smart-glasses landscape](/docs/ecosystem). See [how open it is](/docs/ecosystem/openness) relative to other platforms, [how AI works](/docs/ecosystem/ai) across them, and the full [platform comparison](/docs/ecosystem). --- source: https://extentos.com/docs/ecosystem/platforms/inmo --- --- title: "INMO" seoTitle: "INMO for developers: SDK, app model & distribution (2026)" description: "INMO for third-party developers — SDK access, app model, distribution, capabilities & AI, and where it sits in the 2026 smart-glasses landscape." type: reference verified: "2026-08-24" platform: all vendor: inmo related: - /docs/ecosystem - /docs/ecosystem/openness - /docs/ecosystem/ai - /docs/ecosystem/distribution - /docs/vendors/meta - /docs/concepts/capabilities --- **Openness verdict.** Unusually open at build time — the in-scope INMO Air3 is a standalone Android 14 device that ships (internationally) with the Google Play Store, plus a Unity SDK and a beta n8n-style agent platform — but the openness is qualified: INMO's prose API reference sits behind a login-walled Feishu doc (the narrative guide, the per-version changelogs and the SDK's own shipped XMLDoc are login-free) and the published Air3 kit is slow-moving (v0.8.2 beta, January 2026), app distribution mixes an open Play-Store path with a curated first-party Air3 store, and there is both a reserved first-party wake word ("OK Amu") and a curated INMO-hosted in-app runtime for agent workflows. **Covered here:** INMO Air3. **Not covered here:** INMO GO · INMO GO2 · INMO GO3 · INMO Air2 · INMO X. ## Overview INMO XR (Sichuan INMO Technology / 四川影目科技, Chengdu-based with a secondary Shenzhen office) builds standalone AR + AI eyewear. The in-scope developer device is the **INMO Air3**: an all-in-one, standalone AR headset rather than a phone-tethered viewer. Hardware snapshot ([product page](https://www.inmoxr.com/products/inmo-air3-ar-glasses-all-in-one-full-color-waveguide)): binocular Sony Micro-OLED via 1D diffractive waveguide at 1920×1080, 36° FOV, up to 600 nits, 60/120 Hz; a 16 MP 120° ultra-wide camera with EIS; 4 microphones and 2 speakers; a Qualcomm Snapdragon 8-core (8 GB RAM / 128 GB storage) running **Android 14 / IMOS 3.0**; a 660 mAh battery. On-board compute is **standalone** (a full Android device on your face), not a peripheral. Input is a bundled touchpad, an optional 3DoF smart ring, and a hand-tracking wristband described as a future accessory. No longer listed on INMO's international store: as of 2026-08-24 the Air3 product page 301-redirects to the homepage in every locale, the store feed carries only the GO3 line, and /collections/air is empty. It sold there as a **Global version at $1,099** (the line page was still live on 2026-08-05), and INMO's Chinese site still promotes it with a JD.com buy link. The lighter INMO **GO / GO3** translation glasses are out of scope — INMO publishes no SDK or developer-mode documentation for them. ### What you can buy ## Access The published toolkit is the INMO Unity SDK, whose canonical home is INMO's first-party developer portal at [developer.inmolens.com](https://developer.inmolens.com/). | | | | --- | --- | | **SDK** | `InmoAir3SDK_0.8.2.unitypackage` (v0.8.2 beta, dated 2026-01-16, Unity 2021+), served as a direct download with **no login** | | **What's in it** | The portal package is cleanly **Air3-first** — `Assets/InmoAir3SDK/`, an `air3_core` Android plugin, an `InmoAir3 Player` prefab — and ships an `inmolib_common.xml` API reference, so the core exposed surface is publicly verifiable without a login: 6DoF start/stop plus head pose, camera intrinsics and distortion parameters, 3DoF ring pose and ring input, Bluetooth-beacon scanning, and touchpad/key events. No mic or display-region API appears in that shipped reference | | **Older repos** | The [Gitee INMOLens org](https://gitee.com/inmolens/inmo-sdk-sample-unity) and the [GitHub air3-unity-sdk repo](https://github.com/INMOXR/air3-unity-sdk) hold older Air2+RING2-era samples (min Android API 28, Unity 2020.3 LTS) rather than the current kit | | **You may not need it** | Because Air3 is native **Android 14**, standard Android/Kotlin/Java development also applies — there is no need to go through the Unity SDK to ship an app | | **Maturity** | Preview/early and slow-moving — the portal kit is v0.8.2 beta (2026-01-16), while the GitHub repo carries no releases, tags or license file (license **unknown**) and has had no code push since 2025-09-02 | | **Who can build** | **Open** — any developer enables Developer Mode on-device by tapping the build number 10 times ([support](https://support.inmoxr.com/air3/development/unlock-developer-mode/)). No invite or partner gate to develop | | **A warning that used to apply** | On the earlier Air2, enabling ADB was a separate on-glass step (long-press the right touchpad twice, ~1.5s each, then a factory password) that INMO documented as **voiding the 7-day no-reason return policy**. The Air3 developer-mode page documents no equivalent ADB step or return-policy warning | | **Docs** | [support.inmoxr.com/air3](https://support.inmoxr.com/air3/) plus the portal's own SDK documentation. The narrative guide — Introduction, Update Notes, SDK Integration, QuickStart, Components and Build Settings, in English and Chinese — is served **login-free** from [developer.inmolens.com](https://developer.inmolens.com/), and so are the per-version changelogs. The portal publishes no prose API reference of its own: its 'API Reference' card links out to a [Feishu doc](https://inmolens.feishu.cn/docx/Zu5Vdt4YfoMxgRxLbCccrpOKnHd) that redirects to a Feishu login, as do the FAQ, tutorial-video, sample-code and per-version doc links — so the only API reference readable without an account is the `inmolib_common.xml` shipped inside the package; a community [Unofficial INMO Air 3 Wiki](https://github.com/sam1am/inmo_air_3_wiki), compiled from an INMO Air 3 Discord, covers hardware, setup and troubleshooting but not the API surface | ## App model Code runs **on the glasses** as a standalone Android 14 app — there is no required phone host. **Three developer surfaces** exist. | Surface | What you build | | --- | --- | | **Native Android** | Kotlin/Java apps installed directly on the device | | **Unity** | Apps built with the INMO SDK | | **Multimodal Agent platform** | The **[INMO Multimodal Agent Developer Platform](https://www.inmoxr.com/blogs/news/introduction-inmo-multimodal-agent-developer-platform)** is an n8n-based low-code orchestrator (beta) whose 'apps' are **not standalone Android packages** but visual workflows **published into the on-device INMO Air3 'Super App'** via one-click publish and server-side **hot updates**. Those agent apps run in a curated, INMO-hosted **in-app surface** — rendering into screen regions and controlling volume and progress bars while capturing image/voice/text and returning text/image/audio/video — the closest analog to Meta's Display Web Apps, and distinct from sideloading an APK | Otherwise transport is self-contained (native/Unity apps are on-device); an INMOLens companion app handles casting/management to phone/PC. **Fit with Extentos's phone-companion model:** weak/orthogonal — Extentos assumes a phone app driving a display-limited peripheral over BLE, whereas Air3 runs full Android apps locally. An Extentos-style integration would look like a normal Android app on Air3, not a companion-and-peripheral split. ## Distribution Multiple paths, unusually open for AR. | Path | How it works | | --- | --- | | **Google Play** | The **international Air3 ships with Google Play Store and Chrome** (INMO cites an official Google deal and 90%+ Android-app compatibility, per the [product line page](https://www.inmoxr.com/pages/inmo-air3) and third-party review coverage), so apps reach users the same friction-free way as on a phone | | **INMO Air3 App Store** | A curated store for glasses-optimized apps, submission-based. Acceptance criteria are published on [Gitee](https://gitee.com/inmolens/docs) — security, privacy, performance benchmarks, background behaviour, prohibited content and mandatory signing, with an apply-to-upload → review → list workflow — though that doc set is Air2-era (its README still documents the AIR2 SDK and its version table stops at 1.0.0, 2023-03-30, though the repo itself was last touched 2025-12-04) and no Air3-specific policy or review SLA is published | | **Sideload** | Plain **ADB sideloading** after Developer Mode | | **Super App** | **One-click publish / hot updates** from the Multimodal Agent platform | Public publishing is therefore **open via Google Play** and **submission-gated via the Air3 store**. No preview→GA release-channel invite gate like Meta's DAT exists. ## Capabilities, limits & AI | Capability | Third-party access | | --- | --- | | **Camera** | 16 MP 120° ultra-wide with EIS, video-capable — photo/video capture supported at the OS level, though INMO's own Air3 page footnotes current-firmware photo and video output as 3 MP, with 16 MP output deferred to a later OTA; frame-level API gating is not documented in a public source | | **Microphone and audio out** | 4-mic array and 2 speakers — both available to Android apps | | **On-lens display/UI** | **Yes**, a real binocular 1080p waveguide you render to as a standard Android/Unity full-screen surface (the agent platform can also target screen regions) | | **Sensors** | Head pose and 6DoF are documented in the shipped, login-free `inmolib_common.xml` reference (`ArPoseManager.Start6Dof` / `Stop6Dof` / `arPose` / `arSixDofData`), and INMO's Gitee SDK docs describe an `ARService` 6DoF session; raw IMU/gyro/accelerometer streams go through the standard Android sensor APIs rather than the Unity kit. INMO's own docs list loop-closure, plane detection, occlusion detection, illumination estimation and environment understanding as unsupported, and document no spatial-anchor persistence API | | **Input** | Bundled touchpad, optional 3DoF smart ring, future hand-tracking wristband; no neural band | | **Reserved surfaces** | Air3 ships a first-party AI assistant activated by the reserved wake word **'OK Amu'** (Amu = 影目/INMO), documented in INMO copy and hands-on [reviews](https://www.androidfaithful.com/inmo-air3-review-android-apps-on-your-face-but-the-battery-cant-keep-up/) — a reserved wake surface analogous to Meta's 'Hey Meta'. Whether third-party apps can claim their own wake phrase or must yield to 'OK Amu' is not documented; gesture-monopoly and IMU restrictions are likewise undocumented either way | | **AI & the assistant** | Beyond the first-party Super App, the **Multimodal Agent platform is an explicit bring-your-own-AI path** — developers compose agents against OpenAI, Gemini, Claude, DeepSeek and Llama, plus HTTP/code/database/**MCP** tools, with voice/image/text invocation from the glasses. Around those developer-supplied keys the platform layers a **managed** governance surface — centralized credential management with platform-wide reuse of model API keys, granular credits/token metering, access control, and end-to-end execution logs — a managed-gateway/billing analog over the BYO-model providers. AI execution is **cloud or self-hosted** (models run off-device); on-device local inference is not claimed | Overall the platform reads as broadly open Android rather than a walled first-party runtime. ## Roadmap Air3 was pulled from INMO's international store between 2026-08-05 and 2026-08-24 — inmoxr.com now lists only the GO3 line — while INMO's Chinese site still promotes it with a JD.com buy link. It sold internationally at $1,099 (Android 14 / IMOS 3.0) and was reviewed in the wild in 2026. The **Multimodal Agent Developer Platform is in beta** (introduced 2025-09, still beta), signaling INMO is pushing an agent-first developer story on top of the raw Android/Unity access. The Unity SDK is live but minimal and slow-moving — the portal kit is v0.8.2 beta (January 2026) and the GitHub repo has had no code push since September 2025. Adjacent lineup momentum: **INMO GO3** launched on [Kickstarter (announced 2026-04-02)](https://www.prnewswire.com/news-releases/inmo-unveils-go3-next-generation-everyday-ai-smart-glasses-launching-on-kickstarter-302730972.html) as a translation-focused successor to GO, now shipping at $599 — appliance-class, not a developer target. Announced-but-unshipped for Air3: the hand-tracking wristband accessory (described as tested/future). No public preview→GA gating timeline for app publishing beyond the existing open Play-Store + Air3-store paths. ## In the landscape INMO is one platform in the third-party [smart-glasses landscape](/docs/ecosystem). See [how open it is](/docs/ecosystem/openness) relative to other platforms, [how AI works](/docs/ecosystem/ai) across them, and the full [platform comparison](/docs/ecosystem). --- source: https://extentos.com/docs/ecosystem/platforms/meta --- --- title: "Meta smart glasses" seoTitle: "Meta smart glasses for developers: SDK, app model & distribution (2026)" description: "Meta smart glasses for third-party developers — SDK access, app model, distribution, capabilities & AI, and where it sits in the 2026 smart-glasses landscape." type: reference verified: "2026-08-24" platform: all vendor: meta related: - /docs/ecosystem - /docs/ecosystem/openness - /docs/ecosystem/ai - /docs/ecosystem/distribution - /docs/vendors/meta - /docs/concepts/capabilities --- **Openness verdict.** Semi-open: a real native SDK (the Wearables Device Access Toolkit) plus a hosted Web Apps path ship today, but the platform is preview-narrow — integrations reach only testers you invite by email (every release channel is invite-only, and the per-channel limit is configured per app), public distribution is partner-gated, iOS App Store publishing is blocked, pairing requires Meta's first-party AI app, and the Meta AI assistant and "Hey Meta" wake word stay closed to developers (a few negotiated partnerships aside), so you must bring your own AI. **Covered here:** Ray-Ban Meta (Gen 1, 2023) · Ray-Ban Meta (Gen 2, 2025) · Ray-Ban Meta Optics (prescription/optical) · Oakley Meta HSTN · Oakley Meta Vanguard · Meta Glasses (Meta's self-branded line — Adventurer, Fury, Starfire) · Meta Ray-Ban Display (monocular HUD + Meta Neural Band). **Not covered here:** Meta Quest 3 / 3S / Pro · Meta (Aria) research glasses. ## Overview Meta ships the highest-volume consumer smart glasses on the market, spanning a camera-plus-audio line (Ray-Ban Meta Gen 1/2, Ray-Ban Meta Optics, Oakley Meta HSTN, Oakley Meta Vanguard) and a display-equipped flagship, the [Meta Ray-Ban Display](https://www.meta.com/blog/meta-ray-ban-display-ai-glasses-connect-2025/) — a monocular full-color heads-up display driven by the wrist-worn Meta Neural Band (EMG). The camera/audio glasses have no screen; they capture the wearer's viewpoint (12MP camera, up to 3X zoom on Display), a 5-microphone array, and open-ear speakers. Meta Ray-Ban Display launched in the US on **September 30, 2025 at $799** (glasses + Neural Band included), with expansion to Canada, France, Italy, and the UK announced for early 2026 but [paused that January](https://techcrunch.com/2026/01/06/meta-pauses-international-expansion-of-its-ray-ban-display-glasses/) amid demand outstripping supply, with no new date given; the camera/audio models sell more broadly and at lower price points. Third-party access arrives through the **Meta Wearables Device Access Toolkit (DAT)** — a native Android/iOS SDK, in public developer preview — and a separate **Display Web Apps** path for the Display glasses. Meta's [named launch partners](https://developers.meta.com/blog/introducing-meta-wearables-device-access-toolkit/) skew toward accessibility and media: Be My Eyes, Microsoft Seeing AI, and HumanWare (accessibility), Twitch and 18Birdies (streaming/golf), alongside Disney Imagineering, Logitech Streamlabs, L+R, and Pixel and Texel — the accessibility use case is central to Meta's public positioning. (Building on Meta with Extentos → /docs/vendors/meta.) ### What you can buy ## Access The primary developer surface is the [Wearables Device Access Toolkit (DAT)](https://developers.meta.com/blog/introducing-meta-wearables-device-access-toolkit/). | | | | --- | --- | | **SDK** | Wearables Device Access Toolkit (DAT) — a native SDK, currently at **v0.9.0 (2026-08-03)** | | **Languages** | **Kotlin (Android)** and **Swift (iOS)** | | **Maturity** | Public developer preview, not GA — APIs still shift between minor versions (the [Android SDK is on GitHub](https://github.com/facebook/meta-wearables-dat-android) with a public CHANGELOG) | | **Cost** | Free. Anyone can download the SDK and build against real glasses in Developer Mode; there is no disclosed SDK license fee | | **Account needed** | Only once you distribute beyond Developer Mode: a Meta Wearables Developer Center project, with an `APPLICATION_ID` + `CLIENT_TOKEN` used for app attestation | | **Hard dependency** | [Meta's FAQ states](https://developers.meta.com/wearables/faq/) that **the Meta AI app must be used to pair your glasses** — initial setup, DAT registration, and every permission grant and device-access session run through it | | **Without hardware** | A Mock Device Kit lets you develop and run automated tests, though Meta notes it "currently doesn't support display glasses" | | **Docs** | [wearables.developer.meta.com](https://wearables.developer.meta.com/docs/), backed by a public Wearables MCP server and AI-coding plugins | | **Announced** | [Connect 2025](https://techcrunch.com/2025/09/17/meta-unveils-new-smart-glasses-with-a-display-and-wristband-controller/) (Sept 17–18; the Display + DAT reveal keynote was Sept 17), then [opened as a public preview on 2025-12-04](https://www.uploadvr.com/meta-wearables-device-access-toolkit-public-preview/) | ## App model There are **two independent third-party systems**, and they do not overlap. | | **DAT** | **Display Web Apps** | | --- | --- | --- | | **Where your code runs** | *Your own* native iOS/Android companion app, which pairs to the glasses over Bluetooth (through the Meta AI app) and brokers a session | A standard web app you host at any public **HTTPS** URL, which loads directly onto the glasses | | **Works on** | The camera/audio models and the Display glasses | **Display-only** — no support on the camera/audio glasses | | **How it gets on the device** | Your companion app, installed on the phone as normal | Added on-device via the Meta AI app under **App Settings → App Connections → Web Apps → Add a Web App** | | **Drawing on the lens** | Display-only, via a declarative Kotlin/Swift component API where each `sendContent()` (Android) / `send()` (iOS) call replaces the whole screen | Ordinary web APIs, with no app-store or native build | DAT is the general-purpose path (camera, audio, display); Web Apps is a lighter, display-surface path. ## Distribution During preview, distribution is deliberately narrow on *both* paths — though by different mechanisms, and only the DAT path is invite-bounded. | | **DAT** | **Display Web Apps** | | --- | --- | --- | | **Who can install** | Testers you invite by email, each needing a Meta Account, who accept via [wearables.meta.com/invites](https://wearables.meta.com/invites) | Anyone who receives the link | | **How you share it** | **Invite-only release channels** managed in the Wearables Developer Center | You host at a public HTTPS URL and share a link/QR from the Meta AI app's **Share link** button (Meta's Display build post describes these as password-protected URLs); recipients with Developer Mode enabled add it in one tap | | **Ceiling** | Meta states **"all release channels for Device Access Toolkit are currently invite-only"** and that **"the maximum number of release channels per integration and users per channel are configured per app"** — so the [release-channel docs](https://wearables.developer.meta.com/docs/set-up-release-channels) publish no fixed number, though [Meta's Display-glasses build guide](https://developers.meta.com/blog/build-for-display-glasses/) says you can share DAT builds "with up to 100 testers" in Developer Preview. Check your own project's distribution page for its actual limits | **No invite or release-channel gate** — though every recipient must enable Developer Mode in the Meta AI app before they can add it | | **Review** | **No published content-review criteria or app-review guidelines.** Meta's setup docs name a "publishing review" that Developer Mode lets you skip, but publishing is not open during the developer preview, so no criteria for it have been published. The only review documented in detail is an internal permission-justification check (camera/microphone/voice-invocation), which is not shown to end users | None | | **Publishing to the general public** | **Partner-gated** in preview — only select partners (Meta names the accessibility/media roster above plus Disney Imagineering, Logitech Streamlabs, L+R, Pixel and Texel) could publish broadly — with general availability planned for 2026 | Ungated | | **A store** | — | Meta ships **no first-party store**, but discovery is not nonexistent: **community catalogs have filled the gap**, notably [MetaGlassApps](https://metaglassapps.com/) and [GlassApps](https://glassapps.io/), where Display Web Apps can be browsed and submitted. They are unofficial and unaffiliated with Meta, so treat them as a community index rather than a distribution channel Meta supports | | **iOS App Store** | **Not currently supported**: because DAT uses Apple's `ExternalAccessory`/MFi framework, App Store submission leads to rejection today. The sanctioned iOS path is release channels (and Meta's own signing/TestFlight-style sharing) | n/a — it is a web app | ## Capabilities, limits & AI | Capability | Third-party access | | --- | --- | | **Camera** | Photo capture and live video streaming over DAT at three resolution tiers (LOW 360×640, MEDIUM 504×896, HIGH 720×1280); HEVC is available as an opt-in `StreamConfiguration` codec (the default stream can be raw), and the Display's camera is 12MP with 3X zoom. | | **Microphone / audio in** | The 5-mic array is captured via Bluetooth **HFP** — Meta documents the link as 8 kHz mono voice-band audio, though wideband mSBC at 16 kHz is negotiated on some phone/glasses pairings — and you request it through the OS audio session (there is no raw multi-mic array API). | | **Audio out / TTS** | Playback to the open-ear speakers is done by routing your own audio over Bluetooth **A2DP**; there is no built-in TTS engine — you bring the audio bytes. | | **On-lens display** | Only on Meta Ray-Ban Display — via DAT's declarative layout components (vertical scroll only, full-screen replace) or via Web Apps (fixed **600×600** additive overlay, dark UI recommended). | | **Web Apps API surface** | Motion and orientation data (IMU via `DeviceMotionEvent`/`DeviceOrientationEvent`), phone GPS geolocation, **first-class Meta Neural Band input**, and **client-side local storage**, all through standard web APIs. On the **DAT (native)** side, Meta's permissions docs note that new device capabilities may be added in future iterations, so accelerometer/inclinometer/IMU access is *not* part of the current preview. | | **Input** | on Display, navigation comes from the **Neural Band EMG** and temple **captouch** gestures, surfaced to Web Apps as arrow-key/Enter events (no mouse/touch/keyboard; text input unsupported). | | **Reserved surfaces** | The **"Hey Meta" wake word**, **Meta AI**, and system 'transactions' (notifications, Hey Meta) are system-owned; raw Neural Band gesture signals and IMU-for-gesture are not exposed. | | **AI & the assistant** | Meta AI is **closed as a developer API** — a handful of curated partners aside. Be My Eyes, Spotify, Amazon Music, Audible and iHeart are invocable through the wake word today ("Hey Meta, call a volunteer with Be My Eyes"), but those are Meta AI *feature partnerships*, not DAT apps, and Meta publishes **no application route** to become one. Meta states voice commands and Meta AI capabilities are 'not part of our initial developer preview' and only something it is 'exploring for future updates.' There is **no official bring-your-own-assistant hook into Meta AI and no custom system wake word** — a third-party wake phrase runs phone-side on the mic stream; third parties implement their own AI by capturing mic/camera in their companion app and running their own STT/LLM/TTS off-device (phone or cloud). DAT does expose a **'voice invocation' permission** to start your integration by voice, but that is invocation, not access to Meta AI. There is no third-party **on-device AI** runtime — and Web Apps explicitly lack camera and microphone access. | ## Roadmap Momentum is steady and dated. - **2025-09-17:** Meta Ray-Ban Display ($799) and DAT revealed in the Connect Day-1 keynote (developer coverage spans Sept 17–18). - **2025-09-30:** Display ships in the US. - **2025-10-30:** DAT SDK v0.1.0 (initial public release). - **2025-12-04:** DAT enters public developer preview. - **2026-02-03:** v0.4.0 adds Display support - **2026-04-15:** v0.6.0 adds Ray-Ban Meta Optics + HEVC streaming - **2026-05-14:** v0.7.0 adds the Display capability + the Device Access Toolkit App Model (DAM) - **2026-06-25:** v0.8.0 broadens model support (Oakley Meta HSTN/Vanguard, Meta Glasses) and adds `clearDisplay()`; **2026-08-03:** v0.9.0 consolidates streaming under a new `Camera` capability (`DeviceSession.addCamera` / `Camera.stream` / `CameraState`, replacing `addStream`/`removeStream`), adds `FlexBoxScope.buttonGroup` and local `Bitmap` images, routes taps on display components back to the app, makes the Device Access Toolkit App Model always-on (the `DAM_ENABLED` opt-out is gone), and removes `DisplayComponent` and `VideoScope`. - **Announced / planned:** general-availability public publishing in 2026; Meta's permissions docs note that new device capabilities may be added in future iterations (specifics such as microphone-array or IMU exposure are not committed); and Meta AI/voice-command access is named as an area it is 'exploring for future updates.' Display international expansion (Canada, France, Italy, UK) was slated for early 2026, but Meta paused it on 2026-01-06 citing unprecedented demand and limited inventory, and has given no new date. ## In the landscape Meta smart glasses is one platform in the third-party [smart-glasses landscape](/docs/ecosystem). See [how open it is](/docs/ecosystem/openness) relative to other platforms, [how AI works](/docs/ecosystem/ai) across them, and the full [platform comparison](/docs/ecosystem). --- source: https://extentos.com/docs/ecosystem/platforms/rayneo --- --- title: "RayNeo (TCL)" seoTitle: "RayNeo (TCL) for developers: SDK, app model & distribution (2026)" description: "RayNeo (TCL) for third-party developers — SDK access, app model, distribution, capabilities & AI, and where it sits in the 2026 smart-glasses landscape." type: reference verified: "2026-08-24" platform: all vendor: rayneo related: - /docs/ecosystem - /docs/ecosystem/openness - /docs/ecosystem/ai - /docs/ecosystem/distribution - /docs/vendors/meta - /docs/concepts/capabilities --- **Openness verdict.** Open, standard-based build path (MIT OpenXR Unity ARDK + a native Android ARDK, no partner/invite gate) — but there's a real developer-mode unlock and no consumer publishing surface. X-series apps run standalone on the glasses (not phone-companion), ship as sideloaded Android/Unity APKs, and reach devices via ADB or community sideload tooling (GBox); there is no self-serve public app store, though RayNeo's X-series docs publish a manually brokered submission and review route through its Open Platform. The platform assistant is first-party and closed, and region-split — Google Gemini in Western firmware, Alibaba's Qwen (Tongyi Qianwen) on China-market devices. **Covered here:** RayNeo X2 · RayNeo X3 Pro. **Not covered here:** RayNeo Air 4 Pro / Air 3s / Air 3s Pro · RayNeo Air / Air 2 / Air 2s · RayNeo GT Max · RayNeo V3 / V4 · RayNeo earlier XR viewers. ## Overview RayNeo is the smart-glasses brand of Thunderbird Innovation, a TCL-backed venture. Its **X-series** are standalone binocular AR smart glasses with full-color micro-LED optical-waveguide displays — the in-scope class here. The [**X3 Pro**](https://www.rayneo.com/products/x3-pro-ai-display-glasses) (shipped late 2025) runs a self-contained Android-based OS on a Qualcomm Snapdragon AR1 Gen 1 (4GB RAM / 32GB storage), with a dual-camera rig (Sony IMX681 RGB for 12MP photos / up to 4K video plus an OV monochrome camera for spatial tracking), open-ear speakers and mics, an IMU, and SLAM-based tracking (3DoF at launch — see capabilities); the co-developed waveguide runs ~3,500 nits typical / 6,000 nits peak, 30° FOV, at ~76g. Its predecessor, the [**X2**](https://www.rayneo.com/products/tcl-rayneo-x2) (shipped 2024, Snapdragon XR2, ~1,000-nit micro-LED waveguide), was billed as the first binocular full-color micro-LED waveguide consumer AR smart glasses. Pricing for the X3 Pro is $1,299.00 regular on the [US store](https://www.rayneo.com/products/x3-pro-ai-display-glasses), currently discounted to $1,099.00 in an "End of Summer Sale" ending 31 August 2026 (€1.259,00 against a €1.399,00 regular price on the [EU store](https://eu.rayneo.com/products/x3-pro-ai-display-glasses)), sold via RayNeo's US/EU regional stores; the X2 launched on Indiegogo at ~$699 early-bird. **Out of scope:** the RayNeo **Air** line (Air 3s / Air 3s Pro etc.) are tethered birdbath micro-OLED *viewers* with no on-board compute or app runtime; separately, the display-less [**RayNeo V3**](https://www.xrom.in/post/tcl-rayneo-unveiled-rayneo-v3-ai-glasses) camera-and-audio AI glasses are a distinct Ray-Ban-Meta-class line (scoped out below, though it is the RayNeo device closest to Extentos's target class). ### What you can buy ## Access The developer toolkit is the **RayNeo ARDK**, offered in two flavors: an **[OpenXR Unity ARDK](https://github.com/github-for-rayneo/OpenXR-Unity-ARDK)** (C#/ShaderLab, MIT-licensed on GitHub) and a native **Android ARDK**. | | | | --- | --- | | **SDK** | RayNeo ARDK, in two flavors — OpenXR Unity ARDK and native Android ARDK | | **Languages** | C#/ShaderLab for the Unity flavor; Android native for the other | | **Toolchain** | Per [Qualcomm's X3 Pro getting-started guide](https://www.qualcomm.com/developer/project/get-started-with-rayneo-x3-pro-ar-development), the recommended stack is Unity 2022.3.36f1 (2020.3.20 LTS+ supported) with the Android SDK/NDK, OpenJDK, and ADB; you enable **OpenXR + the 'RayNeo XR feature group'** in Unity's XR Plug-in Management | | **Maturity** | Early — usable and shipping but China-first. The full ARDK is distributed through RayNeo's public, login-free [developer manual](https://rayneo.gitbook.io/rayneo-devdoc/) (with a full [English mirror](https://rayneo-en.gitbook.io/rayneo-devdoc)) as dated, versioned downloads covering the native Android ARDK (v0.2.6, April 2026) and the Unity ARDK (1.1.2), plus a [RayNeo Open Platform](https://open.rayneo.com/) registration portal | | **Who can build** | Effectively open — Qualcomm's guide targets 'AR enthusiasts to professional creators' with no stated invite/partner gate, and the Unity ARDK is public on GitHub | | **Cost** | No documented developer fee; tools are free | | **License** | The OpenXR Unity ARDK repo is MIT; the native Android ARDK's license terms are not clearly published | | **Docs** | The Open Platform and Feishu. X3 Pro hardware capabilities are also documented through Qualcomm's Snapdragon developer program | ## App model Code runs **on the glasses, standalone** — the X-series is a self-contained Android device. | | | | --- | --- | | **What you ship** | An ordinary Android **APK** built in Unity (or native Android) and installed onto the glasses over **ADB/USB-C** | | **Manifest requirement** | There is no phone-hosted application model — the app's main logic always runs on the glasses, and documented phone-side roles are companion-only (the IPC SDK receives phone-pushed GPS; an official sample offloads RTMP livestream relay to a phone app over Wi-Fi P2P): the AndroidManifest must set the launch activity to `com.rayneo.openxradapter.UnityOpenXrActivity` so the app is treated as a glasses (OpenXR) app rather than a phone app | | **Getting a build on** | On consumer firmware ADB itself must first be unlocked (see distribution) | | **The phone's role** | A companion **'RayNeo AR' phone app** ([iOS](https://apps.apple.com/us/app/rayneo-ar/id6476358286) / [Android](https://play.google.com/store/apps/details?id=com.rayneo.mercury)) exists for pairing/management but is not the app-development target | | **Transport** | USB-C/ADB for development; runtime networking/AI is over the glasses' own Wi-Fi | | **How many dev systems** | Effectively **one** for XR apps (OpenXR/Android ARDK) with two front-ends (Unity vs. native Android); the X2 additionally documents a WeChat mini-program path via an embedded mini-program framework, a separate non-APK route that carries no XR content and publishes through WeChat's open platform and then RayNeo's | **Fit with Extentos's phone-companion model: poor / n/a for the X-series** — unlike Meta's DAT (where the app lives on the phone and the glasses are a Bluetooth peripheral), RayNeo X apps execute standalone on the headset, so a phone-hosted-app abstraction doesn't map onto this platform. State this plainly to developers rather than implying parity. ## Distribution An app reaches a device today by **sideload**: build the APK and `adb install` it over USB-C. | | | | --- | --- | | **Where sideloaded apps appear** | The on-device **'App Lab'** launcher (RayNeo's own name — 'App Lab Features' is a toggle in on-device Settings, and the RayNeo AR phone app exposes a 'Glasses App Center') | | **The developer gate** | Real: the consumer ('user version') firmware [does not permit third-party installs by default](https://skarredghost.com/2025/12/17/rayneo-x3-pro-review/), so you must unlock ADB via a hidden gesture (swiping left ~10 times on Settings › General › About Device, which repeating turns back off), and Windows sideload can require an ADB driver downgrade | | **A public store** | **None self-serve** for third-party apps — on Western firmware the on-device menu at launch was dominated by RayNeo's own bundled apps, with no RayNeo-run discovery surface for third-party ones, while China-market X3 Pro units launched with a 'RayNeo AR application virtual machine' carrying a first batch of 30-odd mainstream Android apps (Douyin, Bilibili, Honor of Kings), with an AI Agent store announced but not shipped | | **The brokered route** | RayNeo's X-series dev docs do publish a self-serve submission console followed by a manual, brokered review and listing: register and certify on the [Open Platform](https://open.rayneo.cn/), upload the APK under "Android 应用管理", email `Developer@rayneo.com`, then pass RayNeo functional and test-department sign-off before listing | | **Review** | RayNeo's docs describe an internal functional and listing test rather than a published policy. Preview→GA gating/timeline for a store: not published | | **Community tooling** | Covers sideload plumbing: **GBox**, a compatibility layer that runs the Google Play Store, Chrome, and similar Android apps on the glasses, and [**TapLink**](https://github.com/informalTechCode/TAPLINKX3), a community-built XR web browser for the X3 Pro; its bundled "GlassApps Store" is a dashboard view of the community [glassapps.io](https://glassapps.io) catalog of Meta Ray-Ban Display web apps, which it opens as URLs rather than installing — neither is a RayNeo channel | | **Not a publishing path** | RayNeo's own 'Creator Mode', which unlocks 6DoF SLAM | **Public publishing status: not available** as a self-serve store push — the documented route is the brokered Open Platform submission above, with everyday delivery still sideload/community-grade. ## Capabilities, limits & AI | Capability | Third-party access | | --- | --- | | **Camera** | Dual — Sony IMX681 RGB (12MP photo / up to 4K video) plus an OV monochrome camera for positioning/SLAM; the ARDK exposes camera and sensor data access, though per-frame gating/permission specifics aren't publicly detailed. | | **Microphone / audio capture** | on-board mic array (used for 'Hey RayNeo' and the platform assistant) — and raw third-party capture *is* documented: per [RayNeo's X-series audio docs](https://rayneo.gitbook.io/rayneo-devdoc/x-xi-lie/android-kai-fa/neng-li-jie-shao/yin-pin-kai-fa), an app records with standard Android `AudioRecord` against the `SPEAKER_MIC` device and picks a mic-array mode via `AudioManager.setParameters("audio_source_record=…")` — X3 documents five modes (`OFF`, `RECORD_TRANSLATION`, `CAMCORDER`, `VOICE_RECOGNITION`, `VOICE_COMMUNICATION`), and apps must set `audio_source_record=off` on exit to release the mics. | | **Audio output / TTS** | Open-ear speakers; TTS available through the assistant path. | | **On-lens display/UI** | Yes — binocular full-color micro-LED waveguide (X3 Pro ~3,500 nits typ., 30° FOV); you render via **Unity dual-screen rendering** through the OpenXR/RayNeo feature group. | | **Tracking/sensors** | IMU plus SLAM, but at launch AR tracking is **3DoF only** — a [hands-on review](https://skarredghost.com/2025/12/17/rayneo-x3-pro-review/) found 3D apps 'couldn't keep a stable position in the world' and that 'you can not build proper AR applications with it' today; 6DoF SLAM is Creator-Mode-gated/aspirational, and eye- and hand-tracking have no ARDK API (RayNeo separately markets on-device 'gesture recognition'). Plane and face detection are exposed. | | **Input** | Temple **5-way touchpad**, voice, **phone-as-air-mouse**, and **Apple Watch gesture control**, shipped via the RayNeo AR app 2.1.0 (23 January 2026). | | **AI & the assistant** | The platform assistant is a reserved, **first-party closed** surface invoked by the **'Hey RayNeo'** wake word — and it is **region-split**: Western-market firmware runs **Google Gemini** ('Gemini Live', Gemini 2.5), while China-market devices run **Alibaba's Tongyi Qianwen (Qwen)** ([the V3 was explicitly co-developed with Alibaba](https://www.xrom.in/post/tcl-rayneo-unveiled-rayneo-v3-ai-glasses)). Either way there is **no documented third-party hook** into the system assistant or the reserved wake word, and a developer cannot assume Gemini everywhere. **Bring-your-own-AI is possible but only inside your own ARDK app** — call any cloud model from your APK; it does not integrate with the system assistant. On-device AI: the AR1 Gen 1 can run local inference, but the shipping assistant is cloud-based and no on-device third-party AI API is documented. | ## Roadmap - **Shipped:** X2 (2024, Indiegogo → global); X3 Pro (launched 2025, shipping late 2025 into 2026, US/EU regional stores); the display-less V3 AI glasses (pre-sale Jan 2025). The X3 Pro ARDK is versioned against Unity 2022.3.36f1 and ships through RayNeo's public developer manual; the GitHub package is the X2-era `com.unity.xr.rayneo.openxr` 1.0.1-r3 (Unity 2020.3, last commit October 2024). - **Shipped since launch:** **Apple Watch gesture control** ("X3 Pro & Apple Watch Gesture Interaction System 1.0", RayNeo AR app 2.1.0, 23 January 2026); RayNeo notes it depends on OTA availability and compatible models. - **Announced-but-unshipped / in flight:** reliable **6DoF SLAM** (3DoF-only at launch, Creator-Mode-gated); broader 'AR app ecosystem'/App Lab expansion; deeper assistant features. - **Publishing timeline:** no self-serve third-party app-store path or dated GA for one — submission runs through the brokered Open Platform review described above, and everyday distribution remains sideload/community-run (TapLink/GBox). - **Momentum:** active — TCL/RayNeo iterates hardware yearly (X2→X3 Pro, plus the V3 line) and courts developers via the Qualcomm Snapdragon program and a public GitHub ARDK, but the software/publishing side is early and China-first, with the consumer app-store surface still thin. ## In the landscape RayNeo (TCL) is one platform in the third-party [smart-glasses landscape](/docs/ecosystem). See [how open it is](/docs/ecosystem/openness) relative to other platforms, [how AI works](/docs/ecosystem/ai) across them, and the full [platform comparison](/docs/ecosystem). --- source: https://extentos.com/docs/ecosystem/platforms/rokid --- --- title: "Rokid" seoTitle: "Rokid for developers: SDK, app model & distribution (2026)" description: "Rokid for third-party developers — SDK access, app model, distribution, capabilities & AI, and where it sits in the 2026 smart-glasses landscape." type: reference verified: "2026-08-24" platform: all vendor: rokid related: - /docs/ecosystem - /docs/ecosystem/openness - /docs/ecosystem/ai - /docs/ecosystem/distribution - /docs/vendors/meta - /docs/concepts/capabilities --- **Openness verdict.** Openly documented, registration-free developer suite with a bimodal openness story: the native glasses/phone SDKs ship as AARs with no public source repo (only the glasses SDK publishes a partial sources jar on Rokid's own Maven), while the on-glasses AIUI app framework — a JavaScript + Ink web-style runtime (JSAR) — is open source (Apache-2.0). Four dev axes ship in parallel (phone companion, on-glasses native, on-glasses AIUI web-runtime apps, and cloud/bring-your-own-model agents), with an unusually open third-party-AI path. Distribution is the least-settled part: sideload-to-glasses (phone app, WebUSB, community stores) works today, but the first-party store and review pipeline that has shipped — the Rokid Agent Store, published to via the Rizon platform — covers AIUI agents rather than on-glasses native binaries. **Covered here:** Rokid Glasses (2025) · Rokid AI Glasses Style (2026). **Not covered here:** Rokid Max / Max Pro · Rokid AR Lite / AR Spatial · Rokid Station / Station 2 · Rokid Dock. ## Overview Rokid is a Hangzhou-based XR/AI-wearables maker. **[Rokid Glasses](https://global.rokid.com/products/rokid-glasses)** (launched Aug 27 2025, [newsroom](https://www.prnewswire.com/news-releases/rokid-officially-launches-rokid-glasses-the-worlds-lightest-full-function-ai--ar-smart-glasses-302539782.html)) are 49 g standalone AI+HUD glasses and the in-scope device here. **Hardware snapshot:** dual-eye **monochrome-green Micro-LED** engines on a diffractive optical waveguide (480×400 per eye, up to 1500 nits; the product page cites a 30° FOV while launch coverage cited ~23°), a 12MP Sony IMX681 first-person camera (fixed-focus, no AF), 4 directional microphones, 2 speakers, Wi-Fi 6 / BT 5.3, 2 GB RAM / 32 GB storage, 210 mAh battery. Compute is **standalone** (Snapdragon AR1 Gen 1 for the app OS + an NXP RT600 for low-power voice/wake), not a dumb phone peripheral — but a paired phone (Hi Rokid app) drives setup, heavy AI and most third-party UX. Price is **$699** (list **$799**); shipping globally now (fees and delivery times vary by destination, shown at checkout). Out of scope: the birdbath **Max** viewers and Station-puck-driven **AR Lite/Spatial** (tethered/controller AR). A display-less sibling, **[Rokid AI Glasses Style](https://global.rokid.com/pages/rokid-ai-glasses-style)** ($349, 38.5 g, camera+audio only), shares the ecosystem but has no HUD. ### What you can buy ## Access The developer surface centers on Rokid's connected-XR SDKs, documented at **[x-docs.rokid.com](https://x-docs.rokid.com/docs/en/terminal-sdk/glasses/)** and the **[Rokid Open Platform](https://open.rokid.com/)** (ar.rokid.com/sdk redirects here). | | | | --- | --- | | **SDK** | The official docs frame it as a **Glasses SDK (device side) + Phone SDK (companion)** pairing targeting 'Rokid Glass3' | | **Languages** | The Glasses/Phone SDKs are **Kotlin/Java** (Android) and native **iOS** for companions. **Languages/frameworks are broader than the native SDKs alone:** a second, materially different toolchain exists for on-glasses **AIUI** apps — **JavaScript + the 'Ink' declarative UI framework** (`.ink` single-file components) running on the **JSAR (JavaScript-AR)** web-style runtime | | **How it connects** | The phone companion connects over BLE / Wi-Fi Direct; on-glasses apps run natively on YodaOS-Sprite (an Android-12/API-32-based OS) and talk to the phone over a bidirectional data channel; a standalone variant binds to the on-device Rokid AI service via Android AIDL | | **License** | **Bimodal:** the native Glasses/Phone SDKs ship as **AARs with no public source repo**, though the POMs published on Rokid's own Maven declare Apache-2.0 and sources jars sit next to the binaries: `glass3.open.sdk` ships a partial one (the `GlassSdk` facade, service client and camera helper — 13 files against 282 classes in the AAR), `phone.sdk` is an umbrella whose real modules `phone.sdk.server` and `phone.sdk.api` ship complete sources, and the `com.rokid.cxr` artifacts ship none — whereas the AIUI app tooling — CLI, samples, agent skills — and the underlying JSAR runtime are **open source** ([jsar-project/AIUI](https://github.com/jsar-project/AIUI), Apache-2.0) | | **Maturity** | Shipping/GA product with actively evolving 2.x SDKs (`glass3.open.sdk` 2.2.0-E) | | **Who can build** | **No account at all for the SDKs** — nothing is 'unlocked', because nothing is locked. The Quick Start (English *and* Chinese) publishes the repo `https://maven.rokid.com/repository/maven-public/` with **no `credentials {}` block**, and `com.rokid.security:glass3.open.sdk:2.2.0-E`, `com.rokid.security:phone.sdk:2.2.0-E` and `com.rokid.cxr:client-m:1.0.8` all resolve anonymously (fetched with no token 2026-08-24; the glasses AAR is a real 340 KB artifact carrying `classes.jar`). Neither Quick Start mentions registration, and the Chinese one contains no 实名认证, 注册 or 开通 step. The AIUI tooling is equally open — public GitHub repo plus `@yodaos-pkg/create-aiui-agent` on npm. An Open Platform account exists for the agent console and forum (Rokid cites 30k+ registered developers), but it gates neither SDK. **The one real gate is the cloud OpenAPI**, and it is stricter than self-serve: Rokid's ApiKey page requires you to 开通企业账号 — activate an *enterprise* account by contacting sales/business for OA approval — so the BYO-model agent and chat-log APIs are enterprise-gated. Docs are English + Chinese | | **Not the right SDK** | A separate, older **Unity [UXR SDK](https://github.com/RokidGlass/UXR-docs)** exists but targets the legacy Dock/phone-tethered line (archived 2021) — not Rokid Glasses | | **Sourced to the community, not Rokid** | Rokid's own Open Platform SDK page defines **CXR = Client XR SDK** and names the **CXR-L** (phone, via the Rokid AI App), **CXR-M** (phone, business/contact-sales) and **CXR-S** (on-glasses) SDKs, and the Maven coordinate `com.rokid.cxr:client-m:1.0.8` resolves on Rokid's own repository at `maven.rokid.com`. *Android minSdk 28 is the community mirror [buildwithfenna/rokid-docs](https://github.com/buildwithfenna/rokid-docs)'s figure, not stated on any primary Rokid page — the primary docs cite Android 8.0+ for the Glass3 SDK and API 31 for bare-metal* | ## App model Rokid deliberately exposes **multiple dev systems**, so where your code runs is a design choice. | Path | Where your code runs | | --- | --- | | **Phone-companion** | Your Android/iOS app owns the UX and calls the glasses for camera/mic/display/voice over BLE or Wi-Fi Direct | | **On-glasses native** | Kotlin/Java Android apps running *on* YodaOS-Sprite on the glasses themselves | | **On-glasses AIUI apps** | A distinct web-runtime app framework where you write **JavaScript + Ink** (`.ink` SFCs), scaffold with the official CLI `npm create @yodaos-pkg/aiui-agent`, and render UI to the HUD via the **JSAR** runtime — the Rokid analogue to Meta's Display Web Apps, and publishable without physical hardware ([jsar-project/AIUI](https://github.com/jsar-project/AIUI)) | | **Cloud AI agent** | Via AIUI — an SSE endpoint your glasses call | So the platform is **not** uniformly 'native Android/iOS' — the AIUI/Ink path is an explicitly web-style runtime, and Unity is only the legacy tethered path. **Fit with Extentos's phone-companion model:** strong on the phone-companion axis — a phone app mediating capture/voice/HUD to the glasses over BLE/Wi-Fi is exactly the companion shape Extentos targets, so a phone-side integration is the natural mapping. The on-glasses native, AIUI/JSAR, and cloud-agent paths are Rokid-specific extras with no Extentos analogue. ## Distribution Reaching users is the least-settled part of the platform, and the install landscape is more active-but-fragmented than a single 'community store exists' conveys. | | **AIUI apps / agents** | **On-glasses native binaries** | | --- | --- | --- | | **First-party store** | Yes. Rokid's English AIUI docs describe the full store path end to end — `aix pack` the agent, upload the `.aix` on the Rizon platform, submit for review, then it lists in the **Rokid Agent Store** with VERSION-based hot updates | None. The English SDK docs document no submission or review route for native APKs | | **Review** | Rokid checks performance, interaction compliance and security | Not documented in English | | **Tooling** | The Open Platform's AIUI agent console (AIUI Studio) offers a design→code→debug→deploy flow that publishes without physical hardware | — | | **The everyday route** | Enterprises can instead point agents at their own model endpoint through the Platform OpenAPI, with **no store gate** | **Sideload-to-glasses** over several transports: the **[Hi Rokid companion app](https://apps.apple.com/us/app/hi-rokid-rokid-glasses/id6749669942)** (App Store + Google Play) installs local apps from the phone onto the glasses, and sideload also spans **Bluetooth SPP, Wi-Fi LAN, and USB**, including **browser-based WebUSB installation** where a catalog page (e.g. the EUNG SOFT catalog) flashes an app onto the glasses directly from the web | | **Community channels** | — | **Multiple community stores** exist (RokidBrew plus the store referenced by [awesome-rokid](https://github.com/Anezium/awesome-rokid)) | **Public publishing / content review:** partial — first-party and documented for AIUI apps/agents, but undocumented in English for on-glasses native binaries, where sideload and self-hosted agents remain the reliable routes. No MDM-specific path is documented publicly. ## Capabilities, limits & AI | Capability | Third-party access | | --- | --- | | **Camera** | 12MP photo, video, and frame/media capture exposed through the Glasses SDK's camera-and-media module (fixed-focus, 3°-inward tilt; gated behind SDK connection/permissions). | | **Microphone/audio capture** | 4 directional mics with ASR; **audio output/TTS** via the 2 speakers, all first-class SDK capabilities (the voice/AI module covers ASR, TTS, chat). | | **Recognition/CV** | The Glasses SDK ships a dedicated **recognition module** with packaged **face detection** and **vehicle license-plate recognition** as SDK-exposed primitives — notable for an in-class device (the module set is media, voice/AI, recognition, messaging/files, device-state, Bluetooth/ring, and P2P). | | **On-lens display/UI** | Yes — a **monochrome-green Micro-LED waveguide HUD** for glanceable text/overlays; you render either by running an on-glasses app (native or an AIUI/Ink JSAR app) or by pushing messages/notifications from the phone. It is a low-FOV glanceable HUD, not a rich spatial-AR canvas. | | **Sensors/input** | Temple touchpad, buttons, head gestures, voice, and an optional **smart ring** accessory (the SDK enumerates Bluetooth smart-ring connectivity); IMU is a documented public API on the AIUI side — `Accelerometer`, `Gyroscope` and `AbsoluteOrientationSensor` in W3C Generic-Sensor shape, with a configurable sampling frequency — and location typically comes from the phone. | | **Reserved surfaces** | The **'Hi Rokid' wake phrase/trigger and the default assistant ship first-party**, but third parties are not shut out of voice triggering: the Glasses SDK's `IOfflineCmdService` (`GlassSdk.getGlassOfflineCmdService()`) lets an app register its own offline wake phrases in Chinese or English, each with its own `onVoiceTriggered()` callback, running on-device with no network. Reserving the wake word is a separate axis from extending what the assistant can *do*, though: per Rokid's SDK changelog, Glasses SDK **2.1.9-E** (May 2026) added `GlassSdk.registerLeqiInterceptor()`, a first-party hook where a foreground app implements `LeqiInterceptor.onLeqiInstruction()` and returns `true` to consume a built-in-assistant voice command or `false` to pass it through — superseding the older community workaround of scraping assistant responses/notifications via the Android **Accessibility Service**. | | **AI & the assistant** | Unusually open at the *model* layer even though the wake trigger stays reserved. Via AIUI and the **[Customizable Agent](https://news.aibase.com/news/25479)** feature (announced Feb 2026), developers can **bring their own model** — privately-deployed **DeepSeek R1, Qwen3, Kimi K2.5**, etc. — by registering on the Open Platform and configuring an agent URL + auth key; the glasses call it over **SSE**, with support for tool/agent frameworks (OpenClaw) doing browser/file/Python actions. Voice invocation drives it; on-device compute (the NXP RT600) handles wake/low-power voice while heavier inference is cloud or phone. Net: third-party AI is a supported, first-class path, which is rare in this class. | ## Roadmap - **Recent:** Rokid Glasses launched Aug 27 2025 (NYC event + Kickstarter that topped $400–500k in hours) and are now shipping globally at $699 (list $799); the Glasses/Phone SDKs and the AIUI app tooling are live on the Open Platform. - **Feb 11 2026:** the 'Customizable Agent' bring-your-own-model feature (DeepSeek/Qwen/Kimi via SSE, unreviewed private deployment) was announced — the platform's most notable openness move. The display-less **AI Glasses Style** ($349) expanded the lineup in 2026. - **Shipped since:** the Rokid Agent Store for AIUI agents, surfaced internationally in the Hi Rokid app from v1.10.0. - **Announced-but-maturing:** a first-party app store / public publishing pipeline for on-glasses native binaries (referenced by Rokid but not yet documented in English). - **Momentum:** high — a large existing CN developer base (Rokid cites **30k+ registered developers**), an active global developer push, an open-source AIUI/JSAR app runtime, and frequent SDK iteration; the main gap for outside developers is a clear public distribution/store story for native apps. ## In the landscape Rokid is one platform in the third-party [smart-glasses landscape](/docs/ecosystem). See [how open it is](/docs/ecosystem/openness) relative to other platforms, [how AI works](/docs/ecosystem/ai) across them, and the full [platform comparison](/docs/ecosystem). --- source: https://extentos.com/docs/ecosystem/platforms/snap --- --- title: "Snap Spectacles" seoTitle: "Snap Spectacles for developers: SDK, app model & distribution (2026)" description: "Snap Spectacles for third-party developers — SDK access, app model, distribution, capabilities & AI, and where it sits in the 2026 smart-glasses landscape." type: reference verified: "2026-08-24" platform: all vendor: snap related: - /docs/ecosystem - /docs/ecosystem/openness - /docs/ecosystem/ai - /docs/ecosystem/distribution - /docs/vendors/meta - /docs/concepts/capabilities --- **Openness verdict.** Very open at both build and ship time. Snap ships a free public SDK (Lens Studio 5) plus a second, independent web path (WebXR Browser Lens, no Lens Studio), standalone on-glasses Lenses, and public Lens Explorer distribution open to any developer in the program. The publish gate is narrow: only Experimental APIs (and unsupported permission combinations) cannot ship to a wide audience — camera/mic/location combined with internet access IS publicly publishable via Snap's Transparent Permission system, including bring-your-own-AI Lenses through the Remote Service Gateway. The real access wall is hardware: Snap no longer publishes a sign-up route to 5th-gen Spectacles — the Spectacles Developer Program's announced terms ($99/month US, €110/month in six European countries) are still published, on the original Newsroom announcements and in a Spectacles support article on subscription billing and device return, but no live Snap surface offers a way to join — so the only published purchase path is a Specs pre-order at $2,195, alongside CLAD Device Access — application-gated, NDA-bound on-site sessions with pre-launch SPECS hardware at Snap offices. **Covered here:** Spectacles (5th generation, 2024) · Specs (2026). **Not covered here:** Spectacles v1 / v2 / Spectacles 3 (2016–2019) · Spectacles (4th gen, 2021). ## Overview Snap Inc.'s Spectacles are standalone, see-through binocular AR smart glasses running **Snap OS**, on which third-party **Lenses** (AR apps) execute directly on the device — no tethered phone required ([Snap Newsroom, SPS 2024](https://newsroom.snap.com/sps-2024-spectacles-snapos)). Two models are in scope. The **Spectacles 5th gen (2024)** is the current developer edition: dual Snapdragon SoCs, four cameras driving the Spatial Engine and dual-hand tracking, binocular waveguide displays with a 46° diagonal FOV at ~37 ppd, 13 ms motion-to-photon latency, 226 g, ~45 min battery ([Snap Newsroom, SPS 2024](https://newsroom.snap.com/sps-2024-spectacles-snapos)). **Specs (2026)** is the consumer/pro successor unveiled 16 June 2026 at AWE: fully standalone, two Snapdragon processors (one CV, one for Lenses), proprietary LCoS see-through display, 51° FOV, 16M colors, electrochromic lenses, 7 ms latency, TR90 frames at 132–136 g, up to ~4 h mixed-use battery plus a charging case; **$2,195** with a $200 refundable deposit, shipping **fall 2026** in the **US, UK, France** ([Snap Newsroom, Introducing Specs](https://newsroom.snap.com/introducing-specs-augmented-reality-glasses)). Both are camera+mic+speaker AR smart glasses with on-board compute (standalone, not a peripheral). ### What you can buy ## Access The primary SDK is **Lens Studio 5**, a free, proprietary AR-first IDE from Snap ([ar.snap.com/spectacles](https://ar.snap.com/spectacles)). | | | | --- | --- | | **SDK** | Lens Studio 5 — proprietary, free to download | | **Languages** | **TypeScript and JavaScript** natively, plus **SnapML** for running custom ML models on-device | | **Maturity** | GA and mature — 10 Snap OS updates with 40+ new features/APIs over ~1.5 years, per [Snap Newsroom](https://newsroom.snap.com/introducing-specs-augmented-reality-glasses) | | **First-party frameworks** | Spectacles Interaction Kit (SIK), UI Kit, Sync Kit, Auth Kit, Navigation Kit, Mobile Kit, and Commerce Kit (closed beta), all installed from the Lens Studio Asset Library ([Introduction](https://developers.snap.com/spectacles/get-started/introduction)). Sample projects are public on GitHub and MIT-licensed, in Snap's official SPECS org ([specs-devs/samples](https://github.com/specs-devs/samples)); the older `Snapchat/Spectacles-Sample` repo now carries a deprecation notice pointing there | | **License** | Lens Studio itself is proprietary; sample/community code is typically MIT | | **Second build path** | Spectacles ships a native browser with **WebXR**, so you can author AR with standard web tech (Snap recommends PlayCanvas) and run it in the on-device Browser Lens — no Lens Studio involved ([WebXR](https://developers.snap.com/spectacles/about-spectacles-features/webxr)) | | **Agent tooling** | Documented and usable today, with the SPECS-optimized agent suite (CLAD) in preview. Lens Studio ships a built-in **MCP server** (AI Assistant → MCP → Configure/Start Server) that lets external coding agents — **Claude Code, OpenAI Codex, Cursor** — drive the IDE to generate, debug, optimize, and publish Lenses ([developer mode with Claude Code](https://developers.snap.com/lens-studio/features/lens-studio-ai/developer-mode/with-claude-code)) | | **Hardware access** | **The gate, and Snap no longer publishes how to get it.** The Spectacles Developer Program shipped 5th-gen hardware on subscription — announced at **$99/month in the US with a one-year commitment**, and **€110/month with a 12-month commitment** in Austria, France, Germany, Italy, the Netherlands and Spain ([Snap Newsroom](https://newsroom.snap.com/spectacles-launching-in-first-european-countries)) — but as of 2026-08-10 no live Snap surface offers a way to join: the sign-up URL Snap's own announcement gave now carries no program or price, and `spectacles.com/apply` returns 404. The terms themselves are still published — on the original Newsroom announcements and in a Spectacles support article on subscription billing and device return. Snap now points developers at a free, open-signup **SPECS Developer Program** that grants SDK, docs, learning paths and community access but no device | | **Cost to start** | Anyone can download Lens Studio and read the docs for free. The published purchase route to real hardware is a **Specs pre-order ($2,195)**; Snap also runs **CLAD Device Access** — application-gated, NDA-bound full-day on-site sessions with pre-launch SPECS hardware at Snap offices in LA, London, Amsterdam and Vienna, prioritising developers with a working Lens Studio 5.22.x + CLAD prototype ([CLAD Device Access](https://developers.specs.com/clad-access)) | | **Docs** | [developers.snap.com/spectacles](https://developers.snap.com/spectacles/get-started/introduction) | ## App model Code runs **on the glasses**, not on a phone: a Lens is a Snap OS AR application authored in Lens Studio (TypeScript/JS + scene graph, optionally SnapML), deployed wirelessly to the device ([Introduction](https://developers.snap.com/spectacles/get-started/introduction)). Snap has broadened well past a single authoring path. | | | | --- | --- | | **Native code in a Lens** | A **Native Development Kit (NDK)**, announced at AWE 2026, brings **C and C++** code and libraries directly into Lenses (spatial mapping, physics, audio, networking, navigation) | | **Porting from Unity** | A **Unity-to-Lens-Studio Migration Agent** translates Unity project structures, components, and scenes into native Lens Studio projects — Niantic Spatial ported its VPS via the NDK and Mapbox ported its Navigation Engine in under two hours ([Snap Newsroom, tools for Specs developers](https://newsroom.snap.com/snap-launches-new-tools-for-specs-developers)) | | **The web path** | **WebXR Browser Lenses are an entire second dev-and-distribution system**: standard web AR that runs in the on-device browser with no Lens Studio involved ([WebXR](https://developers.snap.com/spectacles/about-spectacles-features/webxr)) | | **Networking** | On-device networking is available — a WebSocket API, and the Remote Service Gateway for cloud AI | | **Backend** | A first-party managed backend, **Snap Cloud** (powered by Supabase): managed Postgres with Row Level Security, Storage, server-side Edge Functions (TS/JS), and Realtime, with auth bridged from Snapchat identity via OpenID Connect — currently alpha/approval-gated and free during alpha ([Snap Cloud on Supabase](https://eng.snap.com/spectacles_supabase)) | | **The phone's role** | Snap's own **Spectacles phone app** covers auxiliary roles — phone-as-game-controller, Spectator Mode, screen mirroring ([ar.snap.com blog](https://ar.snap.com/blog/lens-studio-cross-platform-ar-development)) — but Snap does ship a phone-side SDK: **Spectacles Mobile Kit**, with documented iOS (Swift, `BondingManager`) and Android (Kotlin, `com.snap.spectacles.kit`) surfaces, lets a developer's own phone app bond over BLE and serve messages, API calls and assets to their Lens ([Spectacles Mobile Kit](https://developers.snap.com/spectacles/spectacles-frameworks/spectacles-mobile-kit/getting-started)) | **Fit with Extentos's phone-companion model: low.** Spectacles/Specs is a self-contained on-glasses AR runtime with its own managed backend and web path; Mobile Kit bonds to a *specific Lens the same developer deployed* (`BondingRequest.SingleLensByLensId`/`ByLensName`) and the Lens drives the exchange, so there is no surface where a phone app drives the glasses' camera, mic or display the way Meta's DAT exposes the device. Extentos's abstraction targets a fundamentally different (phone-tethered) topology. ## Distribution There are two channels, and only one of them has a review step. | | **Lens Explorer** | **WebXR Browser Lenses** | | --- | --- | --- | | **How you ship** | Publishing is built into Lens Studio: hit **Publish**, supply an icon, a 3×4 preview image, description, and release notes ([Publishing Your Spectacles Lens](https://developers.snap.com/spectacles/get-started/start-building/publishing-lens)) | Served from ordinary web hosting by URL — the Snap analog of Meta's Display Web Apps ([WebXR](https://developers.snap.com/spectacles/about-spectacles-features/webxr)) | | **Review** | Snap's moderation team reviews the Lens within **2–3 business days** against community guidelines and, if approved, lists it in **Lens Explorer**, the on-device store. Review covers a defined objective, 60 fps performance, no crashes, and design best practices ([Publishing Your Spectacles Lens](https://developers.snap.com/spectacles/get-started/start-building/publishing-lens)) | **No Lens Explorer submission and no documented moderation review** | | **Who can publish** | **Open to any developer in the program** — no partner/enterprise gate | Anyone with web hosting | | **Size limit** | **25 MB** (vs 8 MB for mobile Snapchat) | — | | **Camera + internet together** | **Snap's Transparent Permission system lets published Lenses combine internet connectivity with sensitive user data (camera, microphone, or location)** as long as they use a supported permission combination (e.g. Internet Module + Camera Module, or Remote Service Gateway + Camera Module), gated at runtime by a per-launch permission prompt plus a hardware capture-LED indicator ([Transparent Permission](https://developers.snap.com/spectacles/permission-privacy/transparent-permission)) | — | | **What still blocks a wide audience** | The **narrow residual block is Experimental APIs** (and unsupported permission combinations / insecure http) — not camera-plus-internet in general | — | Snap also runs **Spectacles Community Challenges** with cash prizes and reports 'hundreds of Lenses already published' for Specs. ## Capabilities, limits & AI | Capability | Third-party access | | --- | --- | | **Camera** | The CameraModule provides on-device camera access to Lenses ([Camera Module](https://developers.snap.com/spectacles/about-spectacles-features/apis/camera-module)). Combining camera (or other sensitive data) with internet access is **publishable** via Transparent Permission with a supported permission combination, and Lenses using the dedicated Remote Service Gateway AI integrations can use camera access and ship publicly **without** Extended Permissions or Experimental APIs ([Transparent Permission](https://developers.snap.com/spectacles/permission-privacy/transparent-permission)); the only publish-blocking path is Experimental APIs. | | **Microphone/audio** | Captured via MicrophoneRecorder; speech-to-text via the **ASR Module** (40+ languages, including mixed language input). | | **Audio output/TTS** | Speakers are available; TTS and generative voice come through the Remote Service Gateway (OpenAI TTS/Realtime, Gemini Live) with DynamicAudioOutput for PCM16 playback. | | **Display/UI** | Full see-through binocular AR — you render a 3D Lens Studio scene (or a WebXR scene) into the wearer's FOV; SIK provides spatial UI. | | **Sensors** | Spatial Engine (SLAM), the Depth Module (an Experimental API, which also disables open internet access for the Lens), dual-hand tracking, plus GPS/location for outdoor location-based Lenses. | | **Input** | Hand tracking with three SIK modes (ray-pointer, pinch, poke), voice, and optionally the phone-as-controller; no dedicated hardware controller or neural band. | | **Persistence/backend** | Beyond on-device WebSocket, **Snap Cloud** (Supabase-backed) offers managed Postgres/RLS, Storage, Edge Functions, and Realtime with Snapchat-identity auth ([Snap Cloud](https://eng.snap.com/spectacles_supabase)). | | **AI & the assistant** | There is **no first-party Snap voice assistant exposed to third parties**; instead Snap offers a strong **bring-your-own-AI** path via the **Remote Service Gateway**, a Snap-brokered proxy to OpenAI (GPT chat, image gen/edit, TTS, Realtime voice), Google (Gemini text/multimodal, Imagen, Lyria), Snap-hosted **DeepSeek R1**, and **Snap3D** (text-to-3D GLB) ([Remote Service Gateway](https://developers.snap.com/spectacles/about-spectacles-features/apis/remoteservice-gateway)). On-device AI is available through SnapML custom models. Voice invocation of AI is a developer-built pattern (mic → LLM → TTS), not a reserved system assistant — and, per Transparent Permission, such camera+internet AI Lenses are publicly publishable. | ## Roadmap **16 Jun 2026 (AWE):** Snap unveiled **Specs**, the 2026 consumer AR smart glasses, at **$2,195** + $200 deposit, shipping **fall 2026** (US/UK/France), alongside a shipping **Native Development Kit** (C/C++ + Unity Migration Agent), an **agentic development** developer preview (Lens Studio's built-in MCP server driving Claude Code, Codex, Cursor), and the **SPECS Spatial Benchmark**; the **Snap3D** API and the location-experience tools (**Fleet Management**, **Guided Mode**, **Guided Navigation**) came a year earlier, in Snap's AWE 2025 wave on 10 Jun 2025 ([Snap Newsroom](https://newsroom.snap.com/introducing-specs-augmented-reality-glasses); [tools for Specs developers](https://newsroom.snap.com/snap-launches-new-tools-for-specs-developers); [investor release, 10 Jun 2025](https://investor.snap.com/news/news-details/2025/Snap-to-Launch-New-Lightweight-Immersive-Specs-in-2026/default.aspx)). Momentum is high: 10 Snap OS updates / 40+ APIs over ~18 months, hundreds of Lenses already published. **Announced-but-unshipped as of this writing:** Specs retail units (fall 2026), Commerce Kit (closed beta), and Snap Cloud (alpha, approval-gated). On 30 Jul 2026 Snap dated the reveal: a **SPECS launch event on 16 September 2026** in Los Angeles, where Evan Spiegel gives the first in-depth look at SPECS, livestreamed at 4:00pm PT ([Snap Newsroom](https://newsroom.snap.com/specs-launch-date)). The 5th-gen Spectacles were the shipping dev hardware via that subscription, but Snap has withdrawn its published sign-up route without issuing an end-of-life notice, leaving a Specs pre-order and application-gated CLAD Device Access as the only routes Snap now lists to hardware. ## In the landscape Snap Spectacles is one platform in the third-party [smart-glasses landscape](/docs/ecosystem). See [how open it is](/docs/ecosystem/openness) relative to other platforms, [how AI works](/docs/ecosystem/ai) across them, and the full [platform comparison](/docs/ecosystem). --- source: https://extentos.com/docs/ecosystem/platforms/solos --- --- title: "Solos" seoTitle: "Solos for developers: SDK, app model & distribution (2026)" description: "Solos for third-party developers — SDK access, app model, distribution, capabilities & AI, and where it sits in the 2026 smart-glasses landscape." type: reference verified: "2026-08-24" platform: all vendor: solos related: - /docs/ecosystem - /docs/ecosystem/openness - /docs/ecosystem/ai - /docs/ecosystem/distribution - /docs/vendors/meta - /docs/concepts/capabilities --- **Openness verdict.** Open to build on (paid but public SDK, standard iOS/Android, commercial use allowed) with an unusual audio-first, no-HUD peripheral model; ship-time is wide open — your own app in the normal app stores, no vendor store or review gate. Access is purchase-based self-service — you buy the kit and SDK access is granted on purchase — with no application step and no invite-only partner gate. **Covered here:** AirGo V2 · AirGo V / Krypton 2 (V1) · AirGo A5 · AirGo 3. **Not covered here:** Legacy Solos cycling smartglasses (Kopin Pupil microdisplay HUD, shipped ~2018 out of a ~2015-16 Kickstarter era) · Kopin microdisplay/optics components · AirGo A6 (announced July 2026, not yet shipping). ## Overview Solos (Solos Technology Limited, spun out of micro-display maker Kopin) makes the **AirGo** line of consumer AI smart glasses positioned as everyday eyewear with hands-free AI. Two families are in scope: **audio-only** models (AirGo 3, AirGo A5 — "Hey Solos" voice AI, directional open-ear speakers, IP54, ~10 h audio / 7 h calls, no camera, [A5 coverage](https://www.androidheadlines.com/2025/06/solos-new-ai-smart-glasses-a5-v2-specs-price.html)) and **camera** models (AirGo V / Krypton 2, and AirGo V2 with a 16MP EIS camera, 2K/30fps video recording and low-power Wi-Fi live streaming, [V2 debut](https://solosglasses.com/blogs/news/solos-airgo-v2-debuts-with-stabilized-16mp-camera-and-live-streaming)). **Hardware snapshot:** open-ear speakers + mic array, motion sensor + e-compass, capacitive touch on the temple, modular "SmartHinge" frames with swappable frame fronts across the line (the electronics live in the temples; an electronics-only Temple Kit is sold for A5 only); camera on the V-series only; **no on-lens display/HUD anywhere in the AirGo line** — output is audio (and camera capture for the app), not visual overlay. Compute is **peripheral/phone-companion** — the glasses pair to a phone; there is no standalone on-glasses app runtime. **Shipping:** all four in scope are shipping; AirGo V2 launched at CES 2026. **Price:** A5 ~$249, V2 from $299 ([Android Central](https://www.androidcentral.com/wearables/solos-airgo-v2-smart-glasses-are-here-with-camera-enabled-ai-for-usd299-at-ces-2026)); the SDK dev kit is a separate $1,999 purchase. **Regions:** direct-webstore availability worldwide, with physical retail only in Hong Kong, Japan and Malaysia — Europe, North America, the Middle East and South America are all still listed "Coming Soon" on the vendor's Retail Shop page (worldwide retail availability for AirGo V2 was announced in July 2026), plus a European technology partnership with Deutsche Telekom (its AI services and cloud, showcased at Digital X). ### What you can buy ## Access Solos runs a formal, **paid, publicly-purchasable** SDK program — the **Solos Smartglasses SDK** ([Developers page](https://solosglasses.com/pages/developers), [SDK Developer Program](https://solosglasses.com/products/sdk-developer-program)). | | | | --- | --- | | **Languages/platforms** | Native **iOS and Android** only — no Unity, Flutter, or web SDK is offered (the glasses are a BLE/Wi-Fi peripheral to a phone app) | | **Maturity** | GA/commercial, single codebase across models with V2 adding new APIs (Wi-Fi Connection, Voice Command, Low Power Mode, Firmware Update; LED Notification listed "coming soon") | | **Who can build** | There is **no invite-only or partner-approval gate** and **no application step** — access is **frictionless paid self-signup**: you buy the kit from the Solos store and SDK access is provided on purchase (SDK Terms 4.1). The project/capability language is a contractual representation accepted at checkout, not a screening step before it | | **Cost** | A **$1,999 kit** bundling two pairs of glasses (one audio + one camera model), one year of Zendesk support + software updates, and a bulk-purchase discount ([FAQ](https://solosglasses.com/pages/sdk-develop-program-faq)). Renewal for continued support/updates is 50%/yr; the license itself is perpetual for the version received | | **License** | **Proprietary**, non-exclusive, non-transferable. You may build and sell commercial apps but only ones that "work only with Solos Smartglasses," and you may **not** redistribute the SDK itself (Solos-designated redistributable runtime components may ship inside your app) or reverse-engineer it except where law permits, while publishing benchmark/test results, using SDK Data to build competing products, and using it to **train or fine-tune machine-learning models for distribution or resale** each require Solos's prior written consent — an AI-scoped restriction that carries weight given these are camera/mic capture glasses feeding exactly that kind of pipeline ([SDK Terms](https://solosglasses.com/pages/sdk-terms-and-conditions), [FAQ](https://solosglasses.com/pages/sdk-develop-program-faq)). Not OSS | | **Docs** | Distributed with the kit (API spec PDFs); no full public API reference, though Solos links a downloadable V2 API specs PDF from the Developers page — inquiries to sdk@solosglasses.com | ## App model **Where code runs:** on the **phone** — this is a phone-companion architecture. | | | | --- | --- | | **The division of labour** | The glasses are a Bluetooth LE (control) + Wi-Fi (V2 camera/video data) peripheral; your native iOS/Android app holds the logic and talks to the glasses through the SDK. There is **no on-glasses app store or standalone runtime** and no web/engine target | | **Transport** | BLE for control/audio/sensors across all models; **Wi-Fi** added on V2 for low-power live video streaming | | **The vendor's own app** | Solos ships a first-party **Solos AirGo** companion app ([App Store since 2020](https://apps.apple.com/us/app/solos-airgo/id1484336694)) — the consumer host for SolosChat, voice search/translation, and "Whisper Messages / Whisper Events" (auto read-aloud of texts, WhatsApp, Teams, calendar). Critically, that official app is **closed to plugins**: there is no app-within-app extension model, so a third-party developer ships a **separate, standalone phone app** (a distinct App Store / Play listing) against the SDK — not an add-in inside AirGo | | **A second build path** | Beyond the native app SDK, Solos exposes an **automation/integration surface** — **webhook / RTMP endpoints and ready-to-use n8n automation templates** so captured audio/video/sensor data can be piped to your cloud or no-code workflows without a full mobile build. Solos advertises these for the capture → parse → route path, wiring glasses capture into tools like Google Sheets, HubSpot and email with no custom backend and no mobile build at all, though it no longer publishes a named sample template on the [Developers page](https://solosglasses.com/pages/developers). It's the closest thing to a genuine second build path — a no-code route where you never touch the iOS/Android SDK | **Fit with Extentos's phone-companion model:** strong conceptual match — like Meta's DAT, Solos is a phone-tethered peripheral where the app runs on the phone and subscribes to capture/audio/sensor primitives. The divergences are that Solos is audio-first with **no display surface to render to**, its SDK is paid + proprietary + iOS/Android-native (no cross-platform layer), and camera streaming rides Wi-Fi rather than BLE. ## Distribution Because the app is an **ordinary phone app**, distribution is via the **normal Apple App Store / Google Play** channels you already own. | | | | --- | --- | | **Solos-side gate** | None — Solos operates **no on-device app store, no release-channel invite system, and no submission/review gate** of its own | | **Public publishing** | **Open:** developers "may create and release commercial applications" provided they are designed to work only with Solos glasses ([FAQ](https://solosglasses.com/pages/sdk-develop-program-faq)) | | **Vendor conditions** | **Contractual, not gatekeeping**: products must carry the attribution "Powered by Solos SDK. Designed for use with Solos Smartglasses.", co-marketing needs prior written approval, and you may not publish test/benchmark results without consent ([SDK Terms](https://solosglasses.com/pages/sdk-terms-and-conditions)) | | **Content review** | None by Solos. Apple/Google review still applies to your app | | **Preview→GA gating/timeline** | n/a — there is no staged rollout program; once you have the kit you ship on your own schedule through your own store listing | ## Capabilities, limits & AI **Camera** (V / V2 only; audio models have none): photo capture and video, exposed as distinctly **named APIs** — a **Photo Taking API** and a **Photo Streaming API**, both listed as V1 camera primitives, with Solos stating **V2 maintains 100% of V1 APIs** and adds **16MP + electronic image stabilization** with **2K/30fps video recording and live streaming** over low-power Wi-Fi; a Voice Command API and app control can trigger capture ([Developers page](https://solosglasses.com/pages/developers)). The documented **Photo Streaming API** is the frame-stream primitive, so continuous frame access is a **named capability**, not merely implied — the exact per-frame grain/latency for real-time AI still isn't published, but the surface itself is documented. | Capability | Third-party access | | --- | --- | | **Microphone/audio capture** | Yes — an **Audio Input API** exposes the mic array to the app. | | **Audio output/TTS** | Yes — an **Audio Output API** drives the open-ear directional speakers (the primary output channel; enhanced-bass, low-leakage). | | **On-lens display/UI** | **none** — no HUD/waveguide on any AirGo model; you cannot render visual UI on the glasses, only audio out + camera in. This is the sharpest limit vs. display-class glasses. | | **Sensors** | **Motion Sensor API** (IMU) and **E-Compass API**; location comes from the paired phone, not the glasses. | | **Input** | Capacitive **Glasses Touch API** (temple tap/swipe) plus voice; no controller or neural band. | | **Reserved surfaces** | The **"Hey Solos" wake word** and the **SolosChat** first-party assistant are Solos-owned; firmware/LED/low-power are managed via dedicated APIs. | | **AI & the assistant** | Solos ships a first-party assistant, **SolosChat 4.0**, that itself routes to multiple LLM providers (**ChatGPT, Claude, Gemini, DeepSeek**) for multimodal image/video/audio/text assistance ([V2 announcement](https://solosglasses.com/blogs/news/solos-airgo-v2-modular-smart-glasses-with-camera-and-accessible-ai)). For **third-party developers the assistant is not an open extension point** — you don't plug into SolosChat; instead you **bring your own AI** by building your own app that pulls camera/audio/sensor data through the SDK (or via the webhook/RTMP/n8n path) and calls whatever model/cloud you choose. Voice invocation of your own app beyond the reserved "Hey Solos" trigger goes through the Voice Command API / your app's own logic. AI inference is **cloud-side**, not on-device (the glasses are a capture/output peripheral). The Envision accessibility integration and a Hong Kong PolyU speech-therapy pilot are cited BYO-AI examples. | ## Roadmap - **Recent (dated):** AirGo A5 + AirGo V2 first unveiled mid-2025 (pre-orders/Q3 2025 for A5 at ~$249); **AirGo V2 formally launched at CES 2026 (January 2026)** at $299 with the 16MP EIS camera, live FHD Wi-Fi streaming, SolosChat 3.0, and an expanded SDK; a charging case (1,100mAh) now planned for August 2026. - **2026-07-15:** at its third annual AI Smartglasses Symposium in Hong Kong, Solos announced the camera-free **AirGo A6**, a **Privacy Kit** ($39/$49/$79), worldwide retail availability for AirGo V2, and **SolosChat 4.0**. - **SDK momentum:** V2 added Wi-Fi Connection, Voice Command, Low Power Mode, and Firmware Update APIs while keeping 100% of the V1 API surface; an **LED Notification API is still listed "coming soon"** on the Developers page. - **Partnerships signaling direction:** Deutsche Telekom (Digital X showcase, Sep 2025), Envision (accessibility), Hong Kong PolyU (speech-therapy pilot). - **Publishing timeline:** n/a in the vendor-store sense — distribution is already open via standard app stores. - **Overall momentum:** active and accelerating on the audio+camera consumer line, but still a small vendor; there is no announced move toward an on-lens display or a standalone on-glasses runtime. ## In the landscape Solos is one platform in the third-party [smart-glasses landscape](/docs/ecosystem). See [how open it is](/docs/ecosystem/openness) relative to other platforms, [how AI works](/docs/ecosystem/ai) across them, and the full [platform comparison](/docs/ecosystem). --- source: https://extentos.com/docs/ecosystem/platforms/vuzix --- --- title: "Vuzix" seoTitle: "Vuzix for developers: SDK, app model & distribution (2026)" description: "Vuzix for third-party developers — SDK access, app model, distribution, capabilities & AI, and where it sits in the 2026 smart-glasses landscape." type: reference verified: "2026-08-24" platform: all vendor: vuzix related: - /docs/ecosystem - /docs/ecosystem/openness - /docs/ecosystem/ai - /docs/ecosystem/distribution - /docs/vendors/meta - /docs/concepts/capabilities --- **Openness verdict.** Build-time: genuinely open — a free, publicly downloadable native SDK (Android + Apple) plus an MIT-licensed, now-GA third-party OS (MentraOS, whose third-party app SDK completed its move from the cloud runtime to a phone-local Miniapp SDK — a public beta — in MentraOS 3.0, which reached Google Play on 2026-08-08 and the App Store on 2026-08-15), no gatekeeping. Ship-time: fully open — a Z100 app is an ordinary phone app shipped through Google Play / the App Store, since the glasses are a display-only Bluetooth peripheral with no on-lens store or review gate. The MentraOS path strengthens this further: it is funded, open-sourced on GitHub, has an app store (rebuilt for local miniapps and due back later in 2026), and is cross-vendor, so a Z100 integration written against it is portable to other makers' glasses. **Covered here:** Vuzix Z100 (Ultralite lightweight smart glasses). **Not covered here:** Vuzix M400 · Vuzix M4000 · Vuzix Blade 2 · Vuzix Shield · Vuzix LX1 · Ultralite OEM platform / waveguide reference designs · Ultralite Pro. ## Overview [Vuzix](https://www.vuzix.com/) (Rochester, NY) is a long-time waveguide-optics and smart-glasses maker whose consumer/prosumer play is the **Z100**, a lightweight heads-up display in a normal-eyewear form factor. This page scopes to the Z100 (the "Ultralite" class); Vuzix's other products — the M400, M4000, Blade 2, Shield and the newer **LX1** — are standalone Android head-mounted *computers* aimed at enterprise, with on-device compute and cameras, covered under a different SDK family (see below). **Hardware snapshot (Z100):** a single-eye (right) transparent [640×480 monochrome microLED waveguide](https://blogs.expandreality.io/vuzix-z100-release-information) display; **no camera, no microphone, and no speaker** (Vuzix positions this as a privacy/battery/weight advantage); tap-on-frame input; ~38 g; up to ~48 h battery; Bluetooth to an Android or iOS phone. It is a **display-only peripheral**, not a standalone computer — all logic and any AI run on the paired phone. **Status:** shipped — [general availability 20 Nov 2024](https://www.prnewswire.com/news-releases/vuzix-announces-general-availability-of-z100-smart-glasses-302311326.html); as of July 2026 it is no longer sold on Vuzix's own store, whose Z100 product URL now resolves to a [smart-glasses index](https://www.vuzix.com/pages/smart-glasses) listing only the LX1, M400 and an in-development "Vuzix AI Smart Glasses", with only Z100 accessories left in the storefront. Vuzix has published no end-of-life notice, and the Z100 support docs and Ultralite SDK remain live. **Price:** $499 at launch. **Regions:** was sold direct and via resellers in North America and Europe; exact regional list not published. ### What you can buy ## Access The official toolkit is the **Vuzix Ultralite SDK**, split by platform. | | | | --- | --- | | **Android** | [ultralite-sdk-android](https://github.com/Vuzix/ultralite-sdk-android) — Java/Kotlin, added via JitPack as `com.vuzix:ultralite-sdk-android`, **v1.9 / Mar 2025**, requires **Android 12+** and the Vuzix Connect app installed | | **Apple** | [UltraliteSDK-releases-iOS](https://github.com/Vuzix/UltraliteSDK-releases-iOS) — Swift, covering **iOS, watchOS and macOS** | | **Maturity** | **GA/stable** — versioned releases, Javadoc, sample apps | | **Who can build** | Open — the SDK is public on GitHub/JitPack with no invite, dev-mode toggle, or partner gate | | **Cost** | Free | | **License** | *Not* open source — use is governed by the proprietary ["VUZIX SOFTWARE DEVELOPMENT KIT LICENSE AND CONFIDENTIALITY AGREEMENT"](https://github.com/Vuzix/ultralite-sdk-android) | | **Docs** | [support.vuzix.com](https://support.vuzix.com/docs/sdk-for-android) plus the [package Javadoc](https://vuzix.github.io/ultralite-sdk-android/javadoc/com/vuzix/ultralite/package-summary.html) | | **Not this SDK** | The [Speech](https://support.vuzix.com/docs/working-with-the-speech-sdk), [Barcode](https://support.vuzix.com/docs/working-with-the-barcode-sdk) and [Connectivity](https://github.com/Vuzix/connectivity-sdk) SDKs target the Android **M400/M4000/Blade/Shield/LX1** HMDs (they run *on* the glasses), **not** the Z100, and are out of scope for this page | ## App model **Phone-companion, natively** — and there are two independent ways to do it. | | **Vuzix Ultralite SDK** | **MentraOS** | | --- | --- | --- | | **Where your code runs** | On the phone/watch/Mac; the app pushes text, images and canvas draw-calls to the glasses over **Bluetooth** | A phone-local runtime: miniapps run inside the Mentra host app on the user's phone. MentraOS 3.0 (Google Play 2026-08-08, App Store 2026-08-15) finished the move off the cloud runtime, now archived as [MentraOS Legacy](https://mentraglass.com/legacy) with support ending October 2026. The Miniapp SDK is what Mentra documents today — a public beta on npm — with its public launch for display glasses including the Z100 slated for September 2026 | | **Language** | Java/Kotlin, Swift | **TypeScript** (`registerMiniapp((session) => ...)`, `session.display.render([...])`) | | **Who brokers the link** | The **Vuzix Connect** app — on Android, Connect owns the Bluetooth link; on iOS the SDK manages Bluetooth directly | — | | **License** | Proprietary | **MIT-licensed, open-source** — [MentraOS](https://mentraglass.com/os), formerly AugmentOS, by Mentra | | **Lock-in** | Vendor-locked to Vuzix | Explicitly [**cross-vendor**](https://docs.mentraglass.com/app-devs/getting-started/overview). Mentra frames it as "Android for smart glasses," and the Z100 is only one supported device alongside Even Realities G1/G2 and Mentra's own hardware (e.g. Mentra Live), with NIMO listed as coming soon. A MentraOS app written against the Z100 therefore runs unchanged on other vendors' glasses — a materially different lock-in profile from the native Ultralite path | The glasses are a thin remote display — no on-glasses runtime, no web/engine layer. Only **one app controls the display at a time** (`requestControl()` / `isAvailable()`), so an integration must arbitrate for control. **Fit with Extentos's phone-companion model:** structurally excellent — the native Ultralite path is exactly a phone-app-drives-a-BT-display architecture. The caveats are that it is a *different vendor and transport* from Extentos's Meta/DAT target, and that the Z100 exposes only a display + tap (no camera/mic/audio), so the capture and audio primitives Extentos centers on have no hardware to bind to on this device. ## Distribution Because a Z100 app **is just a phone app** talking to a Bluetooth peripheral, distribution follows the ordinary mobile path. | | | | --- | --- | | **How you ship** | Publish the Android or iOS app through **Google Play / the App Store** (or sideload/enterprise-distribute it) like any other app | | **Vuzix-side gate** | None — there is **no Vuzix-side app store, review gate, or release-channel invite for Z100 apps**, and no on-glasses store | | **Public publishing** | **Fully open**, and gated only by the phone platform's own store review | | **What the user installs** | Your phone app plus Vuzix Connect, then they pair the glasses | | **Alternate path** | Apps built on **MentraOS** install onto the user's phone — either a [LAN/QR release build](https://docs.mentraglass.com/app-devs/getting-started/distribution) from the miniapp CLI, or a submission to the **Mentra Miniapp Store** through the Developer Console. That store (live since July 2025 for the cloud apps) is being rebuilt for local miniapps, is absent from the initial MentraOS 3.0 release, and returns later in 2026; it is also Mentra's stated monetization surface (developer subscriptions + app sales) | | **Preview→GA gating** | n/a — the SDK is already GA and there is no staged-rollout program for the Z100 | For contrast, the out-of-scope enterprise HMDs distribute Android APKs via the Vuzix App Store and MDM — that model does not apply to the Z100. ## Capabilities, limits & AI | Capability | Third-party access | | --- | --- | | **Camera** | None on the Z100 hardware, so no photo/video/frame API. | | **Microphone / audio capture** | None on the glasses — voice features (e.g., the Connect app's live translation/transcription) run on the **phone's** mic, not the wearable. | | **Audio output / TTS** | None — Vuzix's own Ultralite platform spec lists the Ultralite HUD (the Z100 configuration) as Camera: None, Audio: None, describing it as delivering heads-up information "without introducing audio or camera complexity"; the separate Ultralite Audio OEM platform is the variant that adds integrated stereo audio. | | **On-lens display/UI** | **yes**, this is the whole device — a monochrome 640×480 waveguide you render to via `UltraliteSDK` with `sendText()`, `sendIcon()`, `setLayout()` (`Layout` enum), a direct-draw `Canvas`, `ScrollingTextView`, `Anchor`/`TextAlignment`/`TextWrapMode`, `LVGLImage`, `Icon`, and `UltraliteColor` ([Javadoc](https://vuzix.github.io/ultralite-sdk-android/javadoc/com/vuzix/ultralite/package-summary.html)). | | **Sensors/status** | `BatteryStatus` and connection state via LiveData; **tap-on-frame** input is delivered through the `EventListener` interface. Motion (IMU) and location are **not** exposed to third parties in the documented SDK. | | **Input** | Frame tap and the power button (`EventListener.onTap` / `onPowerButtonPress`) — no touchpad, controller, gesture camera, or neural band. | | **Reserved surfaces** | None in the Meta sense — there is no first-party on-device assistant or reserved wake word on the Z100 itself; the Vuzix Connect app's translation/transcription is a first-party phone feature, not a locked platform surface. | | **AI & the assistant** | There is **no platform assistant on the glasses to open or restrict** — the device is a dumb display. AI is therefore **entirely bring-your-own on the phone side and fully open**: point the display at any model/pipeline you like. MentraOS layers optional **proactive AI, live captions and translation** on top, again running off the phone/cloud. On-device (on-glasses) AI is **not applicable** — the Z100 runs no third-party code on-glasses and has no camera or microphone to feed a model. | ## Roadmap **Shipped:** Z100 **GA 20 Nov 2024** ($499); Ultralite SDK reaching **v1.9 (14 Mar 2025)**. **3 Feb 2025** — Vuzix and Mentra jointly announced **[AugmentOS/MentraOS](https://ir.vuzix.com/news-events/press-releases/detail/2113/vuzix-z100-smart-glasses-and-mentra-augmentos-usher-in-a)** with the Z100 as a launch device, adding an open-source, AI-centric dev platform on top of Vuzix's native SDK. That second dev system has since gone from announcement to a funded, GA platform: Mentra **[raised $8M](https://www.forbes.com/sites/charliefink/2025/07/01/mentra-raises-8-million-to-launch-open-source-os-for-smart-glasses/)** (July 2025; backers include Android co-founder Rich Miner, YouTube co-founder Jawed Karim, and Toyota/Amazon venture arms), **[MentraOS](https://github.com/Mentra-Community/MentraOS)** has been open-source on GitHub since 2023 (originally as AugmentOS) and shipped **MentraOS 3.0** — Google Play served 3.0.0 on 2026-08-08 and the App Store 3.0 on 2026-08-15, completing the migration from its cloud SDK to phone-local miniapps (the GitHub v3.0 tag is still flagged prerelease, with builds attached into late August 2026), and the Mentra app store has been live since July 2025, reportedly with a community of **4,000+ developers** and, by Mentra's own count, over 1,000 apps built on the Cloud SDK (live captions, translation, proactive AI) — the strongest openness signal on this page. **Vuzix direction:** the company is publicly steering toward an **OEM waveguide + AI-glasses** licensing strategy rather than a consumer-hardware treadmill — it has since shown the **Ultralite Pro** OEM waveguide reference platform and introduced the standalone-Android **[LX1](https://ir.vuzix.com/news-events/press-releases/detail/2143/vuzix-introduces-lx1-smart-glasses-designed-specifically)** enterprise HMD (2025), neither of which is a Z100-class device or uses the Ultralite SDK. **No shipping Z100 successor has been announced** as of this writing, though Vuzix now previews an enterprise-framed "AI Smart Glasses" platform (binocular full-colour optics, camera and audio) as "currently in development", with no date, price, or SDK disclosed. **Publishing timeline:** n/a — distribution is already open via standard phone app stores. ## In the landscape Vuzix is one platform in the third-party [smart-glasses landscape](/docs/ecosystem). See [how open it is](/docs/ecosystem/openness) relative to other platforms, [how AI works](/docs/ecosystem/ai) across them, and the full [platform comparison](/docs/ecosystem). --- source: https://extentos.com/docs/getting-started --- --- title: Get started description: Start building with Extentos, the smart-glasses dev layer for Meta Ray-Ban. The recommended path is agent-driven — install the MCP server, it scaffolds the app. type: tutorial platform: all related: - /docs/getting-started/with-agent - /docs/concepts/transport-vs-app - /docs/concepts/architecture - /docs/mcp-server/install - /docs/vendors/meta - /docs/resources/pricing - /docs/mcp-server/agents --- Extentos is a smart-glasses development layer for native iOS and Android apps. **The recommended way to get started is to install Extentos's MCP server in your AI coding agent and let the agent do the rest** — it scaffolds the connection module, writes handler classes against the capability primitives, and provisions a browser simulator session in one continuous flow. This page is the entry point: prerequisites, the recommended path, what to expect, and where to go after the first session is running. > **At runtime, your installed agent has access to the entire MCP tool surface.** Once `@extentos/mcp-server` is registered with your AI coding agent, the agent calls deterministic tools — `getPlatformInfo` for the capability catalog, `getCapabilityGuide(feature)` for per-feature call shapes, `getCodeExample(pattern)` for canonical Kotlin/Swift compositions, `generateConnectionModule` for the scaffold, `validateIntegration` for the correctness gate, `createSimulatorSession` for browser-hosted hardware — plus searches the live documentation via [`searchDocs(topic: ...)`](/docs/mcp-server/tools/search). This page exists for *pre-install evaluation* (so an AI agent or developer can see the path forward before installing) and for SEO / human-search retrieval. The agent's live MCP responses are authoritative once installed; this page is the static reference. ## Prerequisites Three things, all free, all fast: | Requirement | Why | How to get it | |---|---|---| | **Node.js 20 or newer** | The MCP server runs as an `npx` subprocess of your agent | [nodejs.org](https://nodejs.org). Verify with `node -v`. | | **An MCP-compatible AI coding agent** | Operates the Extentos tools on your behalf | Claude Code (terminal CLI), Cursor, Windsurf, or Cline are the verified hosts. See [supported agents](/docs/mcp-server/agents) for capability differences and IDE-portability details. | | **A native mobile project** (iOS Swift / Android Kotlin) | Extentos generates code into your existing app | An empty SwiftUI or Compose app works fine for a smoke test. | **Only if you need camera or display** — voice apps never need these: - A Meta Wearables Developer Center account (free, ~15–30 minute setup) for Meta DAT credentials. See [vendors/meta § Required setup](/docs/vendors/meta#required-setup-for-development). - A paired Ray-Ban Meta variant (Gen 1, Gen 2, Display, or Oakley Meta HSTN). All variants work; see [vendors/meta § Compatible models](/docs/vendors/meta#compatible-models). **Not required** to start: - An Extentos account for the discovery / validation / guidance MCP tools, on-device `LocalSimTransport` simulation, or real-hardware testing — all free, no signup. Two surfaces link a free email-only account: the browser simulator at [extentos.com/s](https://extentos.com/s) (session minting) and `generateConnectionModule`'s scaffold step (it mints your account-bound project key); the device-code flow fires the first time `createSimulatorSession` runs. See [pricing](/docs/resources/pricing). - Real Ray-Ban Meta hardware. The browser simulator at [extentos.com/s](https://extentos.com/s) plus the on-device `LocalSimTransport` (Extentos's in-memory local sim) cover the bulk of the dev loop. ## The recommended path: agent-driven ```text 1. Install the MCP server in your agent (one command) 2. Tell your agent what to build (one prompt) 3. Agent provisions browser simulator session (one tool call) 4. Test your app's flow in the browser (immediate) ``` **First, [choose your path →](/docs/getting-started/choose-your-path)** — voice apps need no vendor setup at all; camera and display do. Then **[continue with the agent quickstart →](/docs/getting-started/with-agent)**. The agent-driven path covers everything the by-hand iOS / Android quickstarts would cover — your agent calls `generateConnectionModule` which writes the SwiftPM dependency or Gradle artifact, the Info.plist or AndroidManifest entries, the bootstrap module, and the integration manifest. You don't have to choose between iOS-by-hand and Android-by-hand; the agent handles whichever platform (or both) you point it at. ### Why agent-driven is the recommended path Five reasons, in order of impact: 1. **Fewer steps.** Install the MCP server once, then describe what you want in natural language. The agent translates to 5–7 tool calls and writes the right files. 2. **Cross-platform reach in one session.** Your agent (e.g., Claude Code in a terminal) can edit your iOS Swift files and your Android Kotlin files simultaneously. By-hand integration on both platforms means doing the wiring twice; agent-driven means once. 3. **Permission derivation handled automatically.** `getPermissions` returns the exact set of Android manifest entries and iOS Info.plist keys for your capability list. You don't have to memorize that subscribing to `transcription_incremental` needs `NSSpeechRecognitionUsageDescription`. See [concepts/permissions](/docs/concepts/permissions). 4. **Validation before testing.** `validateIntegration` checks structural correctness (dependencies declared, bootstrap wired, permissions cover the declared capabilities) before runtime, so you find integration errors at compose time instead of staring at silent no-ops. 5. **Live debugging through `getEventLog`.** When something goes wrong, your agent queries the structured event log directly and tells you which chip the failure landed in — `errors` for everything that failed (regardless of modality), or `voice` / `camera` / `display` / `ai` / `lifecycle` / `custom` to narrow to a single subsystem. The by-hand path means reading raw `Logcat` / `os_log` output yourself. ## Alternative paths: by-hand integration Per-platform quickstarts: | Path | Status | Notes | |---|---|---| | [Android quickstart](/docs/getting-started/android) | ✅ Published | The by-hand Gradle path — install, manifest, init, first capability call | | [iOS quickstart](/docs/getting-started/ios) | ✅ Published | The by-hand SwiftPM path — installs from `github.com/extentos/swift-glasses` (SwiftPM `from: "2.0.0"`), version-locked to the Android release line | In both cases, the agent-driven path produces the same outcome — same files generated, same permissions derived, same simulator session — without you needing to do the wiring by hand. If you specifically want to integrate without an AI agent, the [MCP server install](/docs/mcp-server/install) and [vendors/meta § Required setup](/docs/vendors/meta#required-setup-for-development) cover the per-platform requirements. ## What success looks like at the end of the quickstart When the first session is running, you'll have: - The Extentos MCP server installed and visible in your agent's tool list (run `/mcp` in Claude Code to verify) - The Extentos library wired into your iOS or Android app via `generateConnectionModule` — `Info.plist` keys, AndroidManifest permissions, dependency declarations all in place - A handler class (yours, e.g., `CoachHandler`) that subscribes to capability primitives — typically `glasses.audio.transcriptions()` for a wake phrase, `glasses.audio.recordDiscrete()` for the user's question, then `glasses.audio.speak()` for the response. (For a full voice assistant where the model owns wake / turn-taking / intent, the Phase-4 `glasses.assistant.start { … }` runtime is the canonical path — see [voice assistant](/docs/guides/voice-assistant).) - A browser simulator session open at `extentos.com/s/` showing the wearer-perspective view, with your webcam feeding the camera and your microphone driving transcripts - `validateIntegration` returning ✓ before runtime - Either auto-bind already attached (zero rebuild) or the URL-bake snippet pasted into your project (one rebuild) End-to-end timing: 5–10 minutes if you're following along with an agent, faster on subsequent runs as `npx` cache hits and your agent learns the project shape. ## After the quickstart Once you have a working session, the next stops: | Goal | Read | |---|---| | Understand how Extentos differs from Meta's Mock Device Kit | [Transport vs app simulation](/docs/concepts/transport-vs-app) — the core differentiator framing | | Understand the system architecture | [Architecture](/docs/concepts/architecture) — agent / MCP / library / backend / simulator stack | | Learn what the SDK exposes | [Capabilities](/docs/concepts/capabilities) — audio / camera / hardware-event primitives and how handlers subscribe to them | | See the Meta Ray-Ban capability matrix | [Vendors: Meta Ray-Ban](/docs/vendors/meta) — what's supported, what's preview, distribution state | | Browse the MCP tool catalog | [MCP server overview](/docs/mcp-server) | | Test on real Ray-Ban Meta hardware | [Vendors: Meta Ray-Ban § Required setup](/docs/vendors/meta#required-setup-for-development) — Meta Wearables Developer Center registration, Bluetooth pairing | ## Common gotchas A few things that trip up first-time installs: - **`/mcp` in Claude Code doesn't list `extentos`.** Quit Claude Code completely (Cmd+Q on macOS, full exit on Windows) and reopen. The MCP host loads servers at startup; closing the window isn't enough. - **Node.js too old.** Run `node -v`. Anything below 20 fails with `Unsupported engine`. Update from [nodejs.org](https://nodejs.org). - **First tool call is slow.** Normal — `npx` is fetching `@extentos/mcp-server` on first run. Subsequent calls hit the cache and are instant. - **`createSimulatorSession` returns `auth_required` immediately.** Expected on the first run — the browser simulator needs a free email-only account. The response includes a verification URL — sign up (Google or email + password, no payment), and the agent retries automatically. See [auth](/docs/mcp-server/auth). - **Browser auto-open didn't happen.** Some hosts (headless / SSH / sandboxed) can't auto-open. The simulator URL is always in the agent's response — copy it to your browser manually. See [supported agents § Browser auto-open behavior](/docs/mcp-server/agents#2-browser-auto-open-behavior). For deeper troubleshooting, see [install § Troubleshooting](/docs/mcp-server/install#troubleshooting). ## Frequently asked questions ### How long does the quickstart take? 5–10 minutes for the first run. Subsequent runs are faster as `npx` caches and your agent learns the project layout. The bottleneck is usually whatever you ask the agent to build — adding "voice trigger that captures a photo and speaks the result" is faster than "voice trigger with branching AI logic and a custom video stream pipeline." ### Do I need Meta credentials to build a voice app? No. Smart glasses expose a standard Bluetooth hands-free microphone and speaker to the phone, and Extentos reaches them through the operating system's own audio routing — no vendor SDK in the path. A voice assistant, dictation app, translator or coach needs no Meta Developer account, no App ID or Client Token, and no connection page. Only camera and display require vendor setup. See [choose your path](/docs/getting-started/choose-your-path). ### Do I need an Extentos account to start? Three surfaces need the free account — `createSimulatorSession` (browser-simulator minting), `generateConnectionModule`'s scaffold step (the account-bound project-key mint), and the account-scoped project tools (assistant config, credentials, connection-page writes, analytics). Discovery, validation, guidance, search, on-device simulation, and real-hardware testing are free forever with no account. The first gated call triggers a free email-only account link (Google or email + password — no payment, no card); after linking, sessions are unlimited. See [pricing](/docs/resources/pricing). ### Do I need real Ray-Ban Meta glasses to start? No. The browser simulator at [extentos.com/s](https://extentos.com/s) and the on-device `LocalSimTransport` cover the bulk of the dev loop. Real hardware verifies final-mile fit (camera fidelity, BT latency, real-world audio coexistence) but isn't required for daily iteration. See [transport vs app simulation](/docs/concepts/transport-vs-app). ### Can I start with iOS or Android only and add the other later? Yes. The SDK surface is **designed** to be identical on both platforms — same primitives (`audio.transcriptions`, `camera.capturePhoto`, `audio.speak`, …), same Result-type return shapes, same connection-state model — so handler logic ports closely between Kotlin and Swift. In practice today, **Android is the longer-proven path** on real hardware (published to Maven Central); iOS ships the same surface as a Swift package from `github.com/extentos/swift-glasses` — see the [iOS SDK](/docs/sdk/ios). `generateConnectionModule` takes a `platform` parameter; call it once per platform. ### What if I'm not using one of the listed AI coding agents? Any MCP-compatible agent works — Extentos uses the standard MCP protocol over stdio. The verified hosts (Claude Code, Cursor, Windsurf, Cline) just have the most polished install experience. Generic MCP host setup is in [install § Generic MCP host](/docs/mcp-server/install#generic-mcp-host). ### Can I integrate Extentos without an AI agent at all? Technically yes — the published Android package (Maven Central) can be linked directly via Gradle and the published iOS package via SwiftPM, and the MCP server's tool responses are JSON that you could call from any HTTP client. But the path is rougher, and the productivity wins of agent-driven generation are large. The recommended approach is to use any MCP-capable agent for the initial integration and graduate to direct manipulation if you need it later. ## Related - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — the full step-by-step - **[MCP server install](/docs/mcp-server/install)** — per-host install commands and config - **[Supported agents](/docs/mcp-server/agents)** — Claude Code / Cursor / Windsurf / Cline capability differences - **[Concepts: transport vs app simulation](/docs/concepts/transport-vs-app)** — the core differentiator framing - **[Vendors: Meta Ray-Ban](/docs/vendors/meta)** — the production target, all variants supported - **[Pricing](/docs/resources/pricing)** — free for development; the managed AI gateway is metered --- source: https://extentos.com/docs/getting-started/android --- --- title: Android quickstart description: Add the Extentos Android SDK by hand — Gradle install, manifest permissions, and your first capability call. Voice apps need no vendor token. type: tutorial platform: android related: - /docs/sdk/android - /docs/getting-started/with-agent --- The by-hand path for adding Extentos to an existing Android app. Prefer letting an agent do the wiring? Use the [agent-driven quickstart](/docs/getting-started/with-agent) — same outcome, less typing. Each step below links to the SDK reference for full detail. ## 1. Add the dependencies `com.extentos:glasses` + `com.extentos:glasses-ui` are on Maven Central and carry **no vendor dependency** — nothing else to declare, no token. ```kotlin title="app/build.gradle.kts" android { buildFeatures { // AGP 8 defaults this OFF, and the init snippet below reads // BuildConfig.DEBUG. Without it, step 3 does not compile. buildConfig = true } } dependencies { implementation("com.extentos:glasses:2.8.0") implementation("com.extentos:glasses-ui:2.8.0") // optional — the prebuilt connection page // The SDK's suspend surface needs a dispatcher from your side. implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.9.0") } ``` That's everything a **voice app** needs to build. If your app uses the **camera or the display**, add the Meta vendor module too — it brings the Meta DAT artifacts, which live on Meta's GitHub Packages and need a `read:packages` token: ```kotlin title="app/build.gradle.kts" implementation("com.extentos:glasses-meta:2.8.0") ``` The extra repo block + token setup + toolchain floors are in [Install the Android SDK](/docs/sdk/android/install). Not sure which you need? [Choose your path](/docs/getting-started/choose-your-path). Floors: `minSdk 31`, `compileSdk 35`, Kotlin `2.1.20`, AGP `8.6+`, Gradle `8.7+`, JVM 17. You also need `namespace` set in your `android { }` block — AGP 8 fails the build without it. One more thing before you ship a voice app that uses the assistant: bake your **project key**, or the assistant cannot reach the managed gateway — in any environment, Play Store releases included. It's one line, it never causes a build error, and its absence shows up only at runtime as `AssistantError.NoApiKey`: ```kotlin title="app/build.gradle.kts" android { defaultConfig { buildConfigField("String", "EXTENTOS_PROJECT_KEY", "\"pk_your_project_key\"") } } ``` `generateConnectionModule` mints the key and emits this line; wiring by hand, take it from your dashboard project. Detail: [install § the project key](/docs/sdk/android/install). ## 2. Declare permissions The library auto-merges the Bluetooth + service entries; you add the **capability** permissions your app uses — `CAMERA`, `RECORD_AUDIO`, `INTERNET`, and (for background mic) `FOREGROUND_SERVICE` + `FOREGROUND_SERVICE_MICROPHONE`. Which capability needs which: [Manifest setup](/docs/sdk/android/manifest) and [Permissions](/docs/concepts/permissions). ## 3. Initialize Extentos Create one `ExtentosGlasses` for your app's lifetime: ```kotlin val glasses = ExtentosGlasses.create( ExtentosConfig( applicationContext = applicationContext, environment = if (BuildConfig.DEBUG) ExtentosEnvironment.DEVELOPMENT else ExtentosEnvironment.PRODUCTION, ) ) ``` `TransportChoice.Auto` (the default) picks the browser simulator during development, real glasses when a vendor module claims one, and otherwise the system-audio baseline — mic and speaker over whatever the OS routed, so a voice app works on a phone with nothing paired. Config table + lifecycle: [Initialization](/docs/sdk/android/initialization). ## 4. Make your first capability call Capability calls return `ExtentosResult` — pattern-match, don't `try/catch`: ```kotlin when (val r = glasses.camera.capturePhoto()) { is ExtentosResult.Ok -> showPhoto(r.value) is ExtentosResult.Err -> handle(r.error) // CaptureError } ``` Per-primitive snippets come from the [`getCapabilityGuide`](/docs/mcp-server/tools/discovery) MCP tool; full flows from [`getCodeExample`](/docs/guides). For voice AI, the Phase-4 [assistant runtime](/docs/guides/voice-assistant) is the canonical path (stable since `1.4.0`). ## 5. Verify it in the simulator Run your app against the [browser simulator](/docs/concepts/transport-vs-app) — the same library code, only the transport swapped. Mint a session with the `createSimulatorSession` MCP tool (or from [extentos.com/s](https://extentos.com/s)); the running app auto-attaches via the local bridge. Watch capability events flow with [`getEventLog`](/docs/mcp-server/tools/simulation). ## Next - [Android SDK reference](/docs/sdk/android) — install, manifest, init, lifecycle, threading, runtime internals - [Guides](/docs/guides) — task-shaped recipes (voice, photo, video, display, …) - [From simulator to real glasses](/docs/guides/simulator-to-real-glasses) — the hardware transition --- source: https://extentos.com/docs/getting-started/choose-your-path --- --- title: Choose your path description: Voice apps run on any Bluetooth smart glasses with no vendor setup. Camera and display need vendor credentials. Pick the smaller path first — you can add the other later without rewriting your app. type: concept platform: all related: - /docs/getting-started/with-agent - /docs/concepts/capabilities - /docs/vendors/meta --- Extentos has two integration paths, and most apps need only the first. The difference is not how much of Extentos you get — the voice runtime is identical on both. The difference is whether you need vendor credentials at all. ## Which path are you on? | You are building | Path | Vendor setup | |---|---|---| | A voice assistant, dictation, translation, coaching, notes — anything the user talks to and hears back | **Voice** | None | | Anything that takes a photo, reads a video frame, or draws on the glasses display | **Device** | Required | If you can't tell yet, start on the voice path. Adding camera later is additive — your handlers, tools and conversation logic don't change. ## The voice path Smart glasses with a microphone and speaker present themselves to the phone as a standard Bluetooth hands-free audio device. Extentos reaches them through the operating system's own audio routing, so there is no vendor SDK in the picture. What you need: - The Extentos SDK (`com.extentos:glasses`). `com.extentos:glasses-ui` is **optional** on this path — it only provides the packaged connection page, which a voice app doesn't render. - Microphone permission (`RECORD_AUDIO` / `NSMicrophoneUsageDescription`), requested at runtime before you start the assistant - `INTERNET`, unless you run entirely on [local models](/docs/concepts/local-models) — the managed gateway is a network call - A foreground service if the assistant must keep listening while your app is backgrounded: `FOREGROUND_SERVICE` + `FOREGROUND_SERVICE_MICROPHONE` on Android, `UIBackgroundModes: [audio]` on iOS - On iOS only, `usedCapabilities: [.microphone, .speaker]` in your config — it picks the transport there, and leaving it empty resolves to real glasses - **Your project key**, if you use the assistant: `buildConfigField("String", "EXTENTOS_PROJECT_KEY", …)` on Android, `EXTENTOSProjectKey` in the Info.plist on iOS. Neither causes a build error when missing — the assistant just fails at runtime with `AssistantError.NoApiKey` - On Android, `buildFeatures { buildConfig = true }` — AGP 8 defaults it off and the init snippet reads `BuildConfig.DEBUG` Full sequencing for both platforms is in the [voice-assistant guide](/docs/guides/voice-assistant). What you do **not** need: a Meta Wearables Developer Center account, a Meta App ID or Client Token, a GitHub Packages token, a connection page, or a device pairing flow. None of it applies. **You can build and test the whole voice path with no glasses in the room.** With nothing attached, the SDK captures from the phone's microphone and plays through its **loudspeaker**, with acoustic echo cancellation on so the assistant does not hear itself. Connect a headset or the glasses and both directions follow them automatically — no code change, no route juggling. That makes the phone a complete test rig for everything except the glasses' own microphone and speaker characteristics. **One open question on iOS.** The Android artifact split means a voice app genuinely doesn't pull the vendor SDK. **iOS has no equivalent split yet** — Meta DAT is a dependency of `GlassesCore` on every iOS app, so a voice app links it even though it never initializes it. Meta states that its SDK's use of `ExternalAccessory` currently triggers App Store rejection under the MFi and privacy-manifest rules ([vendor page](/docs/vendors/meta)). **We have not verified whether merely linking DAT, without declaring the accessory protocols or initializing it, is treated the same way.** If you plan an App Store release, treat this as unresolved and budget a review cycle. Android and TestFlight are unaffected. ```kotlin val glasses = ExtentosGlasses.create( ExtentosConfig(applicationContext = applicationContext) ) // `start` and `wake` are suspend functions, so they run inside a coroutine. scope.launch { val session = glasses.assistant.start(provider = AssistantProvider.Managed()) { instructions = "You are a running coach. Speak briefly." tool("log_split", "Record a lap split for the current run.") { ToolResult.Ok("split recorded") } } // Any trigger that calls session.wake() works — here, a wake phrase. glasses.voice.onPhrase("hey coach") { session.wake() } } ``` You don't call `connect()` yourself — `assistant.start(...)` brings the transport up. (`glasses.connection.connect()` exists for apps that want to drive connection explicitly, and is what the packaged connection page calls.) That is the integration — five short items, none of them a vendor account. You get the complete agent runtime: turn-taking, barge-in, tool calling, conversation memory, local and cloud models, and automatic routing between them. You also get the primitives on their own, without an assistant session — including the high-quality **on-device voice**, which reads any text your app has in a natural voice rather than the platform's robotic one: ```kotlin glasses.audio.speak(summary, SpeakConfig(voice = "kokoro-emma")) ``` That works on this path with nothing paired at all. See [on-device models](/docs/concepts/local-models#on-device-voices). ### What it runs on **Anything the phone can route voice through.** The SDK pins whatever exposes a Bluetooth hands-free (HFP) route and does no device-identity checking at all, so all of these are the same code path: | Audio route | Works | Notes | |---|---|---| | Smart glasses with a mic + speaker | ✅ | Including Ray-Ban Meta — their audio was never carried by the vendor SDK either | | Ordinary Bluetooth earbuds or headsets | ✅ | AirPods, any HFP headset, a car kit | | Nothing connected | ✅ | Falls back to the phone's own mic and speaker | That last row matters more than it looks: your app never dead-ends because the glasses are in their case. The phone always has a microphone and a speaker, so the agent stays usable and picks the glasses back up when they reconnect. **You cannot currently tell which microphone is live.** On this transport `glasses.connection.state` reports `Active` whenever audio is available — the phone always has a mic and a speaker — so it does **not** distinguish glasses from earbuds from the phone itself. A dedicated audio-route readout is on the roadmap; until then, don't build a "connected to your glasses" indicator on `connection.state`, because it will claim glasses that aren't there. ## The device path Camera and display go through the vendor's own SDK, which means vendor credentials and a device connection flow. This is where the Meta setup lives. Add the vendor module alongside the core: ```kotlin implementation("com.extentos:glasses:2.8.0") implementation("com.extentos:glasses-meta:2.8.0") // camera + display ``` The vendor transport registers itself, so `TransportChoice.Auto` resolves to real glasses with no extra code. Meta needs `glasses-meta` because its artifacts sit behind Meta's own credentialed repo; [Brilliant Labs](/docs/vendors/brilliant) needs nothing at all — it ships inside the SDK on both platforms, though it remains preview and unverified on hardware. Then follow [Vendors: Meta Ray-Ban](/docs/vendors/meta#required-setup-for-development) for the credential setup, or [Android XR](/docs/vendors/android-xr) for projected glasses. ## Moving from voice to device later **Your conversation logic does not change** — handlers, tools and prompts are written against capabilities (`glasses.audio.*`, `glasses.assistant.*`), never against a vendor. What does change is the bootstrap around them, because the vendor transport needs things a voice app never needed: 1. Add the vendor dependency. 2. Complete the vendor's credential setup. 3. Request `BLUETOOTH_CONNECT` and `CAMERA` **before** `ExtentosGlasses.create(...)`. `TransportChoice.Auto` decides the transport inside `create(...)` and keeps it; the vendor arm can only claim if it can see that glasses are bonded, and that read needs `BLUETOOTH_CONNECT`. 4. Pass an `activityProvider` to `ExtentosConfig` — vendor registration needs a live Activity, and it defaults to `null`. 5. Guard the new calls on the capability: ```kotlin if (glasses.capabilities.camera) { val photo = glasses.camera.capturePhoto() } ``` Steps 3 and 4 are the ones that surprise people; the [voice-assistant guide](/docs/guides/voice-assistant) has the worked bootstrap for a voice **and** camera app. That guard is the normal shape for every capability on every path — one build ships to devices with different hardware, and absent capabilities are safe, never thrown. See [capabilities](/docs/concepts/capabilities). ### What a device app does when there are no glasses Once you're on the device path, your app still has to run for a user whose glasses are in a drawer. It degrades rather than breaking: - The voice agent keeps working over whatever audio the phone has routed — earbuds, or the phone itself. - `glasses.capabilities.camera` reports `false`, so guarded code simply skips the camera branch. - An unguarded `capturePhoto()` returns `CaptureError.PlatformError` with `code = "system_audio_no_camera"`. It does not crash, and it does not hang. (Not `NotConnected` — on this transport the connection genuinely *is* up; there is simply no camera behind it.) **Plugging glasses in later does not move a running app onto them.** `Auto` resolves once, inside `create(...)`. `glasses.capabilities` is re-read live, but it is read *through the chosen transport* — so on the audio baseline `capabilities.camera` stays `false` until the app is restarted with the glasses already bonded. Design for the restart; don't poll expecting it to flip. So a camera app and a voice app are the same app on a phone with no glasses connected — you lose exactly the capabilities the hardware was providing, and nothing else. ### Glasses paired, but in their case This is the common one, and it resolves differently from "never paired" — a Bluetooth *bond* is a stored pairing record that survives the glasses being powered off. So the vendor arm still claims, and `transportChosen` reads `REAL_META` even though there is nothing switched on to talk to. **Voice still works.** The glasses' microphone and speaker were never reached through the vendor SDK on either platform — they are ordinary Bluetooth hands-free audio, routed by the OS. The Meta transport uses the *same* audio path as the vendorless baseline, so when the glasses are off the OS routes to whatever else is available: earbuds, or the phone itself. Your assistant keeps running and the user keeps talking to it. **Camera doesn't, and says so.** There's no device session, so `capturePhoto()` returns `CaptureError.NotConnected` — the connection genuinely isn't `Active`. The two cases stay distinguishable because `capabilities` reports what the *transport's vendor* provides, not what is switched on right now: on `REAL_META` with the glasses in their case, **`capabilities.camera` is still `true`** while `connection.state` is not `Active`. Never-paired is the opposite — `capabilities.camera` is `false` and the state is `Active`. So a camera app guards on both: ```kotlin val canCapture = glasses.capabilities.camera && glasses.connection.state.value is GlassesState.Active ``` When the user takes the glasses out of the case, the device session comes back on its own and capture starts working again — no restart, because the transport was already `REAL_META`. That's the difference from the never-paired case above, which does need a restart. ## Already shipping? Upgrading from an older SDK If your app was built before `com.extentos:glasses-meta` existed, the Meta transport used to live inside `com.extentos:glasses`. It doesn't any more, so a camera or display app needs one added line: ```kotlin title="app/build.gradle.kts" implementation("com.extentos:glasses:2.8.0") implementation("com.extentos:glasses-ui:2.8.0") implementation("com.extentos:glasses-meta:2.8.0") // ← add this if you use camera or display ``` **No Kotlin change is needed.** The module registers itself at app start, so `TransportChoice.Auto` resolves to real glasses exactly as before. **Camera/display apps only — keep your `settings.gradle.kts` as it is.** The Meta DAT repository block and its `read:packages` token are still required — those artifacts now arrive through `glasses-meta` rather than through `glasses`. Removing the block will break your build. **If you skip the new dependency, nothing fails loudly.** Your build succeeds and your voice features keep working, because the SDK falls back to the audio baseline — but `capabilities.camera` becomes `false` and camera calls start returning errors. The SDK logs a warning at startup naming the missing module (`adb logcat -s Extentos:W`), and `glasses.transportChosen` will read `SYSTEM_AUDIO` where you expect `REAL_META`. Check both after upgrading. The honest boundary: the *code* delta is small, but the vendor's own credential setup (developer account, app registration, access token) is real work that Extentos cannot remove. Doing it later rather than up front is the saving. --- source: https://extentos.com/docs/getting-started/ios --- --- title: iOS quickstart description: The by-hand iOS quickstart — add the Extentos Swift package, set the Info.plist keys, initialize, and make your first capability call. type: tutorial platform: ios related: - /docs/sdk/ios - /docs/sdk/ios/install - /docs/getting-started/with-agent --- The Extentos iOS SDK installs from [`github.com/extentos/swift-glasses`](https://github.com/extentos/swift-glasses) via Swift Package Manager — `.package(url: "https://github.com/extentos/swift-glasses", from: "2.8.0")` (current version on the [install page](/docs/sdk/ios/install)). This page is the by-hand path; the [agent-driven flow](/docs/getting-started/with-agent) automates every step below. ## 1. Add the package In Xcode: *File → Add Package Dependencies…* → `https://github.com/extentos/swift-glasses` → dependency rule **Up to Next Major Version**, starting at `2.0.0`. Link **`GlassesCore`**; add `GlassesUI` only if you render the packaged connection page (a voice app doesn't). Other manifest forms: [Install](/docs/sdk/ios/install). Requirements: the Swift 6 toolchain, Meta DAT (`meta-wearables-dat-ios`) 0.8.0+ (resolved transitively), and deployment target **iOS 17.0+**. The package declares that floor for every product, because SwiftPM applies a package's platform floor package-wide and [`GlassesLocal`](/docs/concepts/local-models) pulls MLX, which requires 17. ## 2. Add the Info.plist keys **Which keys you need depends on your path** — see [choose your path](/docs/getting-started/choose-your-path). **Voice apps** need the privacy strings your capabilities imply — `NSMicrophoneUsageDescription` and `NSSpeechRecognitionUsageDescription` — plus `UIBackgroundModes: ["audio"]` if the assistant must keep listening while your app is backgrounded, plus the two Extentos keys `EXTENTOS_APP_ID` and `EXTENTOSProjectKey`. **No Meta keys and no Meta Developer account.** The project key is not a Meta credential. It is how your app authenticates to the managed gateway **in every environment, App Store builds included**, and it is what meters usage to your account. **`generateConnectionModule` mints it and writes it for you, and there is no other source** — an app that skips that call has no key at all, however complete the rest of its integration looks. If you are wiring by hand, still run the tool once and take the value from the Info.plist entries it emits (you can discard the rest of what it generates), or copy the key from your dashboard project. Without it an assistant app fails at runtime with `AssistantError.NoApiKey` after a perfectly clean build. **Camera or display apps** additionally need the Meta DAT set: the `MWDAT` dict (including an explicit `DAMEnabled`), `CFBundleURLTypes` (a custom scheme, not a universal link), `LSApplicationQueriesSchemes: ["fb-viewapp"]`, `UISupportedExternalAccessoryProtocols: ["com.meta.ar.wearable"]`, and the Bluetooth/external-accessory `UIBackgroundModes`. Full set with values: [Info.plist setup](/docs/sdk/ios/info-plist). **Camera apps also need two entitlements, and they are not Info.plist keys.** Camera frames do not travel over Bluetooth: they ride a higher-bandwidth link the Meta SDK obtains by asking iOS to join the glasses' own Wi-Fi network. That request is gated by **Access Wi-Fi Information** (`com.apple.developer.networking.wifi-info`) and **Hotspot Configuration** (`com.apple.developer.networking.HotspotConfiguration`), which must be on your app target *and* enabled on your App ID. An entitlement can only be granted to an app, never to a framework, so no SDK can supply these for you. Camera streaming also needs `NSLocalNetworkUsageDescription` and a non-empty `NSBonjourServices`, which gate the socket layer once the network join has happened. Both are necessary and neither is sufficient on its own. Get them wrong and there is no error to read: the session starts, the glasses report healthy, capture returns no frame, and nothing is logged. `generateConnectionModule` emits the entitlements file for you whenever your footprint includes camera. Confirm what actually shipped with `codesign -d --entitlements - YourApp.app`. Expect a one-time-per-launch iOS prompt asking to join the glasses' network — the wearer must allow it, and the first capture afterwards takes roughly 10-15 seconds while the link comes up. Every capture after that is sub-second. iOS discards the join when your app exits, so the prompt returns on the next cold start; arm the camera during your connect flow if you would rather the wearer not pay that wait on a shutter press. ## 3. Initialize Create the SDK entry point once, at your `App` entry point, and forward the Meta auth callback: ```swift import GlassesCore @main struct MyApp: App { // usedCapabilities is load-bearing on iOS — it picks the transport. // A voice app declares [.microphone, .speaker]; add .camera / .display // if you use them. Leaving it empty resolves to real glasses. let glasses: any ExtentosGlasses = Extentos.create(config: ExtentosConfig( usedCapabilities: [.microphone, .speaker] )) // `debug` means "prefer the browser simulator over real I/O", not "is a // debug build". `debug: true` opens a pending simulator socket: a coding // agent's MCP bridge binds it automatically, and with no bridge the app // shows a 5-character pairing code you type into the simulator. Leave it // at its false default when you want the phone's own mic and speaker. var body: some Scene { WindowGroup { ContentView(glasses: glasses) .onOpenURL { url in Task { _ = await glasses.handleUrl(url) } } } } } ``` `ExtentosConfig`'s full field list (transport choice, environment, consent flags, …) is in the [iOS API reference](/docs/reference/ios-api); details: [Initialization](/docs/sdk/ios/initialization). ## 4. Make your first capability call Every call is `async` and returns an `ExtentosResult` — pattern-match it (`if case .success(let photo) = await glasses.camera.capturePhoto()`); don't wrap in `do/catch`. See [Threading](/docs/sdk/ios/threading). ## 5. Drop in the connection page — camera and display apps only Skip this step if you're building a voice app: it renders no connection page, so it doesn't need `GlassesUI` at all. See [choose your path](/docs/getting-started/choose-your-path). ```swift import GlassesUI ExtentosConnectionPage(glasses: glasses) // status, capabilities, toggles — library-owned UI ``` ## 6. Verify it in the simulator Run against the [browser simulator](/docs/concepts/transport-vs-app) — the same library code as production with only the transport swapped, so a green run here is meaningful. With `debug: true`, the SDK's `Auto` transport resolution picks the simulator transport during development. There are two ways to get a session, and they end at the same place. **With a coding agent.** Ask it to mint one; it calls `createSimulatorSession` and your running app attaches through the local bridge with nothing further to do. **Without one.** Create a simulator from [your projects](https://extentos.com/projects), then pair your app to it: 1. Sign in and hit **New simulator**, then pick iOS. The simulator opens and waits. 2. Run your app with `debug: true`. With no agent to discover, it shows a **5-character pairing code** on the packaged connection page. 3. Type that code into the simulator's **Enter the pairing code** field. The app attaches immediately — no rebuild, no session URL to paste. If you don't render the packaged connection page — a voice app usually doesn't — the same code arrives on `glasses.connection.simulatorHint` as `SimulatorHint.AwaitingPair`. --- source: https://extentos.com/docs/getting-started/with-agent --- --- title: Quickstart with an AI agent description: Install the Extentos MCP server and let your AI agent scaffold Meta Ray-Ban smart-glasses capabilities into a native iOS or Android app. Free to start. type: tutorial platform: mcp related: - /docs/mcp-server - /docs/mcp-server/install - /docs/mcp-server/tools - /docs/concepts/transport-vs-app - /docs/concepts/architecture - /docs/vendors/meta - /docs/resources/pricing --- Extentos is the smart-glasses infrastructure layer for native mobile apps. An AI coding agent installs the Extentos MCP server, the server exposes a tight set of deterministic tools (capability discovery, code-example reference, project scaffold, validation, browser simulator), and the agent uses them to wire a typed Kotlin/Swift SDK into an existing iOS or Android app — your handler classes then subscribe to primitives like `glasses.audio.transcriptions()`, `glasses.camera.capturePhoto()`, and `glasses.audio.speak()`. Targets Meta Ray-Ban (supported in production today), with other vendors on the roadmap. A browser-based simulator covers the bulk of the dev loop without physical glasses. This page is the agent-driven path. If you're integrating by hand, see the [iOS](/docs/getting-started/ios) or [Android](/docs/getting-started/android) quickstart instead. ## Install There's no single host-agnostic install command — every MCP host (Claude Code, Cursor, Windsurf, Cline) stores its server config differently. Three paths, in order of recommended: ### 1. Agent prompt (recommended, host-agnostic) Open your MCP-capable agent and paste: ```text Install @extentos/mcp-server in my MCP config. ``` The agent runs the right command for its host (`claude mcp add ...` for Claude Code, or edits `mcp.json` for Cursor / Windsurf / Cline). Restart the agent when it reports done. The Extentos tools appear in the agent's tool list. ### 2. Claude Code one-liner If you're on Claude Code and prefer a shell command: ```bash claude mcp add extentos -- npx -y @extentos/mcp-server@latest ``` Restart Claude Code. Run `/mcp` — you should see `extentos` listed with the full tool surface. ### 3. Manual JSON For any MCP host, drop this into the host's MCP config file (`~/.cursor/mcp.json` for Cursor, `~/.codeium/windsurf/mcp_config.json` for Windsurf, or equivalent): ```json { "mcpServers": { "extentos": { "command": "npx", "args": ["-y", "@extentos/mcp-server@latest"] } } } ``` Restart the host. The Extentos tools should appear in the agent's tool list. **Prerequisites:** Node.js 20 or newer on `PATH`. That's the only requirement — `npx` fetches and caches the package on first run. **Pinning a version (optional):** Replace `@latest` with an exact version (the current one is on [npm](https://www.npmjs.com/package/@extentos/mcp-server)) if you need a reproducible install across sessions. Extentos is pre-1.0; APIs may shift between minor versions until the hardware test loop closes. ## First prompt to your agent After install, ask the agent to do the actual work. Two prompt patterns that work well: ```text Add Extentos smart-glasses voice triggers to my Android app. The user should be able to say "summarize this" while wearing Meta Ray-Ban glasses, and the app should run my existing summarize() function on whatever the camera sees. ``` ```text Use the Extentos MCP server to add photo capture from Meta Ray-Ban to my iOS app. When the user double-taps the glasses, capture a frame, send it to my analyzeImage() function, and speak the result back. ``` Be specific about: which platform, which capability (voice / photo / video / sensor / audio), and which existing function in your app should consume the data. The agent composes against the SDK primitives — wake-phrase matching, business logic, error handling — all in plain Kotlin / Swift in a handler class you own. ## What the agent does The MCP server has no planning tool — agents are better planners than regex. The agent composes capability primitives itself, then calls a deterministic sequence. You can watch it happen in the tool log: | Step | Tool | What it does | |------|------|--------------| | 0 | `npx @extentos/mcp-server setup` | **Android camera/display path only.** Preflight that checks your environment can resolve `com.extentos:glasses-meta` from GitHub Packages before you build. Skip it for a voice-only app — that path needs no token. Run it from your project root, after the host app exists | | 1 | `getPlatformInfo` | Reads the capability catalog for the target glasses (e.g. `meta_rayban`) — names + categories of every SDK feature | | 2 | `getCapabilityGuide(feature)` | Per-feature minimal Kotlin + Swift snippet + gotchas for each primitive the handler will use | | 3 | `getCodeExample(pattern)` | Full canonical composition — e.g. `assistant_agent_loop` (the Phase-4 voice assistant, where the model owns wake / turn-taking / intent) returns working Kotlin + Swift the agent peels from | | 4 | `generateConnectionModule` | One-shot scaffold: bootstrap module, Gradle/SPM wiring, dependencies, permissions, manifest. Asks where `ExtentosConnectionPage` should live in your app | | 5 | _(agent writes handler classes)_ | The agent writes a Handler class against the SDK primitives — `glasses.audio.transcriptions()`, `glasses.camera.capturePhoto()`, `glasses.audio.speak()`, etc. Updates `extentos.manifest.json`'s `capabilities` array with the feature names used | | 6 | `getPermissions` | Derives the Android permissions / iOS `Info.plist` keys / Meta DAT scopes from the capabilities you actually use. Re-run whenever that set changes | | 7 | `validateIntegration` | Pre-test correctness gate — checks dependencies declared, bootstrap wired, permissions cover the declared capabilities | | 8 | `createSimulatorSession` | Provisions a browser-hosted glasses surrogate at [extentos.com/s](https://extentos.com/s) | **The numbering is a recommended order, not a required one — and minting early is fine.** `createSimulatorSession` sits last because that is when a session is most useful, but agents routinely mint first to see the simulator before committing to a project layout, and nothing stops them. The only difference: a sim minted before `generateConnectionModule` has no `extentos.manifest.json` to key on, so it is held against your account and platform rather than bound to a project, and every unscaffolded directory you mint from resumes that same sim. Scaffolding binds it. Either order works and neither loses your session. **One-time credential setup (Android) — only if you use camera or display.** Those live in `com.extentos:glasses-meta`, which pulls Meta's DAT SDK from GitHub Packages and needs a free `read:packages` GitHub token. **A voice-only app needs none of this** — `com.extentos:glasses` has no vendor dependency and resolves from Maven Central. See [choose your path](/docs/getting-started/choose-your-path). For iteration: edit handler code → rebuild + reinstall → the app auto-attaches to the same simulator session. No remint required. For debugging: `getEventLog` (filter by `errors` first to surface every failure across modalities, then narrow to a capability chip — `voice` / `camera` / `display` / `ai` / `lifecycle` / `custom`), `getSimulatorStatus`. For ship-readiness: `getProductionChecklist`. For credentials (BYOK providers + Meta DAT registration): `getCredentialGuide`. The full tool reference is at [/docs/mcp-server/tools](/docs/mcp-server/tools). ## Verifying it worked When the agent calls `createSimulatorSession`, the MCP server auto-opens a browser tab to your session at [extentos.com/s](https://extentos.com/s) (specifically `extentos.com/s/`). You should see: - A glasses surrogate with camera + mic permission prompts (browser-side; no physical hardware) - A live event log showing your app initializing, the connection opening, and your trigger firing when you exercise it - A WebSocket-synced view of your app's responses, exactly as they'd appear on real Meta Ray-Ban glasses **Auto-bind** is shipped on both Android and iOS: once the Extentos library is in your app, the MCP and library coordinate via a `127.0.0.1:31337/whoami` localhost bridge. Subsequent `createSimulatorSession` calls auto-attach the running app to the new session — no rebuild, no URL paste. ## Pricing Extentos's tools, simulator, code generation, validation, and SDK are free — there's no per-seat or subscription fee to build and ship. The one surface that meters usage is the **managed AI gateway** behind the Phase-4 voice assistant. Specifically: | Tier | Cost | What you get | |------|------|--------------| | **No-account** | Free forever | Every MCP tool. `LocalSimTransport` (Extentos's in-process on-device simulator — no network, no Meta SDK setup). Code generation, validation, scaffolding. Real-hardware testing on Meta Ray-Ban via your own Meta DAT credentials. | | **Free account** | Free, email only | Mint browser-simulator sessions at [extentos.com/s](https://extentos.com/s). Google one-click or email + password — no payment, no card. Once linked, sessions are unlimited. | | **Managed AI gateway** | Metered ($2 free credit) | Voice-assistant AI on Extentos's keys — usage metered on a prepaid-credit model; every account starts with $2 of free credit. | **Why account-required for the simulator?** The browser simulator runs on Extentos's backend (WebSocket hub, voice proxy, event log persistence) — that's the one piece that costs us per-session compute. Asking for a free email-only account is the lightest possible gate. Most other Extentos surfaces (discovery, guidance, validation, on-device sim, real-hardware DAT testing) don't touch the backend and stay free without an account. The exceptions are `createSimulatorSession`, `generateConnectionModule`'s scaffold step, and the account-scoped project tools — all free, but they need the sign-in. See [pricing](/docs/resources/pricing). **Account linking** happens through the device-code flow: the first `createSimulatorSession` call returns `auth_required` with a verification URL, your agent surfaces it, you sign in (Google one-click or email + password + ToS acceptance), and the original tool call retries automatically. No manual token paste, no payment. **Real hardware testing** (running your app on actual Meta Ray-Ban glasses) requires your own Meta Wearables Developer Center registration — a one-time, free, ~15–30 minute setup using your personal Meta credentials. No Extentos account required for this path. Full pricing details: [/docs/resources/pricing](/docs/resources/pricing). ## Multi-platform projects (Android + iOS = one project) Your Android and iOS versions share **one project** on the dashboard when you scaffold both with the same Extentos app id. That id is `config.appId` — conventionally your reverse-DNS package name, but **decoupled from your store bundle id**, so the two platforms share a project even when their store identifiers differ. Give both platforms the same Extentos app id (the `appPackage` you scaffold with — it's `config.appId`, decoupled from your store bundle id) and minting a simulator from each automatically joins the same project: one card, two platform chips, no manual setup. This works even when your Android and iOS store bundle ids differ. The full story — how project identity is derived, what's preserved across a Merge, what the Settings page surfaces — lives at **[/docs/concepts/projects](/docs/concepts/projects)**. ## What's actually supported (be honest with your agent) The Meta DAT public toolkit is the substrate Extentos sits on. It has real boundaries — your agent should know them so it generates apps that actually run: - **Strong public capabilities:** photo capture, video capture, camera streams, sensor data - **Voice triggers:** custom phrases are recognized on the phone (on-device Vosk on Android hardware, Apple's `SFSpeechRecognizer` on iOS; the platform recognizer on the dev path), fed from the glasses mic over Bluetooth (HFP/SCO). Meta AI's "Hey Meta" wake word is **not** accessible to third-party apps - **Audio I/O:** mic capture and TTS playback ride mobile-platform Bluetooth profiles. Extentos wires the audio session; your platform's native STT/TTS does the work (zero Extentos runtime cost) - **Not available:** raw Neural Band gestures and custom gesture recognition. Captouch is system-defined (taps pause/resume sessions); on Meta Ray-Ban Display, wearer interactions reach your `glasses.display.*` UI as abstracted tap events - **Distribution:** public publishing is partner-gated while Meta's toolkit is in developer preview — users enroll as testers via Meta release channels today. Android export works normally; Meta doesn't support iOS App Store publishing yet See [/docs/concepts/capabilities](/docs/concepts/capabilities) for the full matrix, and [/docs/concepts/transport-vs-app](/docs/concepts/transport-vs-app) for the framing of what Meta provides vs what Extentos adds. ## Troubleshooting first-run issues **`/mcp` doesn't list `extentos` in Claude Code.** Restart Claude Code completely (Cmd+Q / quit from system tray, not just close window). The MCP host loads servers at startup. **`npx` errors with "command not found".** Install Node.js 20 or newer from [nodejs.org](https://nodejs.org). Verify with `node -v`. **The agent's first tool call is slow or times out.** `npx` is fetching the package on first run — wait 10–30 seconds and retry. Subsequent calls hit the cache and are instant. **`createSimulatorSession` returns `auth_required` immediately.** That's the expected flow on first run — the browser simulator requires a free account. The response includes a verification URL; open it, sign in (Google one-click or email), and the agent retries automatically. **Auto-bind not connecting on Android.** Check that nothing else is bound to localhost port `31337`. The MCP server logs a warning if it can't bind the bridge (port 31337 in use); the app then falls back to URL-paste mode. **iOS app not picking up the session.** Check that nothing else is bound to localhost port `31337`; the library logs a warning if the bridge fails to start. Falls back to URL-bake (`extentos.session.plist`) if the probe times out. More: [/docs/troubleshooting](/docs/troubleshooting). ## Next steps - **[MCP server reference](/docs/mcp-server)** — full tool catalog, configuration, auth model - **[Concepts: architecture](/docs/concepts/architecture)** — how the MCP, the library, the simulator, and your app fit together - **[Concepts: transport vs app simulation](/docs/concepts/transport-vs-app)** — what Meta provides, what Extentos adds, and why both matter - **[Guides](/docs/guides)** — task-shaped recipes (voice triggers, photo capture, sensor streams, background sessions) - **[From simulator to real glasses](/docs/guides/simulator-to-real-glasses)** — when you're ready for hardware: Meta identity, credentials, and the transport switch - **[Vendors: Meta Ray-Ban](/docs/vendors/meta)** — the primary production target and its capability matrix ## Status Pre-1.0 ([current version on npm](https://www.npmjs.com/package/@extentos/mcp-server)). Android is the longer-proven path on real hardware; both SDKs are published in lockstep (see [iOS install](/docs/sdk/ios/install)). APIs may shift between minor versions until the hardware test loop closes — pin to an exact version if you need cross-session reproducibility. Published on [npm](https://www.npmjs.com/package/@extentos/mcp-server) (MCP server, MIT), [Maven Central](https://central.sonatype.com/artifact/com.extentos/glasses) (Android library), and [github.com/extentos/swift-glasses](https://github.com/extentos/swift-glasses) (iOS Swift package). --- source: https://extentos.com/docs/guides --- --- title: Guides description: Task-shaped Extentos recipes for Meta Ray-Ban — voice triggers, voice assistant, photo/video/audio capture, display, hardware events, and disconnect handling. type: guide platform: all related: - /docs/getting-started/with-agent - /docs/concepts/capabilities - /docs/troubleshooting - /docs/reference/errors --- Task-shaped, vendor-agnostic recipes for both SDKs. Kotlin leads the snippets, with iOS notes inline where the Swift surface differs. - [Voice triggers](/docs/guides/voice-triggers) - [Voice assistant](/docs/guides/voice-assistant) - [Photo capture](/docs/guides/photo-capture) - [Video capture](/docs/guides/video-capture) - [Audio streaming](/docs/guides/audio-streaming) - [Render on the display](/docs/guides/display) - [Sensor streams](/docs/guides/sensor-streams) - [Background sessions](/docs/guides/background-sessions) - [Handle disconnects](/docs/guides/handle-disconnects) - [From simulator to real glasses](/docs/guides/simulator-to-real-glasses) --- source: https://extentos.com/docs/guides/audio-streaming --- --- title: Audio streaming description: Stream raw mic audio off Meta Ray-Ban with audioChunks, get continuous STT with transcriptions(), play TTS with speak(), and push your own realtime model's audio back out with sendAudio. Covers A2DP/HFP coexistence. type: guide platform: all vendor: meta related: - /docs/guides/voice-triggers - /docs/guides/photo-capture - /docs/concepts/capabilities - /docs/reference/errors --- The audio sub-client is symmetric. **In:** `glasses.audio.audioChunks` (raw PCM), `glasses.audio.transcriptions()` (continuous speech-to-text), and `glasses.audio.recordDiscrete(...)` (a silence-bounded clip). **Out:** `glasses.audio.speak(text)` (text-to-speech) with `cancelSpeak()`, and `glasses.audio.sendAudio(...)` (raw PCM) with `stopAudio()` — the pair you want when your own realtime model produces the audio. This page covers each and the one Bluetooth-audio gotcha that bites everyone. > Kotlin leads the snippets and is the source of truth. iOS ships every surface on this page (`for await chunk in ...`, `case .success`). ## Speak (text-to-speech) ```kotlin glasses.audio.speak("Got it.") // suspends until the TTS engine finishes by default ``` `speak()` synthesizes with the phone's platform TTS engine (Android `TextToSpeech` / iOS `AVSpeechSynthesizer`) and routes the audio to the glasses speaker. It returns `ExtentosResult` and **blocks the calling coroutine until the engine finishes** by default — to listen and speak at the same time, launch it in its own scope. ### A better voice, with no assistant involved The platform engine is fine for short confirmations and robotic on longer text. If you've added the on-device voice module, name a voice in `SpeakConfig` and the same call speaks in a natural one: ```kotlin glasses.audio.speak(articleSummary, SpeakConfig(voice = "kokoro-emma")) ``` ```swift _ = await glasses.audio.speak(articleSummary, config: SpeakConfig(voice: "kokoro-emma")) ``` No assistant session is required, and the text can come from anywhere — a cloud model's reply, your own content, a notification. It works on **every transport including the vendorless audio baseline**, so a voice-only app with no glasses paired gets the same voice through the phone or earbuds. Setup and the eleven speakers: [on-device models](/docs/concepts/local-models#on-device-voices). Routing is per call and never errors: an unknown voice id, or a model that hasn't finished downloading, serves the platform voice for that utterance instead. `cancelSpeak()` interrupts a local synthesis exactly as it does platform TTS. `rate` and `pitch` apply to the platform engine only — the local voice speaks at its model's own prosody. To interrupt mid-utterance (barge-in), call `cancelSpeak()` from another coroutine: ```kotlin val job = launch { glasses.audio.speak(longText) } // ...on detected user speech: glasses.audio.cancelSpeak() // fire-and-forget; the speaker falls silent within ~tens of ms job.cancel() ``` The full race-and-cancel barge-in composition is `getCodeExample(pattern: "barge_in_speak")`. > **`speak()` and `earcon()` are never gated by a toggle** — audio *output* always plays. No toggle, not even `privacy_mode`, silences output (the toggles gate audio/camera *input* + STT). A `DisabledByUser` result is therefore impossible for `speak`/`earcon`; build your own mute switch if you need one. > **Speak quality is bound by the phone's TTS engine, not the glasses.** For a premium voice you have three options: name an on-device voice in `SpeakConfig` (above), use the Phase-4 [assistant runtime](/docs/concepts/assistant) (cloud TTS over the [managed gateway](/docs/concepts/ai-gateway)), or synthesize wherever you like and push the bytes yourself with [`sendAudio`](#bring-your-own-realtime-model). ### A2DP vs HFP coexistence `speak()` routes audio over the **Bluetooth HFP profile** (voice-band — the link negotiates wideband speech, mSBC at 16 kHz mono), the same profile the glasses use as a microphone — **not** A2DP. The consequence: while `speak()` is active, high-quality A2DP music playback drops to voice-band HFP mono for the duration of the speech. The same is true for `transcriptions()`, `recordDiscrete()`, `audioChunks`, and `captureVideo(VideoConfig(includeAudio = true))` — any time the glasses act as a mic, you are on HFP and A2DP is suspended. This is a hardware property of Meta DAT, not an Extentos choice. Surface a "listening…" / "speaking…" indicator so the user understands why their music quality dipped, and avoid overlapping a music-playback feature with continuous capture. Coexistence is a real-hardware behavior — the browser simulator has no A2DP music path, so verify this on actual glasses rather than expecting the sim to reproduce it. ## Continuous transcription `transcriptions()` is a `Flow` of partial + final speech-to-text — the primitive behind live captions and string-matched wake phrases: ```kotlin import com.extentos.glasses.core.Transcript import com.extentos.glasses.core.TranscriptionConfig glasses.audio.transcriptions(TranscriptionConfig(language = "en-US")).collect { t -> // Transcript is a sealed class: Partial | Final. Partial fires 5–20x/sec. if (t !is Transcript.Final) return@collect handleFinalText(t.text) } ``` Three things that trip people up: - **Always pattern-match `is Transcript.Final` before reading `t.text`.** `Partial` fires 5–20 times a second and partial text can match a wake phrase mid-utterance, causing double-fires. - **The library does not normalize.** `t.text` is exactly what the recognizer returned, in the casing the user spoke. Match against `t.text.lowercase()` (and strip punctuation/apostrophes) yourself. - **Subscribe once.** Each subscription is a separate recognizer instance — subscribing per-loop or per-trigger means duplicate transcripts and battery drain. Transcription is gated by `listening_mode` (default treated as on; `"off"` is the user's hard kill-switch), `audio_capture_enabled`, and `privacy_mode`. (`transcription_enabled` is declared in the catalog but not consumed by the library yet — don't chase it.) If transcripts stop arriving, check those first — see [Wake phrase not matching](/docs/troubleshooting/voice-trigger-not-firing). For matching a wake phrase you usually want [voice triggers](/docs/guides/voice-triggers) (`glasses.voice.onPhrase`) rather than a raw collector; drop to `transcriptions()` directly only for live-caption feeds or custom (regex / stateful) matching. The live-captions-into-Compose shape is `getCodeExample(pattern: "live_transcription_ui")`. ### Language, and where it is and isn't honored `TranscriptionConfig(language = …)` is the only language control in the SDK, and **it does not take effect on every transport.** **It is honored everywhere on iOS, and only on the simulator transports on Android.** | Transport | Android | iOS | |---|---|---| | Real glasses | ❌ **ignored** | ✅ | | System-audio baseline (no vendor) | ❌ **ignored** | ✅ | | Browser simulator | ✅ | ✅ | | Local in-process sim | ✅ | ✅ | iOS runs Apple's `SFSpeechRecognizer` and builds it from `Locale(identifier: language)`, so every locale Apple supports works on every transport. Android runs a **Vosk** recognizer on real glasses and on the audio baseline — pinned to an en-US acoustic model that never reads the field — and the platform `SpeechRecognizer` (which does read it) only on the simulator transports. **The Vosk model is not in your APK — it downloads on first use, and until it lands you get no transcripts.** It is roughly 40 MB, fetched once per install and kept out of every consumer APK deliberately. While the download is in flight or has failed, starting transcription emits a `TransportError.HardwareUnavailable("stt_engine_unavailable")` and the stream stays silent; the next start retries, so a second attempt after the download completes just works. This is the first thing to check when transcription is silent on a fresh install with no error in your own code. It also explains why an otherwise on-device capability needs `INTERNET`. Budget for it in your first-run UX: watch for that transport error and tell the user speech recognition is still downloading, rather than leaving a dead "Listening…" on screen. **On Android hardware, a non-English `language` is accepted and then discarded.** Passing `language = "de-DE"` on real glasses or the audio baseline gives you English recognition of German speech — not an error. And because the Android *simulator* honors the setting, a multilingual flow can pass in sim and fail on hardware. iOS is unaffected. If your product depends on a non-English language, test it on an Android device before building on it. ## Raw microphone chunks When you need raw PCM — your own on-device STT model, audio analysis, custom keyword spotting — subscribe to `audioChunks`: ```kotlin import com.extentos.glasses.core.AudioChunkConfig glasses.audio.audioChunks( AudioChunkConfig( chunkMillis = 20, // 20ms is the standard opus/CELT frame — don't change unless your model needs it sampleRate = 16000, // REQUESTED rate; see the caveat below ), ).collect { chunk -> // feed chunk.data (ByteArray) to your STT / VAD / classifier. Also chunk.sampleRate, chunk.timestampMs. } ``` Use `transcriptions()` instead if you just want speech-to-text — the SDK's recognizer is cheaper and pre-routed (on Android hardware, on-device Vosk; on iOS hardware, Apple's `SFSpeechRecognizer`; the platform recognizer on the dev/sim path). `audioChunks` is for cases that genuinely need PCM. > **`sampleRate` is a request, not a guarantee.** Meta DAT delivers mic audio over the Bluetooth HFP link, which negotiates **wideband speech (mSBC, 16 kHz mono)** — on real hardware the effective rate is the HFP rate, not necessarily your config value. For any rate-sensitive DSP (FFT bins, VAD windows), read the per-chunk `chunk.sampleRate` rather than assuming the config value. The stream buffers with drop-oldest overflow — consumers that can't keep up lose frames rather than back-pressuring the mic. `audioChunks` is gated by `audio_capture_enabled` and `privacy_mode` (but **not** `listening_mode` — it's a raw-audio path, not STT). ### It is safe to subscribe during a live assistant session `audioChunks` does **not** take the microphone. The SDK holds a single tap on the input node and fans the buffers out to every consumer, because the platform allows only one tap per input — the assistant's own capture, `transcriptions()`, `recordDiscrete()` and `captureVideo(includeAudio = true)` are all already sharing it. So an app can meter or analyse the wearer's own voice *while* `glasses.assistant` is mid-conversation, which is what a live input-level visualiser needs: ```kotlin glasses.audio.audioChunks(AudioChunkConfig(chunkMillis = 40)) .collect { chunk -> meter.value = rms(chunk.data) } // runs happily alongside the assistant ``` This is worth stating because the alternative a developer reaches for — standing up their own `AVAudioEngine` / `AudioRecord` tap — fails on exactly the one-tap rule the shared input exists to solve, and the failure looks like the assistant going deaf rather than like a tap conflict. For the **other** direction — how loudly the assistant itself is currently speaking — see [assistant audio levels](/docs/concepts/assistant#reacting-to-how-loudly-the-assistant-is-speaking). ## Bring your own realtime model If a realtime model already runs your conversation — Gemini Live, an OpenAI Realtime session on your own key, something self-hosted — it hands you finished **audio**, not text. `speak(text)` is the wrong shape for that: it would throw away audio the model already generated so the SDK could re-synthesize the words. `sendAudio` is the outbound half of `audioChunks`, so the loop closes with the SDK owning only the device I/O at either end: ```kotlin import com.extentos.glasses.core.OutgoingAudioFidelity import kotlinx.coroutines.launch // Ask what the speaker path can carry, so your model generates a format // the link will not just resample away. val rate = if (glasses.audio.outputFidelity == OutgoingAudioFidelity.HiFi) 24000 else 8000 // Mic in -> your model. launch { glasses.audio.audioChunks().collect { chunk -> myRealtimeModel.sendUserAudio(chunk.data, chunk.sampleRate) } } // Your model -> the glasses speaker. The mic stays open the whole time, // so the wearer can talk over the reply. launch { myRealtimeModel.audioOut.collect { pcm -> if (myRealtimeModel.wearerInterrupted()) { glasses.audio.stopAudio() // barge-in: drop what is still queued } else { glasses.audio.sendAudio(pcm16 = pcm, sampleRate = rate) } } } ``` **`stopAudio()` matters more than it looks.** A realtime model emits *faster* than realtime, so by the time the wearer starts talking, seconds of reply can already be buffered ahead of the speaker. Without it, playback continues until that backlog drains and the interruption feels ignored. It is the same drain the managed assistant uses for its own barge-in. Note that `cancelSpeak()` does **not** cancel `sendAudio` audio — the two output paths are independent. **You do not have to mute the mic while playing.** Capture runs through the platform's voice-communication unit (Voice-Processing I/O on iOS, `VOICE_COMMUNICATION` on Android), which subtracts the device's own output from the input. That is what makes an always-open mic workable: the echo disappears, a real interruption still arrives. **Format contract.** `pcm16` is signed 16-bit little-endian **mono**. An empty chunk is a no-op rather than an error, and a dangling odd byte is dropped — the same convention the named-sound registry uses. The one rejection is a non-positive `sampleRate`, which returns `AudioError.PlatformError(code = "invalid_sample_rate")`. Those rules live in the shared Rust core, so both platforms behave identically. `outputFidelity` reports `HiFi` on real Meta glasses (the HFP link negotiates wideband mSBC) and in the browser simulator (WebAudio, full band). Read it rather than assuming — it is a per-transport value, and it exists so your model can be asked for the right output format up front. Playback is **not** gated by `audio_capture_enabled` or `privacy_mode`; those govern capture. `speak()` behaves the same way. **This is not a new audio path.** `sendAudio` hands chunks to the same transport call the assistant's own voice already uses on real glasses — the managed assistant, the on-device tier and `speak()` with a local voice all ride it, and `RealMetaTransport` serialises every chunk through one mutex, so ordering holds no matter who is feeding it. Faster-than-realtime producers are the normal case there too, since that is exactly how a realtime model emits. What is new is the public entry point, not the pipe. If you would rather not own the conversation at all, use the [assistant runtime](/docs/concepts/assistant) instead — it brings the model, turn taking, barge-in and error handling with it. The two tiers are deliberate: either Extentos runs the loop, or you do. There is nothing in between to half-configure. ## Bounded capture (record a turn) For a one-shot "ask a question" capture, `recordDiscrete(...)` records until the user goes silent and returns the clip (`rawAudioUri`); on real hardware the `transcript` field is empty today — transcription there is your app's job (the browser sim populates it). It is covered with the voice flows: see `getCapabilityGuide(feature: "record_audio")` for the silence-VAD knobs and `getCodeExample(pattern: "voice_notes")` for a wake → record → persist composition. ## Verify it in the simulator The browser simulator drives transcription from your real laptop microphone and plays `speak()` output through your speakers, so you can rehearse the whole audio loop before hardware. `recordDiscrete` emits a `record_audio_returned` event (`getEventLog(filter: "voice")`). The audio chips in the event log are `voice` (transcripts, recordings, speech) — note there is no `audio` filter. The simulator is designed to behave like the glasses, but the audio substrate differs (24 kHz PCM in sim vs the glasses' wideband 16 kHz HFP) and that fidelity is under active validation — confirm audio behavior on real glasses. ## Related - [Voice triggers](/docs/guides/voice-triggers) — `glasses.voice.onPhrase`, the structured wrapper over `transcriptions()`. - [Capture a photo](/docs/guides/photo-capture) — the vision-LLM flow that pairs capture with `speak()`. - [Capabilities](/docs/concepts/capabilities) — the full audio capability catalog. - [Error reference](/docs/reference/errors#audio-errors) — every `AudioError` variant. - `getCapabilityGuide(feature: "speak")` / `getCapabilityGuide(feature: "transcription_incremental")` / `getCapabilityGuide(feature: "audio_chunks")` — call shapes + full gotchas. --- source: https://extentos.com/docs/guides/background-sessions --- --- title: Background sessions description: Keep a Meta Ray-Ban session alive when your app backgrounds — the Android foreground service for mic, iOS background modes, and which capabilities survive. type: guide platform: all vendor: meta related: - /docs/sdk/android/lifecycle - /docs/guides/handle-disconnects - /docs/concepts/sessions --- By default, capability subscriptions (`glasses.audio.*`, the assistant) run on the SDK's coroutine scope, which is bound to your **process** — not to any Activity. While your app is **foreground**, that's enough. To keep the glasses listening (a voice assistant or live transcription) while your app is **backgrounded**, you need an OS-level keep-alive: a foreground service on Android, background modes on iOS. This guide covers what's required, what survives, and what doesn't. > Kotlin leads the snippets. Android uses the bundled foreground service; iOS uses background modes — a genuinely different OS mechanism, covered in its own section below. ## What "background session" means The glasses connection itself doesn't end the moment your app loses focus — but the OS will reclaim a backgrounded process (and with it your `ExtentosGlasses` instance and its transport) under memory pressure or after enough time, unless you hold a foreground service or background-mode budget. So "keeping the session alive" is really "keeping your process alive and the audio path running while backgrounded." This is a separate concern from *handling* a disconnect when it does happen — see [handle disconnects](/docs/guides/handle-disconnects) for the recovery path. ## Android: the bundled foreground service For background-surviving **microphone** capture, the library ships a foreground service so you don't write one: ```kotlin import com.extentos.glasses.core.service.GlassesForegroundService // AFTER RECORD_AUDIO is granted at runtime — typically in the grant callback: GlassesForegroundService.start(this) // 'this' is any Context // When background capture is no longer needed: GlassesForegroundService.stop(this) ``` The service is declared for you in the library manifest as: ```xml ``` You don't declare it — you **start** it. The rules that matter (see [Android lifecycle](/docs/sdk/android/lifecycle#background-capture-and-the-foreground-service) and [manifest setup](/docs/sdk/android/manifest#the-bundled-foreground-service)): - **Permission first.** `RECORD_AUDIO` must already be granted at runtime. On Android 14+, starting a `microphone`-type foreground service without the grant throws `SecurityException` at service-create time and crashes the app. - **Declare the foreground-service permissions.** Your app's manifest needs `FOREGROUND_SERVICE` and `FOREGROUND_SERVICE_MICROPHONE` (Android 14+) — `getPermissions` derives these from your continuous-capture capabilities. The library declares the `` element; the permissions are app-owned. - **Start from an Activity, not `Application.onCreate`.** The grant can't exist that early. Start it from your permission-grant callback, or after a `checkSelfPermission(...) == PERMISSION_GRANTED` check. - **Idempotent.** Calling `start` again while running just updates the notification. - **Stop it when done** to drop the notification and release the foreground budget. > **Camera in the background is not covered by the bundled service.** `GlassesForegroundService` is `microphone`-type only. Surviving a `videoFrames()` stream in the background requires a **`camera`-type** foreground service, which Android requires the **consumer app to declare itself** (the library can't declare a camera FGS on your behalf without forcing it on every app). Subclass `GlassesForegroundService` and declare your own `` with the matching runtime rules. Most apps only need background mic. ## iOS: background modes iOS doesn't use foreground services — a backgrounded app keeps running only within a declared **background mode** budget. For an audio-driven glasses session, the relevant `UIBackgroundModes` entries (`bluetooth-central`, `bluetooth-peripheral`, `external-accessory` — plus `audio`, which you must add yourself when you stream the mic; `generateConnectionModule` emits only the three Bluetooth/accessory modes) must be present in Info.plist; `generateConnectionModule(platform: "ios")` includes the background-mode and `UISupportedExternalAccessoryProtocols` keys. iOS background execution is more constrained than an Android foreground service — validate background behavior on a real device before relying on it. ## What survives backgrounding | Capability | Backgrounded behavior | |---|---| | `glasses.audio.transcriptions()` / `recordDiscrete()` / the assistant | Survives **with** the foreground service (Android) / audio background mode (iOS) running. Without it, capture stops when the process is reclaimed. | | `glasses.audio.speak()` | Plays while the audio path is alive; same keep-alive requirement. | | `glasses.camera.*` | Needs a **consumer-declared `camera`-type** FGS on Android (not the bundled mic service). | | The connection state | Persists while the process lives; on process death you re-`create(...)` and reconnect — see [process death](/docs/sdk/android/lifecycle#process-death). The dev-loop pairing identity is persisted to `filesDir`, but the live connection is re-established from scratch. | Camera-only or strictly foreground apps don't need any of this — the foreground-service declaration is harmless if unused. ## Honest limits - **The OS can still kill you.** A foreground service and background modes reduce — they don't eliminate — OS reclamation. Under hard memory pressure or aggressive battery-saver/OEM policies, a backgrounded session can still be terminated. Design for it: observe `glasses.connection.state`, handle the `Disconnected` cause, and reconnect cleanly. See [handle disconnects](/docs/guides/handle-disconnects). - **Battery cost is real.** A live mic + Bluetooth audio path in the background drains both the phone and the glasses. Start the service only while you actually need background capture, and stop it as soon as you don't. - **Sim ≠ hardware here.** The **browser simulator does not exercise OS background lifecycle at all** — there's no foreground service, no battery-saver kill, no process reclamation in a browser tab. A session that "stays alive when backgrounded" in the simulator tells you nothing about real-device background survival. Verify backgrounding on real hardware. ## Related - [Android SDK lifecycle](/docs/sdk/android/lifecycle) — the full foreground-service wiring and process-death behavior. - [Handle disconnects](/docs/guides/handle-disconnects) — recovering when a backgrounded session is killed. - [Sessions](/docs/concepts/sessions) — the connection-state machine and what persists. --- source: https://extentos.com/docs/guides/capture-button --- --- title: The capture button description: The touch surface on the Meta Ray-Ban's right temple can pause or stop your camera stream at any moment — a firmware-enforced privacy gesture your app must respect. What your app observes, how to handle it, and how to test it in the simulator. type: guide platform: all vendor: meta related: - /docs/guides/photo-capture - /docs/guides/video-capture - /docs/concepts/transport-vs-app - /docs/reference/errors --- **Camera needs the Meta vendor module.** `com.extentos:glasses` carries no vendor SDK, so add `implementation("com.extentos:glasses-meta")` alongside it — see [install](/docs/sdk/android/install). Without it your build still succeeds and voice still works, but `capabilities.camera` is `false` and every capture returns an error. The SDK logs a warning at startup when it spots that combination. The right temple of Meta's smart glasses carries a capacitive touch surface — the **capture button**. It isn't a button your app owns: it's the *wearer's* control over the camera, enforced by firmware below anything the SDK (or Meta's DAT itself) can override. Every camera-using app inherits its behavior, so this page is worth ten minutes before you ship one. ## What the gestures do Two gestures, hardware-verified: | Gesture | Effect on your camera stream | |---|---| | **Tap** (single touch) | Pause ⇄ resume. Non-destructive — the connection holds. | | **Tap-and-hold** | Stop. The whole device session ends; the connection drops and the SDK auto-recovers over a few seconds. | The privacy LED on the glasses tracks the same state: lit while the camera is live, dark the moment it's paused or stopped — so bystanders always see the truth, whatever the app wants. ## Why it works this way This is a **privacy mechanism, not an API gap**. Meta's model is that the person wearing the camera — not the app — has the final say over whether it's capturing. That's why: - **There is no app-callable pause or resume.** DAT exposes the state (`StreamState.PAUSED`) but no way to change it. We probed the internal resume path on real hardware: the glasses **refuse** an app-initiated resume of a wearer's pause — the firmware ends the session rather than let software un-pause the camera. - **Fighting it makes things worse.** Tearing down and reopening the stream to force a resume drops the whole connection, and doing so shortly after a pause can wedge the glasses for ~20 seconds. Don't. - The gesture always wins. Your app's job is to *observe and respond gracefully*, and the SDK is built around exactly that. ## What your app observes The SDK turns the gesture into typed, actionable signals — the same ones in the simulator and on hardware, because it's the same shared gate: **While paused (tap):** - `capturePhoto()` / `captureVideo()` return `CaptureError.StreamPaused`, whose message already tells the user what to do: *"The camera is paused. Tap the right temple of your glasses to resume the camera, then try again."* - Starting `videoFrames` throws `CameraStreamPausedException`. A pause **mid-stream** is not an error — frame delivery halts (your last frame stays on screen) and resumes on the next tap. - An **in-flight recording stays alive but captures no footage** — end it and the clip contains everything except the paused window, spliced seamlessly. `duration_ms` reflects footage, not wall clock. - [Assistant](/docs/concepts/assistant) tools using `orToolError` automatically speak the tap-the-temple message to the user. **After a stop (hold):** - The stream is gone and **no wearer gesture brings it back for your app** — a tap with no live stream goes to Meta's own capture and your app never sees it. - Your app re-arms the stream with its **next camera use**: the next frame-grab photo, recording, or `videoFrames` subscription starts a fresh stream (on hardware this rides the SDK's automatic session recovery, ~3–5 s after the hold). - An in-flight recording ends with the footage captured so far. The canonical handling pattern is two lines of intent: surface the message, let the wearer tap, retry. Snippets live in [photo capture](/docs/guides/photo-capture#handling-failures) and [video capture](/docs/guides/video-capture#if-the-camera-is-paused-when-you-start). ## Watching the stream itself The signals above all arrive at a *call* you make. If your app forwards frames somewhere else, a live relay, a recorder, an upload pipeline, there may be no call in flight when the wearer taps: frames simply stop arriving and nothing tells you why. Subscribe to `glasses.camera.streamState()` for that. It reports what the stream is doing, independently of any call: ```kotlin glasses.camera.streamState().collect { phase -> when (phase) { CameraStreamState.STREAMING -> relay.resume() CameraStreamState.PAUSED -> relay.hold("Tap the right temple to resume") CameraStreamState.IDLE -> relay.stop() else -> Unit } } ``` ```swift for await phase in glasses.camera.streamState() { switch phase { case .streaming: relay.resume() case .paused: relay.hold("Tap the right temple to resume") case .idle: relay.stop() default: break } } ``` Five phases: `IDLE`, `ARMING`, `STREAMING`, `PAUSED`, `STOPPING`. You get the current one immediately on subscribing, then only genuine transitions, so every value is a change worth acting on. Do not watch the **connection** state for this. It does not move when the wearer taps, and that is deliberate: `Active` is a supertype that internal camera work stays inside, so tearing your pipeline down on any non-`Active` state would kill it during normal arming. A paused stream is a perfectly healthy connection that has stopped producing frames, and `streamState()` is the only place that distinction is visible. The same subscription covers the stop gesture and the five-minute retirement below: both land on `STOPPING` then `IDLE`, and your next camera use re-arms through `ARMING` back to `STREAMING`. ## Streams end on their own, too The gestures aren't the only firmware authority over your stream — the glasses manage stream lifetime themselves. Two behaviors, hardware-measured: - **A long-lived stream is retired at roughly five minutes.** A continuously-armed stream ends the whole device session ~304 seconds after frames start flowing (measured three times: 303.9 / 304.1 / 304.2 s). Your app sees the same session end as a hold (`SESSION_ENDED_BY_DEVICE`), and the handling is identical: the next camera use re-arms everything — a couple of seconds end-to-end, verified without an app restart. - **Thermals ride the same window.** Streaming warms the glasses on a repeatable curve; the wearer's phone shows Meta's own overheating notification around the two-minute mark. The hotter the glasses when the session ends, the rougher the exit — a pre-heated end can also drop the Bluetooth connection for a couple of minutes before the SDK's recovery brings it back. There is no reliable app-side lever to defer or avoid this — software-initiated stream teardown is the destabilizing path described above, and the SDK deliberately leaves stream lifetime to the platform. Treat a stream that ends as a normal lifecycle event, not an error: the [photo](/docs/guides/photo-capture) and [video](/docs/guides/video-capture) handling patterns already cover it. ## Verify it in the simulator The [browser simulator](/docs/concepts/transport-vs-app) reproduces the whole model, because the paused gate is the same code on both substrates: - The **Capture button panel** (right rail) shows the glasses with a marker on the right temple — tap it to pause/resume, press-and-hold to stop. The **capture LED** in the Glasses View header mirrors the privacy light. - Every transition lands in the event log under the camera chip (`camera_stream_opened` / `_paused` / `_resumed` / `_closed`), and a capture denied while paused logs `capture_denied` under errors with the same actionable message. - Agents drive it headless with the `injectHardwareButton` MCP tool — pause the stream mid-test and assert your app's handling end-to-end. One surfaced substrate delta: on real glasses a hold also drops the connection for a few seconds while the SDK recovers; the simulator closes the stream without the blip. ## Related - [Capture a photo](/docs/guides/photo-capture) — `CaptureError.StreamPaused` handling with the full `when` arm. - [Capture video](/docs/guides/video-capture) — mid-recording pause semantics and the footage-splice behavior. - [Transport vs app simulation](/docs/concepts/transport-vs-app) — why the simulator's error is byte-for-byte the hardware error. - [Error reference](/docs/reference/errors#capture-errors) — every `CaptureError` variant. --- source: https://extentos.com/docs/guides/display --- --- title: Render on the display description: "Render UI on glasses with a screen — photo capture, browsable card lists via glasses.display, and full-surface views. One tree, three vendors: Meta Ray-Ban Display, Android XR Display Glasses, and Brilliant Labs Halo and Frame." type: guide platform: all vendor: meta related: - /docs/concepts/display - /docs/concepts/assistant - /docs/concepts/capabilities - /docs/reference/errors - /docs/vendors/meta - /docs/vendors/brilliant --- **Display needs the Meta vendor module.** `com.extentos:glasses` carries no vendor SDK, so add `implementation("com.extentos:glasses-meta")` alongside it — see [install](/docs/sdk/android/install). Without it your build still succeeds and voice still works, but `display.isAvailable` is `false` and display calls are silent no-ops. The SDK logs a warning at startup when it spots that combination. > **Display support is in beta.** The full display surface — the builder DSL, select/back input, per-device gating — ships in both SDKs and is fully tested in the simulator on both platforms. On-glasses rendering on Meta Ray-Ban Display hardware is in its verification phase; APIs are stable, with refinements expected as hardware verification completes. This guide builds a small **media gallery** on the Ray-Ban Display with no LLM at all: capture a photo, save it to your own store, render a browsable list of cards, and show one full-surface when the user pinches it. Then it points at the assistant-driven variant, where the same display DSL is driven by voice. Read [the display capability](/docs/concepts/display) for the full DSL, the additive-light rule, and the input model. This page is the task walkthrough. > **`glasses.display.*` ships in both SDKs and is fully sim-testable on both.** On-glasses rendering via Meta DAT (0.8.0) ships on both platforms, though neither has been confirmed on real Ray-Ban Display hardware yet. The local-media roots (`image(photo)` / `video(clip)`, `prepareVideo`, `forgetHostedVideo` / `forgetHostedImage`) ship on both. On a display-less device every call is a safe no-op. ## Gate on `isAvailable` Display is per-**device**, not per-vendor: Meta Ray-Ban Display has a screen and Ray-Ban Meta doesn't, and the same split exists inside Android XR. Branch on the runtime signal, never on a model name or a vendor, and fall back to voice when there's no screen: ```kotlin if (!glasses.display.isAvailable) { glasses.audio.speak("Saved — you've got ${store.all().size} photos.") return } glasses.display.show { /* ... */ } ``` `show()` on a display-less device is a silent no-op, so the guard is about **UX**, not safety — give the user a voice path. The gate is live in the simulator: switching the device model flips `isAvailable` mid-session with no reconnect. ## Capture → store → browse (no LLM) A self-contained gallery driven by direct display calls plus a button or voice trigger. `show {}` replaces the entire surface each call, so own your nav state app-side (list vs detail) and re-render whole views on every transition. Each `show` registers its own `onBack` because back is view-contextual. What's happening: - **Cards** are clickable `column` containers with a stable `id` (`"photo-" + p.id`). Without an explicit id, a clickable container falls back to a positional `box-0` / `box-1` that shifts on reorder — coupling input routing to render order. - **`image(photo)`** is root-only and full-surface. The glasses fetch only `http(s)` URLs, so the SDK hosts your **local** capture at `show()` time and renders the hosted URL — you never hand-roll hosting. (A photo hosts in well under a second; the video equivalent `video(clip)` takes a few seconds.) - **`onBack`** is registered per `show`: the detail view's `onBack` re-renders the list; the root's clears. The back gesture is the Neural Band mid-pinch on hardware. `onClick` / `onBack` are plain `() -> Unit`, but `show()` / `clear()` are suspend — launch a coroutine inside the handler. - **`forgetHostedImage`** tears down the hosted copy when the user deletes the photo. Rebuild the `Photo` the same way for `show` and `forget` so both resolve to the same copy. It's idempotent and doesn't touch the local file. **This call is not optional housekeeping** — the hosted copy is your user's photo on Extentos's media host with no expiry, so skipping it means a deleted photo isn't really deleted. See [security § hosted display media](/docs/resources/security#hosted-display-media--the-one-content-path) for what to disclose in your own privacy policy, and how to avoid the upload entirely by hosting media yourself. Add a Back **button** to the detail view for discoverability alongside the gesture: ```kotlin button("Back", style = ButtonStyle.OUTLINE, id = "back") { showList() } ``` ## Verify it in the simulator A fresh sim session defaults to `rayban_meta` (**no display**) — set the device first, or every `show` is a silent no-op: ```ts await setSimDevice({ sessionId, device: "rayban_display" }); // Read EXACTLY what's on the glasses — structured tree, no pixels, no browser needed: const d1 = await getDisplayState({ sessionId }); // d1.tree = the rendered DisplayNode tree; d1.interactiveIds = the pinchable card ids // Pinch a card the way the Neural Band would (needs the browser tab — ensureSimulatorBrowser): await injectInput({ sessionId, action: "select", targetId: d1.interactiveIds[0] }); const d2 = await getDisplayState({ sessionId }); // the detail view // Back via the gesture — mid-pinch on hardware, injected back in the sim: await injectInput({ sessionId, action: "back" }); // onBack fires, re-renders the list ``` `getDisplayState` reads the live hub snapshot and works headless; `injectInput` needs the browser tab attached (it owns the rendered tree + focus). The durable trace is `getEventLog(filter: "display")` — `display_show`, `display_select`, `display_navigate`, `display_back` — where event order proves causality. Test **both** branches: `rayban_display` renders; `setSimDevice({ device: "rayban_meta" })` flips `isAvailable` to false and your voice fallback runs. See [the MCP tools reference](/docs/reference/mcp-tools). > **Dead URLs show nothing.** Additive light can't render a failed media fetch — the panel stays dark while the tree checks look green. The simulator shows a "video failed to load" placeholder and a `display_error` on the **`errors`** chip; on hardware a `display.video_error` runtime event fires so you can fall back to a text view. Check the `errors` chip first when a display flow looks green but shows nothing. ([Errors](/docs/reference/errors#there-is-no-displayerror) — there is no `DisplayError` return type.) ## The same tree on Android XR As of 2026-07-27 the display surface has a second vendor: **Android XR Display Glasses**. The tree you just built renders there too, translated to Google's Glimmer toolkit instead of Meta's view builder. Your handler code does not change, and neither does this guide's code — switch the simulator's device and watch the same tree appear. ```ts await setSimDevice({ sessionId, device: "android_xr_display_glasses" }); ``` Four things are worth designing for, because they differ between the two screens: **The panel is smaller, and trees do not scroll.** Meta Ray-Ban Display is 600x600; Android XR display glasses are 450x394. An over-tall tree is clipped, not scrolled, on both. Design for the smaller surface and you fit both; design for Meta's and your bottom row can fall off on XR. The simulator draws each device's real panel, so this is visible rather than theoretical. **Icon names resolve per vendor.** Meta matches its own 116-glyph set; Android XR matches a set of shared concepts (`check`, `close`, `place`, `search`, `settings`, `person`, `phone`, `star`, `play`, `arrow_back`, and about twenty more). A name neither vendor knows renders **nothing** rather than a placeholder, identically on both. If you want the same icon on both screens, stay inside the shared set. **`isAvailable` is answered by the device, after connect.** On Android XR the SDK asks the connected glasses whether they have a screen and reports `false` until the answer arrives — it will never claim a screen it has not confirmed. Read `glasses.display.isAvailable` at the point you need it rather than caching it during startup. **Input is the same actions with a different gesture.** Google's touchpad API is three callbacks — swipe forward, swipe backward, click — which are focus-next, focus-previous and select, the same three Meta's Neural Band delivers as thumb-slide either way and index-pinch. Your `onClick` handlers are unchanged. The one real difference is back: Meta has a back gesture (mid-pinch) that reaches `onBack`, and Google's touchpad API has no back callback at all, so on that vendor back can only come through the system back path. Give every screen a way out the wearer can select. One limit stated plainly: no Android XR glasses of either class are on sale, so nothing on that vendor is hardware-verified, and a `video` node inside a tree plays through the Android framework there but has not been proven on a device. See the [Android XR vendor page](/docs/vendors/android-xr) for the full status. ## The same tree on Brilliant Labs A third vendor renders the same tree: **Brilliant Halo and Frame**. This one is worth designing against even before you can ship for it, because it breaks assumptions the other two let you keep — and all of them are visible in the simulator today. ```ts await setSimDevice({ sessionId, device: "brilliant_halo" }); ``` **The panel is round.** Halo's canvas is 256×256 and circular. The SDK lays your content out inside the inscribed square automatically, so a full-width row does not lose its ends — you write the same tree. What it cannot do is make your tree shorter, and at 256×256 this is the smallest surface in the catalog: a layout that fits Meta's 600×600 shows about a third of its content. The simulator draws the real circle and marks what got clipped. **There is one font size.** Neither Brilliant device's text primitive takes a size argument, so `HEADING`, `BODY` and `CAPTION` render identically. Colour still differentiates — secondary text carries its own colour — but a hierarchy built on type size flattens here. **The wearer cannot move focus.** Halo has a button and an IMU tap and no swipe; Frame has one tap gesture and nothing else. This is the first vendor where the four display actions do not all have a gesture, so build single-action screens rather than browsable lists. The simulator disables the controls the hardware lacks, so a flow you can drive there is one the device could perform. **Frame has a display and no speaker.** If your display path falls back to `audio.speak`, that fallback is silence on Frame. Gate on the speaker capability, not on the display's absence. Brilliant's transport ships inside the SDK on both platforms (nothing to add on Android; `TransportChoice.brilliant` on iOS), but **no hardware has run it** — see the [Brilliant vendor page](/docs/vendors/brilliant) for the full status. The simulator is the part you can rely on today. ## The assistant-driven variant The same display DSL can be driven by **voice** instead of buttons: an assistant tool calls `glasses.display.show(...)`, and "show my notes" / "open the groceries note" / "go back" drive the same app-side state machine the Neural Band does. One state machine, two input rails, no conflict — gate the tool on `isAvailable` and return the reason in `ToolResult.Err` so the model explains it ("These glasses have no display — want me to read it aloud?"). ```kotlin tool( "show_notes", "Show the browsable list of the user's notes on the glasses display. " + "Also call when the user asks to go back to the list from an open note.", ) { if (!glasses.display.isAvailable) { return@tool ToolResult.Err("These glasses have no display. Offer to read the notes aloud instead.") } showBrowse() ToolResult.Ok("Notes are on the display — slide between them, pinch one to open it.") } ``` For the full two-view browse ⇄ detail pattern with both rails wired, see [the assistant runtime](/docs/concepts/assistant) and `getCodeExample(pattern: "display_browse_detail")` via the MCP server. ## Related - [The display capability](/docs/concepts/display) — the full DSL, additive-light rule, hosting, and input model - [The assistant runtime](/docs/concepts/assistant) — drive the display by voice from assistant tools - [Capabilities](/docs/concepts/capabilities) — the per-device capability model and the full SDK vocabulary - [Error reference](/docs/reference/errors) — the no-`DisplayError` model and the display runtime events - [Vendors: Meta Ray-Ban](/docs/vendors/meta) — the device family and the Ray-Ban Display - [Vendors: Brilliant Labs](/docs/vendors/brilliant) — the round panel, the single font size, and the display-without-a-speaker case --- source: https://extentos.com/docs/guides/handle-disconnects --- --- title: Handle disconnects description: Observe glasses.connection.state, branch on the typed Disconnected.cause to tell a user-pulled-the-glasses-off from a Bluetooth drop, and reconnect cleanly. The connection-state spine for any Meta Ray-Ban app. type: guide platform: all vendor: meta related: - /docs/concepts/sessions - /docs/guides/background-sessions - /docs/reference/errors - /docs/troubleshooting/glasses-wont-connect --- Glasses disconnect: the wearer takes them off, walks out of Bluetooth range, the battery dies, or — in the simulator — the session expires. A robust app observes `glasses.connection.state`, reacts to *why* the connection dropped (not just *that* it did), and offers the right recovery. This page is the connection-state spine; the full state machine is documented in [Sessions](/docs/concepts/sessions). > Kotlin (`StateFlow`) leads the snippets and is the source of truth. iOS exposes the same model via `glasses.connection.state.stream` (an `AsyncStream`); the state and cause types are identical. ## Observe the connection state `glasses.connection.state` is a `StateFlow`. `GlassesState` is a sealed type with six buckets — collect it and branch: **`Active` means the transport is up, not that glasses are present.** On the vendorless audio baseline it is reported whenever audio is routable — and the phone always has a mic and a speaker. A "connected to your glasses" indicator built on this state alone will claim glasses that aren't there. Pair it with `glasses.capabilities` for anything hardware-specific. See [choose your path](/docs/getting-started/choose-your-path). ```kotlin import com.extentos.glasses.core.GlassesState @Composable fun GlassesGate(glasses: ExtentosGlasses) { val state by glasses.connection.state.collectAsState() when (state) { GlassesState.NotRegistered -> PairingPromptView() GlassesState.Registered -> Text("Looking for your glasses…") is GlassesState.DeviceDiscovered -> Text("Found them — connecting…") is GlassesState.Connecting -> Text("Connecting…") is GlassesState.Active -> ConnectedView() is GlassesState.Disconnected -> DisconnectedView(cause = (state as GlassesState.Disconnected).cause) } } ``` The six buckets, in lifecycle order: | State | Payload | Meaning | |---|---|---| | `NotRegistered` | — | The install hasn't registered/authenticated yet — show a pairing prompt. | | `Registered` | — | Registered, scanning for a known device. | | `DeviceDiscovered` | `deviceId: String` | A device was found; connection is about to start. | | `Connecting` | `deviceId: String` | Handshake in progress. | | `Active` | `state: ActiveState` | Connected and usable. Capability calls work here. | | `Disconnected` | `cause: DisconnectCause` | The connection ended — `cause` tells you why. | > You usually don't need to build any of this UI yourself — `ExtentosConnectionPage` renders the canonical pairing / connecting / connected / disconnected flow. Reach for `glasses.connection.state` directly only when you need a *custom* in-app affordance (e.g. disabling a "take photo" button while not `Active`). ## Branch on the disconnect cause `Disconnected.cause` is a `DisconnectCause` — a sealed type that tells you whether the drop is the user's intent, a hardware condition, a transient transport blip, or a simulator-session event. Branch on it to give the right message and the right recovery: ```kotlin import com.extentos.glasses.core.DisconnectCause @Composable fun DisconnectedView(cause: DisconnectCause) = when (cause) { // User intent — they asked to disconnect. No auto-retry; show a "Connect" button. DisconnectCause.UserRequested -> Banner("Disconnected.", action = "Connect" to ::reconnect) // Hardware conditions — retrying won't help until the condition clears. DisconnectCause.ThermalCritical -> Banner("Glasses got too hot and disconnected. Let them cool, then reconnect.") DisconnectCause.HingesClosed -> Banner("Glasses were folded. Open them to reconnect.") // Transport blip — Bluetooth dropped / out of range. Retryable; offer (or auto-) reconnect. DisconnectCause.DeviceDroppedConnection -> Banner("Lost the glasses — they may be out of range.", action = "Retry" to ::reconnect) is DisconnectCause.TransportFailure -> Banner("Connection error.", action = "Retry" to ::reconnect) // cause.cause is the TransportError // Simulator-only causes — only seen in the browser simulator, never on real hardware. is DisconnectCause.SimulatorSessionExpired -> Banner("Simulator session expired (${cause.reason}). Mint a new one.") is DisconnectCause.SimulatorBrowserClosed -> Banner("Simulator browser tab closed.") is DisconnectCause.SimulatorMeterExhausted -> Banner("Simulator session limit reached.") } ``` Grouped by how you should treat them: - **`UserRequested`** — the app or user called `disconnect()`. Don't auto-reconnect; show a manual "Connect" affordance. - **`DeviceDroppedConnection` / `TransportFailure`** — treat as **retryable**. The glasses went out of range or the link failed; `TransportFailure.cause` carries the underlying `TransportError` (e.g. `WebsocketClosed`, `Timeout`) if you want to log specifics. Offer a retry or reconnect automatically with backoff. - **`ThermalCritical` / `HingesClosed`** — hardware conditions. Retrying immediately is pointless; tell the user what to do (let them cool / unfold) and reconnect once they act. *(Declared, not yet produced by the core — today these drops arrive as `DeviceDroppedConnection`; keep the branches for forward-compat.)* - **`SimulatorSessionExpired` / `SimulatorBrowserClosed` / `SimulatorMeterExhausted`** — only occur under `BrowserSimTransport` (`SimulatorMeterExhausted` is reserved — never emitted today). You won't see these on real glasses; handle them so your dev builds degrade gracefully (mint a new session). `SimulatorSessionExpired.reason` is a `SessionExpireReason` (idle timeout, lifetime cap, user-ended, backend restart). See the [error reference](/docs/reference/errors) for the full `DisconnectCause` and `TransportError` payload shapes. ## Reconnect To recover, call `glasses.connection.reconnect()`. It disconnects, then connects again, **re-running registration and re-fetching device metadata** (name, firmware, link state) — so a stuck "looking for glasses" recovers, and changes made elsewhere (e.g. renaming the glasses in the Meta AI app) get picked up. It's serialized: overlapping calls run one at a time. ```kotlin import com.extentos.glasses.core.ConnectError import com.extentos.glasses.core.ExtentosResult suspend fun reconnect(glasses: ExtentosGlasses) { when (val r = glasses.connection.reconnect()) { is ExtentosResult.Ok -> { /* state flips to Active via the StateFlow */ } is ExtentosResult.Err -> when (r.error) { is ConnectError.NoDeviceAvailable -> prompt("No glasses found — make sure they're nearby and on.") is ConnectError.PermissionDenied -> requestBluetoothPermissions() is ConnectError.PairingFailed -> prompt("Pairing failed — try again.") // The SDK drives Meta registration itself via your activityProvider — // there is no registration call to make. Surface it and retry. is ConnectError.RegistrationRequired -> prompt("Approve the connection in the Meta AI app, then try again.") is ConnectError.TransportFailure -> prompt("Couldn't connect — try again.") } } } ``` `connect()` and `reconnect()` return `ExtentosResult` — pattern-match, don't `runCatching`. (`disconnect()` returns `Unit`; it always succeeds.) `reconnect()` backs the connection page's pull-to-refresh; call it from your own UI control too. You don't need to read the return value to update UI — the `state` `StateFlow` is the source of truth and flips to `Active` (or back to `Disconnected`) on its own. > **A retry policy that won't thrash:** auto-reconnect once on `DeviceDroppedConnection` / `TransportFailure` with exponential backoff (e.g. 1 s, 4 s, 16 s, then give up and show a manual button). Don't auto-retry `UserRequested` or the hardware causes. Don't spin a tight retry loop — each `reconnect()` re-runs registration. ## In-flight operations A disconnect mid-capture surfaces as the operation's own typed failure, not just a state change: an in-flight `capturePhoto()` returns `CaptureError.NotConnected`, `recordDiscrete()` returns `AudioError.NotConnected`. Handle those at the call site (see [photo capture](/docs/guides/photo-capture#handling-failures) and [audio streaming](/docs/guides/audio-streaming)); they fire independently of the `state` observer, so a robust app handles both — the per-call error for the immediate operation, the state observer for the UI. ## Background and lifecycle Whether a session survives app-backgrounding is a separate concern — see [Background sessions](/docs/guides/background-sessions) for the Android foreground-service and iOS background-mode setup that keeps the connection alive while your app isn't foregrounded. ## Related - [Sessions](/docs/concepts/sessions) — the full connection-state machine and what persists. - [Background sessions](/docs/guides/background-sessions) — keeping the connection alive when backgrounded. - [Error reference](/docs/reference/errors) — `DisconnectCause`, `ConnectError`, and `TransportError` payloads. - [Glasses won't connect](/docs/troubleshooting/glasses-wont-connect) — first-connection failures. - `getCapabilityGuide(feature: "connection_state")` — the call shape + gotchas. --- source: https://extentos.com/docs/guides/migrating-from-meta-dat --- --- title: Migrating from raw Meta DAT description: Move an app you already built against Meta's Device Access Toolkit onto the Extentos SDK — a call-site swap, not a rewrite, because Extentos calls the same DAT underneath. type: guide platform: all vendor: meta related: - /docs/concepts/vs-meta-dat - /docs/getting-started/with-agent - /docs/vendors/meta - /docs/reference/errors --- **Camera and display need the Meta vendor module.** `com.extentos:glasses` carries no vendor SDK, so add `implementation("com.extentos:glasses-meta")` alongside it — see [install](/docs/sdk/android/install). Without it the build still succeeds and voice still works, but `capabilities.camera` is `false` and captures return errors. The SDK logs a warning at startup when it spots that combination. If you already built your app directly against Meta's [Device Access Toolkit](/docs/vendors/meta) (DAT / the Wearables SDK), adopting Extentos is a **call-site swap, not a rewrite** — Extentos's production transport calls the *same* DAT under the hood, so every DAT concept you use has a one-to-one Extentos primitive. This guide maps your DAT symbols to their Extentos replacements and gives the order to do the swap in. For the conceptual "Extentos vs raw DAT" comparison — what each layer owns and when going direct to DAT is still the right call — see [Extentos vs Meta DAT](/docs/concepts/vs-meta-dat). This page is the practical migration. **This is a full cutover, not a side-by-side path.** Running raw DAT and the Extentos SDK in the same process at once is not supported today: the Meta `DeviceSession` is a single-owner handle and there is no public accessor to the DAT session underneath Extentos yet. Migrate an app fully, or keep it on raw DAT — incremental / side-by-side adoption is a roadmap item, not a current capability. ## Do it with the agent The fastest path is to let your coding agent drive it. The `getMigrationGuide` MCP tool returns this same mapping as structured data plus an ordered plan, and most entries name an Extentos `feature` so the agent can drill into `getCapabilityGuide(feature)` for the exact Kotlin/Swift idiom and `getCodeExample(pattern)` for a full composition to peel from. ``` getMigrationGuide({ from: "meta_dat" }) ``` ## The order to migrate in 1. **Scaffold the bootstrap.** Run `generateConnectionModule`. It replaces `Wearables.initialize` / `Wearables.configure` **and** your hand-rolled registration/pairing UI — pick where `ExtentosConnectionPage` lives when prompted. 2. **Derive permissions.** Run `getPermissions` with your capability list, apply the emitted manifest / `Info.plist` entries, then delete the ones you maintained by hand. 3. **Swap call sites,** area by area, using the table below. Drill into `getCapabilityGuide(feature)` for each. 4. **Convert error handling** from `try`/`catch` / `runCatching` to [`ExtentosResult`](/docs/reference/errors) pattern-matching. 5. **Validate and dogfood.** Run `validateIntegration`, then `createSimulatorSession` — you can prove the whole cutover in the browser simulator before pairing glasses. ## The mapping Each row is keyed by the raw DAT symbol you have today. `Feature` is the name to pass to `getCapabilityGuide` for the call-shape. | Area | Your raw DAT | Extentos | Feature | |---|---|---|---| | SDK init | `Wearables.initialize` / `Wearables.configure()` | `ExtentosGlasses.create(...)` / `Extentos.create(config:)` (via `generateConnectionModule`) | — | | Registration / pairing UI | `Wearables.startRegistration` / `registrationState` / `handleUrl` | `ExtentosConnectionPage` + `glasses.connection.state` | `connection_state` | | Device session lifecycle | `Wearables.createSession` / `DeviceSession` / `DeviceSessionState` | `glasses.connection` (Auto transport owns the session) | `connection_state` | | Device discovery / identity | `Wearables.devices` / `Device` / `DeviceType` | `glasses.device.type` / `glasses.device.vendor` | — | | Permissions & Meta scopes | `checkPermissionStatus` / `RequestPermissionContract` / manual manifest+plist | `getPermissions({ capabilities, platform })` | — | | Camera — photo | `PhotoData` / `photoDataPublisher` | `glasses.camera.capturePhoto()` | `capture_photo` | | Camera — frames | `Stream` / `addStream` / `removeStream` / `videoFramePublisher` | `glasses.camera.videoFrames(...)` | `video_frames` | | Camera — video clip | Stream video capture / `VideoQuality` | `glasses.camera.captureVideo(...)` | `capture_video` | | Audio — raw mic | HFP/SCO + `AudioRecord` (hand-wired) | `glasses.audio.audioChunks(...)` | `audio_chunks` | | Audio — transcription & wake | `SpeechRecognizer` / `SFSpeechRecognizer` over BT (hand-wired) | `glasses.audio.transcriptions()` + `glasses.voice.onPhrase` | `transcription_incremental` | | Audio — TTS | `TextToSpeech` / `AVSpeechSynthesizer` + route handling | `glasses.audio.speak()` / `cancelSpeak()` | `speak` | | Display (Ray-Ban Display) | `Display` / `addDisplay` / `DisplayConfiguration` | `glasses.display.show { }` / `clear()` / `isAvailable` | `display` | | Voice-AI loop (if hand-wired) | your own wake → STT → LLM → TTS | `glasses.assistant.start(provider) { tool(...) }` | `assistant_runtime` | | Error handling | raw exceptions / `CaptureError` / `StreamError` | `ExtentosResult` — pattern-match | — | ## What you stop maintaining The value of the cutover is the fragile, per-app plumbing you get to delete: - **The wake-phrase path** — DAT never recognized phrases; you streamed the mic to a phone recognizer and matched strings yourself. That becomes a subscription. - **SCO/HFP mic routing + A2DP playback coexistence** — the SDK owns the audio-route policy. - **Permission derivation** — one capability list drives the manifest, `Info.plist`, and Meta scopes. - **Publisher/stream lifecycle** — `addStream`/`removeStream`/`DeviceSession` ownership is internal; you consume typed primitives. - **The test harness** — you gain the browser simulator and the agent-driven end-to-end loop with no glasses paired. ## Notes - Extentos handler code is **vendor-agnostic** — the SDK is built for multiple vendors, with Meta supported today. Migrating off raw DAT also moves you off Meta-only wiring. - Keep the parts DAT does that Extentos doesn't surface yet on raw DAT; see [when to use DAT directly](/docs/concepts/vs-meta-dat#when-to-use-meta-dat-directly). If you hit a missing primitive, [tell us](mailto:hello@extentos.com) so it gets surfaced. --- source: https://extentos.com/docs/guides/photo-capture --- --- title: Capture a photo description: Take a still photo on the Meta Ray-Ban camera from your handler code, handle CaptureError cleanly, and pipe the image into a vision LLM. Uses glasses.camera.capturePhoto() and the Photos URI helpers. type: guide platform: all vendor: meta related: - /docs/guides/video-capture - /docs/guides/capture-button - /docs/guides/voice-triggers - /docs/concepts/ai-gateway - /docs/reference/errors - /docs/troubleshooting/photo-capture-fails - /docs/guides/audio-streaming --- **Camera needs the Meta vendor module.** `com.extentos:glasses` carries no vendor SDK, so add `implementation("com.extentos:glasses-meta")` alongside it — see [install §1](/docs/sdk/android/install). Without it your build still succeeds and voice still works, but `capabilities.camera` is `false` and every capture returns an error. The SDK logs a warning at startup when it spots that combination. The camera is the strongest public capability on Meta Ray-Ban, and a one-shot still is the simplest way to use it. From any handler you call `glasses.camera.capturePhoto()`, get back an `ExtentosResult`, and either pattern-match the result or take the common `.valueOrNull()` shortcut. This page covers the call shape, the typed failure modes, and the canonical "photo → vision LLM → spoken answer" flow. > Kotlin leads the snippets and is the source of truth. iOS has the same `capturePhoto` surface (`case .success(let photo)`) with fewer bundled helpers (only `photo.loadImage()`) — see the iOS notes inline. ## The minimal call ```kotlin import com.extentos.glasses.core.PhotoConfig import com.extentos.glasses.core.PhotoFormat import com.extentos.glasses.core.Photos import com.extentos.glasses.core.Resolution import com.extentos.glasses.core.valueOrNull suspend fun takePhoto(glasses: ExtentosGlasses) { val photo = glasses.camera.capturePhoto( PhotoConfig(resolution = Resolution.MEDIUM, format = PhotoFormat.JPEG), ).valueOrNull() ?: return // null = the Err branch — see "Handling failures" below val uri = photo.uri ?: return // Photo.uri is String? (nullable) val bitmap = Photos.loadBitmap(uri) // android.graphics.Bitmap? — for Compose } ``` `PhotoConfig` takes **typed enums, not strings**: `Resolution` is `LOW | MEDIUM | HIGH`, `PhotoFormat` is `JPEG | HEIC`. Both live in the flat `com.extentos.glasses.core` package. By default a photo **grabs a frame off the shared camera stream** — the reliable path: a frame-grab can't trip Meta's frame-stall watchdog, and it behaves the same whether or not a recording or live-view is already running. It comes back at stream resolution (`resolution` sets that — 720×1280 / ~0.9 MP at `HIGH`, 504×896 at `MEDIUM`), which is ample for a vision model. For a **separate dedicated still**, set `PhotoConfig(dedicatedCapture = true)`: instead of a frame off the stream, it fires DAT's discrete `capturePhoto`, whose resolution is set by DAT independently of the stream (typically higher, but fixed — DAT's capture takes no resolution argument), traded against reliability (it can intermittently stall while a stream is live). Reach for it on a deliberate, human-facing photo; leave it off for assistant/vision and rapid captures. It's per-call. ### Photo.uri is transport-shaped and nullable `Photo.uri` is a `String?`. Two things to know: - It is typed `String?` — a blank frame on `capturePhoto` is surfaced as `CaptureError.PlatformError("no_image_uri")` rather than an `Ok` with a null uri, but null-check before using it anyway (a `Photo` you construct from other sources can carry null). - **Its scheme varies by transport** — `data:` in the browser simulator, `file:` on real glasses (`RealMetaTransport`) and `LocalSim`. Don't try to `AsyncImage(uri)` or `URL(uri)` it directly. Use the `Photos` helpers, which bridge both schemes uniformly: | Helper | Returns | Use for | |---|---|---| | `Photos.loadBase64(uri)` | `String?` (no `data:` prefix) | vision-LLM request bodies | | `Photos.loadBytes(uri)` | `ByteArray?` | ML pipelines wanting raw source bytes | | `Photos.loadBitmap(uri)` | `android.graphics.Bitmap?` | Compose (`bitmap?.asImageBitmap()`) | | `Photos.copyToFile(uri, dst)` | `Boolean` | saving into your gallery/library dir | | `Photos.mediaTypeFromUri(uri)` | `String?` (e.g. `"image/jpeg"`) | the `media_type` field a vision API needs | All except `mediaTypeFromUri` (synchronous, pure CPU) are `suspend` and run their decode / stream-copy on `Dispatchers.IO`. They return null/false on an unreadable URI, unrecognized scheme, decode failure, or OOM — they never throw. When you save a captured photo, use `Photos.copyToFile(uri, dst)` rather than hand-rolling `loadBytes` + `writeBytes`; it handles `data:` / `file://` / absolute paths, creates parent dirs, and overwrites the destination. > **iOS:** the bundled helper is `photo.loadImage()` (returns a `UIImage`/`NSImage` from either scheme). There is no `Photos.copyToFile` / `mediaTypeFromUri` equivalent yet — you encode the image and infer the media type yourself. ## Handling failures `capturePhoto()` returns `ExtentosResult`. `.valueOrNull()` collapses the failure to null, which is fine for fire-and-forget, but to give the user a meaningful message — or to react to a closed toggle — pattern-match with `when`. **Don't** wrap the call in `runCatching`: it only catches thrown exceptions and would mask the typed error. ```kotlin import com.extentos.glasses.core.CaptureError import com.extentos.glasses.core.ExtentosResult when (val r = glasses.camera.capturePhoto()) { is ExtentosResult.Ok -> use(r.value) is ExtentosResult.Err -> when (val e = r.error) { is CaptureError.NotConnected -> promptToConnectGlasses() // Declared for future transports; the current capture path does not emit // these three. Handle them so the `when` stays exhaustive, but don't rely // on them firing — a denied CAMERA grant arrives as PlatformError. is CaptureError.PermissionDenied -> requestCameraPermission() is CaptureError.ThermalCritical -> glasses.audio.speak("The glasses are too hot — let them cool.") is CaptureError.HingesClosed -> glasses.audio.speak("Open the glasses to take a photo.") is CaptureError.CoexistenceBlocked -> glasses.audio.speak("Camera's busy with a video stream — try again.") is CaptureError.StreamPaused -> glasses.audio.speak("The camera is paused — tap the right temple to resume.") is CaptureError.DisabledByUser -> glasses.audio.speak("Camera is switched off (${e.toggle}).") is CaptureError.PlatformError -> log(e.code, e.message) } } ``` `CaptureError.DisabledByUser` carries the gating `toggle` key as a payload — capture is gated by `camera_streaming_enabled` (default true) and `privacy_mode` (default false); when either closes you get this variant with the toggle name. The full variant list and payload shapes are in the [error reference](/docs/reference/errors#capture-errors). `CaptureError.CoexistenceBlocked` fires when an outgoing video stream holds the camera — capture and `outgoing_video_stream` are mutually exclusive (camera exclusivity). `CaptureError.StreamPaused` is different from a toggle: the wearer paused the camera with the glasses' capture button (a single temple tap) — a deliberate hardware privacy gesture. Meta's DAT gives your app **no way to resume the stream programmatically** (a single tap pauses and resumes; tap-and-hold stops), so the honest, platform-respecting response is to ask the wearer to tap the temple again, then retry. Don't fight it with a teardown-and-reopen — that drops the whole connection. The wearer taps to resume and the next `capturePhoto()` succeeds. You can exercise this whole path **in the simulator, before touching hardware**: the sim page's Hardware-buttons panel has the same capture button (tap = pause/resume, press-and-hold = stop — the capture LED on the glasses view goes dark while paused or stopped), and an agent drives it headlessly with the `injectHardwareButton` MCP tool. It's the same shared gate, so the error your handler sees is byte-for-byte what real glasses produce. After a hold-to-stop, the next capture re-arms the stream — on real glasses that stop also drops the connection for a few seconds while the SDK auto-recovers; the sim skips that blip. ## Trigger it with a voice phrase `capturePhoto()` is just a suspend call — fire it from anywhere: a button, the [assistant](/docs/concepts/assistant) via a tool, or a [voice trigger](/docs/guides/voice-triggers). The voice-trigger shape: ```kotlin glasses.voice.onPhrase(phrase = "take a photo", label = "Take a photo") { val photo = glasses.camera.capturePhoto().valueOrNull() ?: run { glasses.audio.speak("I couldn't take a photo.") return@onPhrase } val uri = photo.uri ?: return@onPhrase val dst = File(context.filesDir, "photo-${System.currentTimeMillis()}.jpg") Photos.copyToFile(uri, dst) glasses.audio.speak("Saved.") } ``` ## Photo + vision LLM The canonical "AI describes what you see" flow is: wake → capture → vision model → speak. You bring the LLM client; the SDK gives you the photo and the URI helpers to feed it. ```kotlin glasses.voice.onPhrase(phrase = "describe what you see", label = "Describe scene") { val photo = glasses.camera.capturePhoto( PhotoConfig(resolution = Resolution.MEDIUM, format = PhotoFormat.JPEG), ).valueOrNull() ?: run { glasses.audio.speak("I couldn't take a photo.") return@onPhrase } val uri = photo.uri ?: return@onPhrase val base64 = Photos.loadBase64(uri) ?: return@onPhrase // bytes for the request body val mediaType = Photos.mediaTypeFromUri(uri) ?: "image/jpeg" // the media_type field // `vision` is YOUR client. capturePhoto gives you the inputs; the model call is yours. val description = vision.describe(base64, mediaType, "Describe this scene briefly.") glasses.audio.speak(description) } ``` Two honest ways to make that `vision.describe(...)` call: - **Bring-your-own-key in handler code.** Write a small client (OkHttp + your provider's `/v1/messages`-style endpoint). `getCodeExample(pattern: "byok_anthropic")` is a paste-ready Anthropic Claude Vision wrapper with retry + typed failure variants; `getCodeExample(pattern: "photo_describe_voice")` is the full composition. - **The assistant's vision tool.** If you're already on the Phase-4 [assistant runtime](/docs/concepts/assistant), don't hand-roll a vision call — capture the photo inside a tool body and hand the URI to `session.includeImage(uri, prompt)`. The model sees the image, speaks about it in its configured voice, and remembers it for follow-ups, all on the [managed AI gateway](/docs/concepts/ai-gateway) (no key in your app). A vision round-trip is 2–8 s p95 and the user is staring at the glasses waiting — play `glasses.audio.earcon(EarconSound.START)` at capture time so they know it worked, and note that `speak()` routes over Bluetooth HFP (see [audio streaming](/docs/guides/audio-streaming)). ## Verify it in the simulator `capturePhoto()` emits a `capture_photo_returned` event when it resolves. Pull it with `getEventLog(filter: "all")` (the `*_returned` row classifies under the `custom` chip) — on success the row carries `duration_ms`, `mechanism`, `width`, `height`, `format`, and `has_uri`. No row means the call is still in flight or never started. `mechanism` is `frame_grab` or `dedicated` — which path your `PhotoConfig` requested. This is especially useful in the **simulator**: the sim returns the same image regardless of `dedicatedCapture` (it has no live-stream vs dedicated-capture distinction to fake), so `mechanism` is how you confirm which path your code *would* take on real hardware, where the resolution and reliability actually differ. The camera-stream lifecycle is visible too: the first frame-grab photo (or video/frames use) arms the shared stream — `camera_stream_opened` lands under the `camera` chip and the sim page's capture LED lights. A capture-button tap emits `camera_stream_paused` / `camera_stream_resumed`, and a capture attempted while paused logs `capture_denied` under the `errors` chip with the same "tap the right temple" message your handler receives. The browser simulator drives the photo from your real webcam, so the whole flow is exercisable before you touch hardware — the simulator is designed to behave like the glasses and that fidelity is under active validation, so confirm capture timing on real glasses too. ## Related - [Capture video](/docs/guides/video-capture) — bounded clips and the continuous frame stream. - [Audio streaming](/docs/guides/audio-streaming) — `speak()`, transcripts, and the HFP routing the snippets above rely on. - [The managed AI gateway](/docs/concepts/ai-gateway) — where assistant vision calls are metered. - [Error reference](/docs/reference/errors#capture-errors) — every `CaptureError` variant and payload. - [Photo capture fails](/docs/troubleshooting/photo-capture-fails) — when `capturePhoto` returns an error you didn't expect. - `getCapabilityGuide(feature: "capture_photo")` — the minimal call shape + the full gotcha list. --- source: https://extentos.com/docs/guides/sensor-streams --- --- title: Hardware events description: What Meta Ray-Ban surfaces today via glasses.runtime.events, connection.state, and toggles — and which hardware-alert streams aren't exposed yet. No IMU API. type: guide platform: all vendor: meta related: - /docs/guides/handle-disconnects - /docs/concepts/capabilities - /docs/vendors/meta - /docs/reference/errors - /docs/concepts/assistant --- > **There is no IMU, head-pose, accelerometer, gyroscope, or gaze stream on Meta Ray-Ban.** The Meta Device Access Toolkit does not expose raw motion sensors to third-party apps, and there is no phone-side workaround. If you've seen those promised elsewhere, it was aspirational — they are not part of the SDK. This page documents what the glasses *do* surface today and what's on the roadmap. What you get today is a runtime **event stream** plus a set of **hardware conditions** that surface through connection state and toggles. This page covers both, honestly scoped to what ships. > Kotlin leads the snippets and is the source of truth; the iOS surface mirrors it via `glasses.runtime.events`. ## The runtime event stream `glasses.runtime.events` is a `Flow` — the on-device event log surfaced to your app. `RuntimeEvent` is a sealed interface; subscribe and `filterIsInstance` for the variant you care about: ```kotlin import com.extentos.glasses.core.RuntimeEvent scope.launch { glasses.runtime.events.collect { event -> when (event) { is RuntimeEvent.ToggleChanged -> // a runtime toggle changed (key, oldValue, newValue, source) onToggle(event.key, event.newValue) is RuntimeEvent.CoexistenceWarning -> // an audio/video coexistence conflict (blocked, reason) warnUser(event.reason) is RuntimeEvent.UnrecognizedUtterance -> // STT produced text that matched no registered phrase (rawTranscript) maybeSuggestHelp(event.rawTranscript) is RuntimeEvent.Log -> // a diagnostic log line (level, message, payload?) Unit is RuntimeEvent.Assistant -> // a Phase-4 assistant lifecycle event — see the assistant runtime Unit } } } ``` The variants that ship today: | Variant | Payload | Fires when | |---|---|---| | `ToggleChanged` | `key, oldValue, newValue, source` | A runtime toggle is mutated (by UI, voice command, or automation). | | `CoexistenceWarning` | `blocked, reason` | **Declared, not yet emitted** — routing conflicts currently surface as `CoexistenceBlocked` errors on the affected call. | | `UnrecognizedUtterance` | `rawTranscript` | **Declared, not yet emitted** — post-pivot the library does no phrase dispatch; match transcripts in your own handler. | | `Log` | `level, message, payload?` | The library emits a diagnostic line. | | `Assistant` | `event: AssistantEvent` | A Phase-4 [assistant runtime](/docs/concepts/assistant) lifecycle event (session start/end, user/assistant spoke, tool called, reconnected, error). | `snapshotEvents()` returns the current buffer as a `List` if you'd rather poll than collect. In the browser simulator the same events flow to the structured [event log](/docs/concepts/transport-vs-app) (chips: errors / voice / camera / display / lifecycle / custom), queryable with `getEventLog`. ## Hardware conditions you can observe today Two physical conditions of the glasses are surfaced — not as a sensor stream, but through the surfaces that already exist: ### Thermal and hinge state → `connection.state` A fold — and, on the phone side, thermal pressure — ends the session. **Today the drop arrives as `Disconnected(DisconnectCause.DeviceDroppedConnection)`**; the dedicated typed causes below are declared but not yet produced by the core: - **Thermal critical** (a phone-side `PowerManager` signal, not a glasses sensor) → reserved cause `DisconnectCause.ThermalCritical`. Flipping `battery_save_mode` on before reconnecting is still a good response to heat. - **Hinges folded** → reserved cause `DisconnectCause.HingesClosed` (DAT doesn't expose hinge position; the fold shows up as the session ending). Observe these the same way you handle any disconnect — branch on `Disconnected.cause`. See [Handle disconnects](/docs/guides/handle-disconnects#branch-on-the-disconnect-cause) for the full pattern. (A continuous, *non-disconnecting* thermal/hinge event stream with severity and temperature is planned — see below — but today these conditions are observable only at the point they drop the connection.) ### Battery-save and stream clamping → toggles + events When `battery_save_mode` is on (set by the user, or by your app in response to thermal pressure), the library clamps any `videoFrames` stream to `LOW` resolution + 2 fps regardless of the config you requested, and emits a `camera_stream_clamped` event carrying the requested-vs-effective resolution/fps so the downgrade is observable. Observe `battery_save_mode` via `glasses.toggles.state`, and watch for the clamp in the event log. This is the signal to pair high-fps [video streams](/docs/guides/video-capture#continuous-frame-stream) with a downgrade path. ## Planned, not yet exposed The Meta DAT SDK and the Extentos transport carry several hardware/lifecycle alerts internally, but **the last hop to customer code is not connected yet** — there is no consumer API for these as of the current release. Don't build against them; they are listed here so you know they're coming and don't mistake their absence for a bug: | Planned event | Intended payload | Status | |---|---|---| | `thermal_warning` | `{ severity, temperatureCelsius }` | Future — today only via `Disconnected(ThermalCritical)` + `battery_save_mode`. | | `hinges_closed` | hinge fold/unfold | Future — today only via `Disconnected(HingesClosed)`. | | `audio_route_changed` | `{ from, to }` | Future — no consumer API yet. | | `incoming_call_detected` | `{ callerNumber, callerName }` | Future — no consumer API yet. | | `app_lifecycle_changed` | `{ state }` | Future — no consumer API yet. | | `location_updated` | `{ latitude, longitude, accuracyMeters }` | Future — no consumer API yet. | | `phone_notification_forwarded` | forwarded phone notification | Future — the `NotificationListenerService` emits it into the transport, but it isn't relayed to customer code yet. | Two other sensor surfaces are earlier-stage research, not committed: `ambient_light_updated` and a `tap_gesture` (custom tap/swipe gestures are **not** in the public DAT toolkit — only phone-side buttons and voice are developer-accessible). Check `getPlatformInfo` for the live status of each, and the [Meta vendor page](/docs/vendors/meta) for the GA capability matrix. ## Related - [Handle disconnects](/docs/guides/handle-disconnects) — branch on `Disconnected.cause` to react to thermal/hinge conditions. - [Capabilities](/docs/concepts/capabilities) — the full capability catalog and what's GA vs future. - [Vendors: Meta Ray-Ban](/docs/vendors/meta) — the GA capability matrix and DAT constraints. - [Assistant runtime](/docs/concepts/assistant) — `RuntimeEvent.Assistant` lifecycle events. --- source: https://extentos.com/docs/guides/simulator-to-real-glasses --- --- title: From simulator to real glasses description: Move a Meta Ray-Ban app from the browser simulator to real glasses — Meta identity, credentials, DAT resolution, the transport switch, and hardware checks. type: guide platform: all vendor: meta related: - /docs/concepts/transport-vs-app - /docs/getting-started/with-agent - /docs/vendors/meta - /docs/troubleshooting/glasses-wont-connect --- You built your app against the [browser simulator](/docs/concepts/transport-vs-app) and the agent-driven test loop closes: you inject an utterance, the right tool fires, the event log shows the expected sequence. That gives you **high confidence the protocol, tool wiring, dispatch, and event reporting are correct** — because those bytes are identical to what real glasses send and receive. It does **not** tell you whether the app will sound clear on a noisy street, recover from a dropped Bluetooth link, or survive a 30-minute session on the glasses' battery. Real hardware is the only thing that proves those. This page is the transition between the two. The simulator needs zero credentials and no Meta involvement — that's the point of it. Moving to real glasses is where your app first acquires a **production identity** and flips off the simulated transport. There are four moves, all of them one-time setup, and an emulator can't do any of it (no real Bluetooth radio, no Meta companion app), so you need a physical Android device. **Voice-only app? This whole page is optional.** Moves 1–3 exist to get you a Meta production identity, and a voice app has no vendor relationship to establish — no Meta Developer Center account, no App ID, no GitHub token, no DAT pairing. Your transition is: build a release variant with no baked session URL and install it. On Android, confirm `glasses.transportChosen` reads `SYSTEM_AUDIO`. iOS has no such property — check that the assistant answers and that `glasses.capabilities.camera` is `false`, which is what the audio baseline looks like there. That's it. See [choose your path](/docs/getting-started/choose-your-path). ## The four moves | Move | What you do | Where | |---|---|---| | **1 — Identity** | Create a Meta app, enable Wearables DAT, get an App ID + Client Token, register your package name + release SHA-256 signature, and enroll your own test account in a release channel | [Meta Wearables Developer Center](https://wearables.developer.meta.com/) | | **2 — Store** | Paste the App ID + Client Token into your project's Credentials section; `generateConnectionModule` bakes them into the build | Extentos dashboard | | **3 — Resolve** | Give Gradle a `read:packages` GitHub token so it can pull Meta's DAT SDK from GitHub Packages | Your shell or `local.properties` | | **4 — Flip & verify** | Build a release variant with no baked simulator URL, pair glasses to an Android device, install, and run the verification pass | Your machine + a paired Ray-Ban Meta | The rest of this page walks each one, then covers what the simulator already proved (so you know what *not* to re-test) and what only hardware can confirm. ## What the simulator already proved — and what it didn't Don't re-test on hardware what the simulator already guarantees, and don't assume the simulator covered what it can't. The split is sharp. **Byte-identical with real glasses** (proven in sim — trust it): | Aspect | Why it's identical | |---|---| | Audio input bytes (16 kHz mono i16 PCM, app → assistant) | Same format the glasses' wideband HFP microphone produces | | Audio output bytes (`tts_audio_chunk`, assistant → speaker) | Same path that drives the glasses' speaker | | The WebSocket to the assistant gateway | The sim opens the *same* connection real hardware does — both go through the Extentos managed gateway, so there is no sim-vs-hardware difference in this path | | Tool dispatch + the `assistant.*` event log | SDK-side, not transport-side — behaves identically everywhere | **Only real hardware can confirm** (the simulator does not model these): | Bucket | What hardware adds | |---|---| | **Audio character** | The glasses' beamforming HFP mic is narrower-band and noisier than a laptop mic. A model that picks tools reliably from clean sim audio may stumble on real capture, especially outdoors. | | **Hardware timing** | The first utterance after the assistant opens is partially clipped while Bluetooth switches A2DP→HFP (~200–500 ms). Music playing over A2DP collides with the assistant and drops to mono. | | **Hardware failure modes** | Bluetooth range loss, an incoming call pausing the session, ~30 min of continuous HFP draining the battery, sustained camera + assistant throttling the SoC. | If your app passes the agent loop in sim, the logic is correct. The verification pass below is about the second table, not the first. ## Move 1 — Get your Meta production identity Real-hardware traffic is gated by Meta. The simulator's placeholder credentials only ever route to the simulator; a real session needs your own Meta registration. 1. Create an app in the [Meta Wearables Developer Center](https://wearables.developer.meta.com/) — the App ID and Client Token are minted there (Manage projects). 2. Copy the **App ID** and **Client Token** from the DAT configuration. 3. Register your app's identity: its **package name** and its **release SHA-256 signing signature**. Meta binds the DAT entitlement to that exact pair — your app will not connect to real glasses until it's registered. (On iOS the pair is bundle ID + Team ID instead.) 4. **Enroll yourself as a tester.** Meta gates registration on release-channel membership: create a version, assign it to a channel, and invite the Meta account that owns the glasses you're about to test with — **including your own**. Until that invite is accepted, registration fails even with correct credentials and a correctly signed build. Requires glasses firmware v125+. See [release channels](/docs/vendors/meta). Because the entitlement is bound to *your* package and signature, this registration is per-app and can't be shared — Extentos can deliver credentials into your build, but it can't substitute its identity into your signed app. ## Move 2 — Store credentials in your dashboard Open your project's **Credentials** section and paste in the **App ID** and **Client Token**. These are *build-time vendor identity* — they ship inside your binary and are extractable, so they're build identity, not a server secret. (That's a different role from your managed-gateway AI key, which stays server-side and never ships. The section separates the two.) You don't hand-edit any files. The next time you run `generateConnectionModule`, it reads your stored credentials over an owner-authenticated request and bakes them in: - **Android** — writes `app/src/main/res/values/extentos_meta_credentials.xml` with the `meta_application_id` and `meta_client_token` string resources, referenced by two `` elements (`com.meta.wearable.mwdat.APPLICATION_ID` / `…CLIENT_TOKEN`) the MWDAT SDK reads at init. (Values go through `@string` refs because a bare numeric App ID is typed as an int by the resource compiler.) - **iOS** — fills the `Info.plist` `MWDAT` dict (`MetaAppID`, `ClientToken`, plus the iOS-only `TeamID` and URL scheme). If you haven't stored them yet, the scaffold leaves `REPLACE_ME` placeholders you can fill by hand. Either way, **the simulator and emulator dev need none of this** — credentials matter only for real hardware and release builds. ## Move 3 — Resolve the Meta DAT SDK (the GitHub token) **Voice-only apps skip this move entirely.** If your app doesn't use the camera or the display, it has no vendor dependency: no Meta DAT artifacts, no GitHub token, and nothing to add to `settings.gradle.kts`. Jump to Move 4. This is the one step that surprises people, so it's worth understanding *why* it exists. Camera and display come from `com.extentos:glasses-meta`, which depends on Meta's DAT SDK (`com.meta.wearable:mwdat-*`) — and **Meta publishes that SDK only through GitHub Packages**, not Maven Central, not Google's Maven repository. `com.extentos:glasses` itself has no vendor dependency and resolves from Maven Central like any normal artifact. Concretely you need two things in your project: 1. **The repository**, in `settings.gradle.kts` (inside `dependencyResolutionManagement.repositories`). `generateConnectionModule` emits this block for you — it's the only repository you add by hand, because the Extentos library itself resolves from its normal Maven coordinates: ```kotlin maven { url = uri("https://maven.pkg.github.com/facebook/meta-wearables-dat-android") credentials { val localProps = java.util.Properties().apply { val f = File(rootDir, "local.properties") if (f.exists()) f.inputStream().use { load(it) } } username = localProps.getProperty("github_username") ?: System.getenv("GITHUB_USERNAME") ?: "token" password = System.getenv("GITHUB_TOKEN") ?: localProps.getProperty("github_token") } } ``` 2. **A GitHub personal access token** with the `read:packages` scope, provided either as a `GITHUB_TOKEN` environment variable or as `github_token` in your `local.properties` (which is in the default Android Studio `.gitignore` — keep it that way). The username can be the literal `token`. Validate the whole chain before your first build — token, scope, repo block, and live reachability — with the bundled diagnostic, run from your project root: ```bash npx -p @extentos/mcp-server extentos-mcp setup ``` It does a ground-truth fetch against the actual mwdat artifact using the same auth scheme Gradle uses, so a green result genuinely predicts a successful build. Running it first saves the 1–3 minute build cycle that otherwise fails late at `:app:checkDebugAarMetadata` with a bare `401 Unauthorized` that never mentions Extentos or Meta. If you skipped it and you're staring at exactly that 401, this is the cause. ## Move 4 — Flip the transport, pair, and run ### Flip off the simulator During development the library auto-selects the browser simulator when a session URL is baked in (`BuildConfig.EXTENTOS_SESSION_URL` on Android, `extentos.session.plist` on iOS). The resolver checks that baked URL **before** it checks for bonded glasses — so an APK built with a simulator URL stays in simulator mode even on a phone with glasses paired. For your hardware build: - Build a **release variant** that leaves `EXTENTOS_SESSION_URL` null (and doesn't set the env var), so the resolver falls through to real Meta DAT when glasses are bonded. A release build also omits the debug-only pairing path. - Or, for a quick spike, pass `TransportChoice.RealMeta` explicitly in `ExtentosConfig`. `validateIntegration` will flag a release build that still has a simulator URL baked, and a missing GitHub Packages repo block — run it before you build. ### Pair and install 1. A physical **Android 12+** device (the library's `minSdk` is 31). No SIM needed. 2. Install the **Meta AI** companion app from the Play Store, sign in, and pair your Ray-Ban Meta. Take a photo from the glasses to confirm the pairing works outside your app. 3. `adb install` your release-variant APK, grant the runtime permissions (microphone, camera, Bluetooth) on first launch, and let the DAT registration flow hand off to the Meta AI app for approval. Once it returns `REGISTERED`, the session opens. ### Run the verification pass Assert the transport first, once, before you judge anything else: ```kotlin Log.i("Extentos", "transport=${glasses.transportChosen} source=${glasses.selectionSource} camera=${glasses.capabilities.camera}") ``` `REAL_META` is what you're after. `SYSTEM_AUDIO` means `Auto` fell through to the vendorless audio baseline — voice will work, capture won't, and nothing will throw. The usual causes are a missing `com.extentos:glasses-meta` dependency, glasses that aren't bonded, or `BLUETOOTH_CONNECT` not granted before `create(...)`. `BROWSER_SIM` means the simulator flip in the previous step didn't take. If the assistant fails to start with `AssistantError.NoApiKey`, that's the gateway, not the glasses — and despite the name it does not mean a missing AI key. Off the simulator, every build authenticates with the baked **project key**, so the usual cause is that no key was ever minted into this build. `generateConnectionModule` is the only thing that mints one, and it does so even for a voice-only app. See [gateway identity](/docs/concepts/ai-gateway#identity-how-the-gateway-knows-whos-calling). Then walk 5–10 real voice interactions covering every tool you registered, and watch `getEventLog(filter: "voice")`. You're confirming two things: that the `assistant.*` sequence matches what the simulator showed (`session_started → user_spoke → tool_called → tool_result → assistant_spoke`), and how the app behaves on the three things the simulator couldn't model: - **Audio character** — do your tools still fire reliably with real HFP audio, including outdoors? If a tool that was solid in sim gets flaky on hardware, the fix is almost always a clearer tool description, not a code change. - **Hardware timing** — does the first utterance feel right, or is it clipped while Bluetooth switches profiles? Playing an earcon or a short `say("ready")` on `assistant.session_started` covers the gap. Check music coexistence if your app needs it. - **Failure modes** — does the app recover from walking out of Bluetooth range, from an incoming call pausing the session, and does it stay usable within the ~30-minute battery ceiling of continuous use? Record what you find in your app's release notes — these are customer-facing UX expectations, not library bugs. When the pass is clean, `getProductionChecklist` covers the remaining ship gates (permissions audit, foreground service, store listing). ## First hardware run — telling the four failures apart Four different things block a first real-glasses run, and three of them can look identical from the app's side. Run `adb logcat -s Extentos:*` while you launch, and read in this order — each line is emitted by the SDK and is greppable. **1. Did you reach the vendor transport at all?** ``` transport=REAL_META source=VENDOR_REGISTRY ``` Anything else and you never got to Meta. `SYSTEM_AUDIO` means the vendor arm didn't claim — missing `com.extentos:glasses-meta`, no bonded glasses, or `BLUETOOTH_CONNECT` not granted before `create(...)`. Fix that first; nothing below can succeed until this line reads `REAL_META`. **2. Did the assistant open?** `AssistantError.NoApiKey` thrown from `assistant.start(...)` is the **project key**, not the glasses, and not a missing AI key. Off the simulator every build authenticates with the baked `EXTENTOS_PROJECT_KEY` — a sideloaded debug build and a store release alike — so the usual cause is that no key was ever minted into this build. This failure happens *before* any camera work and is easy to mistake for a hardware problem — it isn't. See [install § the project key](/docs/sdk/android/install). **3. Did Meta registration complete?** ``` registration: app not registered — starting handoff registration: registered ``` If you see the first line and never the second, the handoff started but didn't land. The SDK also logs, after a stall: ``` registration: still waiting — are the glasses awake and connected in the Meta AI app? ``` Three causes, in the order worth checking: | What to check | How you know | |---|---| | Glasses awake and connected **in the Meta AI app** | Open Meta AI; if the glasses show as disconnected there, Extentos can't reach them either | | Your package + **release** SHA-256 registered with Meta | A debug-signed build never matches the registered signature — reinstall the release variant | | Your own Meta account **enrolled in one of your release channels** | The consent prompt never appears at all. This is the one with no distinct log line, so treat it as the answer when the other two check out | **4. Did the device session start?** ``` device session: STARTED (start gated) ``` If instead you see `device session: did NOT reach STARTED in 12s`, or: ``` device session: createSession failed (dat=NO_ELIGIBLE_DEVICE) ``` …the glasses are registered but the session didn't open. A `createSession failed` line carries a hint when the SDK can identify the cause — for example ` — camera permission not granted; grant Camera access to capture photo/video`. `NO_ELIGIBLE_DEVICE` with no hint usually means the glasses went to sleep between steps; wake them and retry. Only after line 4 does `capturePhoto()` have a session to run on. A capture before that returns `CaptureError.NotConnected`, which is a symptom of one of the four above and never a problem in your capture code. ## Troubleshooting | Symptom | Cause | Fix | |---|---|---| | Build fails at `:app:checkDebugAarMetadata` with `401` | No `read:packages` token, or the GitHub Packages repo block is missing | Run `extentos-mcp setup`; see Move 3 | | App runs on the phone but behaves like the simulator | A simulator session URL is baked into the build | Build a release variant with `EXTENTOS_SESSION_URL` null (Move 4) | | Glasses never reach `REGISTERED` | Meta AI companion app missing, glasses not bonded, or DAT registration not approved | Pair and take a photo in the Meta AI app first; see [Glasses won't connect](/docs/troubleshooting/glasses-wont-connect) | | `Wearables` init fails on a tablet | Some tablets can't install the Meta AI companion app from Play | Use an Android 12+ phone, or verify the companion app installs on your tablet before relying on it | ## Frequently asked questions ### Can I just keep using the simulator? For building and iterating, yes — that's the primary surface, and the agent-driven loop closes there without humans or hardware. But before you put the app on a customer's head, at least one real-hardware pass is required: the simulator is byte-accurate for protocol and logic, and deliberately silent on audio character, Bluetooth timing, and failure modes. ### Can I test on an Android emulator instead of a real phone? No. Emulators have only a virtual Bluetooth stack, so the glasses can't bond, Bluetooth SCO audio can't start, and the DAT registration has no companion app to hand off to. Cloud device farms are real phones but they're in a datacenter — they can't reach the glasses on your desk. Real-hardware testing needs a physical device with the glasses paired to it. ### Do I need to create a GitHub personal access token? You need a token with the `read:packages` scope so Gradle can pull Meta's DAT SDK from GitHub Packages — see Move 3. It can be a brand-new token used only for this; it never goes into your app binary or your git history. If you've already built the library locally, you likely have one set already (`extentos-mcp setup` will tell you). ### Why does Meta's SDK need a token at all? Because Meta distributes the Device Access Toolkit exclusively through GitHub Packages, and GitHub Packages requires authentication even for public packages. There's no unauthenticated mirror today. The token authenticates *to GitHub*, not to Meta or Extentos. ### My build worked all through development and suddenly 401s — what changed? Nothing about your code. The most common trigger is building on a fresh machine or CI runner where the `read:packages` token isn't configured, or a `settings.gradle.kts` where the GitHub Packages repo block was dropped. Run `extentos-mcp setup` to pinpoint which. (Note this applies only to apps that depend on `com.extentos:glasses-meta` for camera or display — those artifacts are transitive through it, so a missing token bites simulator builds too. A voice-only app never pulls them.) ### Is the transition different on iOS? The shape is the same — get a Meta identity, store credentials, flip the transport, pair and verify — but the concrete wiring uses Swift Package Manager and the `Info.plist` `MWDAT` dict instead of Gradle and resource files. See the [iOS quickstart](/docs/getting-started/ios). The assistant-runtime walkthrough on this page is written against the Android library. ## Related - **[Transport vs app simulation](/docs/concepts/transport-vs-app)** — what the browser simulator covers versus Meta's Mock Device Kit, and why the same `glasses.*` code runs against both the simulator and real hardware - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — the MCP flow that scaffolds the project and bakes your credentials - **[Vendors: Meta Ray-Ban](/docs/vendors/meta)** — the capability matrix and what DAT supports on real hardware - **[Glasses won't connect](/docs/troubleshooting/glasses-wont-connect)** — the connection-failure decision tree --- source: https://extentos.com/docs/guides/video-capture --- --- title: Capture video description: Record a bounded video clip from the Meta Ray-Ban camera with glasses.camera.captureVideo(), stop it cleanly with stopVideo(), or subscribe to the continuous videoFrames stream for on-device ML and live preview. type: guide platform: all vendor: meta related: - /docs/guides/photo-capture - /docs/guides/capture-button - /docs/guides/audio-streaming - /docs/guides/voice-triggers - /docs/reference/errors --- **Camera needs the Meta vendor module.** `com.extentos:glasses` carries no vendor SDK, so add `implementation("com.extentos:glasses-meta")` alongside it — see [install](/docs/sdk/android/install). Without it your build still succeeds and voice still works, but `capabilities.camera` is `false` and every capture returns an error. The SDK logs a warning at startup when it spots that combination. Two camera surfaces give you motion: `glasses.camera.captureVideo()` records a phone-side-encoded clip and returns a `VideoClip`, and `glasses.camera.videoFrames(...)` is a continuous `Flow` of camera frames for on-device ML or live preview. This page covers both, the clean stop pattern, and the cross-platform save gotcha. > Kotlin leads the snippets here and is the source of truth. The iOS `VideoConfig` matches it (`includeAudio` defaults `true`, `frameRate` defaults `24`); the gap is the bundled helpers — there are no `Videos.*` helpers on iOS. See the inline notes. ## Record a clip ```kotlin import com.extentos.glasses.core.VideoConfig import com.extentos.glasses.core.VideoFormat import com.extentos.glasses.core.Videos import com.extentos.glasses.core.valueOrNull import java.io.File suspend fun record(glasses: ExtentosGlasses, context: Context) { val clip = glasses.camera.captureVideo( VideoConfig( maxDurationSeconds = 60, // Int? — null (the default) means no time cap includeAudio = true, // forces the HFP route while capturing (see below) format = VideoFormat.MP4_HEVC, // MP4_HEVC | MP4_H264 | MOV — prefer HEVC ), ).valueOrNull() ?: return // ExtentosResult.Err — handle the same CaptureError variants as photo val uri = clip.uri ?: return // VideoClip.uri is String? (nullable, transport-shaped) val dst = File(context.filesDir, "clip-${System.currentTimeMillis()}.mp4") if (!Videos.copyToFile(uri, dst)) return // save to your library dir } ``` The Kotlin `VideoConfig` has five fields: `maxDurationSeconds: Int?` (null = no cap), `includeAudio: Boolean = true`, `format: VideoFormat = MP4_HEVC`, `resolution: Resolution = MEDIUM`, and `frameRate: Int = 24`. The `resolution` / `frameRate` pair takes effect only when the recording is the **first** camera use of the session — video and `videoFrames` share one warm camera stream that can't be reconfigured live. There is **no `stopConditions` field** — older spec metadata advertised one, but it was never wired; the stop primitive is `stopVideo()` (below). `captureVideo()` returns `ExtentosResult` — the failure variants are the same as [photo capture](/docs/guides/photo-capture#handling-failures) (`DisabledByUser` carries the gating toggle, `CoexistenceBlocked` on a camera conflict, etc. — see the [error reference](/docs/reference/errors#capture-errors)). Like `Photo.uri`, `VideoClip.uri` is a nullable `String?` whose scheme is `data:` in the simulator and `file:` on hardware; the `Videos` helpers (`copyToFile`, `loadBytes`, `loadBase64`, `mediaTypeFromUri`) bridge both — and like `Photos`, they never throw. **`Videos` is Android-only today** (iOS parity queued); on iOS read `clip.uri` (a `String?`) via `URL(string:)` + `Data(contentsOf:)` for `file://`, or parse the `data:` scheme by hand. ### Codec: treat `format` as a label, not a guarantee On Android today the clip is written by the phone-side encoder as H.264 in an MP4 container — on real glasses and in the simulator alike — regardless of the `VideoFormat` you request; the `VideoClip.format` field there is a container hint, not a codec promise. On iOS the writer honors the requested format (`MP4_H264` → H.264, otherwise HEVC). Dev/prod parity is on the clip *shape*. ### `includeAudio` forces HFP coexistence When `includeAudio = true`, audio captures over the glasses-as-mic Bluetooth HFP profile, which suspends A2DP playback for the duration — any music drops from hi-fi A2DP to voice-band HFP mono until the clip finishes. If your app plays music while recording, set `includeAudio = false` — it defaults to `true` on both platforms, so opt out explicitly. See [audio streaming](/docs/guides/audio-streaming#a2dp-vs-hfp-coexistence) for the coexistence model. ## Stop a recording cleanly A clip with `maxDurationSeconds = null` runs until you stop it. The clean primitive is `glasses.camera.stopVideo()` — the in-flight `captureVideo()` then **resumes with `Ok` holding the partial clip** recorded up to that point. No coroutine cancellation, no exception, no separate result channel: ```kotlin val capture = scope.async { glasses.camera.captureVideo(VideoConfig(includeAudio = true)) } // ...later, on the user's "stop" intent: glasses.camera.stopVideo() val clip = capture.await().valueOrNull() // natural await — returns the partial clip ``` `stopVideo()` is idempotent (a no-op when nothing is recording). > **Don't** cancel the wrapping `async` Deferred to stop a recording. Once a Deferred is cancelled its state is sticky-Cancelled and `await()` throws even though the body returned a value — you'd lose the clip. Use `stopVideo()`. ### The voice-stop pattern and `NonCancellable` A common shape is a wake phrase that starts recording and a `stops` phrase that ends it. When you stop via the `stops`-cancellation path (rather than `stopVideo()`), the library still recovers the partial: it sends an abort to the transport and drains the partial `video_result` for **up to 10 s** under `NonCancellable`, so `captureVideo()` returns `Ok(VideoClip)` instead of throwing. But the handler coroutine has *already been cancellation-requested* by the time the clip comes back — so your save I/O must run under `withContext(NonCancellable) { ... }` or it throws `CancellationException` and silently drops the clip: ```kotlin glasses.voice.onPhrase( phrase = "start recording", stops = listOf("stop recording"), ) { val clip = glasses.camera.captureVideo( VideoConfig(maxDurationSeconds = 60, includeAudio = true), ).valueOrNull() ?: return@onPhrase val uri = clip.uri ?: return@onPhrase // Post-stop I/O MUST survive the surrounding handler cancellation: withContext(NonCancellable) { Videos.copyToFile(uri, dst) repo.insertRecording(/* ... */) } } ``` On iOS the equivalent is running the post-stop save in a `Task.detached`. The drain ceiling is 10 s — real-hardware `MediaRecorder.stop` → encode can take 3–5 s on a 20-second clip. If the encode exceeds 10 s (very large clips / slow devices) the original `CancellationException` rethrows and the partial is lost; bound your clip lengths, or use a `videoFrames` + `MediaMuxer` custom encoder for clips over ~30 s. ## Continuous frame stream For anything that needs more than one frame per second — on-device object detection / scene classification, a live preview — subscribe to `videoFrames`: ```kotlin import com.extentos.glasses.core.Resolution import com.extentos.glasses.core.VideoFrameConfig glasses.camera.videoFrames( VideoFrameConfig( resolution = Resolution.LOW, // LOW is plenty for most ML frameRate = 2, // 2 fps for ML; 7–15 for preview ), ).collect { frame -> // frame.data is encoded image bytes (the simulator sends JPEG — // BitmapFactory.decodeByteArray before use). Also frame.width / height / timestampMs. } ``` The Kotlin `VideoFrameConfig` is intentionally minimal — `resolution` and `frameRate` only. (iOS additionally exposes `codec` and `backpressure`; Kotlin handles backpressure internally.) Frame rate drives battery hard: 2 fps for ML drains negligibly, 7–15 fps for preview lasts ~1 hr, 24–30 fps drains in ~30 min. Pair any high-fps stream with a thermal listener that downgrades it (see [hardware events](/docs/guides/sensor-streams)). `battery_save_mode = true` clamps the stream to `LOW` + 2 fps regardless of config and emits a `camera_stream_clamped` event so the downgrade is observable. `videoFrames` (and `captureVideo`) require `FOREGROUND_SERVICE` on Android 14+ — see `getPermissions`. ### If the camera is paused when you start The wearer can pause the camera with the glasses' capture button — a single temple tap (tap again to resume; tap-and-hold stops). If you **start** `videoFrames` while it's paused, the flow throws `CameraStreamPausedException` at collection — the streaming analogue of the `CaptureError.StreamPaused` that `capturePhoto`/`captureVideo` return. Catch it and prompt the wearer to tap the temple, then start again: ```kotlin import com.extentos.glasses.core.CameraStreamPausedException try { glasses.camera.videoFrames(config).collect { render(it) } } catch (e: CameraStreamPausedException) { glasses.audio.speak("The camera is paused — tap the right temple to resume.") } ``` A pause that happens **mid-stream** is not an error — the frames simply stop (your last-rendered frame freezes on screen) and resume when the wearer taps. Meta's DAT has no app-callable resume, so never try to force it with a teardown-and-reopen; that drops the whole connection. ### A pause during a recording skips footage A `captureVideo` recording is *made of* the stream, so a mid-recording temple tap doesn't end it — the recording stays alive but **captures no footage while paused**. End it while paused and the clip stops at the pause point; let the wearer resume and keep recording, and the final clip contains everything *except* the paused window, spliced seamlessly. The clip's `duration_ms` reflects footage, not wall clock. A tap-and-hold (stop) while recording ends it with whatever footage was captured so far. The simulator reproduces all of this — drive the capture button from the sim page's Hardware-buttons panel or `injectHardwareButton` and watch the REC pill flip to "paused — no footage". ## Verify it in the simulator `captureVideo()` emits a `capture_video_returned` event when it resolves — `getEventLog(filter: "all")` shows (the `*_returned` row classifies under the `custom` chip) `success`, `duration_ms`, `clip_duration_ms`, and `via`. The `via` value tells you which path produced the clip: `cancelled_with_partial` means a stop fired and the drain recovered a partial; `cancelled` means the drain produced nothing. The simulator is designed to behave like the glasses, but codec and HFP timing are substrate deltas under active validation — confirm long-clip and music-coexistence behavior on real hardware. ## Related - [Capture a photo](/docs/guides/photo-capture) — one-shot stills, the `CaptureError` variants, and the URI-helper model the snippets above reuse. - [Audio streaming](/docs/guides/audio-streaming) — the A2DP/HFP coexistence that `includeAudio` triggers. - [Voice triggers](/docs/guides/voice-triggers) — the `onPhrase` + `stops` shape used for voice-controlled recording. - [Error reference](/docs/reference/errors#capture-errors) — every `CaptureError` variant. - `getCapabilityGuide(feature: "capture_video")` / `getCapabilityGuide(feature: "video_frames")` — minimal call shapes + full gotchas. --- source: https://extentos.com/docs/guides/voice-assistant --- --- title: Build a voice assistant description: Build a wake-word voice assistant on Meta Ray-Ban smart glasses with glasses.assistant.start. A phone-side wake phrase opens the conversation; the model owns turn-taking and intent parsing; you write tool bodies that read and act on your app's state, add a vision tool for "what am I looking at", and let the conversation sleep on intent. Runs on the Extentos managed gateway with no API key in your app. type: guide platform: all vendor: meta related: - /docs/concepts/assistant - /docs/concepts/ai-gateway - /docs/guides/voice-triggers - /docs/guides/display - /docs/concepts/capabilities --- **Camera needs the Meta vendor module.** `com.extentos:glasses` carries no vendor SDK, so add `implementation("com.extentos:glasses-meta")` alongside it — see [install §1](/docs/sdk/android/install). Without it your build still succeeds and voice still works, but `capabilities.camera` is `false` and every capture returns an error. The SDK logs a warning at startup when it spots that combination. This guide builds a working voice assistant: a wake phrase opens a conversation, the model calls **tools** that read and mutate your app's own state (route stats, a clip library, the camera), a **vision** tool answers "what am I looking at," and the conversation **sleeps** when the user is done. A phone-side wake phrase (on-device recognition) opens the conversation; the model handles turn-taking, intent parsing, and confirmation speech — you only write the tool bodies. For the full API and the wake/sleep state machine, read [the assistant runtime](/docs/concepts/assistant) first. This page is the task walkthrough. > `glasses.assistant.*` ships on both platforms — in `com.extentos:glasses` since **1.4.0** ([SDK install](/docs/sdk/android/install)) and in the `Extentos` Swift package — a standard dependency, no snapshot setup. Kotlin leads the snippets here, with the Swift counterpart in a fold under each. The fields and behaviour are the same on both; a handful of spellings differ, and those are called out where they appear rather than left for you to discover at the compiler. ## 1. Bootstrap The assistant needs the microphone, and `RECORD_AUDIO` is a **runtime** permission your app owns — the library never requests it for you. So the order matters, and it is the part people get wrong: **Create the `ExtentosGlasses` handle in `Application.onCreate`, but start the assistant only after the grant.** Creating the handle early is fine and cheap; starting a session before the grant leaves you with an assistant that is running and deaf. ```kotlin title="MyApp.kt" class MyApp : Application() { lateinit var glasses: ExtentosGlasses override fun onCreate() { super.onCreate() // Cheap: resolves the transport, wires the clients. No mic yet. // No OpenAI key anywhere — the assistant runs on the Extentos managed // gateway (there is no option to run it on your own OpenAI account). glasses = ExtentosGlasses.create(ExtentosConfig(applicationContext = this)) } } ``` ```kotlin title="MainActivity.kt" class MainActivity : ComponentActivity() { private val micPermission = registerForActivityResult( ActivityResultContracts.RequestPermission(), ) { granted -> if (granted) onMicGranted() } override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) if (checkSelfPermission(Manifest.permission.RECORD_AUDIO) == PackageManager.PERMISSION_GRANTED) { onMicGranted() } else { micPermission.launch(Manifest.permission.RECORD_AUDIO) } } private fun onMicGranted() { // Only now: the foreground service (so the mic survives backgrounding) // and the assistant handler. GlassesForegroundService.start(this) // Your handler — step 2 defines it. routeTracker / library are your own app state. RunAssistantHandler((application as MyApp).glasses, routeTracker, library).start() } } ``` **Do not start the assistant or the foreground service from `Application.onCreate`.** On first launch the grant cannot exist that early, and Android 14+ rejects a microphone-type foreground service started without it. See [lifecycle](/docs/sdk/android/lifecycle). If you render the packaged `ExtentosConnectionPage` it handles this flow for you as part of pairing — but a voice-only app has no connection page, so the sequence above is yours to write.
The same bootstrap in Swift iOS needs no foreground service — `UIBackgroundModes: ["audio"]` in your Info.plist covers background listening. Two things differ from Android and both matter: - **`usedCapabilities` picks the transport.** Leave it empty and `.auto` resolves to real glasses, so a voice app dies with `NoEligibleDevice` on a phone with none. Declare `[.microphone, .speaker]`. - **The microphone grant is yours.** `requestSpeechRecognitionAuthorization()` covers speech recognition only, not audio capture. ```swift title="InterpreterApp.swift" import AVFAudio import GlassesCore import SwiftUI @main struct MyApp: App { let glasses: any ExtentosGlasses init() { glasses = Extentos.create(config: ExtentosConfig( usedCapabilities: [.microphone, .speaker] )) // `debug` is left at its false default — setting it true opens a pending // browser-simulator socket and waits for the MCP bridge to bind it. } var body: some Scene { WindowGroup { ContentView(glasses: glasses) } } } ``` ```swift title="Starting the assistant, after both grants" func startAssistant(_ glasses: any ExtentosGlasses) async { let mic = await AVAudioApplication.requestRecordPermission() let speech = await Extentos.requestSpeechRecognitionAuthorization() guard mic, speech else { return } // surface your own explanation await RunAssistantHandler(glasses: glasses).start() } ```
### If your app also uses the camera The tools, the session block and the wake phrase are unchanged. **The bootstrap is not** — one thing moves and two things are added, all consequences of the same fact: **`TransportChoice.Auto` picks the transport inside `create(...)` and keeps it for the life of the handle.** Whatever is true at that moment is what your app gets. **What moves: `create(...)` leaves `Application.onCreate`.** In the voice bootstrap above it can run at app start, because nothing it depends on needs a grant. A camera app can't do that — `Auto` needs `BLUETOOTH_CONNECT` in hand to see the glasses, and on first launch no grant exists that early. So the handle becomes `lateinit` and is built from the permission callback instead. Keeping the eager `create()` is the mistake this section exists to prevent: it resolves to the audio baseline, capture returns errors forever, and nothing throws. **1. Grant `BLUETOOTH_CONNECT` before `create(...)`, not after.** The Meta transport claims the session by checking whether Meta glasses are bonded to the phone, and reading the bonded set needs `BLUETOOTH_CONNECT` on Android 12+. Without the grant the check can't answer, `Auto` moves down to the audio baseline, and `capabilities.camera` reports `false` for the rest of the process — voice works, capture doesn't. Restarting the app after the grant fixes it, but you don't want the first run to be the broken one. **2. Give `create(...)` an `activityProvider`.** Meta's registration handoff needs a live `Activity` to present its consent UI, and the SDK requests `BLUETOOTH_CONNECT` and `CAMERA` at runtime through that same Activity. It defaults to `null`, and with no connection page nothing else supplies one. ```kotlin title="MyApp.kt — the camera variant" class MyApp : Application() { lateinit var glasses: ExtentosGlasses private val topActivity = java.util.concurrent.atomic.AtomicReference() override fun onCreate() { super.onCreate() registerActivityLifecycleCallbacks(object : ActivityLifecycleCallbacks { override fun onActivityResumed(activity: Activity) { topActivity.set(activity) } override fun onActivityPaused(activity: Activity) { if (topActivity.get() === activity) topActivity.set(null) } // remaining callbacks: no-op override fun onActivityCreated(a: Activity, b: Bundle?) {} override fun onActivityStarted(a: Activity) {} override fun onActivityStopped(a: Activity) {} override fun onActivitySaveInstanceState(a: Activity, b: Bundle) {} override fun onActivityDestroyed(a: Activity) {} }) } /** Called from the Activity once the permissions below are granted. */ fun initGlasses() { if (::glasses.isInitialized) return glasses = ExtentosGlasses.create( ExtentosConfig( applicationContext = this, activityProvider = { topActivity.get() }, environment = if (BuildConfig.DEBUG) ExtentosEnvironment.DEVELOPMENT else ExtentosEnvironment.PRODUCTION, ) ) } } ``` ```kotlin title="MainActivity.kt — the camera variant" class MainActivity : ComponentActivity() { private val permissions = registerForActivityResult( ActivityResultContracts.RequestMultiplePermissions(), ) { result -> // Create only after the answer, whatever it was — a denied BLUETOOTH_CONNECT // still resolves correctly (to the audio baseline), it just isn't a surprise. val app = application as MyApp app.initGlasses() Log.i("Extentos", "transport=${app.glasses.transportChosen} camera=${app.glasses.capabilities.camera}") if (result[Manifest.permission.RECORD_AUDIO] == true) onMicGranted() } override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) permissions.launch( arrayOf( Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA, Manifest.permission.BLUETOOTH_CONNECT, ) ) } private fun onMicGranted() { GlassesForegroundService.start(this) // Your handler — step 2 defines it. RunAssistantHandler((application as MyApp).glasses, routeTracker, library).start() } } ``` Then assert what you got, once, at startup — this is the check that turns a silent capability loss into a one-line log: ```kotlin Log.i("Extentos", "transport=${glasses.transportChosen} camera=${glasses.capabilities.camera}") ``` A camera app expects `REAL_META` / `true`. `SYSTEM_AUDIO` / `false` means `Auto` fell through to the audio baseline — either the `com.extentos:glasses-meta` dependency is missing, the glasses aren't bonded, or `BLUETOOTH_CONNECT` wasn't granted before `create(...)`. With the base dependency there are no model files or model paths to manage — the assistant is available immediately and runs end-to-end over the gateway's WebSocket. (On-device models are an opt-in you add deliberately: two extra modules and a `register()` call. See [local models](/docs/concepts/local-models).) See [the managed gateway](/docs/concepts/ai-gateway) for the AI plumbing. ## 2. Start the session and register tools The whole assistant is one block. Declare `instructions`, register tools, and pick a greeting. Leave the provider as `AssistantProvider.Managed()` to take your dashboard's model + voice (defaults `gpt-realtime-2` / `alloy`). ```kotlin import com.extentos.glasses.core.ExtentosGlasses import com.extentos.glasses.core.RuntimeEvent import com.extentos.glasses.core.VideoClip import com.extentos.glasses.core.VideoConfig import com.extentos.glasses.core.valueOrNull import com.extentos.glasses.core.assistant.AssistantEvent import com.extentos.glasses.core.assistant.AssistantError import com.extentos.glasses.core.assistant.AssistantException import com.extentos.glasses.core.assistant.AssistantProvider import com.extentos.glasses.core.assistant.AssistantSession import com.extentos.glasses.core.assistant.Greeting import com.extentos.glasses.core.assistant.ToolResult import com.extentos.glasses.core.assistant.tool import kotlinx.coroutines.* import kotlinx.coroutines.flow.filterIsInstance import kotlinx.coroutines.flow.launchIn import kotlinx.coroutines.flow.onEach import kotlin.time.Duration.Companion.seconds class RunAssistantHandler( private val glasses: ExtentosGlasses, private val routeTracker: RouteTracker, // your app state private val library: ClipLibrary, // your app state private val scope: CoroutineScope = CoroutineScope(SupervisorJob() + Dispatchers.IO), ) { private var session: AssistantSession? = null private var activeVideo: Deferred<*>? = null fun start() { scope.launch { session = glasses.assistant.start(provider = AssistantProvider.Managed()) { instructions = """ You are a running companion. Help with route stats and capture moments. Speak briefly — they're running. Don't narrate what you're doing; just do it and confirm. When the user clearly wants to stop talking, call end_conversation. """.trimIndent() // The SDK speaks this greeting automatically on every wake, // generated out-of-band from memory (never continues the prior chat). greeting = Greeting.Custom( "Greet the runner warmly in one short sentence and ask how you can help." ) // Deterministic backup sleep after 30s of user silence. sleepAfterSilence(30.seconds) // Read tools — instant data the model reads aloud. tool("get_route_remaining", "How much of the planned route is left, in km.") { ToolResult.Ok("${routeTracker.kmRemaining()} km remaining") } tool("get_average_pace", "The runner's current average pace in minutes per km.") { ToolResult.Ok("${routeTracker.avgPaceMinKm()} min per km") } // Action tools — side effects on your own state. The model // manages the take/stop pair from conversational context. tool("take_video", "Start recording a video clip of the runner's view.") { if (activeVideo?.isActive == true) { return@tool ToolResult.Err("a recording is already in progress") } activeVideo = scope.async { glasses.camera.captureVideo(VideoConfig(maxDurationSeconds = 30)) } ToolResult.Ok("recording started") } tool("stop_video", "Stop the current video recording.") { val capture = activeVideo ?: return@tool ToolResult.Err("nothing was recording") activeVideo = null glasses.camera.stopVideo() // resumes the await naturally @Suppress("UNCHECKED_CAST") val clip = (capture as Deferred>) .await().valueOrNull() ?: return@tool ToolResult.Err("video capture failed") library.add(clip) ToolResult.Ok("video saved") } } // Wake mechanism — the same voice-trigger system as everywhere. // Defaults to VoiceScope.WhenDormant so it won't double-fire mid-chat. glasses.voice.onPhrase("hey coach") { session?.wake() } } } } ``` **`assistant.start(...)` throws — it's the exception to the "results never throw" rule.** Capability calls return `ExtentosResult`, but an open-time failure raises `AssistantException` wrapping an `AssistantError`. In the bare `scope.launch { … }` above that would kill the coroutine and take the app with it, and the most common trigger is the ordinary one: a sideloaded build reaching the gateway without a baked project key gives you `NoApiKey`. Wrap it: ```kotlin session = try { glasses.assistant.start(provider = AssistantProvider.Managed()) { /* … */ } } catch (e: AssistantException) { Log.e("MyApp", "assistant didn't start: ${e.error}", e) return@launch // surface it in your UI; don't let it propagate } ``` `AssistantException` and `AssistantError` live in `com.extentos.glasses.core.assistant`. Variants and their meanings are in the [error reference](/docs/reference/errors).
The same session in Swift Same fields, same behaviour — four spellings differ. The config block is a closure taking the builder (so `$0.`), `sleepAfterSilence` takes a `TimeInterval` in seconds rather than a `Duration`, tool results are `.ok` / `.err`, and `tool` labels its description. ```swift import GlassesCore final class RunAssistantHandler { private let glasses: any ExtentosGlasses private var session: (any AssistantSession)? init(glasses: any ExtentosGlasses) { self.glasses = glasses } func start() async { do { session = try await glasses.assistant.start( provider: .managed() // .managed(model: "local-auto") to run on-device ) { $0.instructions = """ You are a running companion. Help with route stats and capture moments. Speak briefly — they're running. """ $0.greeting = .custom(directive: "Greet in one short sentence.") $0.sleepAfterSilence(30) // seconds, not a Duration $0.endOnIntent = true $0.tool("log_split", description: "Record a lap split for the current run.") { .ok("split recorded") } } } catch let error as AssistantError { // start() throws on iOS too. .noApiKey on a sideloaded build means the // gateway couldn't identify the caller — see the Info.plist project key. print("assistant didn't start: \(error)") return } catch { return } // No `firesWhen` on Swift — the wake phrase can fire mid-conversation, // so gate re-wakes yourself if that matters. _ = glasses.voice.onPhrase(phrase: "hey coach", label: "Wake", stops: []) { [weak self] in try? await self?.session?.wake() } } } ``` A vision tool is the same shape; note `includeImage` labels both arguments and `prompt` has no default on Swift: ```swift $0.tool("describe_scene", description: "Describe what the user is looking at.") { [weak self] in guard let self else { return .err("handler went away") } let result = await self.glasses.camera.capturePhoto() guard case .success(let photo) = result, let uri = photo.uri else { return .err("couldn't take a photo right now") } try? await self.glasses.assistant.activeSession?.includeImage(uri: uri, prompt: nil) return .ok("looking") } ```
A few things the model does for you here, that you'd otherwise hand-wire: it decides which tool to call from each `description`, speaks a "let me check…" filler while a tool runs (suppress per-tool with `blocking = true`), and ends the conversation via the hidden `end_conversation` tool (`endOnIntent` defaults `true`) when the user wraps up — no rigid "goodbye" phrase required. > **Clean stop pattern.** To stop the recording, call `glasses.camera.stopVideo()` and `await()` the in-flight capture — it resumes with a partial clip. Don't `cancel()` the `Deferred`: Kotlin's cancelled state is sticky and `await()` would throw even when a partial exists. ## 3. Add a vision tool `session.includeImage(uri)` adds a photo to the conversation and auto-triggers a spoken response in the model's voice. Reach the running session from a tool body via `glasses.assistant.activeSession` (non-null while a tool dispatches). Keep "describe" and "save" as **two tools** so the model can disambiguate — and call both back-to-back when the user wants both. ```kotlin import com.extentos.glasses.core.assistant.orToolError tool( "describe_scene", "Describe what the runner is looking at without saving the photo. " + "Call for 'what do you see' / 'describe this' / 'tell me about this'.", ) { val photo = glasses.camera.capturePhoto().orToolError { return@tool it } val uri = photo.uri ?: return@tool ToolResult.Err("photo had no uri") glasses.assistant.activeSession?.includeImage(uri) ToolResult.Ok("looking") } ``` `orToolError { return@tool it }` unwraps the capture, and on failure short-circuits the tool with a `ToolResult.Err` that carries the failure's **actionable** message — so the model relays a real reason instead of a generic "camera failed". If the wearer paused the camera (a temple tap), the model hears *"tap the right temple of your glasses to resume the camera"* and can tell the user exactly that. It's a thin helper over the typed `CaptureError`; pattern-match the `ExtentosResult` yourself if you want custom copy. Camera tools need the simulator browser tab attached when you test them — see step 5. ## 4. Capture transcripts (optional) If you want the user's words for notes, captions, or a journal, subscribe to the shared runtime-event stream. Assistant events carry **verbatim** transcripts (yours to govern; document retention in your privacy policy). ```kotlin glasses.runtime.events .filterIsInstance() .onEach { evt -> (evt.event as? AssistantEvent.UserSpoke)?.let { notes.append(it.transcript) } } .launchIn(scope) ``` ## 5. Verify it in the simulator The agent-driven loop closes without a human or hardware. Drive the **wake** with the real wake path, then inject a user utterance, then assert the tool fired: ```ts // Wake exactly like a real user saying the phrase: await injectTranscript({ sessionId, text: "hey coach" }); // ... wait for assistant.session_started in getEventLog(filter: "voice"), then: const inj = await injectAssistantUtterance({ sessionId, text: "how much further?" }); await assertToolCalled({ sessionId, name: "get_route_remaining", sinceCursor: inj.watchCursor }); ``` - `injectAssistantUtterance` drives **both** the `Mock` provider (deterministic, $0) and the real `OpenAi` provider with the same call — pick via a build flavor, never `useMock` in handler code. - The session must be **Active** when you inject — wake it first and watch for `assistant.session_started`. - Always thread `inj.watchCursor` into `assertToolCalled({ sinceCursor })` — the model fires the tool 0.5–2 s later, and anchoring the wait before the inject avoids a false "tool not called." - Camera/vision tools (`describe_scene`) need the browser tab attached — call `ensureSimulatorBrowser({ sessionId })` first, and budget a longer `timeoutMs` (camera + upload + reasoning). Lifecycle and transcripts land on the simulator's **`voice`** event-log chip (an assistant error climbs to **`errors`**). The **`ai`** chip is only for customer-side BYOK calls wrapped in `glasses.observability.aiCall(...)` — don't watch it for assistant events. See [the MCP tools reference](/docs/reference/mcp-tools) for the full agent-test surface. > **Sim fidelity.** The simulator runs the *same* library code as production with only the transport and audio path swapped, so it's designed to behave identically to hardware. That fidelity is under active validation on Android right now — treat the sim as the iteration loop and a real-hardware run as the final gate, not a guarantee. ## Where to go next - **Render on the glasses screen.** On a Ray-Ban Display, a tool can show results visually instead of (or alongside) speaking — see [render on the display](/docs/guides/display); gate on `glasses.display.isAvailable`. - **Tune the model.** `AssistantProvider.Managed(reasoningEffort = …)`, mid-session `setReasoningEffort` / `updateInstructions`, and `historyCompaction` for long conversations — all in [the assistant runtime](/docs/concepts/assistant). - **Persist memory across sessions.** `persistentMemory = true` (opt-in, consent-gated) — see [memory](/docs/concepts/assistant#memory). --- source: https://extentos.com/docs/guides/voice-triggers --- --- title: Voice triggers description: Wire a voice command on the glasses to an action in your app. Works on Meta Ray-Ban via the phone's speech recognizer over Bluetooth. Phrases auto-surface on the connection page and the simulator's click-to-fire panel. type: guide platform: all related: - /docs/concepts/capabilities - /docs/concepts/assistant - /docs/troubleshooting/voice-trigger-not-firing - /docs/reference/android-api --- **Camera and display need the Meta vendor module.** `com.extentos:glasses` carries no vendor SDK, so add `implementation("com.extentos:glasses-meta")` alongside it — see [install](/docs/sdk/android/install). Without it the build still succeeds and voice still works, but `capabilities.camera` is `false` and captures return errors. The SDK logs a warning at startup when it spots that combination. A voice trigger on Extentos is just a Kotlin lambda or a Swift closure that runs when the user says a phrase. There is no spec file, no trigger / block tree, no special registration step. Your handler class calls `glasses.voice.onPhrase("X") { ... }` and the library does the rest: subscribes to the phone's speech recognizer, matches the phrase against incoming transcripts, runs your handler, and surfaces the phrase on the host-app connection page and the simulator's right-rail click-to-fire panel. This page covers the canonical pattern, the `stops` cancellation primitive, the three usage tiers (when to drop down to raw transcripts), and the per-vendor caveats. ## The canonical pattern ```kotlin class VisionHandler(private val glasses: ExtentosGlasses) { private var registration: VoiceRegistration? = null fun start() { registration = glasses.voice.onPhrase( phrase = "describe what you see", label = "Describe scene", ) { // Your handler — runs when the user says the phrase. val photo = glasses.camera.capturePhoto().valueOrNull() ?: return@onPhrase // ...vision LLM + speak the description... } } fun stop() { registration?.cancel() } } ``` ```swift final class VisionHandler: @unchecked Sendable { private let glasses: any ExtentosGlasses private var registration: VoiceRegistration? init(glasses: any ExtentosGlasses) { self.glasses = glasses } func start() { registration = glasses.voice.onPhrase( phrase: "describe what you see", label: "Describe scene", stops: [] ) { [glasses] in let result = await glasses.camera.capturePhoto() guard case .success(let photo) = result else { return } // ...vision LLM + speak the description... } } func stop() { registration?.cancel() } } ``` That's everything. No `register` table, no callback ID, no spec edit. The library matches the phrase against `glasses.audio.transcriptions()` under the hood — case-insensitive substring on FINAL transcripts. The handler runs under the library's coroutine scope (Android) / Task (iOS). Returning from the handler closes the trigger run; the next utterance of the phrase fires it again. > **Scope note (Android).** On Android, `onPhrase` takes a `firesWhen` parameter (type `VoiceScope`) that defaults to `VoiceScope.WhenDormant`; the iOS `onPhrase` has no such parameter (iOS ships the assistant runtime but hasn't wired `VoiceScope` gating into its voice client yet). A registered phrase therefore fires **only while the assistant is dormant** — it does *not* fire during an active [assistant](/docs/concepts/assistant) session, so your wake phrases don't race with the model's tool dispatch mid-conversation. Pass `firesWhen = VoiceScope.Always` for a utility command (e.g. a "cancel everything" panic phrase) that must fire regardless of assistant state. ## Stop conditions A second voice command that cancels the first one mid-flow: ```kotlin glasses.voice.onPhrase( phrase = "play cat video", label = "Play cat video", stops = listOf("stop the video"), ) { catPlayer.play() // suspending — cancelled when user says "stop the video" } ``` ```swift registration = glasses.voice.onPhrase( phrase: "play cat video", label: "Play cat video", stops: ["stop the video"] ) { [catPlayer] in await catPlayer.play() // cancelled when user says "stop the video" } ``` `stops` is a list of phrases that, while the handler is running, will cancel the handler's coroutine / Task. The cancellation is plain Kotlin structured concurrency / Swift `Task.cancel()` — your `try/finally` or `defer` blocks run normally. For cleanup that itself suspends (releasing a `MediaPlayer`, draining a queue, sending a final `speak`), wrap it in `withContext(NonCancellable) { ... }` (Kotlin) or use a detached cleanup `Task` (Swift). `stops` is also a **UI affordance**: the simulator renders the stop phrases as nested STOP rows under the parent VOICE card, gated by `is_active` so you can only click them while the parent is running. The host-app connection page renders them as indented italic rows under the parent ("Say to me" section). The pattern works for *any* process — playing a video, muting the speaker, starting a recording, running an LLM call. The library doesn't know what the action is; the `stops` list is just metadata + a cancellation signal. ## Three usage tiers | Tier | Customer code | Auto-display | Auto-cancel on stop | |---|---|---|---| | 1 (default) | `glasses.voice.onPhrase(phrase, label, stops) { ... }` | ✅ | ✅ | | 2 (custom match, visible) | raw `transcriptions()` + `glasses.voice.registerHint(phrase, label, stops)` | ✅ | manual | | 3 (raw / hidden) | raw `transcriptions().collect { ... }` only | ❌ (by choice) | manual | Drop to tier 2 when the substring match isn't expressive enough — you need regex, state machines, or per-utterance guards. Drop to tier 3 only when you actively don't want the phrase visible on the connection page or in the simulator (rare). ```kotlin // Tier 2 — raw matching with UI affordance. // // registerHint returns a VoiceRegistration, which exposes ONLY cancel() — // it carries no id. The hint's id lives on the VoiceHint published by the // `glasses.voice.hints` StateFlow; read it from there to call reportFired(). val registration = glasses.voice.registerHint( phrase = "translate this", label = "Translate", ) val hintId = glasses.voice.hints.value .firstOrNull { it.phrase == "translate this" } ?.id glasses.audio.transcriptions().collect { t -> if (t !is Transcript.Final) return@collect if (myRegex.matches(t.text)) { hintId?.let { glasses.voice.reportFired(it) } // keep the sim's "fired N times" honest handleMatch(t.text) } } // ...later, to tear the hint down: registration.cancel() ``` ## Per-vendor caveats (Meta Ray-Ban) - **`Hey Meta` is not exposed.** Meta's system wake word goes to Meta AI; third-party apps cannot intercept it. Your wake phrase *is* whatever string you match against `transcriptions()` (directly or via `onPhrase`). - **The phone is the recognizer.** Audio captures on the glasses, streams over Bluetooth HFP/SCO to the phone, and the phone runs the recognizer — on-device Vosk on Android hardware, Apple's `SFSpeechRecognizer` on iOS hardware, and the platform recognizer (`SpeechRecognizer`/`SFSpeechRecognizer`) on the dev/sim path. No STT on the glasses themselves. - **`listening_mode` toggle gates STT.** When the user flips Voice Activation off on the connection page, `transcriptions()` stops emitting and `onPhrase` handlers never fire. This is intentional — it's the user's hard kill-switch. ## Verify it in the simulator Open `extentos.com/s/[sessionId]` after `createSimulatorSession`. The right rail shows one card per registered phrase: - **VOICE pill** + the phrase + the registration's stable id + `fired N times · Xs ago`. - **Nested STOP rows** under each parent with declared stops, disabled until the parent is `is_active = true`. - **Click the card** to inject the phrase as a synthetic `stt_transcript` — the dispatch path is identical to a real utterance, so any matcher you wrote runs the same way in sim and on real glasses. Clicking a card in the browser injects a synthetic `stt_transcript` and runs *the same dispatch code path* a real spoken phrase would — so the matcher you wrote is exercised identically. The simulator is designed to behave like the glasses here; the substrate still differs (the phone-side recognizer — on-device Vosk on Android hardware, `SFSpeechRecognizer` on iOS — over Bluetooth HFP produces the real transcripts, vs the browser's recognizer in sim), and that fidelity is under active validation on hardware. Treat a green sim run as strong evidence, then confirm phrase recognition on real glasses — see [Wake phrase not matching](/docs/troubleshooting/voice-trigger-not-firing) for the variance that only shows up on hardware. ## Common gotchas - **Substring matching is forgiving.** `onPhrase("stop")` matches "I'm going to stop now" and "let me think, stop right there." Phrase carefully or use tier-2 raw matching with a tighter regex. - **Overlapping phrases each fire.** `onPhrase("start")` and `onPhrase("start recording")` both fire on "start recording now" — each registration is independent. Order your phrases from most-specific to least, and have your handlers guard for the overlap if it matters. - **Stops only apply during handler execution.** Saying "stop the video" before any "play cat video" is dead text — no parent handler running means no listener attached. The simulator UI gates the STOP rows by `parent.is_active` for exactly this reason. - **`registerHint` does NOT auto-cancel.** Stops cancellation is exclusive to `onPhrase`. If you're writing a tier-2 handler, write the cancellation yourself. ## Related - `getCapabilityGuide(feature: "voice_command")` — minimal Kotlin + Swift snippets + the full gotcha list. - `getCodeExample(pattern: "voice_qa_assistant")` — multi-turn wake → speak → record → LLM → speak loop, the canonical voice-glasses flow. - `getCodeExample(pattern: "barge_in_speak")` — cancel TTS the moment the user starts talking. - [Android API reference](/docs/reference/android-api) — the full `glasses.voice` sub-client surface (`onPhrase`, `registerHint`, `reportFired`, the `hints`/`stats` StateFlows, `VoiceScope`). - [Assistant runtime](/docs/concepts/assistant) — when you want the model to own wake / turn-taking instead of matching phrases yourself; `onPhrase { session.wake() }` is the canonical assistant wake. - [Wake phrase not matching](/docs/troubleshooting/voice-trigger-not-firing) — the toggle/casing/recognizer checklist when a phrase never fires. --- source: https://extentos.com/docs/mcp-server --- --- title: MCP server description: The Extentos MCP server (`@extentos/mcp-server`) is an npm package an AI agent (Claude Code, Cursor, Windsurf, Cline) installs once and then uses to add Meta Ray-Ban smart-glasses capabilities to a native iOS or Android app. It exposes a tight set of deterministic tools across 10 categories — discovery, generation, agent configuration, credentials, analytics, guidance, validation, simulation, production-readiness, and documentation — plus a CLI for account linking, telemetry consent, and update checks. This is the agent's operating manual. type: overview platform: mcp related: - /docs/getting-started/with-agent - /docs/concepts/architecture - /docs/concepts/transport-vs-app - /docs/mcp-server/install - /docs/mcp-server/tools - /docs/mcp-server/configuration - /docs/mcp-server/auth - /docs/mcp-server/agents --- The MCP server is how an AI agent — Claude Code, Cursor, Windsurf, Cline, or any [Model Context Protocol](https://modelcontextprotocol.io)-compatible host — operates Extentos. The agent invokes deterministic tools; the server primes the agent on what capabilities the glasses expose, returns canonical SDK code patterns in Kotlin and Swift, scaffolds the project, brokers simulator sessions, and queries debugging traces. The server itself has **no planning tool** — agents are better planners than regex bundles. Tools are typed primitives the agent composes in sequence. This page is the section landing — what the server is, the tools at a glance, the canonical agent-driven flow, configuration knobs, the CLI, and how the auth model works. Sub-pages cover each in depth. ## Install ```bash claude mcp add extentos -- npx -y @extentos/mcp-server@latest ``` For non-Claude-Code hosts, see the [agent prompt or manual JSON](/docs/getting-started/with-agent#install) install paths. Full install reference at [/docs/mcp-server/install](/docs/mcp-server/install). ## The tools, by category The server exposes a deterministic tool surface (verified in `mcp-server/src/tools/definitions.ts`), organized into 10 categories. The categories are the agent's mental map; an agent that understands which category a tool lives in can decide when to call it. The always-current full catalog is the generated [tools reference](/docs/reference/mcp-tools). ### 1. Discovery and SDK reference (4 tools) The first calls in any new task. Cheap, all local, no side effects. | Tool | What it does | |---|---| | `getPlatformInfo` | Returns the static platform catalog — library version, the list of SDK capabilities the glasses expose, per-vendor tiers. Always the right first call. | | `getCapabilityGuide` | Per-feature minimal usage in Kotlin + Swift — call shape, config args, gotchas. Pairs with `getPlatformInfo` (which names the features) to tell the agent how to invoke each. | | `getCodeExample` | Full canonical compositions in both languages. Start with `assistant_agent_loop` (the Phase-4 canonical voice-assistant flow) and `agent_driven_e2e_full_loop` (the agent-driven E2E test). Also covers `voice_qa_assistant`, `barge_in_speak`, `photo_describe_voice`, `live_transcription_ui`, `voice_notes`, `connection_page_setup`, `byok_anthropic`, `display_browse_detail`, `display_media_gallery`, and `video_frames_ml`. Peel from these when writing handler code. The full enumerated list is generated at [/docs/reference/mcp-tools](/docs/reference/mcp-tools). | | `getMigrationGuide` | For apps already built on raw Meta DAT — returns a map keyed by your existing DAT symbols to the Extentos primitives that replace them, plus an ordered cutover plan. | ### 2. Setup and Generation | Tool | What it does | |---|---| | `generateConnectionModule` | One-shot scaffold — bootstrap module, Gradle/SPM wiring, dependencies, permissions, manifest. Two-call flow: first call without `placement` returns a question asking where `ExtentosConnectionPage` should live; second call with the chosen placement returns the full file set. | | `getConnectionPageConfig` / `setConnectionPageConfig` | Read / write the per-project connection-page config (theming tokens + section visibility) the dashboard/server holds. | | `regenerateConnectionPageFile` / `adoptConnectionPageFile` | Sync the committed `extentos.connection-page.json` with the server config — regenerate (server→file) or adopt (file→server). | After scaffolding, the agent writes its own handler classes against the SDK primitives surfaced by `getCapabilityGuide` / `getCodeExample`. Handler code is the customer's authoring surface — there's no `initSpec` or DSL-population step. ### 3. Agent configuration & usage (5 tools) Account-scoped — require a linked account and are gated per-project by the [MCP access grant](/docs/mcp-server/auth) (default Read+Write). | Tool | What it does | |---|---| | `getAssistantConfig` / `setAssistantConfig` | Read or change a project's dashboard-managed assistant settings — the OpenAI Realtime model, the voice, the memory (compaction) model, and the within-session memory mode. `set` is a partial update, validates each value against the catalog, and echoes the cost impact of a model change. | | `getGatewayUsage` | Read the project's managed-gateway usage + exact cost over a recent window — token counts and list-price USD from the billing ledger, broken down by model. Metadata only, never transcripts or content. | ### 4. Credentials (2 tools) Account-scoped + per-project gated. Write-without-knowing — the secret never passes through the agent. | Tool | What it does | |---|---| | `getCredentialStatus` | Read whether a project's Meta DAT build identity is set — a masked hint + updated-at only, never the value. | | `setCredential` | Start write-without-knowing entry of a credential — returns a dashboard link where the signed-in owner pastes the secret straight into the encrypted vault. Takes no secret argument by design. | ### 5. Analytics (1 tool) | Tool | What it does | |---|---| | `getProjectAnalytics` | Read a project's **production** analytics — aggregate telemetry from shipped App Store / Play Store installs (events, active installs, by event / day / vendor / platform). Metadata only, account-scoped, ownership-checked, gated by the Analytics grant. Empty until the app ships and sends prod-attested events (use `getEventLog` for the live dev/sim stream). | ### 6. Implementation Guidance (2 tools) Side-quest tools the agent calls during composition. | Tool | What it does | |---|---| | `getVoiceCommandGuidance` | Analyze proposed wake / command phrases for UX issues (collisions, ambiguity, hard-to-recognize words, Meta wake-word conflicts) before wiring them into a `glasses.audio.transcriptions()` consumer. | | `getPermissions` | Derive exact platform permissions, Meta DAT requirements, and foreground-service needs from the capability list. Run when adding or removing a primitive from your handler. | ### 7. Validation (2 tools) Correctness gates. Run after structural changes (new capability declared, dependency bumped, manifest edited). | Tool | What it does | |---|---| | `inspectIntegration` | Read-only project snapshot — manifest, generated-file hashes, dependency list, connection-page config. Run before manual edits to understand current state. | | `validateIntegration` | Whole-project correctness check — manifest, generated files, dependency declared, permissions cover declared capabilities, bootstrap calls `ExtentosGlasses.create(...)`, toolchain versions, foreground-service hints for continuous-capture flows. The pre-test gate. | ### 8. Simulation Provision and operate browser-based simulator sessions, plus the agent-driven testing tools that close the end-to-end loop without a human. | Tool | What it does | |---|---| | `createSimulatorSession` | Get-or-create a browser-mode session at [extentos.com/s](https://extentos.com/s). Returns the saved sim for this project if one exists (`status: "resumed"`), or mints a new one (`status: "active"`). Auto-attaches the running app via the local bridge when reachable; otherwise emits a `BuildConfig.EXTENTOS_SESSION_URL` snippet (Android) or `extentos.session.plist` payload (iOS). Rotating the sessionId is `deleteSimulatorSession` then mint — there is no force-fresh flag. | | `ensureSimulatorBrowser` | Open + confirm a connected simulator browser tab — the precondition for camera and inject flows. | | `completeAuthLink` | After `createSimulatorSession` returns `status: "auth_required"` (anonymous install needs to link to mint sessions), polls the backend until the user finishes signup, then persists the bearer token to `~/.extentos/auth.json`. | | `getEventLog` | Fetch structured event traces from a session. Filter values: `all` (no filter) plus the seven chips `errors`, `voice`, `camera`, `display`, `ai`, `lifecycle`, `custom` — one chip per event, with `errors` absorbing severity≥warn regardless of modality. Plus `cursor`, `follow`, `limit` for trace-level scope. The primary debugging tool. | | `getSimulatorStatus` | Read a live session's current state — phase, hardware-ready, attached roles, active capability streams, current toggle values. | | `injectTranscript` / `injectAssistantUtterance` / `assertToolCalled` | Drive a wake phrase or an assistant turn, then assert which tool the model called — the agent-driven E2E loop, no human needed. | | `injectHardwareButton` | Press the simulated glasses' hardware capture button — tap pauses/resumes a live camera stream, hold stops it — so the agent can exercise the wearer's privacy gestures (and test the resulting `CaptureError.StreamPaused`) with no human. | | `setSimVideo` / `setSimDevice` | Pipe a test video into the simulated camera; switch the simulated device model (e.g. `rayban_display` to exercise the display path). | | `getDisplayState` / `injectInput` | Read the display tree currently rendered + drive display input (`select` / `navigate` / `back`). | ### 9. Production (2 tools) Pre-ship checks. | Tool | What it does | |---|---| | `getProductionChecklist` | Personalized production-readiness checklist based on declared capabilities + handler names — credential wiring, permission audit, foreground-service requirements (when continuous capture is used), simulator-URL removal from release builds, store-listing readiness. | | `getCredentialGuide` | Step-by-step credential setup for production AI providers — `anthropic`, `openai`, `google_cloud_vision`, `google_translate`, `google_gemini`, `deepl`, `azure_cognitive`, `aws_bedrock`, `huggingface`, or `custom` — plus Meta DAT registration. | ### 10. Documentation & search (1 tool) | Tool | What it does | |---|---| | `searchDocs` | Search Extentos documentation by topic or keyword. For voice assistants read `assistant_runtime` first. Other aligned topics: `voice_integration`, `agent_e2e_testing`, `managed_gateway`, `conversation_memory`, `display`, plus the stable conceptual set — `getting_started`, `custom_handlers` (the canonical SDK-composition doc), `simulator_browser_mode`, `simulator_session_lifecycle`, `event_log_schema`, `toggles`, `library_api`, `permissions`, `multi_platform_projects`. Topic IDs are stable; the live tool input is authoritative. | Full per-tool reference with input schemas, response shapes, and worked examples: [/docs/mcp-server/tools](/docs/mcp-server/tools). ## The canonical agent-driven flow In a fresh project, the agent invokes the tools in this order: ```text 1. getPlatformInfo({ sections: ["version", "capabilities"], glasses: "meta_rayban" }) 2. getCodeExample({ pattern: "assistant_agent_loop" }) // Phase-4 voice assistant; or whatever pattern fits 3. getCapabilityGuide({ feature: "" }) 4. generateConnectionModule({ platform, glasses, appPackage }) → returns "needs_placement" question 5. generateConnectionModule({ ... placement: "" }) → writes scaffold files (ExtentosBootstrap, manifest, etc.) 6. against the SDK primitives 7. validateIntegration() → ✓ all good (or returns structured errors to fix) 8. createSimulatorSession({ glasses }) → returns sessionId; auto-opens browser at extentos.com/s/ → if running app is reachable via local bridge, it auto-attaches 9. 10. getEventLog({ sessionId, filter: "errors" }) → debug getSimulatorStatus({ sessionId }) → status ``` For iteration: edit handler code → rebuild + reinstall → the app auto-attaches to the same simulator session (no remint, the URL is stable). Before shipping: `getProductionChecklist` and `getCredentialGuide`. ## Configuration The MCP server reads these environment variables (verified in `mcp-server/src/`): | Variable | Default | What it does | |---|---|---| | `EXTENTOS_BACKEND_URL` | Production backend | Override the backend URL (`tools/util/backendClient.ts`). For local development of Extentos itself. | | `EXTENTOS_CONFIG_DIR` | `~/.extentos` | Override the config/auth directory (`telemetry/consent.ts`). | | `EXTENTOS_TELEMETRY` | unset (consent-default) | Set to `0` to decline telemetry without running the CLI consent command. | | `EXTENTOS_NO_AUTO_OPEN` | unset | Set to `1` to disable browser auto-open on simulator-session creation (useful in headless environments). | Full configuration reference: [/docs/mcp-server/configuration](/docs/mcp-server/configuration). ## CLI subcommands Running `npx @extentos/mcp-server@latest` with no arguments starts the MCP server over stdio (the path the agent uses). With a subcommand, it acts as a developer CLI: | Subcommand | What it does | |---|---| | `login` | Link this install to an Extentos account via device-code flow (proactive — useful before the first simulator session, or after `logout` to re-link). | | `logout` | Clear `~/.extentos/auth.json`. Install returns to the anonymous tier; the next simulator session call will re-trigger the device-code flow. | | `whoami` | Not yet implemented (Phase-0 stub). Will print `installId`, `accountId` (if linked), tier, auth expiry. | | `setup` | Preflight the local build environment — checks the GitHub Packages PAT (`read:packages`) the Meta DAT artifacts need, for apps that depend on `com.extentos:glasses-meta`, plus other dependency prerequisites. | | `accept-privacy` | Record privacy consent (enables telemetry upload). | | `decline-privacy` | Record privacy decline (disables telemetry upload). | | `status` | Print consent state, install ID, linked account, MCP/library versions. | | `update` | Check for MCP server updates (no-op on `npx @latest` installs). | Full CLI reference: [/docs/mcp-server/auth](/docs/mcp-server/auth). ## Auth model The MCP server is **anonymous-first**. Discovery, capability guides, code examples, validation, docs search, on-device simulation, and real-hardware testing all work without ever signing in. Three things link a free account: minting browser-simulator sessions (`createSimulatorSession`, HTTP 402), the `generateConnectionModule` scaffold step (it mints your account-bound project key — same 402 device-code flow; the informational first call is anonymous), and the account-scoped project tools (assistant config, credentials, connection-page writes, analytics — HTTP 401). The device-code flow: the first gated call returns `status: "auth_required"` with a verification URL. The agent calls `completeAuthLink` to poll the backend; the developer signs up with a free email-only account at the URL (Google or email + password, no payment); the backend issues a token; the original tool call retries automatically. After linking, simulator sessions are unlimited. Extentos's tools, codegen, validation, SDK, and the browser simulator are free — there's no per-seat or subscription fee to build and ship. The one surface that meters usage is the **managed AI gateway** behind the Phase-4 voice assistant. Full auth model: [/docs/mcp-server/auth](/docs/mcp-server/auth); pricing: [/docs/resources/pricing](/docs/resources/pricing). ## Privacy and telemetry On first run, the MCP server injects a one-time privacy notice into the response. Telemetry is anonymous (tagged with `installId`, no source code or personal data) and is dismissed-by-continuing by default — same pattern as Vercel CLI, Astro, Vite. Decline at any time: ```bash npx @extentos/mcp-server@latest decline-privacy # or EXTENTOS_TELEMETRY=0 (env var, persistent for the shell) ``` Privacy notice content is in `mcp-server/src/index.ts` (`PRIVACY_NOTICE` constant). Notice is shown once per install via `claimFirstPrivacyNotice` — never repeats. ## Compatible MCP hosts Verified to work with: - **Claude Code** — primary target. One-line install via `claude mcp add`. - **Cursor** — JSON config in `~/.cursor/mcp.json`. - **Windsurf** — JSON config in `~/.codeium/windsurf/mcp_config.json`. - **Cline** — JSON config in Cline's MCP settings. - **Any MCP-compatible host** — drop the standard `mcpServers.extentos` JSON block in. The MCP server speaks the standard MCP protocol over stdio (`@modelcontextprotocol/sdk`); no host-specific code paths exist on the server side. Per-host install steps: [/docs/mcp-server/agents](/docs/mcp-server/agents). ## The local bridge — auto-bind dev loop When the server starts, it opens a `127.0.0.1:31337/whoami` HTTP listener (`mcp-server/src/localBridge.ts`). The Extentos library on the developer's app probes this endpoint at runtime to learn its host MCP's `installId`. The result: every `createSimulatorSession` call from the agent auto-attaches the running app to the new session — no rebuild, no URL paste. Reach paths: - **Android emulator:** `http://10.0.2.2:31337/whoami` (host-loopback NAT alias) - **iOS Simulator:** `http://localhost:31337/whoami` (shares host network namespace) - **Physical Android phone via USB:** `adb reverse tcp:31337 tcp:31337` once, then `localhost:31337` from the device - **Cellular phone or cloud-hosted agent:** probe times out. The agent uses the URL-bake path instead — `createSimulatorSession` returns a `BuildConfig.EXTENTOS_SESSION_URL` snippet (Android) or `extentos.session.plist` payload (iOS) the developer pastes in, then rebuilds the app once. Less elegant than auto-bind but works on any topology. Bound to `127.0.0.1` only. The `installId` is not a secret — it's the same value the MCP sends to `api.extentos.com` on every tool call. No auth needed at this layer. If port `31337` is in use (rare; another MCP instance already running), startup logs a warning and continues. Auto-bind silently fails for that session; the developer uses the URL-bake path until the port frees up. ## Status - **Package:** [`@extentos/mcp-server`](https://www.npmjs.com/package/@extentos/mcp-server) on npm (MIT license) - **Engines:** Node.js 20+ - **Pre-1.0** — APIs may shift between minor versions until the hardware test loop closes. Pin to an exact version if you need cross-session reproducibility. ## Related - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install the server and walk through a real dev loop - **[Tools reference](/docs/mcp-server/tools)** — full per-tool API - **[Configuration](/docs/mcp-server/configuration)** — environment variables, config files, install-time settings - **[Auth](/docs/mcp-server/auth)** — device-code flow, account linking, CLI auth commands - **[Supported agents](/docs/mcp-server/agents)** — per-host install instructions - **[Architecture](/docs/concepts/architecture)** — how the MCP server fits into the broader Extentos system - **[Transport vs app simulation](/docs/concepts/transport-vs-app)** — what the simulator the MCP brokers actually does --- source: https://extentos.com/docs/mcp-server/agents --- --- title: Supported agents description: Per-host differences for AI coding agents that run Extentos — Claude Code, Cursor, Windsurf, Cline, IDE built-ins. Scope, cross-platform reach, MCP support. type: tutorial platform: mcp related: - /docs/mcp-server - /docs/mcp-server/install - /docs/getting-started/with-agent - /docs/concepts/architecture --- Extentos generates **native mobile app code** (Swift for iOS, Kotlin for Android), so the agents that work with it are the ones that target native mobile development with Model Context Protocol support. Web-first AI builders like Replit Agent, Lovable, Bolt.new, or v0.dev aren't relevant for Extentos work — not because they lack MCP, but because they ship web apps, not native mobile. The MCP-capable agents that DO work are surprisingly portable: most of them run **anywhere you can run their interface** — Claude Code in particular runs in any terminal, which includes the terminal panel inside Android Studio, Xcode, IntelliJ, VS Code, or any IDE you already use. So the question isn't "which IDE supports Extentos?" — it's **"which AI agent supports MCP, and where can I run it?"** This page covers the per-agent capability differences and the IDE-portability picture. ## Quick recommendation | If you're using... | What works | |---|---| | **Claude Code** as your agent | Best general fit. Runs in any terminal (standalone, Android Studio's panel, Xcode's terminal, VS Code's, your favorite shell). Filesystem-wide reach from the working directory; can edit iOS + Android source simultaneously. One CLI command to install. | | **Cursor** or **Windsurf** | Great if you live in a VS Code-style IDE. Multi-folder workspace covers cross-platform Extentos. JSON config install. | | **Cline** (VS Code extension) | Works well. More explicit per-tool approval; good if you want to confirm each MCP call. | | **Android Studio's built-in Gemini, Xcode AI features, JetBrains AI Assistant** | If your IDE's built-in agent has MCP support (verify with the IDE's current docs — this space moves fast), Extentos works there directly. If it doesn't yet, you can run Claude Code (or any MCP agent) in the IDE's terminal panel — your IDE's editor and build tooling still function as normal. Either way, you don't have to leave the IDE. | | **Replit Agent, Lovable, v0.dev, Bolt.new** | Out of scope — these build web apps, not native iOS/Android. Wrong tool for Extentos in the first place. | ## Tier 1 — recommended for Extentos work These four hosts have native, mature MCP support and have been verified end-to-end with Extentos. ### Claude Code (terminal CLI) **Best for:** cross-platform Extentos work where the agent edits files across `ios-app/`, `android-app/`, and any shared library or backend code in one coherent session. | Capability | Behavior | |---|---| | Project scope | Whole filesystem from your current working directory. Agent reads and edits any file in or below `cwd`. | | Cross-platform iOS + Android | ✅ Native. Single agent session edits both platforms simultaneously. | | Shell exec | First-class. Agent runs `npx`, `gradle`, `swift build`, `claude mcp add` directly. | | MCP install | One CLI command — `claude mcp add extentos -- npx -y @extentos/mcp-server@latest` | | Tool output rendering | Inline in terminal. Long-running tools (e.g. `completeAuthLink` polling for 600s) show a clear waiting state. | | Browser auto-open | Native via `open` (macOS) / `xdg-open` (Linux) / `cmd start` (Windows). | | If browser auto-open fails | Extentos prints the verification URL or simulator URL directly in the agent's response. The developer copies it into their browser. | | Auto-bind reach | Always works — MCP server runs on the developer's machine, the same machine the developer's emulator or USB-tethered phone runs on. | | MCP server crash recovery | Auto-restarts the server transparently. | **Recommended setup:** install with the CLI command above, run Claude Code from your project root, you're set. ### Cursor **Best for:** developers who already live in a VS Code-style IDE and want Extentos integrated into their existing editor workflow. | Capability | Behavior | |---|---| | Project scope | VS Code workspace. Agent reads and edits files in any folder added to the workspace. | | Cross-platform iOS + Android | ✅ if both `ios-app/` and `android-app/` are added to the workspace as folders. Cursor handles multi-folder workspaces well. | | Shell exec | Yes, with optional approval. | | MCP install | JSON config in `~/.cursor/mcp.json` (global) or `.cursor/mcp.json` (per-project) | | Tool output rendering | In the agent chat panel. Long-running tools show progress; some tools' raw JSON output may be more verbose than in Claude Code. | | Browser auto-open | Works — Cursor delegates to the OS default browser. | | If browser auto-open fails | Agent surfaces the URL in the chat panel; developer copies it. | | Auto-bind reach | Works — same-machine MCP + library = reach. | | MCP server crash recovery | Auto-restarts. UI surfaces the error in the MCP panel. | **Recommended setup:** add Extentos via the JSON config (see [install](/docs/mcp-server/install#cursor)), open both `ios-app/` and `android-app/` as workspace folders if you're working cross-platform. ### Windsurf **Best for:** same audience as Cursor — IDE-embedded workflow, VS Code-style. Capability profile is essentially identical to Cursor: | Capability | Behavior | |---|---| | Project scope | VS Code workspace, multi-folder supported | | Cross-platform iOS + Android | ✅ if both folders are in the workspace | | Shell exec | Yes, with approval | | MCP install | JSON config in `~/.codeium/windsurf/mcp_config.json` | | Browser auto-open | Works | | If browser auto-open fails | URL appears in the agent chat panel | | Auto-bind reach | Works on same-machine setup | **Recommended setup:** see [install](/docs/mcp-server/install#windsurf). ### Cline (VS Code extension) **Best for:** developers using Cline inside VS Code who want explicit per-tool approval before the agent acts. | Capability | Behavior | |---|---| | Project scope | VS Code workspace | | Cross-platform iOS + Android | ✅ if both folders are open | | Shell exec | Yes, but per-command approval is more explicit than Claude Code or Cursor | | MCP install | Cline UI panel or `cline_mcp_settings.json` config file | | Tool output rendering | In Cline panel. The per-tool approval flow can interrupt long-running tools more often. | | Browser auto-open | Works via VS Code's URL handler | | If browser auto-open fails | Agent prints the URL in the Cline panel | | Auto-bind reach | Works on same-machine setup | | MCP server crash recovery | Manual reload sometimes needed; UI surfaces errors clearly | **Recommended setup:** add Extentos via Cline's MCP Servers tab — see [install](/docs/mcp-server/install#cline-vs-code-extension). **One Cline-specific gotcha:** the per-tool approval flow can be tedious during the canonical Extentos sequence (`getPlatformInfo` → `getCodeExample` → `getCapabilityGuide` → `generateConnectionModule` → `validateIntegration` → `createSimulatorSession`). Set Cline to auto-approve Extentos tools after the first run if you want the smoother flow. ## Tier 2 — works, less battle-tested with Extentos These hosts have MCP support but haven't been verified end-to-end with the full Extentos workflow. They should work — Extentos uses standard MCP — but expect rougher edges. | Host | Notes | |---|---| | **VS Code (native MCP)** | VS Code added native MCP support late 2025. Works with the standard `mcpServers` JSON pattern. Tool output rendering is less polished than the dedicated agent IDEs. | | **Continue.dev** | Open-source AI coding extension for VS Code with native MCP support. Works with the standard JSON config. | | **Zed** | Fast editor with MCP support added in 2025. Lighter than VS Code-based hosts; the agent surface is more minimal. | | **Aider** | Terminal-based git-aware AI coding tool with MCP support. Similar workflow to Claude Code but more constrained to git-tracked files. | For all of these, use the [generic MCP host](/docs/mcp-server/install#generic-mcp-host) JSON snippet. ## Built-in IDE agents — Gemini, Xcode AI, JetBrains AI If you're in Android Studio, IntelliJ, or Xcode, these IDEs have their own built-in AI assistants (Gemini in Android Studio, Xcode AI features, JetBrains AI Assistant). MCP adoption across these built-in agents has been moving — **the only correct answer to "does it support MCP?" is to check the IDE's current documentation**, since this space is shifting month-to-month in 2026. Two scenarios, both of which keep you in your IDE: **If the built-in agent supports MCP** — register Extentos via whatever MCP config mechanism the IDE provides (typically a JSON file path the IDE points at). The agent calls Extentos's tools directly inside your IDE. Same `mcpServers` JSON pattern as Cursor / Windsurf works in any standards-compliant MCP host. **If the built-in agent doesn't support MCP yet** — every one of these IDEs ships with a built-in terminal panel, and Claude Code (or any MCP-capable agent) runs there: - **Android Studio:** open the bottom Terminal tab → run `claude` (or `npx @extentos/mcp-server` directly) → Claude Code now operates on your project. Cross-platform reach because Claude Code reads the filesystem from `cwd`. - **Xcode:** Xcode's built-in terminal access is more limited. Most developers run a separate terminal app side-by-side with Xcode (e.g., iTerm2) and let Claude Code drive the Extentos work there. Xcode keeps doing what it's good at — code reading, builds, simulator runs, debugging. - **IntelliJ / WebStorm / etc.:** same as Android Studio — built-in terminal hosts Claude Code or any other MCP agent. The pattern: **the IDE's built-in agent is one of many possible agents you can use, not the only one**. Either the built-in supports MCP and you call Extentos directly, or you bring your own agent in the same IDE's terminal. Your editor, build tools, breakpoint debugger, and simulator keep working unchanged — only the AI assistant choice differs. > **Verify current MCP support yourself.** The doc above reflects the *project's design intent* (Extentos is built for any MCP-compatible client including Gemini in Android Studio per the architectural plan). What's *currently shipped* by IDE vendors changes faster than this page updates. Check Google's Gemini in Android Studio docs, Apple's Xcode AI release notes, or JetBrains' AI Assistant docs for the current MCP support status before assuming. ## What's actually different per host for Extentos work Distilled into the Extentos-specific capabilities that matter most: ### 1. Cross-platform reach This is the biggest difference. Extentos generates code in: - The developer's iOS app source (Swift) - The developer's Android app source (Kotlin) - The shared integration config (JSON) — `extentos.manifest.json` and `extentos.connection-page.json` A host that can read and edit both platforms simultaneously is unconstrained. A host locked to one platform forces context-switching between two agent sessions. | Host class | Cross-platform reach | |---|---| | Terminal CLI (Claude Code, Aider) | ✅ Whole filesystem from cwd | | VS Code-based (Cursor, Windsurf, Cline, VS Code, Continue.dev) | ✅ if both platform folders are in the workspace | | Single-platform IDE (Android Studio, Xcode) | ❌ Locked to one platform's project tree | ### 2. Browser auto-open behavior When `createSimulatorSession` is called (or the device-code auth flow fires), the MCP server attempts to open a browser tab to the URL — `open` on macOS, `xdg-open` on Linux, `cmd start` on Windows. This is **best-effort**, and **the URL is always also returned in the structured tool response regardless of whether the OS browser command succeeded**. So when browser auto-open *can't* happen — `EXTENTOS_NO_AUTO_OPEN=1` is set, the agent runs in a headless / SSH / sandboxed environment, the OS command isn't available, the agent host doesn't have OS-level browser-launch permission — **the agent simply surfaces the URL in its response and the developer copies it into their browser manually**. Nothing is lost; auto-open is a UX nicety, not a requirement. How the URL appears varies by host: - **Claude Code (terminal)** — printed as plain text in the terminal output, often with surrounding context like *"Open this URL in your browser: https://extentos.com/s/..."* - **Cursor / Windsurf / Cline** — appears inline in the chat panel, typically rendered as a clickable link the developer can click directly - **Headless / cloud-hosted agents** — printed in whatever output channel the agent uses; developer copies and pastes into a local browser The flow is always: tool returns URL in response → agent surfaces URL → developer opens it (whether by auto-open, click, or copy-paste). No failure mode here — the URL is always available. ### 3. Auto-bind reach (the localhost:31337 bridge) Auto-bind requires the MCP server and the developer's app to be reachable on the same `127.0.0.1` interface. In the standard "developer on a laptop, emulator or USB-tethered phone connected to the same laptop" setup, this just works for any of the hosts above. It does **not** work when: - The developer's phone is on a cellular network without USB tethering and `adb reverse tcp:31337 tcp:31337` hasn't been run - The agent host is cloud-hosted (e.g., Claude.ai/code's web sandbox) with the MCP server running in the cloud while the developer's app runs on their local machine - Port `31337` is already in use by another MCP instance When auto-bind isn't reachable, **the agent uses the URL-bake path**: `createSimulatorSession` returns a `BuildConfig.EXTENTOS_SESSION_URL` snippet (Android) or `extentos.session.plist` payload (iOS) that the agent writes into the developer's app, and the developer rebuilds once. One rebuild step versus the zero-rebuild auto-bind happy path. ### 4. Long-running tool handling `completeAuthLink` polls for up to 600 seconds. `createSimulatorSession` can take a few seconds to provision. Hosts vary in how they surface this wait: - **Claude Code:** prints "Polling for up to 600s..." in the terminal; clear and unambiguous - **Cursor / Windsurf:** show a spinner or progress indicator in the chat panel - **Cline:** shows the long-running tool prominently with a cancel button All four handle the wait correctly — none of them time out tool calls during normal Extentos operation. ### 5. Tool approval UX How permissive the host is about letting the agent invoke tools without explicit approval: | Host | Approval UX | |---|---| | Claude Code | Most permissive. Most tools auto-approve after the first run. | | Cursor / Windsurf | Middle ground. First-time tool approvals; subsequent calls auto-approve within the session. | | Cline | Most explicit. Per-tool approval until the user toggles auto-approve for specific tools. | For Extentos, the canonical agent flow uses ~7 tools in sequence on a fresh project. Cline's per-tool approval can be 7 clicks; Claude Code's after-first auto-approve is closer to zero. Both work; the choice is workflow preference. ## Frequently asked questions ### Can I use Extentos with Android Studio? Yes. Open Android Studio's built-in Terminal panel (bottom dock) and run Claude Code there — Claude Code is just a CLI, and it works in any terminal including Android Studio's. The MCP server attaches to your Claude Code session, and you can edit cross-platform code with simultaneous reach to both `ios-app/` and `android-app/`. Android Studio's own Gemini integration doesn't support MCP yet, but you don't have to use Gemini — any MCP-capable agent works in the same terminal panel. ### Can I use Extentos with Xcode? Yes — same idea. Run Claude Code (or another MCP-capable agent) in a terminal window alongside Xcode. Xcode keeps doing what it's good at (code reading, building, running the iOS Simulator); the MCP-capable agent handles the Extentos work in the terminal. They coexist cleanly because they don't compete for the same surface. ### What if my host doesn't support MCP at all? Then it can't run Extentos. The MCP protocol is the integration boundary; without it, there's no way for the agent to call Extentos's tools. Options: switch to an MCP-supporting host (Claude Code is the smoothest install), or use a Tier-1 host alongside your existing tool. ### Can I use multiple agent hosts at the same time? Yes — Extentos's MCP server doesn't care which host calls it. Each host runs its own MCP server instance via npx; the per-machine `installId` is shared (so they share the same install identity and linked account), but otherwise the sessions are independent. Useful pattern: Cursor for primary development, Claude Code in a side terminal for one-off Extentos operations. ### Does the auto-bind dev loop work with cloud-hosted agents? No, by design — auto-bind requires the MCP server's `127.0.0.1:31337` listener to be reachable from the developer's app. A cloud-hosted agent's MCP server runs on a remote machine; the developer's emulator or phone runs locally. The two are in different network namespaces. Use the URL-bake path instead — the agent generates the `EXTENTOS_SESSION_URL` snippet and the developer pastes it locally. ### Does Gemini in Android Studio support Extentos directly? The project's design scope includes Gemini in Android Studio as a target host (per the internal architecture plan that explicitly lists "Gemini in Android Studio" alongside Claude Code, Cursor, and Codex). Whether it currently has MCP host support depends on Google's most recent Android Studio release — Google's AI integration has been evolving fast in 2026, so check the current Gemini in Android Studio docs for MCP support status. If MCP is supported, Extentos works directly. If it's not yet, run Claude Code (or any MCP-capable agent) in Android Studio's Terminal panel — same workflow either way. ### What about Replit, Lovable, v0.dev, Bolt.new? These are AI builders for **web apps** (React, Next.js, etc.), not native mobile. Extentos generates native iOS Swift and Android Kotlin code — wrong target for these tools. Not an MCP support issue; just a different category of product. If you're building a web app, Extentos isn't the right fit at all (your wearer experience would have to be a mobile-web app, which Meta DAT and the public smart-glasses toolkits don't support today). ### What about GitHub Copilot, Cody? | Host | MCP support | |---|---| | GitHub Copilot | MCP support is rolling out across Copilot surfaces (VS Code agent mode first) — check your IDE's Copilot agent-mode docs for current status | | Sourcegraph Cody | Evolving — check Sourcegraph's current docs for MCP server support status | Neither has been verified end-to-end with Extentos yet; where MCP support is present, the standard `mcpServers` JSON pattern should apply. Track the [changelog](/docs/resources/changelog) for updates as new hosts get verified. ### Does Extentos work with web app builders like v0.dev or Lovable? No. Extentos targets native iOS and Android apps that connect to smart glasses over Bluetooth via Meta DAT. Web app builders ship to a different runtime (browser-based React apps, etc.) where the smart-glasses connection isn't possible through public toolkits. The mismatch is at the artifact level, not the integration level — even if Lovable shipped MCP support tomorrow, the apps it generates couldn't talk to Ray-Ban Meta glasses. ## Related - **[Install the MCP server](/docs/mcp-server/install)** — per-host install commands and JSON snippets - **[MCP server overview](/docs/mcp-server)** — the tool catalog and canonical agent flow - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install + first prompt + verify - **[Architecture](/docs/concepts/architecture)** — how the MCP, library, backend, and your app fit together --- source: https://extentos.com/docs/mcp-server/auth --- --- title: Auth description: How Extentos auth works. Discovery, guidance, validation, and search run with no account; a free account (linked via the OAuth 2.0 device-code flow) unlocks browser-simulator sessions, the scaffold step, and the account-scoped project tools. No payment. type: reference platform: mcp related: - /docs/mcp-server - /docs/mcp-server/tools/simulation - /docs/resources/pricing - /docs/getting-started/with-agent --- The Extentos auth model is **anonymous-first, with a free account for the backend-backed surfaces**. Discovery, guidance, validation, docs search, on-device `LocalSimTransport` simulation, and real-hardware testing through `RealMetaTransport` work with **no account, no email, no payment** — forever. Three things require sign-in: minting browser-simulator sessions at [extentos.com/s](https://extentos.com/s), the `generateConnectionModule` scaffold step (it mints your account-bound project key), and the account-scoped project tools (assistant config, credentials, connection-page writes, analytics). The first time your agent makes a gated call (e.g. `createSimulatorSession`), the backend returns `auth_required` and the device-code flow fires to link a free email-only account. The agent handles the handoff programmatically via the `completeAuthLink` MCP tool — you sign in once in your browser (Google one-click or email + password), the bearer token persists to `~/.extentos/auth.json`, and the original tool call retries automatically. From then on, sessions are unlimited. **No manual token paste, no payment at any tier.** This page covers the identity states, the device-code mechanics, the CLI subcommands, the state files, and the env-var dials. ## Identity states Extentos has two operational identity states (plus a placeholder for future paid features): | State | Identifier | What you can do | Cost | |---|---|---|---| | **No account** | `installId` (per-machine, at `~/.extentos/install.json`) | Discovery, guidance, validation, and search tools. `LocalSimTransport`. Real-hardware testing through `RealMetaTransport` (your own Meta credentials). | Free forever | | **Free account** | `accountId` (linked via device-code flow) | Everything above, plus the `generateConnectionModule` scaffold (it mints your account-bound project key), minting browser-simulator sessions, and using the managed AI gateway. Email + Privacy Policy acceptance only. | Free, no payment | | Managed AI gateway | `accountId` + attestation | Voice-assistant AI on Extentos's key. Usage metered on a prepaid-credit model (every account starts with $2 free). | Metered ([pricing](/docs/resources/pricing)) | The `installId` is anonymous and never tied to your identity. The `accountId` is created when you sign up at the verification URL during the device-code flow. The gated surfaces are the ones that touch Extentos's backend — the simulator runs on Extentos infrastructure and the scaffold step mints an account-bound project key; every other surface either runs locally or talks directly to Meta. See [pricing](/docs/resources/pricing) for the full breakdown. ## The device-code flow Extentos uses the **OAuth 2.0 Device Authorization Grant** ([RFC 8628](https://datatracker.ietf.org/doc/html/rfc8628)) — the standard pattern for headless / CLI scenarios where the application can't perform a browser redirect. AI coding agents are exactly that case: the agent runs in a terminal, the developer's browser is somewhere else. ### Sequence diagram ```text Developer Agent MCP server Backend Verification URL (browser) │ │ │ │ │ │ "build app" │ │ │ │ │─────────────▶│ │ │ │ │ │ │ │ │ │ │ createSimulator- │ │ │ │ │ Session(...) │ │ │ │ │───────────────────▶│ │ │ │ │ │ POST /api/.../ │ │ │ │ │ session │ │ │ │ │─────────────────▶│ │ │ │ │ │ │ │ │ │ auth_required, │ │ │ │ │ deviceCode, │ │ │ │ │ userCode, │ │ │ │ │ verificationUrl│ │ │ │ │◀─────────────────│ │ │ │ { status: │ │ │ │ │ "auth_required" │ │ │ │ │ verificationUrl, │ │ │ │ │ userCode, │ │ │ │ │ deviceCode } │ │ │ │ │◀───────────────────│ │ │ │ │ │ │ │ │ │ open browser │ │ │ │ │ at verificationUrl │ │ │ │ │─────────────────────────────────────────────────────────▶ │ │ │ │ │ │ completeAuthLink({ deviceCode }) │ │ │───────────────────▶│ │ │ │ │ │ poll /api/.../ │ │ │ │ │ token │ │ │ │ │◀────────────────▶│ pending, pending │ │ │ │ │ signs up at verificationUrl ────────────────────────────────────────────▶│ │ │ confirms signup │ │ │◀───────────────────│ │ │ │ │ │ │ │ │ poll → │ │ │ │ │ authToken, │ │ │ │ │ accountId, │ │ │ │ │ tier │ │ │ │ │◀─────────────────│ │ │ │ persist to │ │ │ │ │ ~/.extentos/ │ │ │ │ │ auth.json │ │ │ │ │ │ │ │ │ │ retry │ │ │ │ │ createSimulator- │ │ │ │ │ Session │ │ │ │ │───────────────────▶│ │ │ │ │ │ uses bearer token │ │ │ │ session ready │ │ │ │ │◀───────────────────│ │ │ ``` ### What the agent should show the user When `createSimulatorSession` returns `auth_required`, the agent receives both `userCode` (an 8-character XXXX-XXXX string, e.g. ABCD-1234) and `verificationUrl` (`https://extentos.com/activate?code=ABCD-1234` — the URL pre-fills the code). The agent **must surface both** to the user, with explicit guidance to cross-check the code on the verification page matches: ``` Sign in to link this terminal: 1. Open https://extentos.com/activate?code=ABCD-1234 (browser auto-opening) 2. Confirm the code on the page reads ABCD-1234 (matches what's above) 3. Sign in (Google or email) and click Approve I'll detect the link and continue automatically. ``` This is an [RFC 8628 §3.3](https://datatracker.ietf.org/doc/html/rfc8628#section-3.3) anti-phishing cross-check. The verification page renders the user code as a small inline cross-check line ("Verify this matches your terminal: ABCD-1234") above the signup form — it is **not** an entry field, since the URL already carries the code as a query parameter. The user's job is just to confirm what they see on the page matches what their agent printed before clicking Approve. Without this cross-check, a phishing URL with a different valid code would let an attacker have the user approve linking the attacker's device. The agent's `fixHint` in the tool response carries this guidance verbatim — agents that surface `fixHint` directly to the user (Claude Code, Cursor, Cline default behavior) get this for free. ### Backend endpoints Verified from `mcp-server/src/cli/login.ts` and `mcp-server/src/tools/handlers/completeAuthLink.ts`: | Endpoint | Method | Purpose | |---|---|---| | `/api/auth/device/code` | POST | Issue a device code. Body: `{ scope: "account_full" }`. Returns `{ deviceCode, userCode, verificationUrl, expiresInSeconds, pollIntervalSeconds }` | | `/api/auth/device/token` | POST | Poll for completion. Body: `{ deviceCode }`. Returns `{ pending: true, pollIntervalSeconds }` while waiting; `{ authToken, accountId, tier, linkedAt, expiresAt }` on approval. `expiresAt` is `null` — tokens don't expire | ### Defaults and limits | Field | Default | Notes | |---|---|---| | Device-code window | 600 seconds (10 min) | Backend returns the actual `expiresInSeconds`; `completeAuthLink` enforces a max of 600 | | Poll interval | 5 seconds | Backend may bump this via `pollIntervalSeconds` in poll responses; agent honors | | `completeAuthLink.maxWaitSeconds` | 600 | Caller can lower; capped at 600 by the handler | | `completeAuthLink.pollIntervalSeconds` | 5 | Range 1-30 (clamped) | ### HTTP status codes | Status | What it means | Retryable | |---|---|---| | `200` (`pending: true`) | User hasn't finished signup yet — keep polling | yes (continue) | | `200` (with `authToken`) | Approved — token issued | done | | `403` | User declined the signup prompt at the verification page | yes (after asking the user to retry) | | `404` | Unknown deviceCode (never existed or already consumed) | no (need a fresh code) | | `410` | Device-code window elapsed (10-min default) | yes (need a fresh code) | | Other / network failure | Backend unreachable | yes (after the network issue resolves) | ## The `completeAuthLink` MCP tool The agent-facing primitive for the device-code flow. After `createSimulatorSession` returns `status: "auth_required"`, the agent calls `completeAuthLink` with the returned `deviceCode`. ### Parameters | Parameter | Type | Required | Default | Notes | |---|---|---|---|---| | `deviceCode` | string | yes | n/a | From the `auth_required` response | | `maxWaitSeconds` | integer | no | 600 | 1-600 | | `pollIntervalSeconds` | integer | no | 5 | 1-30 | ### Response on success ```json { "linked": true, "tier": "free_account", "accountId": "550e8400-e29b-41d4-...", "linkedAt": "2026-05-01T12:34:56Z", "expiresAt": "2026-07-30T12:34:56Z", "authTokenFile": { "path": "/home/dev/.extentos/auth.json", "written": true, "error": null }, "pollCount": 3, "summary": "Account linked (tier: free_account). Re-invoke createSimulatorSession to use the new bearer token." } ``` The `pollCount` lets the agent know how long it took. The `summary` is the agent-facing reminder. ### Error codes | Code | Cause | Retryable | |---|---|---| | `unknown_device_code` | Backend doesn't recognize the deviceCode | no — call `createSimulatorSession` again for a fresh code | | `device_code_expired` | 10-min window elapsed | yes — call `createSimulatorSession` again | | `device_code_denied` | User declined at the verification page | yes — ask the developer to retry and accept | | `backend_unreachable` | Network failure or backend down | yes | | `backend_invalid_response` | Malformed poll body | no — version mismatch | | `auth_timeout` | Polled past `maxWaitSeconds` without approval | yes — the developer hasn't finished signup yet | Telemetry events fire on every flow outcome: `account.login_initiated`, `account.login_completed` (with outcome `success | code_expired | user_denied | network_error | polling_aborted`), and `account.linked` (only on success). Tagged with `installId`, no PII. ## CLI subcommands `extentos-mcp` is a developer CLI in addition to the MCP server. The auth-related subcommands: | Subcommand | What it does | When to use | |---|---|---| | `login` | Manually trigger the device-code flow. Initiates the flow against `/api/auth/device/code`, prints the userCode + verificationUrl, auto-opens the browser, polls until completion. | When the developer wants to link before the agent's first `createSimulatorSession` call (rare — the agent does this automatically when needed) | | `logout` | Delete `~/.extentos/auth.json`. Install returns to the anonymous tier (the `installId` survives). | When switching machines, debugging auth state, or revoking the link | | `whoami` | Not yet implemented (Phase-0 stub — exits with a not-implemented notice). Will print `installId`, `accountId` (if linked), `tier`, auth expiry | Diagnosing auth issues, once it lands | Run any of them with: ```bash npx @extentos/mcp-server@latest login npx @extentos/mcp-server@latest logout npx @extentos/mcp-server@latest whoami ``` The `login` command output looks like: ```text Extentos account linking User code: ABCD-1234 Verification URL: https://extentos.com/activate?code=ABCD-1234 On the page, confirm the 8-character code matches "ABCD-1234" before approving. Opening browser… (if it doesn't open, paste the URL above) Polling for up to 600s. Ctrl-C to cancel. ``` ## State files Auth state lives in `~/.extentos/` by default (overridable with `EXTENTOS_CONFIG_DIR`): | File | Contents | Created by | |---|---|---| | `install.json` | A per-machine install id (`inst_…`), never expires | First MCP server start | | `auth.json` | `{ authToken, accountId, linkedAt, expiresAt, scope }` after linking. `expiresAt` is `null` for every token minted since `0058` (never expires); a date means a legacy token still carrying its original life | Successful `completeAuthLink` (or `extentos-mcp login`) | | `consent` | Telemetry consent state (`accepted` / `declined` / unset) | First privacy notice acceptance or `extentos-mcp accept-privacy` / `decline-privacy` | `logout` deletes `auth.json` only. The install id in `install.json` persists — wiping the entire `~/.extentos/` directory just creates a fresh anonymous install — there is no usage meter to reset in the account-required model, and the fresh install still has to link an account to mint sessions or scaffold (acceptable at MVP scale because it requires the developer to wipe the agent's MCP context each time). ## Environment variables | Variable | Default | Purpose | |---|---|---| | `EXTENTOS_BACKEND_URL` | Production backend | Override the backend URL the auth endpoints hit. Used for local backend development. | | `EXTENTOS_CONFIG_DIR` | `~/.extentos` | Override the config / auth / consent directory | | `EXTENTOS_NO_AUTO_OPEN` | unset | Set to `1` to disable browser auto-open. The verification URL is still printed; the developer pastes it manually. Useful for headless / SSH / cloud-hosted-agent environments. | ## Browser auto-open When the device-code flow fires, the MCP server attempts to open the verification URL in the developer's default browser via: | Platform | Command | |---|---| | macOS | `open ` | | Windows | `cmd.exe /c start "" ` | | Linux | `xdg-open ` | If auto-open fails (no GUI, no default handler, sandboxed environment), the URL is still printed to the agent's tool response and the CLI's stdout. The developer copies it into their browser. Set `EXTENTOS_NO_AUTO_OPEN=1` to skip the auto-open attempt entirely. ## Telemetry consent (separate from auth) Telemetry consent is a **separate** mechanism from authentication. The MCP server emits anonymous usage telemetry (tool calls, install events, no source code or PII) tagged with the per-machine `installId`. Consent is consent-default with first-run notice — the same pattern used by Vercel CLI, Astro, and Vite. The consent state lives at `~/.extentos/consent`. It's set by: | Action | Effect | |---|---| | First MCP tool call after install | Privacy notice injected once into the response (`PRIVACY_NOTICE` constant in `mcp-server/src/index.ts`); telemetry stays enabled by default | | `extentos-mcp accept-privacy` | Records explicit consent | | `extentos-mcp decline-privacy` | Disables telemetry upload — events still emit locally for debugging but never leave the machine | | Setting `EXTENTOS_TELEMETRY=0` | Same as `decline-privacy` for the current shell, but doesn't persist to the consent file | Telemetry consent has nothing to do with whether you're authenticated. A no-account install can opt out of telemetry; a linked account can opt out too. The two systems are independent. ## Per-project MCP access What an AI agent connected to your MCP can do is governed by **account-held, per-project grants** — not by the auth token's scope. Once your account is linked, an agent can by default read and change a project's dashboard-managed settings; you tighten or turn off individual capabilities per project in the dashboard under MCP access (`/account/mcp-access`). - **Always allowed** (no grant needed): the connection page and the browser simulator. - **Governed capabilities** — set per project to Off / Read / Read + Write (assistant config and credentials default to **Read + Write**; analytics defaults to **Read**): - **Assistant config** — the realtime model, voice, memory model, and reading managed-gateway usage & cost. This one is **per-field**: you can, for example, let an agent change the model but only *read* the voice. (`getAssistantConfig`, `setAssistantConfig`, `getGatewayUsage`) - **Credentials** — check whether a key is set (a masked hint only) and **add keys without the agent ever seeing the value**: `setCredential` returns a dashboard link where you paste the secret yourself, so it never enters the agent's context. (`getCredentialStatus`, `setCredential`) - **Analytics** — read your app's production telemetry (shipped-app sessions, events, errors) via `getProjectAnalytics`. Enforcement is server-side: a grant change takes effect on the agent's next call, with no re-auth. A blocked call returns a `permission_denied` error naming the capability. ## Frequently asked questions ### Why are the simulator and the scaffold account-required when everything else is anonymous? The browser simulator runs on Extentos's own backend infrastructure — a WebSocket hub keeps the live session alive, the voice proxy handles STT/TTS, and the event log persists — and the scaffold step mints an account-bound project key on that same backend. Those are the Extentos surfaces that cost server-side state, so they're gated behind a free account. The discovery, guidance, validation, and search surfaces (plus on-device `LocalSimTransport` and real-hardware testing through your own Meta credentials) either run on your machine or talk directly to Meta — no Extentos compute, no account needed. Sign-in is a free email-only account (Google one-click or email + password); once linked, sessions are unlimited. ### Why device-code instead of OAuth redirect? OAuth redirect requires the application to be a web app that can register a callback URL and receive a redirect. An AI coding agent isn't a web app — it's a process running in the developer's terminal. Device-code (RFC 8628) is the OAuth flow designed for exactly this case: the device shows the user a code and a URL, the user authorizes on a separate browser, the device polls to learn the result. No redirect, no callback URL, no web server. ### Can I use Extentos without ever signing up? The discovery, guidance, validation, and search tools, on-device `LocalSimTransport` simulation, and real-hardware testing through your own Meta credentials all work with no account, no email, no payment — forever. A free account is needed for three things: minting browser-simulator sessions, the `generateConnectionModule` scaffold step, and the account-scoped project tools. The signup ask fires the first time your agent makes one of those calls. ### Does signing up cost anything? No. The free account is the only tier that exists. No payment, no card, no upfront commitment. Email + Privacy Policy acceptance only (or Google one-click). ### What if the device-code expires before I finish signup? Call `createSimulatorSession` again to get a fresh deviceCode. The backend issues a new code with a fresh 10-minute window. Your prior install state is preserved. ### What if I'm on a headless machine (cloud agent, SSH session)? Set `EXTENTOS_NO_AUTO_OPEN=1`. The verification URL prints to the agent's response; the developer copies it to their local browser. The polling continues in the headless environment regardless of where the URL was opened. ### How do I rotate or revoke my auth token? `extentos-mcp logout` deletes `~/.extentos/auth.json` locally. Server-side revocation exists (tokens carry a `revoked_at`), but a self-serve device-revocation UI isn't live yet — email hello@extentos.com to revoke a device. The next API call from the revoked install will get a 401 and prompt re-linking. ### Is the auth token a secret? Yes — treat it like any other API token. It's stored at `~/.extentos/auth.json`, written with owner-only permissions (`0600` on macOS/Linux). Don't commit it to version control. Don't share it across machines. ### Is the `installId` a secret? No — it's a per-machine anonymous identifier. Other apps on the same machine can read `~/.extentos/install.json`; the same value is sent to `api.extentos.com` on every tool call as the telemetry key. It's not a credential. ### What happens if `~/.extentos/auth.json` is corrupted? The MCP server treats it as not-linked. The next `createSimulatorSession` call returns `auth_required` and the device-code flow re-links the install. Or run `extentos-mcp login` proactively to re-link. ## Related - **[Pricing](/docs/resources/pricing)** — what the free tier covers and why the browser simulator needs a free account - **[`completeAuthLink` tool](/docs/mcp-server/tools/simulation)** — the MCP tool reference (full input/output schema) - **[MCP server overview](/docs/mcp-server)** — the broader tool catalog - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — installing the server in the first place --- source: https://extentos.com/docs/mcp-server/configuration --- --- title: Configuration description: Configure @extentos/mcp-server — environment variables, the ~/.extentos state files (install id, auth, consent), and the default vendor. type: reference platform: mcp related: - /docs/mcp-server - /docs/mcp-server/auth - /docs/mcp-server/tools/generation - /docs/mcp-server/install --- The MCP server needs **no configuration to run** — `npx -y @extentos/mcp-server@latest` works out of the box, anonymously, with sensible defaults. The settings below are escape hatches: override the backend for local Extentos development, relocate the state directory, opt out of telemetry, suppress browser auto-open, or feed the `setup` diagnostic your GitHub Packages token. > **This page is about the MCP-server shell**, not the library. The Kotlin/Swift SDK has its own separate config surface — `ExtentosConfig` (`debug`, `telemetryConsent`, …) is set in the host app's bootstrap, not via these environment variables. See the [SDK docs](/docs/sdk/android) for that surface. ## Environment variables All settings are environment variables; there is no config file you author by hand. Every variable is optional. | Variable | Default | What it does | |---|---|---| | `EXTENTOS_BACKEND_URL` | Production backend (`api.extentos.com`) | Override the backend base URL the server talks to. Used when developing Extentos itself against a local backend. Read by `tools/util/backendClient.ts`. | | `EXTENTOS_CONFIG_DIR` | `~/.extentos` | Relocate the state directory (install id, auth token, consent). Useful for sandboxes, CI, or devcontainers that can't write to the home directory. | | `EXTENTOS_TELEMETRY` | unset | Set to `0` to decline anonymous telemetry for the shell without running the CLI consent command. Takes precedence over the on-disk consent file (the escape hatch for environments that can't persist consent). | | `EXTENTOS_NO_AUTO_OPEN` | unset | Set to `1` to suppress the browser auto-open on simulator-session creation (headless / SSH / sandboxed environments). The session URL is still returned in the tool response for the developer to open manually — nothing is lost. | | `GITHUB_TOKEN` | unset | Read by `extentos-mcp setup` to validate the Meta DAT GitHub-Packages PAT, needed only for camera/display apps that depend on `com.extentos:glasses-meta` (those `mwdat-*` artifacts live on GitHub Packages and need a token with `read:packages` scope). This is a **setup-diagnostic input**, not a runtime dial — the running MCP server never reads it; only the `setup` subcommand does. | | `GITHUB_USERNAME` | `token` | Optional companion to `GITHUB_TOKEN` for the same `setup` diagnostic — the Basic-auth username for the GitHub Packages Maven endpoint. Defaults to the literal `token` (GitHub accepts any username when the password is a valid PAT). | `EXTENTOS_TELEMETRY` and `EXTENTOS_CONFIG_DIR` are the only ones that change persistent state; the rest are per-process. ## State files The server keeps a small amount of state under `~/.extentos` (override the directory with `EXTENTOS_CONFIG_DIR`): | File | Holds | |---|---| | `install.json` | The stable anonymous `installId` (`inst_…`) tagged onto every backend request and telemetry event, plus first-run metadata. Not a secret — it's the same value sent to `api.extentos.com` on every tool call. | | `auth.json` | The bearer token written after account linking. Present only once linked; cleared by `extentos-mcp logout`. | | `consent` | One of `accepted` / `declined` — the telemetry consent decision. Absent until you run `accept-privacy` / `decline-privacy`. `EXTENTOS_TELEMETRY=0` overrides it. | The auth and consent lifecycle is documented in full on the [auth page](/docs/mcp-server/auth). You never edit these files by hand — the CLI subcommands (`login`, `logout`, `accept-privacy`, `decline-privacy`, `whoami`, `status`) manage them. ## Default vendor The only glasses vendor in the current MVP is `meta_rayban` — every tool that takes a `glasses` argument requires it. There is no vendor-selection setting; the value is passed per tool call (e.g. `getPlatformInfo({ glasses: "meta_rayban" })`, `generateConnectionModule({ glasses: "meta_rayban", … })`). Additional vendors will surface as new accepted values on those tools when they ship. ## Related - **[Auth](/docs/mcp-server/auth)** — the device-code flow, `auth.json`, and the CLI auth commands that manage the state files above - **[MCP server overview](/docs/mcp-server)** — the tool catalog, the canonical agent flow, and the CLI subcommands - **[Setup and Generation tools](/docs/mcp-server/tools/generation)** — `generateConnectionModule`, which references the `GITHUB_TOKEN` setup diagnostic for the Meta DAT repo - **[Install](/docs/mcp-server/install)** — per-host install commands and JSON snippets --- source: https://extentos.com/docs/mcp-server/credentials-checklist --- --- title: Credentials checklist description: Provider-keys checklist for Extentos. Simulator needs none; Phase-4 assistant defaults to the managed gateway; handler calls are BYOK; Meta creds for hardware only. type: reference platform: mcp related: - /docs/mcp-server/auth - /docs/concepts/ai-gateway - /docs/mcp-server/configuration - /docs/mcp-server/tools/production - /docs/resources/pricing --- Extentos itself needs **no AI-provider keys**. Discovery, guidance, and validation tools run with no signup; the browser simulator and the `generateConnectionModule` scaffold step take a free account — the [auth flow](/docs/mcp-server/auth) handles the link the first time it's needed. There is exactly one Extentos credential, and it is not on the list below because it is not a provider key: your **project key**, minted by `generateConnectionModule` and baked into the build. Any app that uses the assistant needs it, in every environment including store releases. An app that only uses `transcriptions()` / `speak()` on real glasses does not — those are phone-native and never reach the gateway. AI shows up in an Extentos app two distinct ways, and only one of them needs a key in your code: - **The Phase-4 voice assistant runs on the [managed AI gateway](/docs/concepts/ai-gateway) — no key in your app, but not free.** Gateway usage is metered against a prepaid balance; "no credentials" on this page means no *keys to obtain*, not no cost. `glasses.assistant.start(...)` routes through Extentos's gateway on Extentos's provider key; you supply nothing. (There is no option to run it on your own OpenAI account.) No provider key lives in your handler for the assistant path. See [the gateway page](/docs/concepts/ai-gateway). - **Handler-code calls to your own AI provider need your key.** If your handler calls Anthropic / OpenAI / DeepL / etc. directly (e.g., to run a vision model on a captured photo or summarize a transcript outside the assistant), that key lives in the developer's process. Extentos never sees it. The rest of this page is the checklist for **this** path. Plus one non-AI credential set: - **Real Meta Ray-Ban hardware, for camera or display** — those capabilities come from `com.extentos:glasses-meta` and go through Meta's Device Access Toolkit, which requires the developer's own Meta Wearables Developer Center registration. **A voice-only app needs none of it, even on real glasses**: the microphone and speaker are reached through the phone's own Bluetooth audio routing, with no vendor SDK in the path. Surface the relevant entries from this page while writing the handler, not at deploy time. If a developer is writing a translation handler, mention DeepL or Google Translate in the same exchange — not three days later when they try to ship. ## Quick decision table | If your handler calls… | You need… | |---|---| | `glasses.assistant.start(...)` (managed gateway, the default) | **No AI-provider key** — the [gateway](/docs/concepts/ai-gateway) runs on Extentos's. You do need your **project key** (`EXTENTOSProjectKey` / `EXTENTOS_PROJECT_KEY`), minted by `generateConnectionModule` and baked into the build; it authenticates the app to the gateway in every environment and bills usage to your account. Not free either: usage draws down your prepaid credit ($2 granted, then top-up), simulator included. See [pricing](/docs/resources/pricing). | | `glasses.audio.speak(...)` (simulator only) | Nothing — Extentos voice proxy handles dev-time TTS | | `glasses.audio.speak(...)` (real hardware) | Nothing — phone-native TTS; optional premium TTS provider | | `glasses.audio.transcriptions()` (simulator only) | Nothing — Extentos voice proxy + browser STT | | `glasses.audio.transcriptions()` (real hardware) | Nothing — phone-native STT | | An LLM **from your own handler code** (text answer, summarization, classification) | Anthropic, OpenAI, Google Gemini, or another LLM provider | | A vision API (photo description, OCR via vision) | OpenAI Vision, Google Vision, Anthropic, Azure, or AWS Bedrock | | A translation API | DeepL, Google Translate, or another translation provider | | A dedicated OCR API | Google Vision, Azure, or another OCR provider | | Premium voice synthesis in production | ElevenLabs, Azure Neural, Play.ht, or similar | | Camera or display on real glasses | Meta Wearables Developer Center app registration | | Voice-only on real glasses | *no vendor credential* — no vendor SDK, no Meta account, no token. Mic and speaker ride the phone's own Bluetooth audio routing, and `transcriptions()` / `speak()` are phone-native, so they need nothing at all. If the app uses the **assistant**, that goes through the gateway and needs the project key (row 1). | The simulator and managed-assistant paths intentionally need zero credentials — that's the point of Extentos's anonymous-first model and the [managed gateway](/docs/concepts/ai-gateway). Provider keys come up only for direct handler-code AI calls, and Meta credentials only when the developer leaves the simulator for real hardware. ## LLM providers Only one is required, pick whichever the developer prefers. All have free tiers usable for development. ### Anthropic (Claude) - **Portal:** [console.anthropic.com/settings/keys](https://console.anthropic.com/settings/keys) - **Env var convention:** `ANTHROPIC_API_KEY` - **Free tier:** $5 of credit on signup; pay-as-you-go after. - **Best for:** Long-context reasoning, tool use, code generation in handlers. ### OpenAI - **Portal:** [platform.openai.com/api-keys](https://platform.openai.com/api-keys) - **Env var convention:** `OPENAI_API_KEY` - **Free tier:** Limited free credits on signup; pay-as-you-go after. Multi-modal (text + vision + speech) under one key. - **Best for:** Vision through GPT-4 Vision, transcription via Whisper, generic chat. ### Google Gemini - **Portal:** [aistudio.google.com/app/apikey](https://aistudio.google.com/app/apikey) - **Env var convention:** `GEMINI_API_KEY` - **Free tier:** Generous free quota (1500 requests/day on Flash, lower on Pro). Best free option for prototyping. - **Best for:** Cost-sensitive prototyping, multi-modal, fast iteration before committing to a paid provider. ## Vision APIs If your app captures photos and the handler needs to interpret them, pick one: | Provider | Portal | Env var | |---|---|---| | OpenAI Vision (GPT-4o) | [platform.openai.com/api-keys](https://platform.openai.com/api-keys) | `OPENAI_API_KEY` | | Google Vision | [console.cloud.google.com/apis/credentials](https://console.cloud.google.com/apis/credentials) (enable Cloud Vision API) | `GCLOUD_VISION_API_KEY` or service account | | Anthropic Claude Vision | [console.anthropic.com/settings/keys](https://console.anthropic.com/settings/keys) | `ANTHROPIC_API_KEY` | | Azure Computer Vision | [portal.azure.com](https://portal.azure.com) → Computer Vision resource | `AZURE_COGNITIVE_KEY` + endpoint | | AWS Bedrock (Claude/Nova) | [console.aws.amazon.com/bedrock](https://console.aws.amazon.com/bedrock) | AWS access key + secret + region | For most apps, just reuse the LLM provider's key — GPT-4o, Claude, and Gemini all do vision under the same key. Separate vision-only providers only matter when you need their specific capabilities (Google's high-accuracy OCR, Azure's domain models). ## Translation For multilingual apps: | Provider | Portal | Env var | |---|---|---| | DeepL | [deepl.com/pro-api](https://www.deepl.com/pro-api) | `DEEPL_AUTH_KEY` | | Google Translate | [console.cloud.google.com/apis/credentials](https://console.cloud.google.com/apis/credentials) (enable Translation API) | `GCLOUD_TRANSLATE_API_KEY` | | HuggingFace Inference | [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) | `HF_TOKEN` | DeepL is generally higher quality for European languages; Google Translate has wider language coverage. HuggingFace Inference works for self-hosted or community models. ## OCR If the handler reads text out of camera frames: | Provider | Portal | Env var | |---|---|---| | Google Vision (text detection) | [console.cloud.google.com](https://console.cloud.google.com) (enable Cloud Vision API) | `GCLOUD_VISION_API_KEY` | | Azure Computer Vision (Read API) | [portal.azure.com](https://portal.azure.com) | `AZURE_COGNITIVE_KEY` + endpoint | | AWS Textract | [console.aws.amazon.com/textract](https://console.aws.amazon.com/textract) | AWS access key + secret + region | Most LLMs with vision (GPT-4o, Claude, Gemini) can do OCR-style tasks under their existing key — separate OCR services matter for high-volume document workflows. ## Premium voice synthesis (production) Optional. Production apps use phone-native TTS by default (`TextToSpeech` on Android, `AVSpeechSynthesizer` on iOS) at zero cost. If the developer wants higher-quality voices: | Provider | Portal | Env var | |---|---|---| | ElevenLabs | [elevenlabs.io/app/settings/api-keys](https://elevenlabs.io/app/settings/api-keys) | `ELEVENLABS_API_KEY` | | Azure Neural TTS | [portal.azure.com](https://portal.azure.com) → Speech resource | `AZURE_SPEECH_KEY` + region | | Play.ht | [play.ht/studio/api-access](https://play.ht/studio/api-access) | `PLAYHT_API_KEY` + user ID | Premium voice synthesis isn't wired into the shipped Android SDK yet — phone-native TTS is the only built-in path today. When it ships, Extentos won't sit in the runtime path; the library will call the provider directly from the phone. ## Meta Wearables Developer Center (real hardware) **Skip this until the developer is ready to test on physical Meta Ray-Ban glasses.** The browser simulator and `LocalSimTransport` don't need any of it. When real hardware comes into play: 1. Create an app in the [Meta Wearables Developer Center](https://wearables.developer.meta.com/). 2. Register a Wearables app under the developer's account. 3. Bind the app to platform-specific identifiers: - **Android:** package name + release signing SHA-256. - **iOS:** Bundle ID + Team ID + a custom URL scheme of the developer's choosing. 4. Retrieve the **Meta App ID** and **Client Token** from the Wearables Developer Center (Manage projects). 5. Set your Meta credentials in the **AndroidManifest ``** (Android) / **`Info.plist` `MWDAT` dict** (iOS) — the MWDAT SDK reads them from there at init (`Wearables.initialize(applicationContext)` takes only the context). There is no runtime credential-injection path through `ExtentosConfig`. ```xml title="Android (AndroidManifest.xml)" ``` ```xml title="iOS (Info.plist)" MWDAT MetaAppID$(META_APP_ID) ClientToken$(META_CLIENT_TOKEN) TeamID$(DEVELOPMENT_TEAM) ``` 6. **Enroll your own Meta account as a tester.** Create a version in the Wearables Developer Center, assign it to a release channel, and invite the account that owns the glasses you'll test with — **including yours**. Until that invite is accepted, registration fails even with correct credentials and a correctly signed build, and it is the one blocker with **no distinct log line**, so it's diagnosed by elimination. Needs glasses firmware v125+. See [release channels](/docs/vendors/meta) and the [first-hardware-run ladder](/docs/guides/simulator-to-real-glasses). 7. **Register the RELEASE signing SHA-256, and install the release variant.** Meta binds the entitlement to package name + release signature, so a debug-signed `installDebug` build never matches and never registers. Your normal debug loop works against the simulator; real glasses need the release variant. 8. Source the values from `local.properties` → string resource (Android dev) or an `.xcconfig` substitution into `Info.plist` (iOS) — never hardcode them in source. The MCP `getCredentialGuide` tool emits the exact manifest/Info.plist entries and the local.properties-first wiring for the developer's platform. The MCP `getCredentialGuide` tool walks through this interactively with the developer when they call it — that's the agent-facing entry point if step-by-step is preferred over this static reference. ## Voice in the simulator (no keys needed) The browser simulator's STT and TTS run on Extentos-managed infrastructure (the voice proxy described in [accounts and pricing](/docs/resources/pricing)). Developers don't sign up for Whisper, OpenAI, or any voice provider to use voice in the simulator — that's deliberate, so the first speak/listen experience works out of the box. Voice in the simulator is unmetered for linked accounts (the browser simulator itself requires sign-in — see [pricing](/docs/resources/pricing)) — there's no separate voice quota. On real hardware, voice runs entirely on the phone (zero Extentos cost, zero provider keys). ## Where to put keys **Never check provider keys into source control.** - **Android:** `local.properties` for dev (read at Gradle config time, deterministic across IDE-launch + daemon-reuse), with `System.getenv(...)` as the CI/release fallback baked into the same snippet. The `getCredentialGuide` response embeds the exact `Properties().apply { ... }` block to paste into `app/build.gradle.kts`. Keys surface at runtime via `context.getString(R.string.)` (NOT `BuildConfig.X` — that path inlines stale keys across rotations). - **iOS:** `.xcconfig` files (gitignored) or environment variables on the build runner. Read at runtime via `Bundle.main.object(forInfoDictionaryKey:)`. - **Local dev:** for the dev-machine path, `local.properties` (Android) / `Secrets.local.xcconfig` (iOS) — both gitignored by default. `.env` files work for non-build runtime use. For real production deployments, prefer build-time secret injection over runtime env vars where possible — it keeps the keys out of the running app's process memory. ## When to surface this If you're an AI agent scaffolding an integration for a developer: - **At handler-scaffolding time**, look at which BYOK providers the handler will call. Surface the matching rows from the [Quick decision table](#quick-decision-table) inline so the developer knows what they'll need before they get deep into handler code. - **At code-generation time**, call `getCredentialGuide(services, platform)` and apply the returned `build.gradle.kts` snippet directly — the canonical local.properties-first pattern is already baked in. - **Before the first real-hardware test**, surface the Wearables Developer Center section as a separate setup step — it's a longer process (~15-30 min) and the developer wants to do it once, not interleaved with handler code. Don't dump the full table on a developer who's still in the simulator; it adds noise. The simulator path is meant to need no keys at all. ## Related - **[Auth](/docs/mcp-server/auth)** — Extentos account auth, separate from provider keys - **[Configuration](/docs/mcp-server/configuration)** — env vars and config-file settings - **[Production tools](/docs/mcp-server/tools/production)** — `getCredentialGuide`, `getProductionChecklist` - **[Pricing](/docs/resources/pricing)** — what the free tier covers --- source: https://extentos.com/docs/mcp-server/install --- --- title: Install the MCP server description: How to install the Extentos MCP server (@extentos/mcp-server) in any MCP-compatible AI coding agent — Claude Code, Cursor, Windsurf, Cline, and others. Per-host install commands, config file locations, copy-pasteable JSON snippets, restart and verify steps, version pinning, updating, troubleshooting common errors, and uninstall instructions. Verified install paths for each supported host. type: tutorial platform: mcp related: - /docs/mcp-server - /docs/mcp-server/agents - /docs/mcp-server/configuration - /docs/mcp-server/auth - /docs/getting-started/with-agent --- The Extentos MCP server (`@extentos/mcp-server`, MIT) is published on npm and installs into any [Model Context Protocol](https://modelcontextprotocol.io)-compatible AI coding agent. This page is the per-host install reference — Claude Code, Cursor, Windsurf, Cline, and the generic MCP-host pattern that covers everything else. Each section gives the verified config file path, the exact JSON snippet to paste, the restart-and-verify steps, and links to troubleshooting if it doesn't work the first time. ## Before you start **Prerequisites:** - **Node.js 20 or newer** on `PATH`. Verify with `node -v`. The MCP server is published as an npm package and runs through `npx`; older Node versions will fail with `Unsupported engine` errors. - **Any MCP-compatible AI coding agent.** Claude Code, Cursor, Windsurf, Cline are all officially verified. Any host that speaks the standard MCP protocol over stdio works — see the [generic MCP host](#generic-mcp-host) section. - **No Extentos account needed for install.** Discovery, validation, and guidance tools work anonymously; the [auth flow](/docs/mcp-server/auth) fires the first time you call `createSimulatorSession` — or `generateConnectionModule`'s scaffold step, which mints your account-bound project key (same free device-code login). **Three install paths**, ranked by simplicity: 1. **Agent prompt** — paste a sentence into your agent and let it install Extentos using its own filesystem-edit / shell tools. Host-agnostic, recommended. 2. **Host-specific command or config** — direct install via the host's CLI (Claude Code) or config file (Cursor, Windsurf, Cline). Per-host instructions below. 3. **Generic MCP JSON** — drop the standard `mcpServers` block into the host's MCP config file by hand. Universal fallback. ## Path 1 — Agent prompt (recommended, host-agnostic) Paste this into your AI coding agent: ```text Install @extentos/mcp-server in my MCP config. ``` The agent will: - Run the right command (`claude mcp add ...` for Claude Code) or edit the right config file (`~/.cursor/mcp.json` for Cursor, etc.) - Tell you to restart your agent host - Verify the tools appear after restart This is the simplest path because the agent figures out the host conventions itself. Use it unless you specifically want to control the install yourself. ## Path 2 — Host-specific install ### Claude Code The cleanest install — Claude Code has a native MCP CLI: ```bash claude mcp add extentos -- npx -y @extentos/mcp-server@latest ``` **Restart Claude Code** (Cmd+Q on macOS, quit from system tray on Windows / Linux — closing the window isn't enough). **Verify:** run `/mcp` in Claude Code. You should see `extentos` listed with all of Extentos's tools available. Or `claude mcp list` from the terminal. The command writes the server entry to your Claude Code MCP config (typically `~/.claude.json`). To remove: `claude mcp remove extentos`. ### Cursor Cursor uses an `mcp.json` file. Two scopes: - **Global** (all projects): `~/.cursor/mcp.json` - **Project-only** (this repo): `.cursor/mcp.json` at the project root Create or edit the file with: ```json { "mcpServers": { "extentos": { "command": "npx", "args": ["-y", "@extentos/mcp-server@latest"] } } } ``` If the file already has other MCP servers, add `"extentos"` as a sibling key inside `mcpServers`. **Restart Cursor** (full quit, not just close window). **Verify:** open Cursor's MCP settings panel — `extentos` should appear in the server list with a green status indicator and Extentos's tools. Or ask the agent: *"List the MCP servers you have available"* — Extentos's tools should be enumerated. ### Windsurf Windsurf uses a JSON config at `~/.codeium/windsurf/mcp_config.json`. Create or edit it: ```json { "mcpServers": { "extentos": { "command": "npx", "args": ["-y", "@extentos/mcp-server@latest"] } } } ``` **Restart Windsurf** completely. **Verify:** open Windsurf's MCP / tools panel. Extentos should be listed. Or invoke the agent and ask it to call any Extentos tool (e.g., *"Run getPlatformInfo for meta_rayban"*) — it should succeed. ### Cline (VS Code extension) Cline ships as a VS Code extension. Two ways to add the MCP server: **Via Cline's UI** (recommended): 1. Open the Cline panel in VS Code 2. Click the **MCP Servers** tab 3. Click **Edit MCP Settings** (or **Configure MCP Servers** depending on Cline version) 4. Paste the JSON below and save: ```json { "mcpServers": { "extentos": { "command": "npx", "args": ["-y", "@extentos/mcp-server@latest"] } } } ``` **Via the config file directly** — `cline_mcp_settings.json`, typically at: | Platform | Path | |---|---| | macOS | `~/Library/Application Support/Code/User/globalStorage/saoudrizwan.claude-dev/settings/cline_mcp_settings.json` | | Windows | `%APPDATA%\Code\User\globalStorage\saoudrizwan.claude-dev\settings\cline_mcp_settings.json` | | Linux | `~/.config/Code/User/globalStorage/saoudrizwan.claude-dev/settings/cline_mcp_settings.json` | **Restart VS Code** (or reload the Cline extension via the command palette: *Developer: Reload Window*). **Verify:** Cline's MCP Servers tab should show `extentos` with a green status. Tools enumerate when you ask Cline anything Extentos-related. ### Generic MCP host Any host that supports the standard Model Context Protocol can use the same JSON shape. The host's config file path varies, but the server entry is identical: ```json { "mcpServers": { "extentos": { "command": "npx", "args": ["-y", "@extentos/mcp-server@latest"] } } } ``` Find your host's MCP config file (consult the host's documentation). Common locations follow the pattern `~/./mcp*.json` or somewhere under `~/.config//`. Drop the `mcpServers.extentos` entry in, restart, verify. ## Pinning a version By default, `@latest` resolves to the most recent published version on every npm fetch. For reproducibility across sessions, pin a specific version: ```json { "mcpServers": { "extentos": { "command": "npx", "args": ["-y", "@extentos/mcp-server@"] } } } ``` Why pin: Extentos is pre-1.0 (find the current version on [npm](https://www.npmjs.com/package/@extentos/mcp-server)). Minor version bumps may shift APIs until the hardware test loop closes. Pinning means a known-good install survives across MCP server restarts even if a new version ships overnight. The tradeoff is missing bug fixes — re-pin to a newer version periodically. For Claude Code's CLI, pinning looks the same: ```bash claude mcp add extentos -- npx -y @extentos/mcp-server@ ``` ## Updating If you used `@latest`, every host restart re-resolves the version through npm — you're always on the newest one as long as your network reaches the npm registry. No manual update step required. If you pinned a version, edit the config to a newer version number and restart your host. Run `npx @extentos/mcp-server@latest update` to check whether a newer version exists (no-op on `@latest` installs). You can also force a fresh fetch by deleting the npx cache: ```bash # macOS / Linux rm -rf ~/.npm/_npx # Windows rmdir /s "%LocalAppData%\npm-cache\_npx" ``` The next host restart re-downloads the package. ## Verifying the install worked The fastest universal check: ask your agent to call any Extentos tool. For example: ```text Run getPlatformInfo with sections ["version", "capabilities"] and glasses "meta_rayban" ``` A successful response includes the library version, the Meta Ray-Ban capability catalog (compact form, ~2 KB), and a `summary` field. If the agent reports the tool isn't available, the server didn't load — see [troubleshooting](#troubleshooting) below. Per-host verification methods: | Host | Verify command | |---|---| | Claude Code | `/mcp` — lists active MCP servers | | Cursor | MCP settings panel shows `extentos` with green status | | Windsurf | MCP / tools panel shows `extentos` | | Cline | MCP Servers tab shows `extentos` with green status | | Any | Ask the agent: *"Run getPlatformInfo for meta_rayban"* | The full tool list is the generated [MCP tools reference](/docs/reference/mcp-tools); see [`/docs/mcp-server`](/docs/mcp-server) for the catalog overview. ## Troubleshooting ### `/mcp` doesn't list `extentos` (Claude Code) Restart Claude Code completely — Cmd+Q on macOS, quit from system tray on Windows. Just closing the window isn't enough; the MCP host loads servers at startup. ### `extentos` shows red / errored status (Cursor / Windsurf / Cline) Click into the server's status to see the error. Most common causes: - **Node.js too old.** Run `node -v`. Extentos requires Node.js 20+. Update from [nodejs.org](https://nodejs.org). - **`npx` command not found.** Node.js installed without npx — reinstall a recent Node.js from the official installer. - **Network blocked on first run.** `npx -y @extentos/mcp-server@latest` fetches the package on first run; corporate firewalls or air-gapped environments can block it. Pin a version that's already in the npx cache. - **Config JSON syntax error.** Validate with `cat ~/.cursor/mcp.json | python -m json.tool` (or your tool of choice). A trailing comma or missing bracket prevents the host from parsing the entry. ### First tool call hangs for 10–30 seconds Normal — `npx` is downloading the package on first run. Subsequent tool calls hit the npx cache and respond in milliseconds. ### `EACCES` or permission errors on first run Some sandboxed environments restrict npm's cache directory (which `npx` uses). Point it at a writable location: ```bash npm config set cache "$HOME/.cache/npm" # or, per-session: export npm_config_cache="$HOME/.cache/npm" ``` Then restart your host. ### `auth_required` returned by `createSimulatorSession` immediately That's the expected flow on first use — the browser simulator requires a free Extentos account. The response includes a `verificationUrl`; open it in your browser, sign in (Google one-click or email + password — no payment), and the agent retries automatically. See the [auth page](/docs/mcp-server/auth) for the full flow. ### Auto-bind not connecting on Android Check that nothing else is bound to localhost port `31337`. The Extentos library probes the MCP server's local bridge at `127.0.0.1:31337/whoami`; if the port is in use, the bridge logs a warning at startup and continues without auto-bind. The developer can still target a session via the URL-bake path (paste the `EXTENTOS_SESSION_URL` snippet Android-side or write the `extentos.session.plist` iOS-side that `createSimulatorSession` returns, then rebuild). To check what's on the port: `lsof -i :31337` on macOS / Linux, `netstat -ano | findstr 31337` on Windows. ### Privacy notice keeps appearing on every tool call The notice is meant to fire once per install. If it's repeating, the consent file at `~/.extentos/consent` isn't being persisted (permissions issue, or `EXTENTOS_CONFIG_DIR` pointing somewhere unwritable). Run `npx @extentos/mcp-server@latest accept-privacy` to set consent explicitly. ## Uninstalling | Host | Command | |---|---| | Claude Code | `claude mcp remove extentos` | | Cursor / Windsurf / Cline / generic | Delete the `"extentos"` entry from the host's `mcp.json` (or equivalent), then restart | To also wipe local state (install ID, auth token, consent): ```bash # macOS / Linux rm -rf ~/.extentos # Windows rmdir /s "%USERPROFILE%\.extentos" ``` This returns the install to a fresh anonymous state on next install — the next `createSimulatorSession` call will trigger the device-code flow again. ## Frequently asked questions ### Which MCP host should I use? Whichever you already use. Extentos is host-agnostic — the same tools work identically across Claude Code, Cursor, Windsurf, and Cline. If you don't have a preference, **Claude Code** has the cleanest install (one CLI command). ### Do I need to install anything else besides the MCP server? No, not for the MCP-driven dev loop. The MCP server itself includes everything the agent needs to scaffold your project, generate code, and provision simulator sessions. Only when you go to ship to **real Ray-Ban Meta hardware** do you need a Meta Developer Center account (free, ~15-30 minutes of setup) — see [vendors/meta](/docs/vendors/meta). ### Can I use Extentos in a CI environment? Yes. The MCP server runs over stdio and `npx` works headlessly. Set `EXTENTOS_NO_AUTO_OPEN=1` to disable browser auto-open during simulator sessions. For air-gapped / no-network CI, install the package globally (`npm install -g @extentos/mcp-server`) so `npx` doesn't try to fetch. ### Does Extentos work in a Docker container? Yes — install Node.js 20+ in the container, mount `~/.extentos` as a volume to persist the `installId` across runs, and treat the rest like any host. Browser-simulator sessions need network access to `wss://api.extentos.com`. ### Can I install multiple Extentos versions side-by-side? Each MCP host has one `extentos` entry at a time, and that entry pins one version. To run multiple versions, register them with different keys: `extentos-stable` (pinned to an exact version) and `extentos-canary` (`@latest`) as separate entries in the same `mcpServers` block. ### What's the difference between `@latest` and a pinned version? `@latest` resolves to whatever's newest on npm at fetch time. Pinning an exact version (`@extentos/mcp-server@`, with the current version from the [npm page](https://www.npmjs.com/package/@extentos/mcp-server)) locks it. Use `@latest` for development (you get bug fixes); pin a version when you need reproducibility (e.g., a team is debugging a specific issue and everyone needs the same MCP build). ### Will I get a popup or modal during install? Only the agent-host's standard "approve this MCP server?" dialog the first time you start it after adding the server. Extentos itself doesn't run any GUI. The first browser-simulator session opens a browser tab to `extentos.com/s/` — you can disable that with `EXTENTOS_NO_AUTO_OPEN=1`. ### Why do I need `npx` instead of `npm install`? `npx` is the standard pattern for MCP servers — it lets the agent host spawn the server without a global install, ensures version isolation per-host, and makes uninstall as simple as removing the config entry. `npm install -g @extentos/mcp-server` works too if you prefer a global install (point your `command` at `extentos-mcp` directly instead of `npx`). ## Related - **[MCP server overview](/docs/mcp-server)** — what the tools do, the canonical agent flow - **[Supported agents](/docs/mcp-server/agents)** — per-host capability notes beyond install - **[Configuration](/docs/mcp-server/configuration)** — environment variables, config files, dial reference - **[Auth](/docs/mcp-server/auth)** — the device-code flow that fires on the first browser-simulator session - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install + first prompt + verify in the simulator end-to-end --- source: https://extentos.com/docs/mcp-server/tools --- --- title: Tools reference description: The Extentos MCP server exposes a tight set of deterministic tools across deterministic categories — discovery + SDK reference, generation, guidance, validation, simulation, production, agent configuration, and search. Per-tool reference covering input parameters, response shapes, when to call each tool, and worked examples. Verified from the actual tool definitions and handler implementations in @extentos/mcp-server. type: reference platform: mcp related: - /docs/mcp-server - /docs/mcp-server/tools/discovery - /docs/mcp-server/tools/generation - /docs/mcp-server/tools/guidance - /docs/mcp-server/tools/validation - /docs/mcp-server/tools/simulation - /docs/mcp-server/tools/production - /docs/mcp-server/tools/search --- The MCP server exposes a deterministic tool surface an AI agent calls to operate Extentos. Each tool is a typed primitive — clear input schema, predictable response shape, no hidden state. The agent composes them itself; there's no planning tool. Tools are organized by purpose: | Category | Tools | What they're for | |---|---|---| | **[Discovery and SDK reference](/docs/mcp-server/tools/discovery)** | `getPlatformInfo`, `getCapabilityGuide`, `getCodeExample`, `getMigrationGuide` | First calls in any new task — read the capability catalog, look up per-feature call shape, peel from canonical patterns, map raw Meta DAT symbols to Extentos primitives | | **[Setup and Generation](/docs/mcp-server/tools/generation)** | `generateConnectionModule`, `getConnectionPageConfig`, `setConnectionPageConfig`, `regenerateConnectionPageFile`, `adoptConnectionPageFile` | Scaffold the project + manage the managed connection-page config. After scaffolding, the agent writes handler classes against the SDK primitives. | | **[Agent configuration & usage](/docs/reference/mcp-tools)** | `getAssistantConfig`, `setAssistantConfig`, `getGatewayUsage` | Read or change a project's dashboard-managed assistant settings (model/voice/memory), and read managed-gateway usage & cost. Gated per-project by the MCP access grant (default Read+Write). | | **[Credentials](/docs/reference/mcp-tools)** | `getCredentialStatus`, `setCredential` | Check whether a project's API keys are set (masked hint only) + start write-without-knowing entry — a dashboard handoff where the owner pastes the secret, which never passes through the agent. Gated per-project by the MCP access grant. | | **[Analytics](/docs/reference/mcp-tools)** | `getProjectAnalytics` | Read a project's production analytics — aggregate shipped-app telemetry (events, active installs, by event/day/vendor/platform). Metadata only; ownership-checked; gated per-project by the MCP access grant. | | **[Implementation Guidance](/docs/mcp-server/tools/guidance)** | `getVoiceCommandGuidance`, `getPermissions` | Side-quest helpers during composition | | **[Validation](/docs/mcp-server/tools/validation)** | `inspectIntegration`, `validateIntegration` | Correctness gates — run after structural changes | | **[Simulation](/docs/mcp-server/tools/simulation)** | `createSimulatorSession`, `ensureSimulatorBrowser`, `completeAuthLink`, `getEventLog`, `getSimulatorStatus`, `injectTranscript`, `injectAssistantUtterance`, `assertToolCalled`, `injectHardwareButton`, `setSimVideo`, `setSimDevice`, `getDisplayState`, `injectInput` | Provision and operate browser-mode simulator sessions, plus the agent-driven testing tools that close the end-to-end loop without a human — including `injectHardwareButton` (tap pauses/resumes a live camera stream, hold stops it, so `CaptureError.StreamPaused` is testable) | | **[Production](/docs/mcp-server/tools/production)** | `getProductionChecklist`, `getCredentialGuide` | Pre-ship checks | | **[Search](/docs/mcp-server/tools/search)** | `searchDocs` | Search the Extentos conceptual documentation | The full, always-current tool list with input schemas is generated at [/docs/reference/mcp-tools](/docs/reference/mcp-tools). Three **Agent-configuration** tools manage a project's voice agent over MCP: `getAssistantConfig` reads the dashboard-managed assistant settings (Realtime model, voice, memory model + mode), `setAssistantConfig` changes them and `getGatewayUsage` reads managed-gateway usage + exact cost. All are account-scoped (they require a linked account). Writes are **Read+Write by default** so they work out of the box; the project owner can restrict any capability per-project in the dashboard's MCP access controls, and `setAssistantConfig` echoes the cost impact of a model change. Two **Credentials** tools manage a project's API keys without the agent ever seeing a secret: `getCredentialStatus` reports whether the Meta DAT identity is set — a masked hint only, never the value — and `setCredential` starts **write-without-knowing** entry: it returns a link to the project's dashboard credentials page where the signed-in owner pastes the secret, which goes straight into the encrypted vault. The agent can't read a stored secret back. Both are account-scoped and gated by the per-project Credentials grant. One **Analytics** tool, `getProjectAnalytics`, reads a project's **production** telemetry — aggregate metrics (events, active installs, by event/day/vendor/platform) from the app's shipped (App Store / Play Store) installs, metadata only. It reads the prod-attested warehouse (the same data the dashboard Analytics page shows), so it stays empty until the app ships and sends prod-attested events; for the live dev/sim event stream use `getEventLog` instead. It's account-scoped, **ownership-checked** (you can only read your own projects), and gated by the per-project Analytics grant. ## Conventions A few conventions hold across every tool: - **All tool inputs and outputs are JSON.** Tools that take an object pass it via the MCP `arguments` field; the schemas live in `mcp-server/src/tools/definitions.ts`. - **`additionalProperties: false`.** Unknown parameters fail validation with `code: "invalid_arguments"` and a `fixHint` telling the agent what to drop. - **Successful responses include a `summary` string** — a short human-readable description of what the tool returned. Agents can surface this directly to the developer. - **Errors return a structured shape** — `{ code, message, fixHint?, retryable? }` — instead of throwing. The agent can decide whether to retry or surface the error. - **Most tool calls require no account.** Discovery, guidance, validation, the simulator-driving tools, and search all work anonymously — no signup, no key, no payment. Three groups need a linked (free) account: minting browser-simulator sessions (`createSimulatorSession`), the `generateConnectionModule` scaffold step (HTTP 402 — it mints the account-bound project key; the informational first call is anonymous), and the account-scoped project tools — assistant config (`getAssistantConfig` / `setAssistantConfig` / `getGatewayUsage`), credentials (`getCredentialStatus` / `setCredential`), connection-page config writes (`setConnectionPageConfig` / `adoptConnectionPageFile`), and analytics (`getProjectAnalytics`). Those return error code `account_required` until you link — the account-scoped project tools on HTTP 401, and `createSimulatorSession` on HTTP 402 with a structured `auth_required` body that kicks off the device-code login. ## The canonical agent-driven flow For a fresh project, the typical tool sequence is: ```text getPlatformInfo → getCodeExample(pattern) → getCapabilityGuide(feature) → generateConnectionModule → → validateIntegration → createSimulatorSession → → getEventLog ``` For iteration: edit handler code → rebuild + reinstall → the app auto-reattaches to the same simulator session. No mutation tools needed for handler iteration; the simulator URL is stable. Before shipping: `getProductionChecklist` → `getCredentialGuide`. ## Retired tools Pre-pivot the MCP server also exposed `initSpec`, `updateSpec`, `validateSpec`, `getExampleSpec`, and `generateConsumer`. All five are retired with the pure-SDK pivot — the library no longer interprets a spec, so DSL authoring, patching, callback-dispatch generation, and detached spec validation are all dead surface. Handler code is now hand-authored against the SDK directly; see `getCodeExample` for canonical patterns and `getCapabilityGuide` for per-feature call shape. ## Per-category reference The deep reference per tool lives on the category pages: - **[Discovery and SDK reference →](/docs/mcp-server/tools/discovery)** - **[Setup and Generation →](/docs/mcp-server/tools/generation)** - **[Implementation Guidance →](/docs/mcp-server/tools/guidance)** - **[Validation →](/docs/mcp-server/tools/validation)** - **[Simulation →](/docs/mcp-server/tools/simulation)** - **[Production →](/docs/mcp-server/tools/production)** - **[Search →](/docs/mcp-server/tools/search)** ## Source of truth Every claim on these pages is verified from `mcp-server/src/tools/definitions.ts` (input JSON Schemas) and the per-tool handler implementations under `mcp-server/src/tools/handlers/`. If a doc claim conflicts with the source, the source wins — file an issue and we'll reconcile. --- source: https://extentos.com/docs/mcp-server/tools/discovery --- --- title: Discovery and SDK reference tools description: The Extentos MCP server's discovery + reference tools — getPlatformInfo (the static catalog of vendor capabilities), getCapabilityGuide (per-feature Kotlin + Swift call shape with gotchas), and getCodeExample (full canonical compositions for the common voice-glasses patterns). These are the first calls an AI agent makes in any new task — cheap, all local, no side effects, free and anonymous. type: reference platform: mcp related: - /docs/mcp-server/tools - /docs/mcp-server/tools/generation - /docs/mcp-server/tools/search - /docs/concepts/capabilities --- These discovery + reference tools answer "what does Extentos offer for this target?" and "how do I actually call it?" before any project mutation — plus `getMigrationGuide` for apps already built on raw Meta DAT. All are pure-read, no side effects, no state changes — free and anonymous. > **For your installed agent:** these tools are invoked directly via the MCP — once the server is registered with your agent, it calls `getPlatformInfo` / `getCapabilityGuide` / `getCodeExample` and gets live, structured responses. This page exists so an agent evaluating Extentos pre-install can see the tool surface, and so human developers and search engines can find the documentation outside an MCP context. The live MCP response is authoritative; the parameter tables and example responses below are illustrative. ## `getPlatformInfo` Returns the static platform catalog — library version + the list of SDK capabilities the glasses expose (audio.transcriptions, audio.recordDiscrete, audio.speak, camera.capturePhoto, camera.videoFrames, audio.audioChunks, toggles, connection.state, …). The right first call for any new task. ### When to use - At session start, before scaffolding or writing handler code - When the agent needs to know which capabilities a vendor supports - When the agent is building a capability declaration for the manifest ### When NOT to use - For *how to call* a primitive — use `getCapabilityGuide(feature)` for the per-feature minimal usage + gotchas - For end-to-end compositional examples — use `getCodeExample(pattern)` - For project-installed state — use `inspectIntegration` ### Parameters | Parameter | Type | Required | Description | |---|---|---|---| | `sections` | `array<"version" \| "capabilities">` | yes | Which catalog sections to return. At least one. | | `glasses` | `"meta" \| "meta_rayban" \| "android_xr" \| "brilliant"` | no | Optional — defaults to `meta` (canonical token; `meta_rayban` is a legacy alias for the same vendor). `android_xr` is experimental/preview — it returns the Android XR capability catalog for discovery, while scaffolding and simulator sessions remain Meta-only. | | `expand` | `array<"schema" \| "capabilities.full" \| "capabilities.advanced">` | no | Opt-in to detail. See response-size table below. | ### Response The default response is **compact (~2 KB)** — names, categories, tier markers — sized to fit comfortably in the agent's context. Detail is opt-in via `expand`: | Response variant | Approx size | What's included | |---|---|---| | Default (compact capabilities) | ~2 KB | Vendor, DAT version, feature names + categories + tiers, toggle keys, global constraints | | `expand: ["capabilities.full"]` | ~10 KB | All of the above plus per-feature params, payload shapes, requires lists, per-feature constraints | | `expand: ["capabilities.advanced"]` | + ~1 KB | Adds `droppedPrimitives` and `futurePrimitives` informational fields | The compact-by-default design is intentional. Returning the full ~10 KB on every first call burns the agent's context window before any work. The `summary` field hints at what's available and how to expand. ### Example — discovery at session start ```json { "sections": ["version", "capabilities"], "glasses": "meta_rayban" } ``` Response (compact): ```json { "version": { "latestStable": "2.8.0", "...": "..." }, "capabilities": { "vendor": "meta_rayban", "datVersion": { "android": "0.8.0", "ios": "0.8.0" }, "features": [ { "name": "capture_photo", "category": "media", "tier": "dat_native" }, { "name": "capture_video", "category": "media", "tier": "phone_side_workaround" }, { "name": "video_frames", "category": "stream", "tier": "dat_native" }, { "name": "record_audio", "category": "media", "tier": "phone_side_workaround" }, { "name": "audio_chunks", "category": "stream", "tier": "dat_native" }, { "name": "transcription_incremental", "category": "stream", "tier": "phone_side_workaround" }, { "name": "speak", "category": "voice", "tier": "sdk_method" }, "...etc" ], "toggles": [{ "key": "listening_mode", "gates": ["transcription_incremental", "..."], "description": "..." }, { "key": "camera_streaming_enabled", "gates": ["..."], "description": "..." }], "globalConstraints": ["...", "..."] }, "summary": "Returned library + DAT versions and the meta_rayban capability catalog (compact: N audio/camera primitives, 8 toggles)." } ``` ### Errors - **`invalid_arguments` — `sections must be a non-empty array`** — pass at least one of `"version"`, `"capabilities"` --- ## `getCapabilityGuide` Per-feature SDK usage guide — minimal Kotlin + Swift snippet, config args, gotchas, and which `getCodeExample` patterns exercise the feature. Pairs with `getPlatformInfo` (which lists feature names + categories) by adding the actual idiom for using each. ### When to use - When you know which feature you need (from `getPlatformInfo`) and want the canonical call shape - When you're hitting a confusing failure — the gotchas list typically covers it - When mapping out the manifest's `capabilities` array — confirm each entry's call shape before declaring ### When NOT to use - For a complete compositional pattern — use `getCodeExample` - For capability discovery — use `getPlatformInfo` ### Parameters | Parameter | Type | Required | Description | |---|---|---|---| | `feature` | enum | yes | Feature name from `getPlatformInfo.features[].name`. Plus `connection_state` and `toggles` for the cross-cutting SDK surfaces. | Accepted feature names span the capability primitives (`capture_photo`, `capture_video`, `record_audio`, `transcription_incremental`, `speak`, `video_frames`, `audio_chunks`, `voice_command`, `earcon`), the glasses-display surface (`display`), the cross-cutting SDK surfaces (`connection_state`, `toggles`), and the **assistant runtime** (`assistant_runtime`, `assistant_start`, `assistant_tool`, `assistant_provider_openai`, `assistant_vision`, `assistant_session_runtime`). The live input-schema enum is derived from the guide catalog, so the schema is the authoritative list. ### Response ```json { "feature": "", "summary": "", "kotlin": "", "swift": "", "gotchas": ["", "", "..."], "relatedExamples": ["voice_qa_assistant", "barge_in_speak", "..."] } ``` --- ## `getCodeExample` Full canonical SDK compositions in Kotlin + Swift for the common voice-glasses patterns. Each pattern is a complete handler implementation the agent can peel from when writing real customer code. ### When to use - When you're about to write handler code and want the canonical shape - When a customer asks for a feature that maps cleanly to one of the patterns - When you need a paste-ready BYOK client (currently: Anthropic via `byok_anthropic`) — the LLM-client wrapper that the voice-Q&A and vision patterns reference as `AnthropicClient` ### When NOT to use - For capability discovery — use `getPlatformInfo` - For per-feature minimal usage — use `getCapabilityGuide` ### Parameters | Parameter | Type | Required | Description | |---|---|---|---| | `pattern` | enum (see the pattern table below — the live schema enum is derived from the served catalog) | yes | Which compositional pattern to fetch | The patterns: | Pattern ID | Composition | |---|---| | `assistant_agent_loop` | **Phase 4 canonical — start here for new voice-assistant work.** `glasses.assistant.start(provider) { tool(name, description) { body } }` plus wake wiring; the model owns wake/turn-taking/intent/confirmation, the customer writes tool bodies against app state. | | `agent_driven_e2e_full_loop` | The agent-side E2E test workflow for Phase 4 assistants — explicit two-step wake, multi-tool sweep, four-channel verification (event log + adb logcat + screencap + library state). Drives the REAL OpenAI Realtime provider as well as Mock. Companion to `assistant_agent_loop`. | | `display_browse_detail` | The glasses-display two-view navigation pattern (Ray-Ban Display): browse cards ⇄ full-screen detail, Neural-Band slide/pinch/mid-pinch and assistant tools driving one app-side state machine, with `show(onBack = ...)` back routing and the agent verification recipe. | | `display_media_gallery` | The glasses-display media-gallery pattern — paged media cards (image + caption) on the Ray-Ban Display with Neural-Band navigation (slide to page, pinch to select), one app-side paging state machine, and the agent verification recipe. | | `video_frames_ml` | Continuous `videoFrames()` → an on-device / remote ML model, with the throttling + backpressure gotchas (drop-frames-when-busy, don't queue, decode off the main loop) that keep an ML pipeline from falling behind the frame stream. | | `voice_qa_assistant` | Wake transcript match → speak prompt → recordDiscrete (silence-VAD) → LLM call → speak answer. The canonical multi-turn voice-Q&A flow. | | `barge_in_speak` | Speak with cancellation — TaskGroup-style race between `speak()` completion and a fresh Final transcript that interrupts. | | `photo_describe_voice` | Wake transcript match → speak prompt → capturePhoto → vision LLM → speak description. | | `live_transcription_ui` | Subscribe to `transcriptions()`, stream partials/finals into a Compose / SwiftUI live caption UI. | | `voice_notes` | Wake transcript match → recordDiscrete with longer max-duration → persist transcript + audio bytes to local storage. | | `connection_page_setup` | Minimum-viable scaffold — instantiate `ExtentosGlasses`, mount `ExtentosConnectionPage`, log connection state. | | `byok_anthropic` | Paste-ready Anthropic Claude API client (text + Vision, OkHttp / URLSession). Returns a sealed `LlmResult` with `Ok` / `KeyMissing` / `KeyRejected` / `Connectivity` / `Transient` / `ClientBug` / `Empty` variants so the handler can give a distinct user-facing message per failure mode. Referenced by `voice_qa_assistant` and `photo_describe_voice` as `AnthropicClient`. | | `agent_test_loop` | LEGACY (pre-Phase-4) three-surface verification recipe — for Phase 4 assistants use `agent_driven_e2e_full_loop` instead. | ### Response ```json { "pattern": "", "title": "", "description": "<2-3 sentence framing>", "code": { "kotlin": "", "swift": "" }, "explanation": "", "gotchas": ["", "", "..."], "relatedFeatures": ["transcription_incremental", "record_audio", "speak", "..."], "requiredDependencies": { "android": [{ "configuration": "implementation", "notation": "", "appliedIn": "app/build.gradle.kts" }], "iosSwiftPackages": [{ "url": "", "from": "", "products": [""] }] }, "requiredPlugins": { "android": [{ "id": "", "version": "", "appliedIn": "root build.gradle.kts", "applyFalse": true }] } } ``` `requiredDependencies` and `requiredPlugins` are optional fields populated when a pattern needs Gradle / SPM deps beyond the SDK + stdlib (currently only `byok_anthropic`, which needs OkHttp + kotlinx-serialization-json on Android plus the Kotlin serialization plugin in BOTH the root and app `build.gradle.kts`). Agents iterating multiple patterns merge the union of all `requiredDependencies` / `requiredPlugins` into the host app's build files. ### Example — peel the voice-Q&A composition ```json { "pattern": "voice_qa_assistant" } ``` The response returns a full Kotlin `CoachHandler` class (and matching Swift) that registers a wake phrase via `glasses.voice.onPhrase` (sugar over `glasses.audio.transcriptions()`), calls `glasses.audio.speak("What would you like to know?")`, then `glasses.audio.recordDiscrete(...)` with silence-VAD, calls an LLM, and speaks the answer. ### Errors - **`invalid_arguments`** — `pattern` must be one of the ids in the pattern table above. The live input-schema enum is derived from the served catalog, so the schema (not this page) is the authoritative list. --- ## `getMigrationGuide` The entry point for a developer who **already built their app against raw Meta DAT** and is adopting the Extentos SDK. Because Extentos's production transport calls the same DAT underneath, migration is a call-site swap, not a rewrite. Returns a map **keyed by your existing DAT symbol** (`Wearables.createSession`, `photoDataPublisher`, `addStream`, `Display`, raw `CaptureError`, …) → the Extentos primitive that replaces it, each naming the `feature` to drill into with `getCapabilityGuide`, plus an ordered cutover plan. ### When to use - When the project already imports `com.meta.wearable.dat.*` (Android) / `MWDATCore` / `MWDATCamera` (iOS) and you're moving it onto Extentos ### When NOT to use - For a greenfield project with no existing DAT code — start at `getPlatformInfo` → `generateConnectionModule` ### Parameters | Parameter | Type | Required | Description | |---|---|---|---| | `from` | `"meta_dat"` | no | Which SDK you're migrating from. Defaults to `meta_dat` — the only mapped source today; the enum is forward-compatible for other wrappers as they land. | ### Response Returns `approach` (`"full_cutover"`), a `coexistenceNote` (side-by-side operation is **not** supported today — the Meta `DeviceSession` is single-owner), an ordered `plan`, and `mappings[]` (each `{ area, datSymbols, extentosCall, extentosFeature, whatChanges }`). The conceptual companion is [Migrating from raw Meta DAT](/docs/guides/migrating-from-meta-dat); see also [Extentos vs Meta DAT](/docs/concepts/vs-meta-dat). --- ## Why these are the first calls `getPlatformInfo` answers "what's possible on this vendor?" — without that catalog, the agent is composing blind. `getCapabilityGuide` answers "how do I call this specific feature?" — the agent calls it once per primitive it plans to use. `getCodeExample` answers "what does a complete real flow look like end-to-end?" — useful for the catalogued shapes; for ad-hoc business logic, the agent composes from primitives directly. A typical first-time flow: ```text 1. getPlatformInfo({ sections: ["version", "capabilities"], glasses: "meta_rayban" }) 2. getCodeExample({ pattern: }) 3. getCapabilityGuide({ feature: }) 4. generateConnectionModule(...) 5. 6. validateIntegration() ``` ## Related - **[Setup and Generation tools](/docs/mcp-server/tools/generation)** — what the agent calls *after* discovery - **[Search tools](/docs/mcp-server/tools/search)** — `searchDocs` for conceptual topics - **[Capabilities](/docs/concepts/capabilities)** — the conceptual layer behind what `getPlatformInfo` returns - **[Tools overview](/docs/mcp-server/tools)** — back to the full tool catalog --- source: https://extentos.com/docs/mcp-server/tools/generation --- --- title: Setup and Generation tools description: The Extentos MCP server's Setup and Generation surface — generateConnectionModule (the one-shot project scaffold that emits the bootstrap module, build-script changes, dependencies, permissions, and the integration manifest) plus the four connection-page-config tools (getConnectionPageConfig, setConnectionPageConfig, regenerateConnectionPageFile, adoptConnectionPageFile). After scaffolding, the customer (or their agent) writes handler classes against the SDK primitives surfaced by getCapabilityGuide / getCodeExample. type: reference platform: mcp related: - /docs/mcp-server/tools - /docs/mcp-server/tools/discovery - /docs/mcp-server/tools/validation - /docs/mcp-server/tools/search --- Post pure-SDK pivot, the Setup and Generation category is **five tools**: `generateConnectionModule` (the one-shot project scaffold, below) plus four connection-page-config tools that manage the per-project theming / section-visibility config the dashboard holds. Handler code is the customer's authoring surface — there is no `initSpec` / `updateSpec` / `generateConsumer` because there's no spec to populate, no callback dispatch table to generate. The agent peels from `getCodeExample(pattern)` to write the handler in Kotlin/Swift directly. > **For your installed agent:** call `generateConnectionModule` once per fresh project, then write handler classes. Subsequent code iterations don't need any more generation tool calls — rebuild + reinstall the app and the simulator auto-reattaches. This page is the reference for the one tool that does emit files. ## `generateConnectionModule` One-shot project scaffold. Emits the bootstrap module that wires `Extentos.create(...)` into the host app, build-script changes (Gradle / SPM), dependencies, permissions, and `extentos.manifest.json`. Two-call flow: the first call returns a placement question; the second call (after the developer picks where `ExtentosConnectionPage` should live) returns the full file set. ### When to use - Exactly once per platform per project — Android scaffolding on first Android target, iOS scaffolding on first iOS target - After Extentos is removed (e.g., the dev wiped the `/extentos` directory) and you need to re-scaffold ### When NOT to use - When Extentos is already installed (use `inspectIntegration` to read current state) - For per-handler code generation — there is no per-handler tool; write the handler against the SDK primitives by hand ### Parameters | Parameter | Type | Required | Description | |---|---|---|---| | `platform` | `"android" \| "ios"` | yes | Target platform for this scaffold call | | `glasses` | `"meta" \| "meta_rayban" \| "android_xr"` | yes | Vendor. **Canonical: `meta`**; `meta_rayban` is a legacy alias for the same vendor. `android_xr` is accepted by the schema but scaffolding is Meta-only today, so it is rejected with a dedicated message. | | `appPackage` | string | yes | The **Extentos app id** for this project (e.g. `com.example.myapp`) — baked into the scaffold as `config.appId`, which keys telemetry, agent config, and sounds. Conventionally your Android `applicationId` / iOS bundle identifier, but deliberately **decoupled** from the store id: pass the SAME value on both platforms and they share one Extentos project even when the store bundle ids differ. | | `libraryVersion` | string | no | Pin to a specific library version. Defaults to latest stable. | | `placement` | enum | conditional | `dedicated_route` / `settings_subscreen` / `bottom_tab` / `modal_sheet` / `headless`. Omit on the first call to receive the placement question. | | `projectPath` | string | no | Absolute path to the project root. **Android:** when given, the handler reads `AndroidManifest.xml` and, if you already have an `Application` subclass, emits an init-helper `object` + a `manual_patch` for it instead of clobbering your class. Omit → greenfield (emits a full `Application` subclass). | | `capabilities` | string[] | no | The SDK capabilities your app uses (feature names from `getPlatformInfo`, e.g. `["capture_photo", "transcription_incremental", "speak", "display"]`). Recorded in the manifest **and** emitted as `ExtentosConfig.usedCapabilities` on both platforms (Kotlin `setOf(CapabilityKind.X, …)` / Swift `[.camera, …]`) so the connection page shows one tile per capability — lit when the connected glasses provide it, dimmed when they don't. Omit → no capability tiles and no derived camera/mic/speaker permissions (declare nothing, show nothing — valid for a connection-only app; re-run with `capabilities` set to add them). | | `responseFormat` | `"concise" \| "detailed"` | no | `detailed` (default) includes per-step `wireInstructions` + `notes`. `concise` omits the per-step `wireInstructions` + `notes`; it still returns the full files / dependencies / rendering payload. | ### Two-call flow **Call 1 — without `placement`:** ```json { "platform": "android", "glasses": "meta_rayban", "appPackage": "com.example.myapp" } ``` Returns `status: "needs_placement"` plus the placement question and the 5 valid IDs. The agent reads the question to the developer, gets their pick, and calls again. **Call 2 — with `placement`:** ```json { "platform": "android", "glasses": "meta_rayban", "appPackage": "com.example.myapp", "placement": "dedicated_route" } ``` Returns the full file set: `files[]` (the bootstrap module to write), `manifest` (the `extentos.manifest.json` content), `dependencies[]`, `repositories[]`, `buildConfigFields[]`, `permissions[]`, `suggestedRendering` (the Compose / SwiftUI snippet for the chosen placement), plus `developerInstructions` walking the agent through applying everything. ### What gets emitted - **`files[]`** — `ExtentosBootstrap.kt` (Android) or `ExtentosBootstrap.swift` (iOS), with `action: "create"` (fully tool-managed; safe to overwrite on re-run). On iOS, also an `action: "manual_patch"` entry for the developer's `App.swift` with `onOpenURL` + `scenePhase` wiring instructions. - **`manifest`** — `extentos.manifest.json` v2 content: library version, installedAt, platform / glasses / appPackage metadata, `gradle` or `spm` build-config record, empty `permissions[]` + `capabilities[]` + `handlerNames[]` arrays for the agent to populate as the handler grows. - **`dependencies[]`** — `com.extentos:glasses` + `com.extentos:glasses-ui` (Android), **plus `com.extentos:glasses-meta` when the declared `capabilities` include camera or display** — that module is what carries the vendor SDK; a voice-only footprint omits it - **`repositories[]`** — the Meta DAT GitHub Packages repo (Android only), for the `mwdat-*` artifacts that arrive through `com.extentos:glasses-meta`. **Emitted ONLY for a camera/display footprint** — a voice-only scaffold returns an empty `repositories[]` and needs no token. - **`buildConfigFields[]`** — `EXTENTOS_SESSION_URL = null` placeholder (Android). Default null routes to auto-bind; only a URL-bake fallback (from `createSimulatorSession`) replaces it. - **`permissions[]`** — base Bluetooth + INTERNET permissions. Capability-specific permissions get added later when the agent calls `getPermissions(capabilities, platform)` after declaring capabilities in the manifest. - **`suggestedRendering`** — the Compose (Android) or SwiftUI (iOS) snippet for the chosen `placement`. Same shape `searchDocs(topic: "connection_ui_placement")` returns. - **`developerInstructions`** — numbered steps for the agent to apply the scaffold + know what to do next (write handler code, update manifest.capabilities, validateIntegration, …). ### What does NOT get emitted - Handler classes. Those are the customer's authoring surface. The agent peels from `getCodeExample(pattern)` and writes them directly to the customer's repo. - An `extentos.spec.json`. There is no spec runtime post-pivot. - Callback dispatch stubs. There is no `app_callback` dispatch. ### Errors - **`invalid_arguments` — `platform must be "android" or "ios"`** - **`invalid_arguments` — `glasses must be "meta" — or the legacy alias "meta_rayban"`** — `android_xr` gets its own message explaining that code scaffolding is Meta-only - **`invalid_arguments` — `appPackage is required and must be a non-empty string`** - **`invalid_arguments` — `placement must be one of dedicated_route, settings_subscreen, bottom_tab, modal_sheet, headless`** — only those 5 are valid - **`account_required` (HTTP 402)** — the scaffold step (with `placement`) mints an account-bound project key; anonymous installs get the device-code login flow, same as `createSimulatorSession`. The informational first call (no `placement`) works anonymously --- ## Connection-page config tools The connection page (the screen the end user sees to pair their glasses) can be themed and have its sections toggled per project. These four tools manage that config: the dashboard/server is the single source of truth when a project is **managed**, and a committed `extentos.connection-page.json` mirrors it into the repo so the app themes correctly offline. The writes are account-scoped (run `extentos-mcp login` once). | Tool | Direction | What it does | |---|---|---| | `getConnectionPageConfig` | read | Read the per-project config (theming tokens + section visibility) the dashboard holds. Returns `{ managed, config }` — `managed: true` means the server is the single source of truth (the SDK applies `server.overlay(defaults)` at render time). | | `setConnectionPageConfig` | write (server) | Persist theming tokens + section visibility to the dashboard, making the project **managed**. Validated against the connection-page schema; unknown keys surface as a warning, not a block. | | `regenerateConnectionPageFile` | server → file | Regenerate the committed `extentos.connection-page.json` from the server config. Returns `{ path, content }` to write — it carries a generated `_comment` marker, so don't hand-edit it; edits belong in the dashboard. | | `adoptConnectionPageFile` | file → server | Seed an existing committed `extentos.connection-page.json` up to the dashboard. Drift-protected: if a differing managed config already exists it returns `needs_confirmation` and does **not** clobber — re-call with `confirm: true` to let the file win. | The file-sync pair (`regenerate` / `adopt`) is Android-only today (iOS file-sync is Phase 4). The web dashboard's **Connection** section is the human-facing equivalent of `set`. --- ## Retired generation tools Pre-pivot the MCP server also exposed `initSpec`, `updateSpec`, and `generateConsumer`. All three are retired with the pure-SDK pivot: - `initSpec` populated the first `extentos.spec.json` from agent-composed primitives → no spec to populate post-pivot. - `updateSpec` mutated an existing spec and live-pushed the change to the simulator → no spec, no live-push. - `generateConsumer` stubbed `app_callback` / stream-consumer handler classes from spec declarations → handler code is now hand-authored against the SDK directly. Calls to these tool names against the post-pivot MCP server return `unknown_tool`. Use `getCodeExample(pattern)` for canonical compositions and write handler code directly. ## Related - **[Discovery tools](/docs/mcp-server/tools/discovery)** — `getPlatformInfo` / `getCapabilityGuide` / `getCodeExample` (what the agent calls *before* this) - **[Validation tools](/docs/mcp-server/tools/validation)** — `inspectIntegration` / `validateIntegration` (what the agent calls *after* this + after writing handler code) - **[file_actions topic](/docs/mcp-server/tools/search)** — how to apply each `action` value (`create` / `manual_patch`) --- source: https://extentos.com/docs/mcp-server/tools/guidance --- --- title: Implementation Guidance tools description: The Extentos MCP server's guidance tools — getVoiceCommandGuidance (analyze proposed wake / command phrases for UX issues) and getPermissions (derive exact platform permissions, Meta DAT requirements, and foreground-service needs from the declared SDK capability list). Side-quest helpers the agent calls during composition. type: reference platform: mcp related: - /docs/mcp-server/tools - /docs/mcp-server/tools/generation - /docs/concepts/capabilities - /docs/concepts/permissions --- Two helpers the agent calls between scaffolding and validation. Both are pure-read, no side effects. ## `getVoiceCommandGuidance` Analyze proposed wake / command phrases for UX issues — length, homophones, digit-usage, Meta wake-word collision (`hey meta`, `hey facebook`, `ok meta` are reserved), ambiguity against existing phrases — before the agent wires them into a `glasses.audio.transcriptions()` consumer. **Parameters:** `phrases: string[]` (required), `existingPhrases: string[]` (optional, so the tool can flag ambiguity against phrases the handler already matches), `glasses: "meta_rayban"`. **Response:** per-phrase analysis with `issues[]` (length / homophones / digits / wake-word collision), `suggestions[]` (concrete rewrites), `collisions[]` (other phrases too similar), plus a roll-up `summary`. Post-pivot voice matching is plain string contains/regex inside the customer's handler — this tool catches the issues the customer can't see at runtime ("didn't match"-style failures with no signal from the recognizer). ## `getPermissions` Derive Android manifest permissions, iOS Info.plist keys, and Meta DAT scopes from the list of SDK capabilities the handler uses. Run after the agent decides which primitives (`capture_photo`, `transcription_incremental`, `speak`, …) the handler will subscribe to. **Parameters:** `capabilities: string[]` (required; feature names from `getPlatformInfo.features[].name`), `platform: "android" | "ios"`. **Response:** `{ android: { permissions[], manifestEntries[], foregroundService, ... }, ios: { plistKeys[], ... }, metaDat: { scopes[], registrationRequired, ... }, runtimeGrants: { grants[], note }, summary }`. The manifest halves drop in directly to `AndroidManifest.xml` / `Info.plist`. **Applying every returned key is not the whole job.** `runtimeGrants` is the half that is not a manifest entry: the OS grants nothing until the app *asks* at runtime. Each entry names who asks — `requestedBy: "app" | "sdk"` — the exact call when it is yours, and what silence looks like when nobody makes it. On iOS the **microphone** is the app's job and its absence is invisible: no prompt, no error, no transcripts, every voice command dead on hardware while working perfectly in the simulator, where transcripts arrive from the browser and never touch the phone's microphone. Speech recognition, by contrast, the SDK requests for you. Found the hard way, 2026-08-04. The capability → permission map is centralized in `mcp-server/src/tools/util/permissions.ts`. `transcription_incremental` adds `RECORD_AUDIO` + `NSMicrophoneUsageDescription` + `NSSpeechRecognitionUsageDescription`; `capture_photo` adds `CAMERA` + `NSCameraUsageDescription`; continuous-capture capabilities (`video_frames`, `audio_chunks`, `transcription_incremental`) additionally require `FOREGROUND_SERVICE` on Android 14+. See **[concepts/permissions](/docs/concepts/permissions)** for the full mapping table. Update `extentos.manifest.json`'s `capabilities` array as your handler grows; re-run `getPermissions` and write the diff into the manifest / Info.plist. ## Related - **[Capabilities](/docs/concepts/capabilities)** — the SDK primitive vocabulary - **[Permissions](/docs/concepts/permissions)** — the full per-capability permission mapping - **[Tools overview](/docs/mcp-server/tools)** — back to the full tool catalog --- source: https://extentos.com/docs/mcp-server/tools/production --- --- title: Production tools description: The Extentos MCP server's production tools — getProductionChecklist (personalized ship-readiness checklist based on declared capabilities + handler names) and getCredentialGuide (step-by-step credential setup for Anthropic, OpenAI, Google Cloud, DeepL, Azure, AWS Bedrock, HuggingFace, or custom providers). Pre-ship gates. type: reference platform: mcp related: - /docs/mcp-server/tools - /docs/mcp-server/tools/validation - /docs/resources/security --- Two tools the agent invokes at the end of the dev loop, before the developer cuts a release. Both are pure-read, no side effects. ## `getProductionChecklist` Personalized production-readiness checklist based on what the handler uses. The agent passes the manifest's `capabilities` array + `handlerNames` array; the response is a categorized list of "before you ship" items the agent walks through with the developer. **Parameters:** `capabilities: string[]` (required; SDK feature names the handler uses), `handlers: string[]` (optional; the customer's handler class names — labels the API-keys step's affected handlers), `services: string[]` (optional; the BYOK provider names the app actually integrates, e.g. `["anthropic", "openai"]` — **this is what triggers the API Keys step**, not `handlers`), `platform: "android" | "ios"`, `projectPath` (optional). **Response:** a personalized checklist spanning categories such as Meta Developer Account, DAT Registration, Credential Swap, API Keys (only when `services` are declared — a DAT-native or managed-gateway app declares none and skips it), Permissions Audit, Foreground Service / Background Modes (skipped if no continuous-capture capabilities), Transport Config, Real-Hardware Verification (when the app uses voice / video / audio capture), Privacy (when the app can render a local capture on the display), and Store Listing. Each step is personalized to the developer's actual app: a no-camera app skips camera entries, an app with `speak` but no `transcription_incremental` skips STT-related steps. The Real-Hardware Verification step is the load-bearing one — Extentos validates simulator integration cleanly, but final-mile glass-fit (Bluetooth latency on the developer's specific glasses, real-world A2DP/HFP coexistence, camera/mic fidelity in actual wearing conditions) only verifies on hardware. The checklist surfaces it as required before cutting a release. The **Privacy** step appears only when the declared capabilities include `display` *and* a capture capability — the combination that lets the app call `display.show { image(photo) }` with a local capture, which uploads a copy of the end user's media to Extentos's media host. It's marked `needed`, and it covers the two things that are easy to miss: wiring `forgetHostedImage` / `forgetHostedVideo` into the app's delete path, and disclosing the upload in the app's own privacy policy. See [hosted display media](/docs/resources/security#hosted-display-media--the-one-content-path). **Store Listing** is `needed` rather than `optional` whenever the app declares a camera or microphone capability, since both stores hard-gate those on a published privacy policy. ## `getCredentialGuide` Step-by-step credential setup for the BYOK AI providers the handler uses, plus Meta DAT registration. The agent passes the list of providers it's about to call from the handler; the response walks the developer through each. **Parameters:** `services: string[]` (**optional** — omit or pass `[]` for the Meta-DAT-only guide; otherwise one or more of `anthropic`, `openai`, `google_cloud_vision`, `google_translate`, `google_gemini`, `deepl`, `azure_cognitive`, `aws_bedrock`, `huggingface`, `custom`), `handlers: string[]` (optional; handler class names — used to label which handler each provider is for), `platform: "android" | "ios"` (required). **Response:** per-provider numbered steps (sign-up URL, account-creation cost if any, where to find the API key, what to paste where, how to test). Plus a Meta DAT registration walkthrough — required at first real-hardware test (one-time, free, ~15–30 min): create Meta Developer account, enable Wearables DAT, register package name + signing signature (Android) or bundle ID + Team ID (iOS), copy App ID + Client Token into the manifest (Android) / Info.plist MWDAT dict (iOS). **Android storage guidance:** the response recommends a **`local.properties`-first** pattern reading the key at Gradle config time and injecting via `resValue` + `R.string` at runtime. CI/release builds fall back to environment variables baked into the same snippet via `?: System.getenv(...)`. This is deliberately NOT env-var-first: Gradle daemons cache the env at startup, so a rotated key in your current shell isn't visible until you `./gradlew --stop` and restart the IDE — the failure mode the R2 dogfood F-R2-9 finding surfaced as "I couldn't read the cards." `resValue` is preferred over `buildConfigField` because BuildConfig String constants are compile-time-inlined by kotlinc, and stale keys silently survive rotations (the original `BuildConfig` API-key gotcha that landed in dogfood F22). **BYOK client return shape:** the canonical Anthropic client emitted by `getCodeExample('byok_anthropic')` returns a sealed `LlmResult` type with `Ok` / `KeyMissing` / `KeyRejected` / `Connectivity` / `Transient` / `ClientBug` / `Empty` variants — NOT a bare `String`. Handler code pattern-matches each variant into a distinct user-facing line so the end user can tell "fix your API key" from "the AI couldn't see well enough" from "try again, network's flaky." Flattening these into one message (the dogfood F-R2-10 finding) makes the failure invisible to non-technical end users. ## Related - **[Validation tools](/docs/mcp-server/tools/validation)** — what to run before the production checklist - **[Security](/docs/resources/security)** — what Extentos collects vs what stays in the developer's app - **[Tools overview](/docs/mcp-server/tools)** — back to the full tool catalog --- source: https://extentos.com/docs/mcp-server/tools/search --- --- title: Search tools description: The Extentos MCP server's searchDocs tool — the bundled conceptual docs corpus. Phase-4 topics (assistant_runtime, managed_gateway, conversation_memory, display, agent_e2e_testing) plus the stable set (getting_started, custom_handlers, voice_integration, library_api, toggles, simulator_browser_mode). type: reference platform: mcp related: - /docs/mcp-server/tools - /docs/mcp-server/tools/discovery --- `searchDocs` is the conceptual / narrative layer of the MCP tool surface. The other tools answer "what / how / where" with structured data; `searchDocs` carries the prose that explains the *why* and the *idiom*. ## `searchDocs` Search the Extentos documentation corpus by topic or keyword. Topic IDs are stable; the corpus is regenerated when the library or backend changes shape. ### When to use - To learn how the SDK is meant to be composed (`custom_handlers`, `voice_integration`) - To read the conceptual model for a specific area (`toggles`, `connection_state_model`, `permissions`) - To look up the simulator dev-loop semantics (`simulator_browser_mode`, `auto_bind_session_lifecycle`, `local_bridge_discovery`) - To understand the manifest schema (`manifest_format`), file-action contract (`file_actions`), event-log chips (`event_log_schema`) ### When NOT to use - For the live capability list — use `getPlatformInfo` - For per-feature call shape — use `getCapabilityGuide` - For complete compositional patterns — use `getCodeExample` - For project-installed state — use `inspectIntegration` ### Parameters | Parameter | Type | Required | Description | |---|---|---|---| | `topic` | string | conditional | The topic ID to fetch. Pass alone to retrieve full topic content (most common). | | `query` | string | conditional | Keyword(s) to search. Pass alone for cross-topic search; pass with `topic` to narrow within a topic. | | `mode` | `"full" \| "snippets"` | no | Response shape. `full` (default) returns each matching topic's full body. `snippets` returns only the paragraphs containing the query terms (up to ~3 per topic) — best when searching across topics so a single search doesn't pull in unrelated content. `snippets` requires a `query`. | **At least one of `topic` or `query` is required** — calling with neither returns `invalid_arguments`. ### Aligned post-pivot topics Read these to learn the current model: - **`getting_started`** — the full topic index. Read first to see what's available. - **`custom_handlers`** — the canonical SDK-composition doc. How to write a Handler class that subscribes to capability primitives. The post-pivot replacement for the retired `app_callback_guide`. - **`voice_integration`** — STT + TTS architecture, what the simulator routes through vs what production routes through. - **`connection_ui_placement`** — five placement options (`dedicated_route` / `settings_subscreen` / `bottom_tab` / `modal_sheet` / `headless`) with Compose + SwiftUI snippets. - **`host_app_scaffold`** — copy-pasteable empty-app template for devs who don't have an existing app yet. - **`auto_bind_session_lifecycle`** — how the running app reattaches to a persistent simulator session via the MCP local bridge. - **`local_bridge_discovery`** — the `127.0.0.1:31337/whoami` probe protocol. - **`device_code_flow`** — account-linking lifecycle when `createSimulatorSession` returns `auth_required`. - **`connection_state_model`** — the 6 `GlassesState` buckets + transitions. - **`permissions`** — full Android manifest + iOS Info.plist + Meta DAT scope derivation. - **`production_checklist`** — categories, what's personalized vs static. - **`concurrency_modes`** — running multiple handler subscriptions concurrently, restart semantics, scope lifetime. - **`multi_platform_projects`** — how project identity comes from `appPackage` (`config.appId`, decoupled from the store bundle id) and how the same app id shares one project across platforms. - **`library_api`** — the post-pivot SDK reference: `ExtentosGlasses` and its typed sub-clients (connection / camera / audio / display / runtime / toggles / voice / assistant / telemetry / observability / device / …), the `ExtentosResult` + error types, and the Photo / Videos URI helpers. - **`toggles`** — all 8 runtime toggles, their enforcement status, the canonical read / write patterns. - **`audio_video_coexistence`** — A2DP / HFP profile conflict policy. - **`simulator_browser_mode`** — the persistent-sim model, get-or-create semantics, iteration model. - **`simulator_session_lifecycle`** — mint once, reuse, delete deliberately: why a project has exactly one sim per platform, why sims never expire, and the two-step rotation. - **`event_log_schema`** — every event chip (`errors` / `voice` / `camera` / `display` / `ai` / `lifecycle` / `custom`) and event type the structured log carries. - **`manifest_format`** — `extentos.manifest.json` v2 shape (post-pivot — no `spec` field, no `integration.{blocks,streams,triggers,handlerNames,togglesOverridden}` sub-object). - **`file_actions`** — the two post-pivot file-action values (`create` / `manual_patch`) and how to apply each. - **`connection_ui`** — `ExtentosConnectionPage` customization layers (drop-in / themed / `@ExtentosEscapeHatch`). - **`connection_page_schema`** — the `extentos.connection-page.json` theming + section-visibility schema the connection-page-config tools read and write. - **`credentials`** — the credential model: the dashboard vault, write-without-knowing entry, BYOK vs. Meta DAT identity, and what's stored where. - **`simulator_local_mode`** — the `LocalSimTransport` dev loop: Extentos's own in-memory deterministic transport (no DAT SDK, no network), the fastest inner loop. - **`voice_ux_guide`** — wake-phrase UX: there's no DSL matcher, the canonical case/punctuation normalizer, and the phrase-collision pitfalls `getVoiceCommandGuidance` catches. - **`constraints_and_limitations`** — the Meta DAT constraints (A2DP/HFP exclusivity, no custom gestures, "Hey Meta" reserved, background limits). This list is illustrative, not exhaustive — fetch **`getting_started`** for the live topic index. ### Phase 4 / assistant + gateway The current voice-assistant and managed-AI surface. Read `assistant_runtime` first for any voice-assistant work: - **`assistant_runtime`** — the Phase-4 assistant runtime. `glasses.assistant.start(provider) { tool(...) }`: the model owns wake / turn-taking / intent / confirmation; the customer writes tool bodies against app state. Read this first for voice assistants. - **`agent_e2e_testing`** — the agent-driven end-to-end test loop for assistants: inject a transcript / utterance, drive the real provider, assert tool calls, and verify across the event log + adb + screencap + library state with no human in the loop. - **`managed_gateway`** — the Extentos managed AI gateway: the default zero-config path (no provider key in your app), no bring-your-own-key option, metering, and attribution via the project key. - **`conversation_memory`** — the assistant memory model: within-session working memory plus the persistent per-wearer profile. - **`display`** — the glasses-display capability (Ray-Ban Display): runtime-detected, additive `show(...)` trees, Neural-Band navigation, and the never-throw degrade on devices without a display. ### Retired topics `trigger_types`, `action_types`, `block_types`, `stream_types`, `spec_format`, `template_syntax`, `app_callback_guide`, `spec_validation_rules` — all retired with the pure-SDK pivot. Calling `searchDocs` with these IDs returns an `unknown_topic` error (`Topic "" is not one of the N indexed topics`). The conceptual ground each covered is now in `custom_handlers` (for the composition layer) and `library_api` (for the SDK surface). Pre-pivot users searching for them should be steered to those two. ### Response ```json { "results": [ { "topic": "", "title": "", "content": "", "relatedTools": ["getPlatformInfo", "getCapabilityGuide", "..."] } ], "totalResults": 1, "mode": "full", "summary": "Returned full content for topic \"\"." } ``` `relatedTools` steers the agent to the structured tools that complement the prose. ## Related - **[Discovery tools](/docs/mcp-server/tools/discovery)** — the structured tools that pair with `searchDocs` - **[Tools overview](/docs/mcp-server/tools)** — back to the full tool catalog --- source: https://extentos.com/docs/mcp-server/tools/simulation --- --- title: Simulation tools description: Extentos MCP simulation tools — provision and operate browser-mode simulator sessions, plus agent-driven testing tools that close the E2E loop with no human. type: reference platform: mcp related: - /docs/mcp-server/tools - /docs/mcp-server/tools/validation - /docs/concepts/transport-vs-app - /docs/mcp-server/tools/search --- These tools orchestrate the browser-simulator dev loop — plus the agent-driven testing tools (below) that close the end-to-end loop without a human. The simulator hosts your customer-built app running against a Meta-DAT-shaped transport stub — `glasses.audio.transcriptions()`, `glasses.camera.capturePhoto()`, `glasses.audio.speak()` and every other capability primitive flow through it exactly as on real hardware. The full parameter set for every tool is the generated [MCP tools reference](/docs/reference/mcp-tools). ## `createSimulatorSession` **Get-or-create** a browser-based simulator session for the current project. Persistent-simulations model: each `(account, project, platform)` has at most one saved sim. Calling this returns that sim's existing URL (`status: "resumed"`) instead of minting a new one. **Parameters:** `glasses: "meta_rayban"` (required), `platform: "android" | "ios"`, `projectPath`, `recordBinary`, `autoOpenBrowser`, `autoLink` (default true; auto-poll the device-code flow if `auth_required` returns), `autoLinkSeconds` (default 30). **Response:** `{ status, sessionId, sessionUrl, transportEndpoint, expiresAt, autoBind: "attached" | "no_device" | "error", capabilities, developerInstructions, summary }`. Auto-bind tries to attach the running app via the local MCP bridge; falls back to a URL-bake snippet (Android `buildConfigField` patch / iOS `extentos.session.plist` write) when the bridge can't reach. **Iteration model — do NOT call this per change:** app-code edits rebuild + reinstall and reattach automatically. The simulator URL is stable. Calling this again on the same project just returns the same sim — cheap, idempotent. **Sims do not expire.** There is no 24-hour cap, no idle timeout and no cleanup sweep — yesterday's sim is still there this morning, and a brand-new agent session in the same project resumes it. The `expiresAt` field in the response is a far-future sentinel kept for wire compatibility; don't branch on it. ## `deleteSimulatorSession` Retire a session on purpose. This is the **first half of a rotation**: delete, then mint. There is deliberately no one-call force-fresh, because a silent replace is what produced sim churn — discarded sessions leave device registrations that outlive them, so the next mint could auto-bind to an app that was already gone and report it as attached. **Parameters:** `sessionId` (required). **Use it only when you genuinely need a different session identity** — the URL leaked, or you want a deliberate clean-slate id. Not to "start over" when something looks wrong: deleting disconnects any attached app and browser, and a URL-baked app needs a rebuild against the new URL. A clean event log doesn't need a new sim either — `getEventLog` is filterable. Idempotent: deleting a session that doesn't exist returns `alreadyGone: true` rather than an error. ## `ensureSimulatorBrowser` Guarantee a **connected** simulator browser tab — open one if needed and confirm its WebSocket actually attached before returning. The browser tab is the hardware surrogate's viewport: `setSimVideo` returns `browser_not_attached` without it, and the camera flows the assistant runtime drives fail. **Idempotent** — a no-op if a browser is already connected, so it's safe to call before every camera flow. **When to call:** right after `createSimulatorSession` before a camera-driven flow; after a backend deploy (which severs every sim WebSocket — the tab does not auto-reconnect); whenever the tab was closed; or any time the browser role shows as detached. **Parameters:** `sessionId` (required), `timeoutMs` (default 20000, clamped `[5000, 60000]`), `autoOpen` (default true). **Response:** `{ alreadyOpen, opened, browserConnected, browserClientId, appConnected, sessionUrl, waitedMs, summary }`. On an auto-open that doesn't attach in time, errors with `browser_not_connected` (carrying `sessionUrl`). It reads the **authoritative in-memory hub liveness** — the same signal `setSimVideo`/inject enforce, so it can't disagree with them (unlike `getSimulatorStatus.connectedRoles`, which is persisted state that can lag a dead socket by ~25–50s). When a tab is needed it opens the session URL in your default browser **cross-platform, from the MCP server itself** (no shell command for you to run), then polls until the WebSocket attaches. **Headless / remote agents:** pass `autoOpen: false` to get the `sessionUrl` back with no spawn and no wait — surface it to the developer to open on a machine with a display, then re-run with `autoOpen: true` to confirm. ## `completeAuthLink` Poll the backend until the developer finishes signup at the verification URL, then persist the bearer token to `~/.extentos/auth.json`. Use only when `createSimulatorSession` returns `status: "auth_required"` AND you opted out of `autoLink` (or the inline poll timed out). **Parameters:** `deviceCode` (from the prior `createSimulatorSession` response), `maxWaitSeconds`, `pollIntervalSeconds`. ## `getEventLog` Fetch structured event trace inside a simulator session. The primary debugging tool — answers "why didn't my handler see what I expected?" **Parameters:** `sessionId` (required), `filter` (`all` / `errors` / `voice` / `camera` / `display` / `ai` / `lifecycle` / `custom`), `limit`, `cursor` (opaque keyset cursor — pass it back to fetch only newer events), `follow` (block until new events land — live-tail), `timeoutMs`, `responseFormat` (`concise` / `detailed`), `redactBinary` (default true — large data: URIs and >2KB strings collapse to a `[redacted: N chars]` marker so a single capture_photo response doesn't blow the agent's context). **Response:** `{ events: EventRecord[], sessionStatus, totalEvents, hasMore, summary }`. Each event carries `layer` + `severity` + `type` + `message` + optional `details`. Consecutive payload-identical events collapse into one entry with `repeat: N` + `lastTimestamp` (timestamps excluded from the identity); `totalEvents` counts the raw events, and `collapseRepeats: false` returns the uncollapsed rows. Post-2026-05 redesign uses a 7-chip taxonomy. One label per event: severity≥warn absorbs into `errors` regardless of modality (so `camera_error` lives in `errors`, not `camera`); then prefix-match into `voice` / `camera` / `display` / `ai` / `lifecycle`; then `custom` for app-emitted opens. Pre-pivot filter values (`triggers` / `blocks` / `callbacks` / `spec`) AND the intermediate layer values (`transport` / `audio` / `speak` / `toggles` / `streams` / `system`) are retired. See `searchDocs("event_log_schema")` for the complete rules + event-type catalog per chip. ## `getSimulatorStatus` Read a live simulator session's current state — phase (active / paused / closed), hardware-ready, attached roles (app + browser), the **`testVideos`** available to drive `setSimVideo`, and a per-role **freshness** advisory that flags a connection gone stale after a rebuild or backend redeploy. **Parameters:** `sessionId` (required). **Response:** `{ sessionId, sessionState, hardwareReady, connectionState, connectedRoles, expiresAt, recording, freshness, summary }`. Use during testing to confirm the session is healthy, a toggle is set, or the app role has actually attached. Distinct from `getEventLog` (event trace) — this is the snapshot view. This tool does **not** report which capability streams are open. It carried `activeStreams` and `lastEventTimestamp` until 0.11.55, but nothing ever populated them — across 251 production sessions neither field was ever set — so the response always claimed zero open streams even while a subscription was running. They were removed rather than left misreporting. To check whether a subscription is live, read `getEventLog` or your own app's state. ## Agent-driven testing tools Eight more tools close the end-to-end test loop without a human or hardware — they drive input and read back what happened. Full parameters are in the [MCP tools reference](/docs/reference/mcp-tools#simulation--agent-driven-testing); in brief: - **`injectTranscript`** — inject a synthetic STT transcript to drive a wake phrase or a `glasses.voice.onPhrase` matcher. - **`injectAssistantUtterance`** — drive a Phase-4 assistant turn (Mock or real OpenAI); returns a `watchCursor`. - **`assertToolCalled`** — wait for an `assistant.tool_called` event (pass the `watchCursor` from the inject so the wait anchors before it). - **`setSimVideo`** — pipe a test video into the simulated camera so `capturePhoto` / `videoFrames` run against a known scene. - **`setSimDevice`** — switch the simulated device model (e.g. `rayban_display` to exercise `glasses.display.isAvailable`). - **`getDisplayState`** — read the display tree currently rendered plus its selectable node ids. - **`injectInput`** — drive display input (`select` / `navigate` / `back`) to fire a display node's `onClick`. - **`injectHardwareButton`** — press the hardware capture button (the right temple): tap = pause/resume the live camera stream, hold = stop it, so paused/stopped-camera handling (`CaptureError.StreamPaused`, re-arm after a stop) is testable end-to-end. Mirrors the sim page's Hardware-buttons panel and its capture LED. **A vendor is simulatable only on a platform that could host it.** `createSimulatorSession({ platform: "ios", glasses: "android_xr" })` is refused, and so is switching an iOS session onto an `android_xr_*` device (`vendor_platform_unsupported`). Android XR is Android-only *structurally* — Google's model is a projected activity, and a projected activity is an Android activity — so no iOS transport can exist, and simulating one would teach your app a combination no device can be. `meta` and `brilliant` simulate on either platform. This is a hard refusal, unlike a missing vendor *module*: you fix a missing module with one dependency line, and you cannot fix a platform that has no such API. Together with `getEventLog(follow: true)` these are the agent-driven E2E loop — see the [voice-assistant guide](/docs/guides/voice-assistant) and `getCodeExample('agent_driven_e2e_full_loop')`. ## Related - **[Transport vs app simulation](/docs/concepts/transport-vs-app)** — what the simulator actually simulates and why - **[searchDocs topic: simulator_browser_mode](/docs/mcp-server/tools/search)** — deep dive on the iteration model + persistent-sim semantics - **[searchDocs topic: simulator_session_lifecycle](/docs/mcp-server/tools/search)** — mint once, reuse, delete deliberately - **[searchDocs topic: event_log_schema](/docs/mcp-server/tools/search)** — every event layer + type the log carries - **[Tools overview](/docs/mcp-server/tools)** — back to the full tool catalog --- source: https://extentos.com/docs/mcp-server/tools/validation --- --- title: Validation tools description: The Extentos MCP server's validation tools — inspectIntegration (read-only project snapshot) and validateIntegration (whole-project correctness gate before testing). The pre-test sanity checks the agent runs after structural changes. type: reference platform: mcp related: - /docs/mcp-server/tools - /docs/mcp-server/tools/generation - /docs/mcp-server/tools/simulation --- Two validation tools answer "is this integration structurally correct?" before the agent points the simulator or a real device at it. Both are pure-read, no side effects. ## `inspectIntegration` Read-only snapshot of the project's Extentos integration state — manifest, generated-file hashes, dependency status, connection-page config. Use when the agent needs to know what's wired before making decisions. **When to use:** before manual edits to understand current state; as a "what's already installed?" probe. **When NOT to use:** for correctness checking (use `validateIntegration` instead). **Parameters:** `projectPath` (optional; defaults to cwd). **Response:** `{ found, manifest, generatedFilesStatus, dependency, drift, connectionPageConfig, gaps, summary }`. `gaps[]` lists obvious missing pieces (no manifest, missing dependency, etc.); `drift.fileDrift[]` flags generated files whose normalized hash doesn't match the recorded value. The drift check is cosmetic-tolerant: line-ending differences (CRLF vs LF), trailing whitespace, and block-comment edits (e.g., trimming the KDoc atop `ExtentosBootstrap.kt` for readability) do NOT trip drift. Structural changes — rename, body edits, added/removed declarations — still do. Pre-R2-F-R2-5 the drift hash was byte-strict and tripped on whitespace + Windows line endings; the validator now accepts both the normalized hash and the legacy byte-hash for backward compatibility with manifests written by older scaffolds. ## `validateIntegration` Whole-project correctness check — manifest exists + parses, generated bootstrap file is on disk with the right marker + matching hash, dependency declared, library version current, declared `permissions` cover the manifest's declared `capabilities`, bootstrap actually calls `ExtentosGlasses.create(...)` / `Extentos.create(...)`, AGP + Gradle versions meet the library's floor (Android), foreground-service hint is present when continuous-capture capabilities are declared. **When to use:** after structural changes — new capability declared, dependency bumped, manifest edited. Before testing. **When NOT to use:** for reading current state (use `inspectIntegration`). **Parameters:** `projectPath` (optional). **Response:** `{ valid: boolean, checks: Check[], summary }`. Each check carries `name`, `passed`, `severity` (`error`/`warn`/`info`), `details`, optional `fix` hint. The agent reads the `fix` field and acts on it directly. ### The vendor-transport checks Two checks fire only when the project declares `camera` or `display` — a voice-only app needs neither, and flagging one would push you onto the credentialed Meta path the vendorless baseline exists to spare you. | Check | What it verifies | |---|---| | `glasses_meta_declared` | `com.extentos:glasses-meta` is in the build files — a direct coordinate or a `gradle/libs.versions.toml` entry. `com.extentos:glasses` is vendorless, so camera and display have no transport without it. | | `meta_dat_repo_declared` | `settings.gradle.kts` declares the Meta DAT GitHub Packages repository the `mwdat-*` artifacts resolve from. | Both are **warnings, not errors**, and they are independent: a project upgraded from 1.x already declares the repository, so `meta_dat_repo_declared` passes while the dependency is still missing. A missing `glasses-meta` **does not fail in the browser simulator.** The simulator serves camera and display from the SDK itself and never touches the Meta SDK, so every capability works there. It fails on real glasses, where the session resolves to the vendorless system-audio baseline and captures return a typed refusal. This check, and the startup advisory the SDK logs in simulator mode, exist because the simulator alone cannot tell you. --- ## Retired validation tool Pre-pivot the MCP server also exposed `validateSpec` — a pure JSON check against the AppSpec JSON Schema (`extentos://schema/v1`). Retired with the pure-SDK pivot: there is no spec to validate. Structural correctness lives in `validateIntegration` (does the project compile and link against the SDK), and per-call shape lives in the SDK type system itself (Kotlin/Swift compiler will reject a wrong call at build time). ## Related - **[Setup and Generation tools](/docs/mcp-server/tools/generation)** — what the agent calls *before* validation - **[Simulation tools](/docs/mcp-server/tools/simulation)** — what the agent runs *after* validation passes - **[Tools overview](/docs/mcp-server/tools)** — back to the full tool catalog --- source: https://extentos.com/docs/reference --- --- title: Reference description: Generated API reference — the Android SDK (Dokka), iOS SDK (DocC), MCP server tools (JSON Schema), and a flat error-code index for troubleshooting. type: reference platform: all related: - /docs/reference/errors - /docs/reference/android-api - /docs/reference/ios-api - /docs/reference/mcp-tools --- Two of these pages are generated from source — don't hand-edit them. The SDK API references are still being set up. **Generated now (do not hand-edit):** - [MCP tools](/docs/reference/mcp-tools) — emitted from the tool registry (`definitions.ts`) - [Error codes](/docs/reference/errors) — flat index from the Rust error enums, cross-linked from troubleshooting **Pending generation:** - [Android API](/docs/reference/android-api) — Dokka reference (placeholder today) - [iOS API](/docs/reference/ios-api) — entry point, `ExtentosConfig`, and the `GlassesUI` surface (hand-maintained until the generated DocC reference) --- source: https://extentos.com/docs/reference/android-api --- --- title: Android API reference description: Symbol-level reference for the Extentos Android SDK — ExtentosGlasses, ExtentosConfig, every typed sub-client, the assistant builder, and the core data types, with exact Kotlin signatures. type: reference platform: android related: - /docs/reference/errors - /docs/reference/ios-api - /docs/sdk/android - /docs/sdk/android/initialization --- Exact public signatures for `com.extentos:glasses`. Defaults shown are the real ones — where a parameter has a default you can omit it. For *how* to use each area the SDK guides read better: [initialization](/docs/sdk/android/initialization), [lifecycle](/docs/sdk/android/lifecycle), [threading](/docs/sdk/android/threading). Error variants and payloads are in the generated [error reference](/docs/reference/errors). ## Entry point ```kotlin public class ExtentosGlasses { public companion object { public fun create(config: ExtentosConfig = ExtentosConfig()): ExtentosGlasses } // Transport outcome — fixed for the life of the handle public val transportChosen: TransportChosen public val selectionSource: TransportSelectionSource // Sub-clients public val connection: ConnectionClient public val camera: CameraClient public val audio: AudioClient public val display: DisplayClient public val runtime: RuntimeClient public val toggles: ToggleClient public val voice: VoiceClient public val assistant: AssistantClient public val telemetry: TelemetryClient public val observability: ObservabilityClient public val device: DeviceIdentityClient public val connectionConfig: ConnectionConfigSource public val capabilities: GlassesCapabilities // computed on every read public val usedCapabilities: Set public suspend fun shutdown() } ``` The constructor is `internal` — `create(...)` is the only way in. The property is `toggles`; its type is **`ToggleClient`** (singular). There is no root `connect()` on Android — go through `glasses.connection.connect()`. ## ExtentosConfig ```kotlin public data class ExtentosConfig( val appId: String? = null, val accountId: String? = null, val transport: TransportChoice = TransportChoice.Auto, val logLevel: LogLevel = LogLevel.WARN, val preferredStreamConfig: PreferredStreamConfig? = null, val debug: Boolean = false, val usedCapabilities: Set = emptySet(), val telemetryConsent: Boolean = true, val dataSharingConsent: Boolean = true, val environment: ExtentosEnvironment = ExtentosEnvironment.DEVELOPMENT, val telemetryEndpoint: String? = null, val applicationContext: Context? = null, val activityProvider: (() -> Activity?)? = null, val hasBondedMetaDevice: (() -> Boolean)? = null, ) public sealed interface TransportChoice { public data object Auto : TransportChoice public data object RealMeta : TransportChoice public data object SystemAudio : TransportChoice public sealed interface Simulated : TransportChoice { public data object Local : Simulated public data class Browser(val url: String? = null) : Simulated } } public enum class LogLevel { VERBOSE, DEBUG, INFO, WARN, ERROR } ``` `debug` is a config field defaulting to `false` — **not** your `BuildConfig.DEBUG` build type. See [runtime internals](/docs/sdk/android/runtime-internals) for what it changes. ## Sub-clients ### ConnectionClient ```kotlin public val state: StateFlow public val simulatorHint: StateFlow public suspend fun connect(deviceId: DeviceId? = null): ExtentosResult public suspend fun disconnect() public suspend fun reconnect(deviceId: DeviceId? = null): ExtentosResult ``` `DeviceId` is `typealias DeviceId = String`. ### CameraClient ```kotlin public suspend fun capturePhoto(config: PhotoConfig = PhotoConfig()): ExtentosResult public suspend fun captureVideo(config: VideoConfig = VideoConfig()): ExtentosResult public suspend fun stopVideo() public fun videoFrames(config: VideoFrameConfig = VideoFrameConfig()): Flow public fun activeStreamInfo(): ActiveStreamInfo? // a function; null when nothing is armed public fun streamState(): Flow // the activity axis public var preferredStreamConfig: PreferredStreamConfig? ``` `streamState()` reports what the camera stream is **doing** — `IDLE`, `ARMING`, `STREAMING`, `PAUSED`, `STOPPING`. It emits the current phase immediately on subscription, then only on real transitions, so every value is something that changed. Pair it with `activeStreamInfo()`, which reports the config a running stream locked at. This is a different axis from the connection state's `camera: CameraStatus`, which is **health** (will a capture work, and should the UI prompt the wearer). Both matter at once: a latched capture failure while a stream is running is a real state, which is why they are not one enum. Follow `streamState()` — not the connection state — for anything that forwards frames onward, such as a live relay, a recorder, or a frame pipeline. A stream can stop delivering while the connection stays perfectly healthy: the wearer folds the glasses, taps the temple, or another app takes the device session. `PAUSED` is the case with no other signal at any layer, because frames simply stop. ```kotlin glasses.camera.streamState().collect { phase -> when (phase) { CameraStreamState.STREAMING -> relay.resume() CameraStreamState.PAUSED -> relay.hold("folded, or another app took the camera") CameraStreamState.IDLE -> relay.stop() else -> Unit } } ``` `videoFrames` is the one primitive that reports failure by **throwing into the flow** rather than returning `ExtentosResult` — a `Flow` has no error channel, so `collect` is where the reason arrives. Two exceptions, both at collection start: | Exception | Meaning | |---|---| | `CameraStreamPausedException` | The wearer paused the camera with the temple button. Prompt a tap and retry. A pause *mid*-stream is not an error — frames halt and resume. | | `CameraUnavailableException` | This transport has no camera path, so no frame will ever arrive. Carries `.error`, the same typed `CaptureError` `capturePhoto` would return. The usual cause on Android is a build declaring camera without `com.extentos:glasses-meta` — `com.extentos:glasses` is vendorless, so camera needs a vendor transport. | ```kotlin try { glasses.camera.videoFrames().collect { render(it) } } catch (e: CameraUnavailableException) { Log.e(TAG, "no camera on this transport: ${e.error}", e) } catch (e: CameraStreamPausedException) { showHint("Tap the right temple of your glasses to resume the camera") } ``` To branch *before* starting a stream, read `glasses.capabilities.camera`. ### AudioClient ```kotlin public suspend fun recordDiscrete(config: AudioRecordConfig = AudioRecordConfig()): ExtentosResult public fun audioChunks(config: AudioChunkConfig = AudioChunkConfig()): Flow public fun transcriptions(config: TranscriptionConfig = TranscriptionConfig()): Flow public suspend fun speak(text: String, config: SpeakConfig = SpeakConfig()): ExtentosResult public suspend fun cancelSpeak() public suspend fun earcon(sound: EarconSound, volume: Double = 1.0) public suspend fun sendAudio(pcm16: ByteArray, sampleRate: Int): ExtentosResult public suspend fun stopAudio() public val outputFidelity: OutgoingAudioFidelity // Extension, not a member: public fun AudioClient.startPushToTalk( scope: CoroutineScope, config: TranscriptionConfig = TranscriptionConfig(language = "en-US", partial = false), ): PushToTalkSession ``` ### VoiceClient ```kotlin public val hints: StateFlow> public val stats: StateFlow> public fun onPhrase( phrase: String, label: String? = null, stops: List = emptyList(), firesWhen: VoiceScope = VoiceScope.WhenDormant, handler: suspend () -> Unit, ): VoiceRegistration public fun registerHint(phrase: String, label: String? = null, stops: List = emptyList()): VoiceRegistration public fun reportFired(id: String) public enum class VoiceScope { Always, WhenDormant, WhenActive } public interface VoiceRegistration { public fun cancel() } ``` `firesWhen` is Android-only — the iOS `onPhrase` has no scope parameter. ### AssistantClient ```kotlin public suspend fun start(provider: AssistantProvider, block: AssistantConfigBuilder.() -> Unit): AssistantSession public fun createSession(config: AssistantConfig): AssistantSession public suspend fun stop() public val activeSession: AssistantSession? ``` `start(...)` **throws** `AssistantException` on an open-time failure — it does not return an `ExtentosResult`. See [the voice-assistant guide](/docs/guides/voice-assistant#1-bootstrap). ### The rest ```kotlin // RuntimeClient public val events: Flow public suspend fun snapshotEvents(): List // ToggleClient public val state: StateFlow> public fun put(key: String, value: JSONValue, source: ToggleSource = ToggleSource.UI) public fun get(key: String): JSONValue? // TelemetryClient public fun setUserSegment(segment: String?) public fun trackEvent(name: String, properties: Map = emptyMap()) public val consent: Boolean // ObservabilityClient — wrap your own AI calls so they land in the event log public suspend fun aiCall(label: String, metadata: Map = emptyMap(), block: suspend () -> T): T // DeviceIdentityClient public val type: GlassesDeviceType public val vendor: GlassesVendor // ConnectionConfigSource public fun cached(): JsonObject? public suspend fun fetch(): JsonObject? // DisplayClient — guard every call on isAvailable public val isAvailable: Boolean public suspend fun show(onBack: (() -> Unit)? = null, content: DisplayRootScope.() -> Unit) public suspend fun clear() public suspend fun prepareVideo(clip: VideoClip): Boolean public suspend fun forgetHostedVideo(clip: VideoClip) public suspend fun forgetHostedImage(photo: Photo) ``` ## The assistant builder Everything settable inside `assistant.start(provider) { … }`. The receiver type is `AssistantConfigBuilder`. ```kotlin public var instructions: String = "" public var startActive: Boolean = false public var endOnIntent: Boolean = true public var greeting: Greeting = Greeting.Default public var historyCap: Int = 100 public var historyCompaction: HistoryCompaction = HistoryCompaction.Auto public var compactionModel: String? = null public var persistentMemory: Boolean = false public var memoryUserId: String? = null public var memoryStore: MemoryStore? = null public var localConductFloor: Boolean = true public var fallbackGreeting: String? = null public fun sleepAfterSilence(duration: Duration) // kotlin.time.Duration; must be positive public fun sleepOnPhrase(phrase: String) // additive public fun onWake(block: (suspend AssistantSession.() -> Unit)?) public fun onSleep(block: (suspend AssistantSession.() -> Unit)?) // tool() — three public overloads public fun AssistantConfigBuilder.tool( name: String, description: String, blocking: Boolean = false, body: suspend () -> ToolResult, ) public inline fun AssistantConfigBuilder.tool( name: String, description: String, blocking: Boolean = false, noinline body: suspend (Args) -> ToolResult, // schema inferred from @Serializable ) public fun AssistantConfigBuilder.tool( name: String, description: String, schema: JsonObject, blocking: Boolean = false, body: suspend (args: JsonObject) -> ToolResult, ) ``` **`includeDeviceInfoTool`, `deviceInfoNote` and `withinSessionMemory` are not on this builder.** They exist only on the `AssistantConfig` data class, so setting them inside `start(provider) { … }` does not compile. Use the raw form: `assistant.createSession(AssistantConfig(provider = …, includeDeviceInfoTool = false))` then `session.start()`. Same on both platforms. ## AssistantSession ```kotlin public val config: AssistantConfig public val state: StateFlow public suspend fun start() public suspend fun wake() public suspend fun sleep() public suspend fun stop() public suspend fun say(text: String) public suspend fun greet(prompt: String? = null) public suspend fun includeImage(uri: String, prompt: String? = null) public suspend fun cancelSpeak() public suspend fun setReasoningEffort(effort: ReasoningEffort) public suspend fun setVoice(voice: String) public suspend fun setModel(model: String) public suspend fun updateInstructions(instructions: String) public suspend fun sendVideoFrame(jpegBytes: ByteArray, mimeType: String = "image/jpeg") public val modelSupportsVideoInput: Boolean public fun conversationHistory(limit: Int = 100): List public fun clearHistory() public fun appendHistory(turn: Turn) public fun replaceHistory(turns: List) public enum class AssistantState { Idle, Dormant, Activating, Active, Reconnecting, Sleeping, Stopping, Stopped } ``` ## Providers and results ```kotlin public sealed class AssistantProvider { // One type for every model — the MODEL ID picks the vendor: // gpt-* OpenAI, grok-* xAI, gemini-* Google, local-* on-device. public data class Managed( val model: String? = null, // "local-auto" or a local-* id runs on-device val voice: String? = null, val turnDetection: TurnDetection = TurnDetection.ServerVad(), val reasoningEffort: ReasoningEffort = ReasoningEffort.Low, ) : AssistantProvider() public data class Mock( val behavior: MockBehavior = MockBehavior.MatchToolDescriptions, ) : AssistantProvider() // Deprecated since 2.0.0, still fully functional — same shape, same // behaviour. Existing code compiles unchanged; migrate when convenient. @Deprecated("Renamed to AssistantProvider.Managed") public data class OpenAi(/* identical to Managed */) : AssistantProvider() } public enum class ReasoningEffort { Minimal, Low, Medium, High, Xhigh } public sealed class TurnDetection { public data class ServerVad( val threshold: Double = 0.5, val prefixPaddingMs: Int = 300, val silenceDurationMs: Int = 500, ) : TurnDetection() public object SemanticVad : TurnDetection() } public sealed class ToolResult { public data class Ok(val output: String) : ToolResult() public data class Err(val message: String) : ToolResult() } public sealed class Greeting { public object Default : Greeting() public data class Custom(val directive: String) : Greeting() public object Off : Greeting() } // Unwrap a capture result inside a tool body, or bail with the SDK's own message. // Constrained to CaptureError — it does NOT apply to AudioError or ConnectError. public inline fun ExtentosResult.orToolError( onError: (ToolResult.Err) -> Nothing, ): T ``` ## Data types ```kotlin public data class GlassesCapabilities( val camera: Boolean, val microphone: Boolean, val speaker: Boolean, val display: Boolean, ) public sealed interface CapabilityKind { public data object Camera; public data object Microphone public data object Speaker; public data object Location public data object Notifications; public data object Display public data class Other(val name: String) : CapabilityKind } public enum class TransportChosen { REAL_META, BROWSER_SIM, LOCAL_SIM, PROJECTED_XR, SYSTEM_AUDIO } public enum class TransportSelectionSource { BUILD_CONFIG, ENV_VAR, BONDED_DEVICES, FALLBACK_DEFAULT, EXPLICIT_CONFIG, PAIRING, VENDOR_REGISTRY } public enum class EarconSound { CONFIRMATION, ERROR, NOTIFICATION, START, STOP, COMPLETE } public data class Photo( val uri: String?, val width: Int, val height: Int, val format: PhotoFormat, val exif: String?, ) public data class VideoClip( val uri: String?, val durationMs: Long, val format: VideoFormat, val width: Int, val height: Int, ) ``` `Photo.uri` and `VideoClip.uri` are **nullable** — elvis them. Use the `Photos.*` helpers rather than parsing the URI yourself; the scheme differs between simulator and hardware. ### Capture and audio config ```kotlin public data class PhotoConfig( val resolution: Resolution = Resolution.MEDIUM, val format: PhotoFormat = PhotoFormat.JPEG, val dedicatedCapture: Boolean = false, ) public data class VideoConfig( val maxDurationSeconds: Int? = null, val includeAudio: Boolean = true, val format: VideoFormat = VideoFormat.MP4_HEVC, val resolution: Resolution = Resolution.MEDIUM, val frameRate: Int = 24, ) public data class VideoFrameConfig( val frameRate: Int = 2, val resolution: Resolution = Resolution.LOW, val format: VideoFrameFormat = VideoFrameFormat.JPEG, ) public data class AudioRecordConfig( val maxDurationSeconds: Int? = null, val silenceTimeoutSeconds: Double? = null, val quality: AudioQuality = AudioQuality.STANDARD, ) public data class AudioChunkConfig(val chunkMillis: Int = 20, val sampleRate: Int = 16000) public data class TranscriptionConfig(val language: String = "en-US", val partial: Boolean = true) public data class SpeakConfig( val voice: String? = null, val rate: Double = 1.0, val pitch: Double = 1.0, val volume: Double = 1.0, val waitForCompletion: Boolean = true, ) ``` `TranscriptionConfig.language` is honored on the simulator transports and **ignored on real glasses and the audio baseline** — see [audio streaming](/docs/guides/audio-streaming#language-and-where-it-is-and-isnt-honored). `SpeakConfig` has no language field; platform TTS follows the device locale. ## Events ```kotlin public sealed interface RuntimeEvent { public data class ToggleChanged(val key: String, val oldValue: JSONValue, val newValue: JSONValue, val source: ToggleSource) : RuntimeEvent public data class CoexistenceWarning(val blocked: String, val reason: String) : RuntimeEvent public data class Log(val level: LogLevel, val message: String, val payload: JSONValue? = null) : RuntimeEvent public data class UnrecognizedUtterance(val rawTranscript: String) : RuntimeEvent public data class Assistant(val event: AssistantEvent) : RuntimeEvent } public sealed interface AssistantEvent { public data class SessionStarted(val provider: String, val model: String?, val voice: String?) public data class SessionEnded(val reason: EndReason, val message: String? = null) public data class UserSpoke(val transcript: String) public data class AssistantSpoke(val transcript: String) // transcript — generation done, NOT playback public object AssistantAudioStarted // started talking public object AssistantAudioFinished // stopped talking public data class ToolCalled(val name: String, val args: JsonObject, val callId: String) public data class ToolResultEvent(val callId: String, val name: String, val output: String, val isError: Boolean, val durationMs: Long) public data class Reconnected(val reason: ReconnectReason, val downtimeMs: Long) public data class Error(val kind: String, val message: String) public data class LocalModelRequired(val modelId: String, val reason: LocalModelRequirement.Reason, val spokenMessage: String) public data class AutoModelResolved(val servedModelId: String, val isLocal: Boolean, val cloudReason: CloudFallbackReason?, val downloadTarget: String?) public object WentDormant } ``` `AutoModelResolved` is how you verify an on-device session actually stayed on-device. `LocalModelRequired.Reason` is `NotDownloaded` or `DoesNotFit`. `AssistantAudioStarted` / `AssistantAudioFinished` strictly alternate and are what to watch for "is it talking right now" — `AssistantSpoke` is the transcript and fires seconds earlier, because the model streams audio faster than real time. The finished edge spans a whole turn including any tool round trip inside it, and fires immediately on a barge-in. See [knowing when the assistant has stopped talking](/docs/concepts/assistant#knowing-when-the-assistant-has-stopped-talking). ## Connection state ```kotlin public sealed class GlassesState { public object NotRegistered public object Registered public data class DeviceDiscovered(val deviceId: String) public data class Connecting(val deviceId: String) public data class Active(val state: ActiveState) public data class Disconnected(val cause: DisconnectCause) } ``` `Active` means the transport is up, not that glasses are present — see [sessions](/docs/concepts/sessions). `DisconnectCause` variants and payloads are in the [error reference](/docs/reference/errors). ## Foreground service ```kotlin public open class GlassesForegroundService : Service() { public companion object { public const val CHANNEL_ID: String = "extentos_capture" @JvmStatic @JvmOverloads public fun start(context: Context, title: String? = null, text: String? = null, smallIconResId: Int = 0) @JvmStatic @JvmOverloads public fun start(context: Context, serviceClass: Class, title: String? = null, text: String? = null, smallIconResId: Int = 0) @JvmStatic @JvmOverloads public fun stop(context: Context, serviceClass: Class = GlassesForegroundService::class.java) } } ``` Start it **after** `RECORD_AUDIO` is granted — Android 14+ rejects a microphone-type foreground service started without the grant. Subclass it and override `extentosForegroundServiceType` for a camera-type service. ## Local tier Separate artifacts — see [on-device models](/docs/concepts/local-models). ```kotlin // com.extentos.glasses.local (com.extentos:glasses-local) public object ExtentosLocalTier { public fun register(context: Context) // before ExtentosGlasses.create(...) public fun models(context: Context): List public suspend fun download( context: Context, modelId: String, onProgress: (bytesDownloaded: Long, totalBytes: Long) -> Unit = { _, _ -> }, ): Boolean public fun delete(context: Context, modelId: String): Boolean public fun autoChoice(context: Context): AutoChoice public fun deviceFit(context: Context, modelId: String): DeviceFit public data class LocalModelInfo(val id: String, val displayName: String, val totalBytes: Long, val isInstalled: Boolean) public data class AutoChoice(val modelId: String?, val isLocal: Boolean, val cloudReason: CloudFallbackReason?, val downloadTarget: String?) public data class DeviceFit(val requiredMb: Long, val availableMb: Long, val fits: Boolean) } // com.extentos.glasses.localvoice (com.extentos:glasses-local-voice) public object ExtentosLocalVoice { public fun register(context: Context) // registers BOTH the voice and the recogniser public fun modelDirectory(context: Context): File public fun sttModelDirectory(context: Context): File } // On-device voice (~330 MB) and on-device transcription (~73 MB). Identical // contract: size up front, byte progress, resumable, deletable, and never // downloaded unless the app asks. Until installed, the system voice / // recogniser serves. public object KokoroVoiceModel { public val totalBytes: Long public fun isInstalled(context: Context): Boolean public fun delete(context: Context): Boolean public suspend fun download( context: Context, onProgress: (bytesDownloaded: Long, totalBytes: Long) -> Unit = { _, _ -> }, ) } public object ExtentosSttModel { public val totalBytes: Long public fun isInstalled(context: Context): Boolean public fun delete(context: Context): Boolean public suspend fun download( context: Context, onProgress: (bytesDownloaded: Long, totalBytes: Long) -> Unit = { _, _ -> }, ) } ``` Every local-tier call takes a `Context` on Android; the Swift equivalents take none. ## Source of truth The library ships as a compiled artifact on [Maven Central](/docs/sdk/android/install); its API surface and KDoc are what you consume. Signatures here are transcribed from the shipped source — if one disagrees with your IDE's completion, trust the IDE. --- source: https://extentos.com/docs/reference/errors --- --- title: Error reference description: Every typed error the Extentos SDK can return — ConnectError, CaptureError, AudioError, TransportError, the ExtentosError umbrella, and the Meta-DAT DeviceSessionError — with their payload fields and meaning. Lifecycle operations return ExtentosResult with these concrete failure variants rather than throwing; pattern-match them. Generated from the Rust core. type: reference platform: all related: - /docs/reference - /docs/sdk/android - /docs/troubleshooting - /docs/concepts/permissions --- {/* EXTENTOS-GENERATED — do not hand-edit. Source: core/extentos-core/src/types/errors.rs + real_meta/mod.rs. Regenerate: cd mcp-server && npm run docs:errors. The AssistantError section is hand-authored (shell-level Phase-4 type). */} > **Generated from source.** The error enums below are emitted from the Rust core (`core/extentos-core/src/types/errors.rs`), which is codegen'd into the Kotlin and Swift SDKs. The same variants exist on both platforms; Kotlin uses the PascalCase names shown, Swift the lowerCamelCase equivalents (`TransportFailure` → `.transportFailure`). ## How errors surface Lifecycle operations don't throw — they return an **`ExtentosResult`** with a typed failure variant. This is DAT's own convention and avoids coroutine-cancellation footguns. Pattern-match the result; **don't** wrap it in `runCatching` (which only catches thrown exceptions and would mask the typed error): ```kotlin when (val r = glasses.camera.capturePhoto()) { is ExtentosResult.Ok -> render(r.value) // Photo is ExtentosResult.Err -> when (val e = r.error) { // CaptureError is CaptureError.PermissionDenied -> requestCameraPermission() is CaptureError.ThermalCritical -> toast("Glasses too hot — let them cool") is CaptureError.DisabledByUser -> explainToggle(e.toggle) else -> log(e) } } ``` Helpers on the Kotlin `ExtentosResult`: `.map { }`, `.flatMap { }`, `.valueOrNull()`, `.errorOrNull()`. The Swift shape differs in names, not model — `ExtentosResult` is `.success(Success)` / `.failure(Failure)`, with `map` / `flatMap` and optional `success` / `failure` properties instead of `valueOrNull()` / `errorOrNull()`: ```swift switch await glasses.camera.capturePhoto() { case .success(let photo): render(photo) case .failure(let error): // CaptureError switch error { case .permissionDenied: requestCameraPermission() case .thermalCritical: toast("Glasses too hot — let them cool") default: log(error) } } ``` The `code` + `message` carried by the `PlatformError` / `TransportFailure` variants is the shell's stable machine tag for a native-origin failure (the original native exception is logged to the event log at the catch site, not carried across the FFI). ## Connection errors ### `ConnectError` Returned by `glasses.connection.connect()`. | Variant | Payload | Meaning | |---|---|---| | `NoDeviceAvailable` | — | No paired or otherwise known device is available to connect to. | | `PermissionDenied` | — | A runtime permission required to connect was denied. | | `PairingFailed` | — | Pairing / handshake with the device failed. | | `RegistrationRequired` | — | The install must register / authenticate before it can connect. | | `TransportFailure` | `code: String, message: String` | A native transport-layer failure surfaced while connecting. | ## Capture errors ### `CaptureError` Returned by `glasses.camera.capturePhoto()` / `captureVideo()`. | Variant | Payload | Meaning | |---|---|---| | `NotConnected` | — | — | | `PermissionDenied` | — | — | | `ThermalCritical` | — | — | | `HingesClosed` | — | — | | `CoexistenceBlocked` | — | — | | `StreamPaused` | — | The user paused the camera with the glasses' capture button (temple tap), so the live stream this capture needs is held, not flowing. | | `DisabledByUser` | `toggle: String` | A user-facing toggle is gating this capture; `toggle` is the toggle key (e.g. `camera_streaming_enabled`) so a handler can explain it. | | `PlatformError` | `code: String, message: String` | A native-origin capture failure — structured `code` + `message`. | ## Audio errors ### `AudioError` Returned by `glasses.audio.recordDiscrete()` / `speak()`. | Variant | Payload | Meaning | |---|---|---| | `NotConnected` | — | — | | `PermissionDenied` | — | — | | `CoexistenceBlocked` | — | — | | `DisabledByUser` | `toggle: String` | As `CaptureError.DisabledByUser` — a toggle (e.g. `audio_capture_enabled`). | | `PlatformError` | `code: String, message: String` | — | ## Transport errors ### `TransportError` Surfaced by the underlying transport (simulator WebSocket or Meta DAT). `WebsocketClosed.code` is the WebSocket close code. | Variant | Payload | Meaning | |---|---|---| | `WebsocketClosed` | `code: i32, reason: Option<String>` | — | | `HandshakeFailed` | `reason: String` | — | | `SchemaMismatch` | `server_version: String, client_version: String` | — | | `Timeout` | — | — | | `HardwareUnavailable` | `reason: String` | — | | `PlatformError` | `code: String, message: String` | — | ## The umbrella type ### `ExtentosError` `ExtentosError` wraps the four per-area errors plus a catch-all, for APIs that surface any failure. | Variant | Payload | Meaning | |---|---|---| | `Connect` | `inner: ConnectError` | — | | `Capture` | `inner: CaptureError` | — | | `Audio` | `inner: AudioError` | — | | `Transport` | `inner: TransportError` | — | | `Unexpected` | `message: String` | — | ## Meta DAT device-session errors ### `DeviceSessionError` Surfaced by the Meta DAT bridge (`RealMetaTransport`) when starting a device session on real hardware. | Variant | Payload | Meaning | |---|---|---| | `NoEligibleDevice` | — | — | | `PlatformError` | `code: String, message: String` | — | ## Assistant errors (Phase 4) The Phase-4 assistant runtime (`glasses.assistant.*`) uses a shell-level `AssistantError` type (not a Rust core enum) with the same six variants on both platforms: a Kotlin sealed class thrown wrapped in `AssistantException` (`com.extentos:glasses`, since 1.4.0 — see [SDK install](/docs/sdk/android/install)) and a Swift `AssistantError` enum in `GlassesCore` (lowerCamelCase cases — `.noApiKey`, `.providerError(code:message:)`). | Variant | Payload | Meaning | |---|---|---| | `NoApiKey` | — | The Extentos managed gateway isn't reachable. Legacy name: there is no API key to supply — the assistant always runs on the managed gateway. Beta and production builds attest automatically (Play Integrity / App Attest) and need no setup. A sideloaded **development** build cannot attest, so it reaches the gateway one of two other ways — bound to a browser-simulator session, or through the dev-tier project-key lane. If you hit this on real hardware from a debug build, that binding is what's missing: see [how the gateway identifies a caller](/docs/concepts/ai-gateway#identity-how-the-gateway-knows-whos-calling). | | `AlreadyActive` | — | An assistant session is already active (the runtime is singleton-active). | | `SessionEnded` | — | `start()` was called on a session that has already stopped. | | `NotReady` | — | `wake` / `sleep` / `say` / etc. called on an Idle or non-Active session. | | `NetworkError` | `cause: Throwable` | A network failure reaching the gateway or provider. | | `ProviderError` | `code, message` | The upstream AI provider returned an error. | ## There is no `DisplayError` Display (`glasses.display.*`) has **no returned error type** — `show()` degrades gracefully and never throws (a transport with no display path simply no-ops). Display delivery problems surface as **runtime log events** on `glasses.runtime.events` (`display.video_delivery_failed`, `display.image_delivery_failed`, `display.video_error`) and on the `errors` chip in the simulator [event log](/docs/mcp-server/tools/simulation) (they are warn severity). See the [display capability](/docs/concepts/capabilities) docs. --- source: https://extentos.com/docs/reference/ios-api --- --- title: iOS API reference description: The Extentos iOS SDK API reference — entry point, ExtentosConfig fields, and the GlassesUI surface (ExtentosConnectionPage, theming, escape hatch). Hand-maintained until the generated DocC reference replaces it. type: reference platform: ios related: - /docs/sdk/ios - /docs/sdk/ios/install - /docs/sdk/ios/initialization - /docs/getting-started/with-agent - /docs/sdk/ios/threading - /docs/sdk/ios/info-plist - /docs/sdk/ios/lifecycle - /docs/reference/android-api --- The iOS SDK installs from [`github.com/extentos/swift-glasses`](https://github.com/extentos/swift-glasses) ([install](/docs/sdk/ios/install)). This page is a **hand-maintained** reference of the surface you touch first — the entry point, the configuration struct, and the `GlassesUI` types; a generated symbol-level DocC reference will replace it as the package line grows. For per-primitive call shapes (camera, audio, voice, …), your agent's `getCapabilityGuide(feature)` serves Swift snippets generated from source. ## Entry point ```swift import GlassesCore // usedCapabilities picks the transport on iOS — declare your real footprint. let glasses: any ExtentosGlasses = Extentos.create(config: ExtentosConfig( usedCapabilities: [.microphone, .speaker] // + .camera / .display if used )) // Camera/display apps forward the Meta auth callback from their App scene. // A voice app has no DAT pairing, so it needs neither of these lines: .onOpenURL { url in Task { _ = await glasses.handleUrl(url) } } ``` `ExtentosGlasses` exposes the typed sub-clients (`connection`, `camera`, `audio`, `runtime`, `toggles`, `voice`, `telemetry`, `observability`, `assistant`, `display`), plus `capabilities: DeviceCapabilitySet`, `usedCapabilities: [DeclaredCapability]`, and a root `connect()` convenience — what the generated bootstrap calls, equivalent to `connection.connect()`. See [initialization](/docs/sdk/ios/initialization). Return shapes vary by kind of call: lifecycle and capture ops return an `ExtentosResult` to pattern-match ([threading](/docs/sdk/ios/threading)); continuous primitives are `AsyncStream` / `AsyncThrowingStream`; display `show` / `clear` return `Void`; and assistant methods `throw`. `camera.videoFrames(config:)` is an `AsyncThrowingStream`, and the two errors it can throw both arrive at the start of iteration: | Error | Meaning | |---|---| | `CameraStreamPaused` | The wearer paused the camera with the temple button. Prompt a tap and retry; a pause *mid*-stream is not an error — frames halt and resume. | | `CameraUnavailable` | This transport cannot serve a live frame STREAM, so no frame will ever arrive. Carries `.error`, the same typed `CaptureError` the discrete paths return. A transport can throw this and still have a working `capturePhoto` — Brilliant is exactly that: stills work, while neither device has a video primitive or codec. | ```swift do { for try await frame in glasses.camera.videoFrames() { render(frame) } } catch let e as CameraUnavailable { log("no camera on this transport: \(e.message)") } catch is CameraStreamPaused { showHint("Tap the right temple of your glasses to resume the camera") } ``` To branch *before* starting a stream, read `glasses.capabilities.camera`. Note the Android counterpart adds one cause iOS cannot have — a build missing `com.extentos:glasses-meta` — because on iOS every vendor transport ships inside `GlassesCore`, so there is no vendor dependency to omit. `camera.streamState()` is an `AsyncStream` reporting what the camera stream is **doing** — `.idle`, `.arming`, `.streaming`, `.paused`, `.stopping`. It yields the current phase immediately on iteration, then only on real transitions. Pair it with `activeStreamInfo()`, which reports the config a running stream locked at. This is a different axis from the connection state's `camera: CameraStatus`, which is **health** (will a capture work, and should the UI prompt the wearer). Both matter at once: a latched capture failure while a stream is running is a real state, which is why they are not one enum. Follow `streamState()` — not the connection state — for anything that forwards frames onward, such as a live relay, a recorder, or a frame pipeline. A stream can stop delivering while the connection stays perfectly healthy: the wearer folds the glasses, taps the temple, or another app takes the device session. `.paused` is the case with no other signal at any layer, because frames simply stop. ```swift for await phase in glasses.camera.streamState() { switch phase { case .streaming: relay.resume() case .paused: relay.hold("folded, or another app took the camera") case .idle: relay.stop() default: break } } ``` ## ExtentosConfig All fields have defaults — `ExtentosConfig()` is a valid development configuration. | Field | Type (default) | What it does | |---|---|---| | `appId` | `String?` (`nil`) | App identity for telemetry + the dashboard. `nil` → falls back to the bundle identifier | | `accountId` | `String?` (`nil`) | Optional Extentos account identity | | `transport` | `TransportChoice` (`.auto`) | Transport selection; `.auto` resolves simulator vs real glasses vs the vendorless audio baseline ([how](/docs/concepts/transport-vs-app)) | | `progressiveResponse` | `ProgressiveResponseConfig` (`.default`) | Advanced: progressive-response delivery tuning | | `logLevel` | `LogLevel` (`.warn`) | SDK log verbosity | | `externalTaskGroup` | `(any TaskGroupHandle)?` (`nil`) | Advanced: run SDK tasks in a host-owned task group | | `interceptors` | `[any ExtentosInterceptor]` (`[]`) | Advanced: request/event interception hooks | | `debug` | `Bool` (`false`) | Development mode — feeds `.auto` transport resolution (production builds leave this `false` so bonded glasses resolve to the real transport) | | `telemetryConsent` | `Bool` (`true`) | Master telemetry switch — `false` sends no events at all | | `dataSharingConsent` | `Bool` (`true`) | Developer-level Do-Not-Sell: `false` keeps your dashboard analytics but excludes your events from cross-account vendor aggregates ([security](/docs/resources/security)) | | `environment` | `ExtentosEnvironment` (`.development`) | `.development` / `.beta` / `.production` — routes analytics to the right bucket. Production apps must set `.production` explicitly | | `telemetryEndpoint` | `URL?` (`nil`) | Override the telemetry ingest endpoint; `nil` → production endpoint | | `premiumVoice` | `PremiumVoiceConfig` (`.none`) | Premium voice configuration for the assistant | | `usedCapabilities` | `[DeclaredCapability]` (`[]`) | The app's declared capability footprint. **On iOS this decides the `.auto` transport**: non-empty and free of `.camera`/`.display` → the vendorless system-audio baseline; empty or vendor-declaring → real glasses. Also drives the connection page's capability tiles. Cases: `.camera`, `.microphone`, `.speaker`, `.display`, `.location`, `.notifications`, `.custom(String)`. A voice app should pass `[.microphone, .speaker]`. | | `hasBondedMetaDevice` | `(@Sendable () -> Bool)?` (`nil`) | Advanced: override bonded-device detection during `.auto` resolution. Note the semantics differ from Android: on iOS `nil` means "no bonded device", because there is no platform API to enumerate bonds. On Android `null` means "use the SDK's built-in bonded-device detector". | ## Display `glasses.display` renders declarative trees on display-capable glasses (Ray-Ban Display). `display.isAvailable` reports whether the connected device has a display — a display call on a no-display device is a silent no-op, never a crash. Display ships in both SDKs and is fully simulator-testable; on-glasses rendering via Meta DAT ships on both platforms, not yet confirmed on Display hardware. ```swift if glasses.display.isAvailable { await glasses.display.show(onBack: { /* back gesture for this show */ }) { scope in scope.flexBox(direction: .column, gap: 8) { col in col.text("Hello from the app") col.button("Done") { /* fires on a real tap or a sim-injected select */ } } } } await glasses.display.clear() ``` Each `show` replaces the whole display (latest wins); tappable nodes get auto-assigned ids, stable within a show. The node vocabulary is `text`, `image(url:)` (the glasses fetch http(s) themselves), `button`, `icon`, and `flexBox(direction:)` containers; the root scope adds `video(url:)` — a hosted video as the sole, full-surface node. The local-media roots (`image(photo:)`, `video(clip:)`) and the hosted-media helpers (`prepareVideo`, `forgetHostedVideo`, `forgetHostedImage`) behave exactly as on Android — the SDK hosts the local capture at `show()` time and renders the hosted URL, because the glasses fetch `http(s)` only. ## Named sounds `glasses.audio` also pushes raw PCM to the glasses speaker alongside `speak` and `earcon`: | Call | Shape | |---|---| | `sendAudio(_:sampleRate:)` | `async` → `ExtentosResult`; non-positive rate → `.platformError(code: "invalid_sample_rate")` | | `stopAudio()` | `async` — drops everything still queued (barge-in) | | `outputFidelity` | Synchronous — what the transport's speaker path can carry | See [bring your own realtime model](/docs/guides/audio-streaming#bring-your-own-realtime-model). ## VideoConfig and Photo.loadImage `captureVideo(config:)` takes a `VideoConfig` with five fields: `resolution`, `maxDurationSeconds` (`nil` = no time cap — the capture ends on cancellation), `format`, `includeAudio` (default `true`), and `frameRate` (default 24 — video and live view share one warm camera stream, so it takes effect only when the recording is the session's first camera use). `Photo.loadImage()` is the one iOS photo helper: `async`, decodes `file://` and `data:` URIs uniformly across transports, and returns a `PlatformImage?` (`UIImage` on iOS) — `nil` on unrecognized scheme, missing file, or decode failure. ## Assistant `glasses.assistant` is the voice-AI runtime, routed through the Extentos managed gateway (no API key in code). Two forms: the sugar `start(provider:_:)` (builds the config inline, returns a started session) and the raw `createSession(config:)` + `session.start()`. Sessions are singleton-active per instance; assistant methods `throw` (`AssistantError`). ```swift let session = try await glasses.assistant.start(provider: .managed()) { $0.instructions = "You are a helpful assistant on smart glasses." $0.tool("take_picture", description: "Take a photo with the glasses camera.") { if case .success = await glasses.camera.capturePhoto() { return .ok("photo saved") } return .err("camera failed") } } ``` An `AssistantSession` has a wake/sleep cycle inside it: `start()` lands it Dormant, `wake()` opens the realtime connection, `sleep()` closes it but keeps history/tools ready, `stop()` tears it down for good. In-session ops: `say(_:)`, `greet(_:)`, `includeImage(uri:prompt:)`, `setVoice(_:)`, `setModel(_:)`, `updateInstructions(_:)`, `cancelSpeak()` (barge-in), and the history ops (`conversationHistory(limit:)`, `clearHistory()`, `appendHistory(_:)`, `replaceHistory(_:)`). `session.state` observes the 8-state lifecycle: `idle`, `dormant`, `activating`, `active`, `reconnecting`, `sleeping`, `stopping`, `stopped`. ## Voice and audio helpers - `glasses.voice.onPhrase(phrase:label:stops:handler:)` — the library matches the phrase (case-insensitive substring on final transcripts) and runs your handler. While it runs, hearing any `stops` phrase cancels the handler's `Task` — plain structured concurrency, so your `defer` cleanup runs. Returns a `VoiceRegistration` whose `cancel()` removes the hint. - `glasses.voice.registerHint(phrase:label:stops:)` — announce the phrase to the simulator + connection-page UI while doing your own matching. - `glasses.audio.cancelSpeak()` — barge-in: stops in-flight `speak` TTS immediately; idempotent. - `glasses.audio.startPushToTalk()` — returns a `PushToTalkSession`; `await session.stopAndFlush()` ends the capture and returns the concatenated final transcripts. ## GlassesUI | Type | Shape | What it is | |---|---|---| | `ExtentosConnectionPage` | `init(glasses:config:)` — `config` defaults to `ConnectionPageConfig()` | The drop-in SwiftUI connection page: status, capability tiles, toggles | | `ConnectionPageConfig` | `init(sections: SectionVisibility = .init())` | Structural visibility control for the page | | `SectionVisibility` | `init(capabilities: Bool = true, voiceCommands: Bool = true, toggles: Bool = true)` | Which page sections render | | `ExtentosTheme` | `init(appearance: Appearance = .default) { content }` | Wrap Extentos views to restyle them with brand tokens | | `Appearance` | `init(colors:typography:shapes:)` | The token set `ExtentosTheme` applies | | `ExtentosPairingScreen` | `init(code:expiresAtMs:appearance:)` | The simulator pairing-code screen — shown automatically by the connection page when the localhost auto-bind probe doesn't resolve; the developer enters the code in the browser simulator to claim the socket | | `rememberExtentosState(_:)` | `(any ExtentosGlasses) -> ExtentosUiState` | Escape hatch: snapshot the UI state and build fully custom UI — you forfeit library-driven UI updates | The escape hatch — `rememberExtentosState(_:)` and its `ObservableObject` form `ExtentosStateModel` — is gated behind an SPI import: `@_spi(ExtentosEscapeHatch) import GlassesUI`. The opt-in is deliberate: once you bypass `ExtentosConnectionPage`, you're responsible for rendering any surfaces the library adds later. ```swift import GlassesUI ExtentosConnectionPage(glasses: glasses) // Restyled + trimmed: ExtentosTheme(appearance: myAppearance) { ExtentosConnectionPage( glasses: glasses, config: ConnectionPageConfig(sections: SectionVisibility(voiceCommands: false)) ) } ``` ## Local tier Separate SPM products — see [on-device models](/docs/concepts/local-models). Linking `GlassesLocal` raises the package's deployment floor to **iOS 17**. ```swift // GlassesLocal public enum ExtentosLocalTier { public static func register() // BEFORE Extentos.create(...) public static func models() -> [LocalModelInfo] public static func autoChoice() -> AutoChoice public static func deviceFit(for dashboardId: String) -> DeviceFit public static func download( modelId: String, onProgress: @escaping @Sendable (Double) -> Void = { _ in } ) async throws @discardableResult public static func delete(modelId: String) throws -> Bool public struct LocalModelInfo { public let id: String public let displayName: String public let requiredMb: Int public let isInstalled: Bool public let autoEligible: Bool } public struct AutoChoice { public let modelId: String? public let isLocal: Bool public let cloudReason: CloudFallbackReason? public let downloadTarget: String? } public struct DeviceFit { public let modelId: String public let requiredMb: Int public let availableMb: Int public var fits: Bool } public enum LocalTierError: Error { case unknownModel(String) } } // GlassesLocalVoice public enum ExtentosLocalVoice { public static func register() public static var modelDirectory: URL? { get } } public enum KokoroVoiceModel { public static var totalBytes: Int { get } public static func isInstalled() -> Bool public static func delete() throws public static func download( progress: @escaping @Sendable (Double) -> Void ) async throws } ``` Two shape differences from Android: **no `Context` parameter** on any call, and `download`'s progress is a single `Double` fraction rather than Android's `(bytesDownloaded, totalBytes)` pair. `register()` must run **before** `Extentos.create(...)`. Because Swift runs stored-property initializers ahead of `init()`, declare the handle without an initializer and assign it inside `init()` after the `register()` calls — see [on-device models](/docs/concepts/local-models#setup). ## Differences from Android Shipping both platforms? These are the ones that bite. Names and shapes first: | | Swift | Kotlin | |---|---|---| | OpenAI provider | `.managed(model:voice:turnDetection:reasoningEffort:)` | `AssistantProvider.Managed(...)` | | Assistant config block | `(AssistantConfigBuilder) -> Void` — `$0.instructions = …` | receiver lambda — bare `instructions = …` | | Typed tool | `schema:` is **required** | schema inferred from `@Serializable` | | Tool result | `.ok` / `.err` | `ToolResult.Ok` / `.Err` | | Capability dial | `DeviceCapabilitySet` | `GlassesCapabilities` | | `usedCapabilities` | `[DeclaredCapability]` (array) | `Set` | | `sleepAfterSilence` | `TimeInterval` seconds | `kotlin.time.Duration` | | Toggles | `state` + `update(_:)` | `state` + `put(...)` / `get(...)` | | Layout builder | `flexBox(direction:…)` | `column(…)` / `row(…)` | | Local tier | no `Context` parameter | every call takes `context: Context` | | Streams | `AsyncStream` / `AsyncThrowingStream` | `Flow` | | `includeImage` | `prompt` has no default | `prompt: String? = null` | **Two defaults differ silently — same call, different behaviour.** - `earcon(_:volume:)` defaults to **`0.8`** on Swift and **`1.0`** on Kotlin. - `SpeakConfig.pitch` defaults to **`0.0`** on Swift and **`1.0`** on Kotlin (and Swift's `SpeakConfig` fields are `Float`, Kotlin's are `Double`). Neither produces an error — your iOS build just sounds different. Pass the value explicitly on both platforms if it matters to you. **Swift has, Android doesn't:** `progressiveResponse`, `premiumVoice`, `interceptors`, `externalTaskGroup` on `ExtentosConfig`; a root `glasses.connect()`; `Extentos.default()`; `Extentos.requestSpeechRecognitionAuthorization()`. (`KokoroVoiceModel` — `isInstalled` / `download` / `delete` for the on-device voice — exists on **both** platforms; Swift takes no `Context`.) **iOS has no `transportChosen`, so the one-line startup assertion the Android guides recommend has no direct equivalent.** Use what iOS does expose: `glasses.capabilities` (a voice app on the audio baseline reads `camera: false`) together with `glasses.connection.state`. If you need certainty about which transport resolved, force it — set `transport: .systemAudio` or `.realMeta` explicitly rather than `.auto` — and let `connect()`'s typed error tell you when it can't be served. **Android has, Swift doesn't:** `VoiceScope` / `onPhrase(firesWhen:)` — so an iOS wake phrase can re-fire mid-conversation and you gate it in your handler; `camera.stopVideo()`; `camera.preferredStreamConfig`; `display.prepareVideo` / `forgetHostedVideo` / `forgetHostedImage`; the `video(clip:)` and `image(photo:)` display overloads; `applicationContext` / `activityProvider`; and `transportChosen` / `selectionSource` / `device` / `connectionConfig` on the root handle. Neither platform has an assistant-runtime language setting — see [what language does the assistant speak](/docs/concepts/assistant#what-language-does-the-assistant-speak-and-hear). ## Related - [Initialization](/docs/sdk/ios/initialization) — entry points, sub-clients, auth handoff, speech authorization - [Threading](/docs/sdk/ios/threading) — the concurrency contract - [Info.plist setup](/docs/sdk/ios/info-plist) and [lifecycle](/docs/sdk/ios/lifecycle) - [Android API](/docs/reference/android-api) — the same surface in Kotlin --- source: https://extentos.com/docs/reference/mcp-tools --- --- title: MCP tools reference description: The complete, always-current catalog of @extentos/mcp-server tools — 36 deterministic tools across 10 categories for discovering smart-glasses capabilities, scaffolding native iOS/Android integrations, driving the browser simulator, and gating production readiness. Generated from the live tool registry. type: reference platform: mcp related: - /docs/mcp-server - /docs/mcp-server/tools - /docs/getting-started/with-agent - /docs/reference --- {/* EXTENTOS-GENERATED — do not hand-edit. Source: mcp-server/src/tools/definitions.ts. Regenerate: cd mcp-server && npm run docs:reference. The category map + coverage assertions live in scripts/generate-docs-reference.mjs. */} > **Generated from source.** This page is emitted from `mcp-server/src/tools/definitions.ts` — the exact tool registry your agent loads. It lists every tool currently shipped. The **authoritative live surface** is what your installed agent sees via the MCP `tools/list` call; this page is the static mirror for pre-install evaluation and search. The Extentos MCP server exposes **36 deterministic tools** (zero LLM inference on the Extentos side) across 10 categories. Install it with: ```bash claude mcp add extentos -- npx -y @extentos/mcp-server@latest ``` Current published version and changelog: [npmjs.com/package/@extentos/mcp-server](https://www.npmjs.com/package/@extentos/mcp-server). Each tool's parameters below are generated from its input schema; for the full narrative on when to reach for each, follow the per-category guide links. ## Discovery & SDK reference Learn what the glasses can do and the canonical call shape for each primitive before writing handler code — plus the migration path for apps already built on raw Meta DAT. Narrative guide: [/docs/mcp-server/tools/discovery](/docs/mcp-server/tools/discovery). ### getPlatformInfo Return static platform metadata: library version + the list of SDK capabilities the glasses expose. | Parameter | Type | Required | Description | |---|---|---|---| | `sections` | string[] (enum) | no | One of: `version`, `capabilities`. | | `glasses` | string (enum) | no | Target glasses vendor. One of: `meta`, `android_xr`, `brilliant`, `htc`, `meta_rayban`. | | `expand` | string[] (enum) | no | Optional opt-in expansions on top of the compact default. One of: `schema`, `capabilities.full`, `capabilities.advanced`. | ### getCodeExample Reference library — retrieve a complete SDK code example (Kotlin + Swift) for a use case. | Parameter | Type | Required | Description | |---|---|---|---| | `pattern` | string (enum) | yes | Which compositional pattern to fetch. One of: `agent_driven_e2e_full_loop`, `agent_test_loop`, `assistant_agent_loop`, `barge_in_speak`, `byok_anthropic`, `connection_page_setup`, `display_browse_detail`, `display_media_gallery`, `live_transcription_ui`, `photo_describe_voice`, `video_frames_ml`, `voice_notes`, `voice_qa_assistant`. | ### getCapabilityGuide Per-feature SDK usage guide — minimal Kotlin + Swift snippet + gotchas + which getCodeExample patterns exercise the feature. | Parameter | Type | Required | Description | |---|---|---|---| | `feature` | string (enum) | yes | Feature name from getPlatformInfo.features[].name. One of: `assistant_provider_openai`, `assistant_runtime`, `assistant_session_runtime`, `assistant_start`, `assistant_tool`, `assistant_vision`, `audio_chunks`, `camera_stream_state`, `cancel_speak`, `capture_photo`, `capture_video`, `connection_state`, `display`, `earcon`, `outgoing_audio_stream`, `record_audio`, …(+5). | ### getMigrationGuide The entry point for a developer who ALREADY built their app against raw Meta DAT (the Device Access Toolkit / Wearables SDK) and wants to move onto the Extentos SDK. | Parameter | Type | Required | Description | |---|---|---|---| | `from` | string (enum) | no | Which SDK the app is migrating FROM. One of: `meta_dat`. | ## Setup & generation Scaffold the connection module into a host app and manage the per-project connection-page config. Narrative guide: [/docs/mcp-server/tools/generation](/docs/mcp-server/tools/generation). ### generateConnectionModule One-shot project scaffold — emits the bootstrap module that wires `Extentos.create(...)` into the host app, build-script changes, dependencies, permissions, and the integration manifest. | Parameter | Type | Required | Description | |---|---|---|---| | `platform` | string (enum) | yes | One of: `android`, `ios`. | | `glasses` | string (enum) | yes | Target glasses vendor. One of: `meta`, `android_xr`, `brilliant`, `htc`, `meta_rayban`. | | `appPackage` | string | yes | The Extentos project identity (reverse-DNS). | | `libraryVersion` | string | no | — | | `responseFormat` | string (enum) | no | One of: `concise`, `detailed`. | | `placement` | string (enum) | no | Where ExtentosConnectionPage should live in the app. One of: `dedicated_route`, `settings_subscreen`, `bottom_tab`, `modal_sheet`, `headless`. | | `projectPath` | string | no | Absolute path to the project root. | | `capabilities` | string[] | no | The SDK capabilities your app uses (feature names from getPlatformInfo — e.g. | ### getConnectionPageConfig Read the per-project ExtentosConnectionPage config (theming tokens + section visibility) the dashboard/server holds for an app. | Parameter | Type | Required | Description | |---|---|---|---| | `appPackage` | string | yes | The app's package / bundle id (project_install_id), e.g. | ### setConnectionPageConfig Persist the per-project ExtentosConnectionPage config (theming tokens + section visibility) to the dashboard/server, making the app MANAGED — the dashboard/server becomes the single source of truth and the SDK fetches it at render time (se… | Parameter | Type | Required | Description | |---|---|---|---| | `appPackage` | string | yes | The app's package / bundle id (project_install_id). | | `config` | object | yes | The config to persist: { schemaVersion?, sections?, appearance? | ### regenerateConnectionPageFile Regenerate the committed extentos.connection-page.json FROM the dashboard/server config (file ← server) — the generated mirror that makes the dashboard's tokens real in the repo and lets the app theme correctly offline. | Parameter | Type | Required | Description | |---|---|---|---| | `appPackage` | string | yes | The app's package / bundle id (project_install_id). | | `platform` | string (enum) | no | Defaults to android. One of: `android`, `ios`. | | `projectPath` | string | no | Absolute path to the project root. | ### adoptConnectionPageFile Adopt the committed extentos.connection-page.json UP to the dashboard/server (file → server) — the deliberate one-time seed of an existing code/file theme into the managed config. | Parameter | Type | Required | Description | |---|---|---|---| | `appPackage` | string | yes | The app's package / bundle id (project_install_id). | | `platform` | string (enum) | no | Defaults to android. One of: `android`, `ios`. | | `projectPath` | string | yes | Absolute path to the project root (required — reads app/src/main/assets/extentos.connection-page.json). | | `confirm` | boolean | no | Set true to overwrite an existing, DIFFERING server config with the committed file. | ## Agent configuration & usage Read or change a project's dashboard-managed assistant settings (Realtime model, voice, memory model + mode), and read its managed-gateway usage + exact cost. Account-scoped; writes are gated per-project by the MCP access grant (default Read+Write) and echo the cost impact. Narrative guide: [/docs/mcp-server/tools](/docs/mcp-server/tools). ### getAssistantConfig Read the per-project "Agent" (assistant) settings the dashboard holds for an app — the OpenAI Realtime model, the voice, the memory (compaction) model, and the within-session memory mode — plus the catalog of valid options for each, the da… | Parameter | Type | Required | Description | |---|---|---|---| | `appPackage` | string | yes | The app's package / bundle id (project_install_id), e.g. | ### setAssistantConfig Change a project's dashboard-managed "Agent" (assistant) settings via MCP — any of the OpenAI Realtime model, voice, memory (compaction) model, or within-session memory mode (PARTIAL update; pass only what you're changing). | Parameter | Type | Required | Description | |---|---|---|---| | `appPackage` | string | yes | The app's package / bundle id (project_install_id). | | `realtimeModel` | string (enum) | no | OpenAI Realtime model id. One of: `gpt-realtime-2.1`, `gpt-realtime-2.1-mini`, `gpt-realtime-2`, `gpt-realtime-1.5`, `gpt-realtime-mini`, `grok-voice-think-fast-2.0`, `grok-voice-think-fast-1.0`, `gemini-3.1-flash-live-preview`, `gemini-2.5-flash-native-audio-preview-12-2025`, `local-auto`, `local-qwen3-8b`, `local-qwen3-4b`, `local-qwen3-1.7b`, `local-qwen25-1.5b`, `local-qwen3-0.6b`. | | `voice` | string (enum) | no | Voice id. One of: `marin`, `cedar`, `ash`, `ballad`, `coral`, `sage`, `verse`, `alloy`, `echo`, `shimmer`, `eve`, `ara`, `rex`, `sal`, `leo`, `Puck`, …(+19). | | `compactionModel` | string (enum) | no | Memory (compaction) model id. One of: `gpt-4o-mini`, `gpt-5.4-nano`, `gpt-5.4-mini`, `gpt-5.4`, `gpt-5.5`, `gemini-3.1-pro-preview`, `gemini-3.5-flash`, `gemini-3.1-flash-lite`, `gemini-2.5-flash-lite`, `claude-opus-4-8`, `claude-sonnet-4-6`, `claude-haiku-4-5`. | | `withinSessionMemory` | string (enum) | no | Within-session memory mode. One of: `smart`, `basic`. | ### getGatewayUsage Read account-scoped managed-AI-gateway usage + exact cost for a project over a recent window. | Parameter | Type | Required | Description | |---|---|---|---| | `appPackage` | string | yes | The app's package / bundle id (project_install_id). | | `days` | number | no | Look-back window in days. | ## Credentials Read whether a project's API keys are set (masked hint only, never the value) and start write-without-knowing entry — the agent hands off a dashboard link where the owner pastes the secret, which never passes through the agent. Account-scoped; gated per-project by the MCP access grant. Narrative guide: [/docs/mcp-server/tools](/docs/mcp-server/tools). ### getCredentialStatus Read whether a project's Meta DAT build identity is set — plus a MASKED hint and when it was updated. | Parameter | Type | Required | Description | |---|---|---|---| | `appPackage` | string | yes | The app's package / bundle id (project_install_id). | ### setCredential Start WRITE-WITHOUT-KNOWING entry of the project's Meta DAT build identity. | Parameter | Type | Required | Description | |---|---|---|---| | `appPackage` | string | yes | The app's package / bundle id (project_install_id). | | `credentialType` | string (enum) | yes | Which credential to set. One of: `meta_dat`. | ## Analytics Read a project's production analytics — aggregate telemetry from shipped App Store / Play Store apps (events, active installs, by event/day/vendor/platform) over a window. Metadata only, never content. Account-scoped + ownership-checked; gated per-project by the MCP access grant. Narrative guide: [/docs/mcp-server/tools](/docs/mcp-server/tools). ### getProjectAnalytics Read a project's PRODUCTION analytics — aggregate telemetry from the app's shipped (App Store / Play Store) installs over a recent window. | Parameter | Type | Required | Description | |---|---|---|---| | `appPackage` | string | yes | The app's package / bundle id (project_install_id). | | `days` | number | no | Look-back window in days. | ## Implementation guidance Derive permissions and pressure-test voice phrases before wiring them in. Narrative guide: [/docs/mcp-server/tools/guidance](/docs/mcp-server/tools/guidance). ### getVoiceCommandGuidance Analyze proposed voice phrases for UX issues (length, homophones, digit-usage, Meta wake-word collision, ambiguity with existing phrases) before wiring them into a wake trigger. | Parameter | Type | Required | Description | |---|---|---|---| | `phrases` | string[] | yes | The proposed wake / trigger phrases to analyze (at least one). | | `existingPhrases` | string[] | no | Phrases already wired into the app. | ### getPermissions Derive the Android permissions, iOS Info.plist keys, Meta DAT scopes, AND the runtime grants a list of SDK capabilities requires. | Parameter | Type | Required | Description | |---|---|---|---| | `capabilities` | string[] | yes | Capability names from getPlatformInfo.features[].name. | | `platform` | string (enum) | yes | Selects how the one-line `summary` is phrased (Android vs iOS framing). One of: `android`, `ios`. | ## Validation Read the current integration state and gate correctness before you test. Narrative guide: [/docs/mcp-server/tools/validation](/docs/mcp-server/tools/validation). ### inspectIntegration Read-only snapshot of the current Extentos integration at a project path. | Parameter | Type | Required | Description | |---|---|---|---| | `projectPath` | string | no | Absolute (or cwd-relative) path to the project root holding extentos.manifest.json. | ### validateIntegration Pre-test gate that runs ~16 severity-tiered checks over the whole project — manifest present + parses, generated-file markers + hashes, dependency declared, library-version freshness, Android Meta-DAT repo declaration, bootstrap wiring (Ex… | Parameter | Type | Required | Description | |---|---|---|---| | `projectPath` | string | no | Absolute (or cwd-relative) path to the project root. | ## Simulation & agent-driven testing Drive the browser simulator and close the agent-driven end-to-end loop — inject input, read the event log and display tree, assert tool calls — without a human or physical hardware. Narrative guide: [/docs/mcp-server/tools/simulation](/docs/mcp-server/tools/simulation). ### createSimulatorSession Get-or-create: provision a browser-based simulator session for this project, OR return the existing saved one. | Parameter | Type | Required | Description | |---|---|---|---| | `glasses` | string (enum) | yes | Target glasses vendor. One of: `meta`, `android_xr`, `brilliant`, `htc`, `meta_rayban`. | | `platform` | string (enum) | no | One of: `android`, `ios`. | | `projectPath` | string | no | Absolute path to the app project — the SAME value you pass to generateConnectionModule and validateIntegration. | | `recordBinary` | boolean | no | — | | `autoOpenBrowser` | boolean | no | — | | `autoLink` | boolean | no | When true (default), if the backend returns auth_required, this handler polls completeAuthLink internally for `autoLinkSeconds` and re-mints on success. | | `autoLinkSeconds` | number | no | Max seconds to wait for the user to approve the verification URL when autoLink is true. | ### ensureSimulatorBrowser Ensure a **connected** simulator browser tab for the session — opens one if needed and confirms the browser's WebSocket actually attached before returning. | Parameter | Type | Required | Description | |---|---|---|---| | `sessionId` | string | yes | — | | `timeoutMs` | number | no | Max ms to wait for the browser tab's WebSocket to attach after auto-opening. | | `autoOpen` | boolean | no | When true (default): if no browser is connected the MCP server opens the session URL in the developer's default browser and polls until it attaches. | ### completeAuthLink Poll the backend until the user finishes signup at the verificationUrl, then persist the bearer token to ~/.extentos/auth.json. | Parameter | Type | Required | Description | |---|---|---|---| | `deviceCode` | string | yes | The `deviceCode` from createSimulatorSession's auth_required (HTTP 402) response — the opaque handle the backend correlates the developer's signup with. | | `maxWaitSeconds` | integer | no | Max seconds to keep polling for the developer to finish signup. | | `pollIntervalSeconds` | integer | no | Seconds between backend polls. | ### getEventLog Fetch structured event trace inside a simulator session. | Parameter | Type | Required | Description | |---|---|---|---| | `sessionId` | string | yes | — | | `filter` | string (enum) | no | Chip to filter the event stream by. One of: `all`, `errors`, `voice`, `camera`, `display`, `lifecycle`, `custom`. | | `limit` | integer | no | — | | `cursor` | string | no | Opaque keyset cursor from a prior getEventLog response. | | `follow` | boolean | no | When true, turns this call into a live watch: if no events are newer than `cursor`, the call BLOCKS until something happens (or `timeoutMs` elapses) instead of… | | `timeoutMs` | integer | no | Max time (ms) a `follow` call blocks waiting for new events. | | `responseFormat` | string (enum) | no | One of: `concise`, `detailed`. | | `redactBinary` | boolean | no | — | | `collapseRepeats` | boolean | no | When true (default), consecutive payload-identical events (same layer/severity/type/message/details — timestamps excluded) collapse into ONE entry carrying `re… | ### injectTranscript Inject a synthetic STT transcript into a live simulator session, the same way the simulator browser tab's click-to-fire chips do. | Parameter | Type | Required | Description | |---|---|---|---| | `sessionId` | string | yes | — | | `text` | string | yes | The transcript text to inject. | | `isFinal` | boolean | no | When true (default for type=stt_transcript), the frame is a final transcript that fires onPhrase matchers. | | `partial` | boolean | no | Convenience: when true, send as stt_partial (a not-yet-final transcript). | | `confidence` | number | no | STT confidence value (0-1). | ### injectAssistantUtterance Phase 4: drive an assistant turn from outside the live session for agent-driven E2E tests of `glasses.assistant.start { tool(...) { ... | Parameter | Type | Required | Description | |---|---|---|---| | `sessionId` | string | yes | — | | `text` | string | no | Utterance text the agent wants the assistant to react to. | | `audioWavBase64` | string | no | DEFERRED to v1.1 — passing this field returns `not_implemented`. | ### assertToolCalled Phase 4: wait for an `assistant.tool_called` event matching `name` (and optionally `argsMatch` partial-match). | Parameter | Type | Required | Description | |---|---|---|---| | `sessionId` | string | yes | — | | `name` | string | yes | Exact tool name to wait for. | | `sinceCursor` | string | no | Strongly recommended.: The `watchCursor` value returned by the `injectAssistantUtterance` that should trigger this tool. | | `argsMatch` | object | no | Optional top-level partial match against the tool's parsed args. | | `timeoutMs` | number | no | Max time to wait, in ms. | ### getSimulatorStatus Read a live simulator session's current state — phase (active/paused/closed), hardware-ready, attached roles (app + browser), and the **testVideos** list available to drive `setSimVideo` (defaults bundled with the platform plus any MP4s up… | Parameter | Type | Required | Description | |---|---|---|---| | `sessionId` | string | yes | — | ### deleteSimulatorSession Retire a simulator session on purpose. | Parameter | Type | Required | Description | |---|---|---|---| | `sessionId` | string | yes | — | ### setSimVideo Pipe a Test Video into the simulator's camera input so capture_photo / capture_video / videoFrames run against a known scene instead of a blank viewport — closes the agent-driven test loop for camera-driven flows the same way injectTranscr… | Parameter | Type | Required | Description | |---|---|---|---| | `sessionId` | string | yes | — | | `videoId` | string | yes | ID of the test video to load. | ### setSimDevice Switch the simulated glasses DEVICE MODEL. | Parameter | Type | Required | Description | |---|---|---|---| | `sessionId` | string | yes | — | | `device` | string (enum) | yes | The device model to simulate. One of: `rayban_meta`, `oakley_meta_hstn`, `oakley_meta_vanguard`, `rayban_meta_optics`, `meta_glasses`, `rayban_display`, `android_xr_audio_glasses`, `android_xr_display_glasses`, `brilliant_halo`, `brilliant_frame`, `vive_eagle`. | ### getDisplayState Read what's currently rendered on the simulated glasses DISPLAY (track 5 — the native `glasses.display.*` capability). | Parameter | Type | Required | Description | |---|---|---|---| | `sessionId` | string | yes | — | ### injectInput Drive the simulated glasses' DISPLAY input — the agent's way to "click" the display with no human and no hardware (track 5). | Parameter | Type | Required | Description | |---|---|---|---| | `sessionId` | string | yes | — | | `action` | string (enum) | yes | 'select' fires the target (or focused) node's onClick — pass targetId (from getDisplayState.interactiveIds) for a deterministic click. One of: `navigate`, `select`, `back`. | | `targetId` | string | no | The node id to target. | ### injectHardwareButton Press the simulated glasses' hardware CAPTURE BUTTON (the right-temple controls) — the agent's way to exercise the wearer's hardware privacy gestures with no human and no hardware. | Parameter | Type | Required | Description | |---|---|---|---| | `sessionId` | string | yes | — | | `button` | string (enum) | no | Which hardware button to press. One of: `capture`. | | `action` | string (enum) | no | The press gesture. One of: `tap`, `hold`. | ## Production Ship-readiness checklist and credential setup (Meta DAT + any BYOK providers). Narrative guide: [/docs/mcp-server/tools/production](/docs/mcp-server/tools/production). ### getProductionChecklist Return a personalized production-readiness checklist based on the SDK capabilities the app uses, the handlers it declares, and the BYOK services it integrates. | Parameter | Type | Required | Description | |---|---|---|---| | `capabilities` | string[] | yes | Capability names from getPlatformInfo.features[].name. | | `handlers` | string[] | no | Names of the handlers the app declares (labels the API Keys step's affectedHandlers). | | `services` | string[] | no | BYOK provider names the app actually integrates (e.g. | | `platform` | string (enum) | yes | One of: `android`, `ios`. | | `projectPath` | string | no | Absolute (or cwd-relative) path to the project root. | ### getCredentialGuide Return step-by-step credential setup for the Meta DAT build identity plus each BYOK AI provider the app integrates. | Parameter | Type | Required | Description | |---|---|---|---| | `services` | string[] (enum) | no | BYOK providers the app integrates — one per integration. One of: `anthropic`, `openai`, `google_cloud_vision`, `google_translate`, `google_gemini`, `deepl`, `azure_cognitive`, `aws_bedrock`, `huggingface`, `custom`. | | `handlers` | string[] | no | Optional list of handler names. | | `platform` | string (enum) | yes | One of: `android`, `ios`. | ## Documentation Search the in-session conceptual docs corpus by topic or keyword. Narrative guide: [/docs/mcp-server/tools/search](/docs/mcp-server/tools/search). ### searchDocs Search Extentos documentation by topic or keyword. | Parameter | Type | Required | Description | |---|---|---|---| | `query` | string | no | Optional keyword(s). | | `topic` | string | no | Optional topic id. | | `mode` | string (enum) | no | Response shape. One of: `full`, `snippets`. | ## Keeping this page honest The tool list and parameters above are generated from `toolDefinitions` in `mcp-server/src/tools/definitions.ts`. The generator (`scripts/generate-docs-reference.mjs`) asserts that every shipped tool is categorized, so a new tool can't be added without appearing here. Counts are derived, never hand-typed. --- source: https://extentos.com/docs/resources --- --- title: Resources description: Extentos resources — pricing, data-handling and security, support, service status, license, public roadmap, FAQ, and changelog. type: resource platform: all related: - /docs/resources/pricing - /docs/resources/security - /docs/resources/faq - /docs/resources/roadmap --- - [Pricing](/docs/resources/pricing) - [Security](/docs/resources/security) - [Support](/docs/resources/support) - [Status](/docs/resources/status) - [License](/docs/resources/license) - [Roadmap](/docs/resources/roadmap) - [FAQ](/docs/resources/faq) - [Changelog](/docs/resources/changelog) --- source: https://extentos.com/docs/resources/changelog --- --- title: Changelog description: Release notes for Extentos — the MCP server (@extentos/mcp-server), the Android library (com.extentos:glasses), and the iOS library (swift-glasses). Newest first. The MCP server is pre-1.0; APIs may shift between minor versions until the hardware test loop closes. The 2.x line covers the vendorless core split, the display overflow policy, the iOS on-glasses display path, the minified-release fix, the removal of the automatic wake sound, and capture telemetry; the managed AI gateway, the Phase-4 voice assistant, the first stable iOS release, the pure-SDK pivot, the browser simulator, getEventLog, auto-bind, and device-code auth landed earlier. type: resource platform: all related: - /docs/resources/roadmap - /docs/concepts/ai-gateway - /docs/resources/pricing - /docs/resources/license --- Release notes for Extentos, newest first. The MCP server is **pre-1.0** (the SDK line is 2.x on both platforms — Maven Central and the swift-glasses SPM package, released in lockstep) — minor versions of the MCP server may introduce breaking changes until the hardware test loop closes. For the current published MCP version see the [npm package](https://www.npmjs.com/package/@extentos/mcp-server); for the exact live capability state of any primitive, have your agent call `getPlatformInfo` (more accurate than any docs page). Dates are given where they're verifiable. Items grouped under **Recent** shipped in the same period but aren't pinned to a precise date — we'd rather say "recent" than invent a date. ## Recent These landed across recent releases (mid-2026) and represent the current shape of the platform. ### Managed AI gateway The [Phase-4 voice assistant](/docs/concepts/assistant) now routes its voice AI through Extentos's [managed gateway](/docs/concepts/ai-gateway) on Extentos's provider key — a developer can ship a voice-AI glasses app with zero AI configuration (no key to obtain, store, or rotate). Conversation content (audio, transcripts, model responses) is relayed but **never stored**; only token-usage metadata and computed cost are recorded per project. Metering **and billing are live** — a prepaid-credit model with a $2 free grant per account (see the milestone below). (There is no bring-your-own-key option.) See [pricing](/docs/resources/pricing). ### Phase-4 assistant runtime The `glasses.assistant.*` runtime — a real-time voice loop with a wake/sleep state machine, `session.say()`, barge-in, intent-based end, and tool calling — is **stable on both platforms**: on Android since `com.extentos:glasses` **`1.4.0`** (Maven Central), on iOS in the `1.7.0` Swift package (see the release entries below). Setup and API surface: [the assistant page](/docs/concepts/assistant). ### Meta DAT 0.8.0 The Android library now builds on **Meta Device Access Toolkit 0.8.0** (the iOS package followed on 2026-07-03 — its DAT floor is now 0.8.0 too). The one developer-visible addition is the generic **`META_GLASSES`** device type, surfaced as `glasses.device.type` on Android; it's handled by the same capability dial as every other camera + audio model, so no app changes are needed. Camera, audio, voice, and display behavior are unchanged. ### Display capability Meta opened heads-up display content to third parties; Extentos exposes it as `glasses.display.*` — a UI tree plus full-surface video — **shipped in both SDKs** and fully testable in the simulator on both. On-glasses rendering via Meta DAT (DAT 0.8.0) ships on both platforms, not yet confirmed on Display hardware. Per-variant display status is on the [Meta vendor page](/docs/vendors/meta). ## Library releases The Android and iOS libraries version in lockstep — one release number covers both. ### 2.7.0 — 2026-08-28 **Breaking, in two lines: `audio.playSound` / `registerSound` / `soundNames` are gone, and so is `AssistantConfig.wakeSoundEnabled`.** Both are source-breaking. If you called either, read on — the replacement is one method. **Bring your own realtime model.** If a realtime model already runs your conversation, it hands you finished **audio**, not text. `speak(text)` was the wrong shape for that: it would throw away audio your model already generated so the SDK could re-synthesize the words. The audio client is now symmetric — ```kotlin val rate = if (glasses.audio.outputFidelity == OutgoingAudioFidelity.HiFi) 24000 else 8000 glasses.audio.sendAudio(pcm16 = chunk, sampleRate = rate) // your model's audio, out glasses.audio.stopAudio() // barge-in: drop what is queued ``` `stopAudio()` matters more than it looks. A realtime model emits faster than realtime, so seconds of reply can already be buffered when the wearer starts talking; without it, playback continues until the backlog drains and the interruption feels ignored. `outputFidelity` tells you what the link can carry so your model generates the right format instead of guessing. And you do not have to mute the mic while playing: capture runs through the platform's voice-communication unit, which subtracts our own output from the input. This is not a new audio path — it is the same transport call the managed assistant, the on-device tier and local-voice `speak()` already use on real glasses. What is new is the public entry point. Full guide: [bring your own realtime model](/docs/guides/audio-streaming#bring-your-own-realtime-model). **The named-sound system is removed** — `playSound`, `registerSound`, `soundNames`, the dashboard Sounds section, the clip library and its upload flow, the `listProjectSounds` / `addProjectSound` MCP tools, and the `assistant_config.sounds` grant. It let you upload a clip in the dashboard and play it by name from code. Across its whole life it held eight clips and four slots belonging to one account, and shipping `sendAudio` made it redundant: **read your own bytes and send them.** A chime that used to be `playSound("shutter")` is now `sendAudio(shutterPcm, sampleRate = 24000)`, with no upload, no name, and no round trip through us. **`AssistantConfig.wakeSoundEnabled` is removed** (it was already inert after 2.3.0 stopped the SDK playing a chime). Delete the line and your build is fixed. The wake **phrase** engine is untouched — `glasses.voice.onPhrase` and on-device wake detection are unchanged. Only the wake *sound* is gone. ### 2.6.0 — 2026-08-26 **Telemetry that can answer why a capture failed.** `capture.completed`, `speak.completed` and `device.condition_changed` join the event vocabulary. Photo capture is the flagship capability of the product and emitted no telemetry at all before this, so a failed capture left nothing behind to diagnose. - **Error codes moved into the shared core.** Kotlin was reflecting over class names and Swift kept a hand-maintained parallel switch, which is two copies of one vocabulary and one rename away from drift. Both platforms now read the same codes from the core. - **iOS reports which glasses were used.** The device model was hardcoded empty on iOS, so vendor breakdowns had been Android-only even though the core held the fact the whole time. - **The event registry shrank from 35 events to 29.** Five were dead since the pivot and could not fire. Four more were defined as carrying verbatim transcripts or tool payloads as properties; nothing emitted them, and they are gone rather than left as standing permission. No application change is required. ### 2.5.0 — 2026-08-16 **A level for the audio the assistant is playing.** `AssistantSession` gains `audioLevels()`, so your UI can show the assistant speaking the same way it already shows the wearer speaking. Additive if you *call* the SDK. **If you implement `AssistantSession` yourself**, most likely as a hand-rolled test fake, the protocol gained a member and your implementation needs it. ### 2.4.0 — 2026-08-16 **The assistant says when it stopped talking.** `AssistantEvent` gains `assistantAudioStarted` and `assistantAudioFinished`, which is what you want if your UI needs to know the difference between "the model finished generating" and "the wearer stopped hearing it". Additive, with one caveat worth stating plainly: adding a case to a public Swift enum or Kotlin sealed class **breaks an exhaustive switch that has no `else` branch**. If yours is exhaustive, that is the one edit this release asks of you. ### 2.3.1 — 2026-08-15 **A refused gateway session now says why.** Every failure to open the realtime connection used to surface as the same ten-second `realtime WebSocket open timed out`. A quota refusal, a bad credential, a DNS failure and a TLS failure were indistinguishable, because the failure signal was gated on a flag that stays false for the whole connect window, leaving the timeout as the only reachable outcome. Your app now receives a prefixed reason it can branch on, for example `insufficient_credits: the gateway refused the session (HTTP 429)`. No public API change on either platform, and not source-breaking. ### 2.3.0 — the SDK stops playing a sound you didn't ask for **Superseded in 2.7.0.** The named-sound system this entry tells you to lean on (`playSound` / `registerSound` / `soundNames`, the dashboard Sounds section and its clip library) was removed. Push your own bytes with `sendAudio` instead — the snippets below have been updated to the current API so they still compile. See [bring your own realtime model](/docs/guides/audio-streaming#bring-your-own-realtime-model). **Breaking, in one line: `AssistantConfig.wakeSoundEnabled` is gone.** Delete it and your build is fixed. Nothing else in the sound system changed. Until now the SDK synthesized a chime and played it the moment `wake()` began. It fired *before* the realtime connection opened, which is exactly the window where the glasses audio link may not be up yet — so it went quiet precisely when the wearer needed the confirmation. A cue that only sometimes plays is worse than no cue, so it's removed rather than patched. **If you want an activation sound, you now own it — and it's one line.** Play your own clip from your own wake path: ```kotlin // Android — wherever you wake the session glasses.voice.onPhrase("hey coach") { glasses.audio.sendAudio(wakeChimePcm, sampleRate = 24000) // your clip, your timing session?.wake() } ``` ```swift // iOS _ = await glasses.audio.sendAudio(wakeChimePcm, sampleRate: 24000) try await session.wake() ``` Because it's your call now, you can `await` it, skip it on a button press but not a wake phrase, or use a different clip per trigger — none of which the built-in chime allowed. **Nothing in the dashboard changed** at the time of this release. (That changed in 2.7.0, which removed the Sounds section along with the rest of the named-sound system.) ### 2.2.4 — 2026-08-13 **Read this if you ship a minified Android release build.** On 2.2.3 and earlier, every Android release build with `isMinifyEnabled = true` crashed on launch, and a second defect silently stripped the gateway credential out of the same build. Both are fixed in the SDK's own consumer rules, so upgrading is the whole fix and your `proguard-rules.pro` needs no change. - **The crash.** R8 renamed `com.sun.jna.*`, JNA resolves those classes by name from native code, and the SDK's consumer rules only kept `com.extentos.glasses.**`. The SDK died during transport resolution before doing anything, on every app regardless of which capabilities it used. - **The credential.** Your project key is read by reflection out of your app's generated `BuildConfig`. Nothing kept that class, so in a minified build it was absent from the dex entirely and the app authenticated as though no key had been set. Neither could appear in debug, in the browser simulator, or in any test, because none of those minify. If you ship minified, treat 2.2.4 as your floor. Also in this cut: 11 MB of orphaned model files removed from the core artifact, taking the Android AAR from 14.1 MB to 5.1 MB. ### 2.2.3 — 2026-08-09 **The connection page stops offering toggles the app cannot use.** The toggle list returned a hardcoded voice-activation entry to every integration, so a camera-only app showed its user a microphone kill-switch for hardware it never opens. The list now derives from your declared capability footprint, the way the capability tiles on the same page already did. ### 2.2.2 — 2026-08-09 **iOS raw camera frames match the documented I420 contract.** Meta hardware hands iOS an NV12 buffer with 16-byte-aligned rows, where the documentation, Android and the browser simulator all promise packed I420. The two layouts are byte-identical in size, so no length check could catch it: every computer-vision integrator received a valid-looking buffer, rendered nothing, and saw no error at any layer. Also in this cut: the iOS scaffold now emits the two camera Wi-Fi entitlements and declares the session plist as a build-phase input, without which a generated project does not build at all under Xcode's default script sandboxing. Android now warns once on a raw size mismatch rather than passing an unverified buffer through. ### 2.2.1 — 2026-08-08 **The first 2.2.x an iOS integrator can install.** The Swift package published for 2.2.0 did not compile, so 2.2.1 is the iOS floor for everything in that release. Android was unaffected and republished only to hold lockstep. ### 2.2.0 — 2026-08-07 **The per-device display overflow policy on both platforms, and the iOS on-glasses display path**, which the iOS package had never carried because the Meta display module was simply never pulled in. Display content written once now renders through the same policy on either platform instead of each shell guessing at a row limit. **On iOS, install 2.2.1 rather than this version** (see above). ### 2.1.2 — 2026-07-31 **Behaviour change worth checking before you upgrade: `recordDiscrete` no longer returns a transcript in the browser simulator.** It returns an empty transcript, which is what real hardware has always returned. An app that read that field demoed perfectly in the simulator and went silent on glasses, so the simulator was corrected to match rather than the other way round. ### 2.1.1 — 2026-07-31 **Two fixes a fresh-agent audit turned up**, both on paths a first-time integrator hits. - `capabilities` no longer takes the host app down when the vendor SDK never initialised. It is a property getter read during Compose composition, so an uninitialised vendor SDK crashed the app outright on the most ordinary first-run path there is: the placeholder credentials the scaffold itself emits. - `observability.aiCall` gained a failure reason, so a typed failure stops being recorded as a success. Additive; existing call sites are unchanged. ### 2.1.0 — 2026-07-31 **Brilliant stops costing a dependency line.** The Brilliant transport folded into `com.extentos:glasses`, so a vendor that charges nothing to support now costs nothing to add. The Android XR module self-registers on the same pattern, and the video container check landed on both platforms. ### 2.0.2 — 2026-07-29 **Android on-device transcription and the iOS Brilliant transport become installable.** Both had landed in source; this is the release that makes them resolve. The on-device recogniser is Android-only, and that is a deliberate asymmetry rather than a missing feature. Feeding audio buffers to the system recogniser needs a workaround on Android and is only available from Android 13, and a given device may still refuse it. On iOS the system recogniser has always accepted buffers directly, so iOS reaches the same place with no model to download and no OS floor. ### 2.0.1 — 2026-07-29 **`glasses-brilliant` self-registers.** At 2.0.0 the module required a second explicit registration call from your code. Maven Central is immutable, so the fix needed a new version, and the whole line moves together by the lockstep rule. The module's published POM also declared the core artifact at runtime scope rather than compile scope, so the core types its own public surface hands back were missing from a consumer's compile classpath unless that consumer happened to declare the core itself. ### 2.0.0 — the vendorless baseline (Maven Central + Swift Package) **Read this before upgrading if your app uses the camera or the display.** This is the one release in Extentos's history that can disable a working feature without a build error. **What changed.** `com.extentos:glasses` no longer contains the Meta transport. Camera, display and the Meta device session moved into a new optional artifact, `com.extentos:glasses-meta`. The core artifact now carries **no vendor SDK at all**. **If your app uses camera or display — add one line:** ```kotlin title="app/build.gradle.kts" implementation("com.extentos:glasses:2.0.0") implementation("com.extentos:glasses-ui:2.0.0") implementation("com.extentos:glasses-meta:2.0.0") // ← add this ``` No Kotlin change is needed. The module registers itself at app start, so `TransportChoice.Auto` resolves to real glasses exactly as before. **Keep your `settings.gradle.kts` as-is** — the Meta DAT repository and its `read:packages` token are still required, because those artifacts now arrive through `glasses-meta`. **If you skip it, nothing fails loudly.** Gradle resolves, the build succeeds, voice keeps working and the connection still reports as up — only capture breaks. That shape survives CI and a smoke test, so check explicitly after upgrading: `glasses.transportChosen` reads `SYSTEM_AUDIO` where you expect `REAL_META`, `glasses.capabilities.camera` is `false`, and the SDK logs a warning naming the missing module (`adb logcat -s Extentos:W`). **Android only, for now.** The artifact split is an Android change: SwiftPM has no credentialed-repository problem to solve, and splitting would mean restructuring the published `swift-glasses` package. On iOS the DAT modules remain unconditional dependencies of `GlassesCore`, so an iOS voice app still links them — it just never uses them, needs no Meta credentials, and no Info.plist DAT keys. The vendorless *runtime* behaviour is identical on both platforms; only the dependency hygiene differs. **Why.** The Meta DAT artifacts live on a credentialed GitHub Packages repository. While they sat inside the core artifact, *every* customer had to accept Meta's terms and mint a GitHub token — including voice-only apps that never touch a camera. Skipping it failed the build with a 401 that never mentioned Extentos. That toll is now charged only to apps that actually use vendor hardware. **What this unlocks — voice apps need no vendor setup whatsoever.** A new `SystemAudioTransport` serves the complete agent runtime (turn-taking, barge-in, tool calling, conversation memory, local and cloud models) over the phone's own Bluetooth audio routing. No vendor SDK, no Meta Developer account, no App ID, no GitHub token, no connection page, no pairing flow. It works on smart glasses, on ordinary Bluetooth earbuds, and — with nothing connected — on the phone's own microphone and speaker, so a voice app never dead-ends because the glasses are in their case. See [choose your path](/docs/getting-started/choose-your-path). `TransportChoice.Auto` gains a rung for it, placed below every vendor arm and below the simulator arms: an app with bonded glasses still resolves to `RealMeta`, and the browser-simulator dev loop is unchanged. **Renamed: `AssistantProvider.OpenAi` → `AssistantProvider.Managed`** (Swift `.openAI` → `.managed`). The type was named after one vendor while the **model id** picks the vendor — `gpt-*` OpenAI, `grok-*` xAI, `gemini-*` Google, `local-*` on-device. So `OpenAi(model = "local-auto")` ran an on-device model with no network, which reads like a bug and isn't one. **Your code keeps working.** The old name stays as a fully functional deprecated case on both platforms — same shape, same behaviour, with a deprecation warning and an IDE fix-it pointing at the new one. Migrate when convenient: ```kotlin AssistantProvider.Managed(model = "local-auto") // was AssistantProvider.OpenAi(...) ``` **Also in this release:** bring-your-own-key is fully removed. The assistant runs on the Extentos managed gateway, always; there is no option to supply a provider key and none is on the roadmap. ### 1.11.1 — 2026-07-26 (Maven Central + Swift Package) Android-only cut that restores lockstep: `ExtentosLocalTier.autoChoice(context)` landed for iOS parity just after 1.11.0 was tagged, so on Android it wasn't callable until this release. If you use the local tier's automatic rung selection, this is your floor. ### 1.11.0 — 2026-07-26 (Maven Central + Swift Package) The **local tier ships whole**. `com.extentos:glasses-local` (on-device text models) and `com.extentos:glasses-local-voice` (on-device speech) are both on Maven Central, with `GlassesLocal` / `GlassesLocalVoice` as the Swift-package equivalents. `glasses-local-voice` is the first public release of the high-quality on-device voice. **Automatic** joins the model picker — the assistant selects a local rung sized to the phone and falls back to the gateway when the device can't serve one. See [local models](/docs/concepts/local-models). ### 1.10.0 — 2026-07-26 (Maven Central + Swift Package) The first release carrying `com.extentos:glasses-local` — the on-device text tier, published alongside the existing artifacts on both platforms. ### 1.9.0 — 2026-07-19 (Maven Central + Swift Package) Adds OpenAI's two newest realtime models to the assistant catalog, so they're selectable from the dashboard's Agent tab, from `AssistantProvider.Managed(model = …)`, and from `AssistantCatalog.realtimeModels` in your own in-app picker. - **`gpt-realtime-2.1`** — a drop-in successor to `gpt-realtime-2`: same voices, same `reasoning.effort` knob, same price on every dimension. - **`gpt-realtime-2.1-mini`** — distilled reasoning at mini pricing. **Despite the name it is not a newer `gpt-realtime-mini`**: it *does* accept `reasoning.effort` (the older mini does not) and carries the full 128k context window (the older mini is 32k). Its image-input rate also differs from the rest of the realtime line. - Both are token-billed through the [managed gateway](/docs/concepts/ai-gateway) at OpenAI's published rates; the dashboard's cost estimate and the exact per-response ledger both price them from day one. Existing models are unchanged, and the SDK default stays `gpt-realtime-2` — upgrading does not move any project onto a new model. ### 1.8.0 — 2026-07-17 (Maven Central + Swift Package) **Google Gemini Live joins the assistant**, alongside OpenAI Realtime and xAI Grok — the model id picks the vendor (`gemini-*`), so switching providers is a string change. Gemini adds **streaming video input** (`session.sendVideoFrame`) on the video-capable models: gate it with `session.modelSupportsVideoInput`, because an ungated send on a text-only model surfaces an error to your app *and* tells the model it can't see, rather than dropping the frame silently. All of it bills through the [managed gateway](/docs/concepts/ai-gateway) at list price. ### 1.7.0 — 2026-07 (Maven Central + Swift Package — first stable iOS release) The first **joint Android + iOS release**, and the iOS library's stable debut: the Swift package is published from [`github.com/extentos/swift-glasses`](https://github.com/extentos/swift-glasses), version-locked to the Android line. `.package(url: "https://github.com/extentos/swift-glasses", from: "1.7.0")` resolves the binary core and builds with no workarounds — see [install](/docs/sdk/ios/install). - **iOS SDK, stable.** The capability surface ships on iOS: camera (photo, video **with audio**, live frames), the Phase-4 voice assistant on the managed gateway (wake/sleep, barge-in, tool calling), named sounds (`playSound` / `registerSound` + dashboard slots), the display API, connection page, telemetry. A season of real-hardware dogfooding landed reliability work across the voice loop (silence endpointing, mic-subscription persistence across STT restarts, barge-in flush) and the camera/video path (AAC video audio at the encode ceiling, stream re-arm, auto-finalize when the stream stops mid-record). - **Core (both platforms):** `CameraStatus` (`ready` / `starting` / `broken`) replaces the boolean capture gate; connection-state fixes — no transient `Disconnected` while the glasses stay reachable, no `Active → Connecting` blip when the camera starts; strictly monotonic frame timestamps across stream re-arms; a shared named-sound registry behind both shells. - Published as `com.extentos:glasses:1.7.0` + `com.extentos:glasses-ui:1.7.0` (Maven Central) and the `swift-glasses` `1.7.0` tag (SPM). ### 1.6.0 — 2026-07-11 (Maven Central) Camera-capture refinements and browser-simulator fidelity for the temple-tap capture button. - **`PhotoConfig.fullResolution` → `dedicatedCapture`** (breaking rename, no deprecation alias — pre-adoption). DAT's discrete `capturePhoto` takes no resolution argument, so the non-frame-grab path is a *dedicated* capture whose resolution DAT sets independently of the stream — not a caller-chosen resolution. The old name was a misnomer. - **Capture mechanism in the simulator event log.** `capture_photo_returned` now carries `mechanism` = `frame_grab` | `dedicated`, so a developer testing in the browser simulator — where the returned image is identical either way — can confirm which path their code *would* take on real hardware. - **The capture button is reproducible in the simulator.** The wearer's temple-tap pause/resume/stop and the resulting `CaptureError.StreamPaused` are now testable without hardware: `BrowserSimTransport` honors a simulated paused state through the same shared gate real glasses use, and a shell-declined capture records `capture_denied` in the event log. Published as `com.extentos:glasses:1.6.0` + `com.extentos:glasses-ui:1.6.0`. ### 1.5.0 — 2026-07 (Maven Central) Camera reliability on real Meta hardware, and honest handling of the wearer's temple-tap pause. - **`CaptureError.StreamPaused`** — a capture attempted while the wearer has paused the camera (a single temple tap) returns this typed error carrying an actionable "tap the right temple to resume" message, instead of fighting the platform. DAT 0.8 exposes no app-callable resume; the honest response is to prompt the wearer, then retry. - **Camera-streaming reliability fixes** from a full audit of the frame-grab / video / live-frames paths (cold-start re-arm, stream-state teardown on disconnect). - **Capture failures funnel to the assistant by default** — an assistant tool that captures a photo surfaces the failure's actionable message to the model, so the assistant tells the user exactly what to do. ### 1.4.0 — 2026-07-02 (Maven Central) The **Phase-4 assistant runtime graduates to stable**: `glasses.assistant.*` (wake/sleep state machine, `session.say()`, barge-in, intent-based end, tool calling) is now in the public Maven Central release — the `1.4.0-phase4-dogfood` preview snapshot and its `mavenLocal()` setup are retired. Published as `com.extentos:glasses:1.4.0` + `com.extentos:glasses-ui:1.4.0`. ### 1.3.0 — 2026-05-24 (Maven Central) Added the Phase-3 conversation runtime (`glasses.conversation.onWake { listen() / speak() / cancelSpeak() / ai.complete() }`). **Superseded by the Phase-4 assistant runtime** (see Recent) for new voice-AI work, but still resolvable from Maven Central. Published as `com.extentos:glasses:1.3.0` + `com.extentos:glasses-ui:1.3.0`. ### 1.2.0 — 2026-05-22 (Maven Central) **First public release** of the Android library on Maven Central, via the Sonatype Central Portal. `com.extentos:glasses` (core) + `com.extentos:glasses-ui` resolve from the default `mavenCentral()` repository with no workaround. > The iOS Swift package ships from [`github.com/extentos/swift-glasses`](https://github.com/extentos/swift-glasses), version-locked to the Android release line — stable since `1.7.0` (see the entry above). Some primitives landed Android-first; per-primitive status: `getPlatformInfo`. ## Platform milestones ### Managed-gateway billing — 2026-06-26 The [managed AI gateway](/docs/concepts/ai-gateway) went from metered-only to **live prepaid billing**. Every account gets **$2 of free managed-gateway credit**; past that, usage draws down a prepaid credit balance bought in the dashboard's account **Billing** hub (any amount, with optional auto-reload), and the gateway stops when the balance hits zero — prepaid, so no surprise invoice. Pricing is the provider's list price with no markup — tokens for token-billed models, connected minutes for per-minute voice models. All managed usage counts (simulator, dev, and real hardware). Nothing outside the managed gateway is affected — the free surfaces stay free. Full detail on [pricing](/docs/resources/pricing). ### Pure-SDK pivot — 2026-05 The library stopped interpreting a declarative runtime and became a **pure Kotlin/Swift SDK**. The trigger was evidence rather than taste: 31 items from our own dogfood logs were developers fighting the authoring layer to reach capabilities the SDK already exposed, so the layer was removed rather than improved. Customer code now writes its own handler classes directly against capability primitives (`glasses.audio.transcriptions()`, `glasses.camera.capturePhoto()`, `glasses.audio.speak()`, …) — there is no intermediate authoring layer to fight through. The MCP server, simulator, and backend all moved post-pivot in the same period. The canonical voice-glasses patterns are available via the `getCodeExample` MCP tool. ### Earlier foundation These shipped before the pivot period and remain core to the developer loop: - **Browser simulator** at `extentos.com/s` — the hardware surrogate the library connects to in BrowserSim mode, running the same library code as production with only the transport and I/O swapped. - **`getEventLog`** — the structured event log (queryable per chip: `errors` / `voice` / `camera` / `display` / `ai` / `lifecycle` / `custom`), mirrored to the backend during simulator mode so an AI agent can debug a run. - **Auto-bind dev loop** — shipped on **Android and iOS**; the library's debug-build probe finds the agent's local MCP server and binds a running app to a simulator session automatically (the `extentos.session.plist` URL-bake path remains as the iOS fallback). - **Device-code auth** — the OAuth-style flow (RFC 8628) that links an anonymous install to a free email-only account. It's required for browser-simulator session minting, project scaffolding (`generateConnectionModule`), and the managed AI gateway; discovery, validation, guidance, and on-device simulation stay anonymous. ## Notes - **Last updated: 2026-08-28.** Entries marked "Recent" are honest about approximate timing; precise dates appear only where verifiable. - **Versioning:** the Android and iOS libraries version in lockstep; the MCP server versions independently (always install `@latest` unless you're pinning for reproducibility — see the [npm package](https://www.npmjs.com/package/@extentos/mcp-server)). - For *where Extentos is going* rather than what shipped, see the [roadmap](/docs/resources/roadmap). ## Related - **[Roadmap](/docs/resources/roadmap)** — forward-looking direction - **[The managed AI gateway](/docs/concepts/ai-gateway)** — how the assistant's AI runs, and metering - **[Pricing](/docs/resources/pricing)** — free surfaces and the one metered surface - **[License](/docs/resources/license)** — MIT for the MCP server; proprietary libraries (public packages) --- source: https://extentos.com/docs/resources/faq --- --- title: Frequently asked questions description: The Extentos FAQ — what it is, install, cost (mostly free; the AI gateway is metered), supported Ray-Ban/Oakley Meta hardware, and Meta DAT vs Extentos. type: resource platform: all related: - /docs/getting-started/with-agent - /docs/concepts/transport-vs-app - /docs/concepts/ai-gateway - /docs/vendors/meta - /docs/resources/pricing - /docs/resources/security --- **Extentos is the smart-glasses development layer for Meta Ray-Ban: install its MCP server, let your AI agent scaffold a native iOS/Android app, and test in a browser simulator — free, except the managed AI gateway is metered.** The questions below go deeper. The most-asked questions about Extentos, organized by topic and cross-linked to the deep documentation. If you don't find your question here, see the [support page](/docs/resources/support) or email [hello@extentos.com](mailto:hello@extentos.com). ## Getting started ### Is Extentos free? Almost entirely. The discovery, validation, and guidance MCP tools, on-device simulation (`LocalSimTransport`), and real-hardware testing on Ray-Ban Meta are free with no account, forever. Code generation via `generateConnectionModule` and the browser simulator both link a free email-only account. The browser simulator at [extentos.com/s](https://extentos.com/s) requires a free email-only account from the first session — Google one-click or email + password, no payment, no card. Once linked, sessions are unlimited. The **one** surface that meters usage is the [managed AI gateway](/docs/concepts/ai-gateway) behind the [Phase-4 voice assistant](/docs/concepts/assistant): usage is metered today, and billing runs on a prepaid-credit model — every account starts with **$2 of free** managed-gateway credit, then draws down a prepaid balance. There is no bring-your-own-key option — you cannot supply a provider key, and none is on the roadmap. Apps that don't use the managed gateway never touch a metered surface. See [pricing](/docs/resources/pricing) for the full breakdown. ### Do I need an Extentos account to start? Mostly no. Discovery, guidance, validation, and real-hardware testing on Ray-Ban Meta hardware work without ever creating an Extentos account. A free email-only account (no payment) links the first time you mint a browser-simulator session or run the `generateConnectionModule` scaffold step (it mints your account-bound project key). See [auth](/docs/mcp-server/auth) for the full flow. ### How do I install Extentos in my AI coding agent? The shortest path is the agent prompt: paste *"Install @extentos/mcp-server in my MCP config"* into Claude Code, Cursor, Windsurf, or Cline. The agent edits the right config file or runs the right command for its host. For Claude Code specifically, the one-liner is `claude mcp add extentos -- npx -y @extentos/mcp-server@latest`. See [install](/docs/mcp-server/install) for per-host instructions and config-file locations. ### What prerequisites do I need? Node.js 20 or newer on `PATH`. That's it for the MCP server — `npx` fetches and caches the package on first run. To target real Ray-Ban Meta hardware later, you'll also need a Meta Wearables Developer Center account (free, ~15-30 minute setup) for the Meta DAT credentials. See [vendors/meta](/docs/vendors/meta) for the Meta-specific setup. ## Hardware support ### What smart glasses does Extentos support? As of 2026-07, Extentos is **production-supported on Meta Ray-Ban only** — every variant in market (Gen 1 Wayfarer, Headliner, Skyler frame styles; Gen 2; Ray-Ban Meta Optics; Meta Ray-Ban Display) plus Oakley Meta HSTN, Oakley Meta Vanguard, and the 2026 Meta Glasses line. **Android XR is in preview**: the projected transport is built and proven on Google's Android XR emulator, and simulator sessions accept an Android XR device identity, so you can check how your app behaves under it today. The module is deliberately unpublished and no Android XR glasses are buyable yet, so nothing there is production-supported or hardware-verified — see [Android XR](/docs/vendors/android-xr) for the exact status. Apple smart glasses remain a tracked roadmap vendor. See [vendors](/docs/vendors) for the full status table. ### Does Extentos work with Ray-Ban Meta Display? Yes. Your Extentos-built app runs on a Meta Ray-Ban Display the same way it runs on Gen 1 or Gen 2 — full camera, voice trigger, and audio support. You can **also render UI to the heads-up display** via `glasses.display.*` — it ships in both SDKs and is fully sim-testable on both; on-glasses rendering via Meta DAT ships on both platforms, not yet confirmed on Display hardware. The wearer drives it with captouch + the Neural Band, delivered to your app as abstracted tap events. What's not exposed is raw Neural Band gesture data and the "Hey Meta" engine. See [vendors/meta § Compatible models](/docs/vendors/meta#compatible-models). ### Why do Meta's own Display apps look richer than what I can build? Third-party display rendering **is** available — Extentos exposes it as `glasses.display.*` in both SDKs (on-glasses rendering via Meta DAT 0.8.0 ships on both platforms, not yet confirmed on Display hardware). The full-screen experiences on Meta Ray-Ban Display — pedestrian navigation (Meta's own app, not Google Maps), WhatsApp, live captions — are Meta first-party system apps; your surface is the display session. See [vendors/meta § Partner integrations](/docs/vendors/meta#partner-integrations-the-how-did-they-get-that-question). ### Do I need real Ray-Ban Meta glasses to develop? No. Extentos's on-device `LocalSimTransport` (in-memory local sim) and the Extentos browser simulator (`BrowserSimTransport` at [extentos.com/s](https://extentos.com/s)) cover the bulk of the dev loop. Real hardware verifies the final-mile fit (camera fidelity, BT latency, real-world coexistence) but isn't required for daily iteration. See [transport vs app simulation](/docs/concepts/transport-vs-app) for what each simulator does. ### When will Android XR / Brilliant Labs / Apple be supported? No committed timelines. A vendor becomes production-supported when adding it makes a real product — transport implementation, simulator coverage, and tested hardware — which depends on each vendor's SDK maturity and hardware availability. Android XR has the first two: the transport is built and proven on Google's emulator, and the simulator carries the device identity. What it lacks is hardware, so the module stays unpublished until Google's platform leaves developer preview and glasses are buyable ([full status](/docs/vendors/android-xr)). Apple is earlier still — no third-party smart-glasses SDK exists to build against. See the [changelog](/docs/resources/changelog) for actual release activity. **Brilliant Labs** is the inverse, and worth understanding as a contrast. You can order a Halo today, though the vendor says the first units ship in early August 2026, and nothing has run our transport yet — and unlike Android XR there is no emulator standing in either. The BLE transport, the on-device Lua bundle and the display translation are all built, the wire protocol is verified byte-for-byte against Brilliant's own implementation, and it ships inside the SDK on both platforms (nothing to add on Android, `TransportChoice.brilliant` on iOS) precisely so that someone with a Halo can be the first to run it. Still capture and the microphone work; video does not and never will, because neither device has a video primitive or a codec. Both devices are selectable in the simulator now. See the [Brilliant vendor page](/docs/vendors/brilliant). ## Pricing and accounts ### Why does the browser simulator need an account? The browser simulator runs on Extentos's backend (WebSocket hub, voice proxy for STT/TTS, event log persistence). It costs Extentos per-session compute, so it takes the same free email-only account as the other backend-provisioned surfaces (the `generateConnectionModule` scaffold step, which mints your project key, and the account-scoped project tools). Discovery, validation, and guidance run on your machine or talk directly to Meta — no Extentos compute, no account needed. See [pricing](/docs/resources/pricing#free-with-a-free-account). ### Does my shipped app cost me anything? It depends on whether it uses the managed AI gateway. A DAT-native app (camera, audio, sensors, voice triggers, display) or one that calls its own AI provider from handler code runs against real Ray-Ban Meta hardware via `RealMetaTransport`, which doesn't talk to the Extentos backend — runtime cost from Extentos is zero per end-user. If your app ships the [Phase-4 voice assistant](/docs/concepts/assistant) on the [managed gateway](/docs/concepts/ai-gateway), that AI usage is metered and billed on a prepaid-credit model, with no bring-your-own-key option. See [pricing](/docs/resources/pricing#real-hardware-and-production-cost). ### Will Extentos add a paid tier later? The free surfaces stay free. The one metered surface — the [managed AI gateway](/docs/concepts/ai-gateway) — is metered and billed today on a prepaid-credit model, with a $2 free grant per account. Existing accounts will not be retroactively gated on the free surfaces; material changes are communicated via the [changelog](/docs/resources/changelog). ### Can I share an Extentos account across my team? Single accounts only at MVP. Multi-seat billing and team workspaces are deferred post-launch. Each developer should run their own MCP install with their own account — the per-machine `installId` is anonymous and doesn't conflict with collaboration. ## Capabilities and limits ### Can my third-party app trigger on "Hey Meta, [my app], do X"? No. The "Hey Meta" wake word is system-level and not exposed to third-party apps. Your custom voice trigger is recognized on the phone (on real hardware: on-device Vosk on Android, Apple's SFSpeechRecognizer on iOS; the platform recognizer on the dev path) over Bluetooth audio — the wake word *is* your custom phrase, not "Hey Meta" + your phrase. See [vendors/meta § Capabilities NOT in the public toolkit](/docs/vendors/meta#capabilities-not-in-the-public-toolkit). ### Can my app use custom gestures (tap, swipe)? There's no general gesture API in the public Meta DAT toolkit as of 2026-07. Captouch is system-defined (taps pause/resume your session), and on Meta Ray-Ban Display wearer interactions reach your `glasses.display.*` UI as abstracted tap events. Session lifecycle signals (pause, resume, stop) work normally. ### What can I actually build for Meta Ray-Ban today? Photo capture, video capture, camera streams, voice-triggered apps (custom phrases via phone-side recognition), always-listening assistants, TTS-driven audio apps — and on Meta Ray-Ban Display, your own UI via `glasses.display.*`. There is no motion-sensor (IMU) API in DAT. The full capability matrix is at [vendors/meta § What's in the Wearables Device Access Toolkit](/docs/vendors/meta#whats-in-the-wearables-device-access-toolkit-dat). ### Why doesn't Meta's Mock Device Kit simulate voice triggers? Meta's Mock Device Kit simulates the BLE/SDK layer — your app *thinks* it's connected to glasses. Voice-trigger recognition runs on the *phone's* speech recognizer, not on the glasses, so simulating the glasses doesn't simulate the recognizer. Extentos's browser simulator replaces the *whole* path with the developer's webcam + microphone, which is why it can exercise voice triggers end-to-end. See [transport vs app simulation](/docs/concepts/transport-vs-app). ## Distribution ### Can I publish my Meta Ray-Ban app to the Meta App Store? As of 2026-07, public publishing is partner-gated while Meta's toolkit is in developer preview (Meta says broader publishing arrives in 2026). Everyone else distributes the phone app normally (APK sideload or enterprise on Android; TestFlight on iOS — Meta doesn't support iOS App Store publishing yet) and enrolls users as testers via Meta release channels — no Meta content review. See [vendors/meta § Distribution state](/docs/vendors/meta#distribution-state-2026-07). ### How do users install my Meta Ray-Ban app? Users install your *phone* app (Android via APK or Play Store; iOS via TestFlight or App Store). The first time they open it, the Meta DAT registration flow runs and pairs the app with their Ray-Ban Meta. Your app is a regular phone app that talks to the glasses over Bluetooth — there's no "install on the glasses" step. ## Privacy and data collection ### Does Extentos collect my users' photos, video, voice transcripts, or AI prompts? Almost never, and never as telemetry — but there is one exception worth knowing before you answer this question for your own users. For handler-code paths (photo bytes, video frames, audio samples, your own AI calls), Extentos's library is architecturally incapable of inspecting the payloads at all. For the [managed AI gateway](/docs/concepts/ai-gateway) (the Phase-4 voice assistant on Extentos's key), the gateway *relays* the audio/transcripts/model responses but **never persists them** — the only thing recorded is the provider's token-usage metadata and computed cost. So Extentos is in the path for managed-gateway traffic but never stores the content. **The exception: display media.** If you render a *captured* photo or clip on a Ray-Ban Display, the SDK uploads a copy to Extentos's media host, because the glasses can only fetch media from an `http(s)` URL. It persists until your app calls `forgetHostedImage` / `forgetHostedVideo`. Capture alone never uploads, and passing a URL you host yourself uploads nothing. See [hosted display media](/docs/resources/security#hosted-display-media--the-one-content-path) for the full picture and what to put in your own privacy policy. The full never-collect list is in [security § What we never collect](/docs/resources/security#what-we-never-collect). ### Does Extentos see my AI provider API keys? It depends on the path. For **handler-code AI calls** (you call Anthropic / a vision model / a translation API from your own handler), the keys live in your app and your provider's auth headers — Extentos has no callback or hook between the SDK read and your request; no inspection, no proxy. For the **managed AI gateway**, the assistant runs on Extentos's *own* provider key (you supply none — that's the zero-config point); there is no option to run it on your own key. See [the gateway page](/docs/concepts/ai-gateway) and [security](/docs/resources/security). ### What does Extentos collect, then? Aggregate metadata — tool calls, install events, vendor adoption rates, capability usage, error rates, library/MCP versions, device models, country (derived from IP, then the IP is dropped). Tagged with anonymous IDs that can never be reversed back to a person. This data powers a developer-facing analytics dashboard (coming soon), Extentos's product roadmap, and aggregate ecosystem signals shared with glasses vendors as market data (10-account aggregation floor). Full schema in [security § What we collect](/docs/resources/security#what-we-collect). ### Will Extentos sell data about my specific app to my competitors? No. Aggregate ecosystem signals (vendor market share, capability popularity) are sold to glasses vendors only as aggregates over **at least 10 distinct accounts**, with a second floor that suppresses vendor-level data when fewer than 25 developers ship to that vendor. Individual-developer or individual-end-user data is never shared. These two floors are the structural guarantees against competitive leakage. Developers can also opt their app out of inclusion in vendor aggregates via `ExtentosConfig.dataSharingConsent = false` (a shipped SDK field; the vendor aggregate product it gates is forthcoming) — separate from disabling telemetry collection. ### Can I opt out of telemetry? Yes — three independent surfaces. (1) `EXTENTOS_TELEMETRY=0` env var disables MCP-side telemetry for the current shell. (2) `npx @extentos/mcp-server@latest decline-privacy` disables it permanently for the install. (3) `ExtentosConfig.telemetryConsent = false` in your shipped app disables library-runtime telemetry per-app. Any combination works. See [security § Three opt-out surfaces](/docs/resources/security#three-opt-out-surfaces). ## Architecture and AI agents ### Is Extentos a wrapper around Meta's Device Access Toolkit? Partially. Extentos has four internal transport implementations: `RealMetaTransport` wraps the real Meta DAT SDK for production hardware; `LocalSimTransport` is Extentos's own in-memory transport for fast on-device testing (no DAT SDK); `BrowserSimTransport` is an Extentos-original architecture (browser simulator at extentos.com/s, WebSocket protocol, no DAT involvement) for the headline dev experience. See [transport vs app simulation](/docs/concepts/transport-vs-app) for the deep dive. ### Does my code change when targeting different vendors? No, by design. Your handler subscribes to vendor-agnostic SDK primitives (`glasses.audio.transcriptions()`, `glasses.camera.capturePhoto()`, `glasses.audio.speak()`, etc.). When a vendor gets a transport implementation in Extentos, your existing handler code runs on it without rewrites — Android XR landed that way, and the same handlers run under an Android XR device identity in the simulator today. See [capabilities § Why a shared vocabulary](/docs/concepts/capabilities#why-a-shared-vocabulary-is-the-right-design). ### Which AI coding agent works best with Extentos? All MCP-compatible agents (Claude Code, Cursor, Windsurf, Cline) work — Extentos's tools are protocol-standard. Claude Code has the smoothest install (single CLI command); the others need a small JSON config edit. Tool capabilities are identical across hosts. See [install](/docs/mcp-server/install) for per-host setup. ### What does the AI agent actually do with the MCP server? The agent calls Extentos's deterministic tools to discover the SDK capability list, fetch canonical Kotlin/Swift code patterns, scaffold a connection module, validate the integration, and provision a browser simulator session — all without you typing any setup code. The canonical sequence is `getPlatformInfo` → `getCodeExample(pattern)` → `getCapabilityGuide(feature)` (per primitive) → `generateConnectionModule` → (agent writes handler classes against the SDK) → `validateIntegration` → `createSimulatorSession`. See [MCP server overview](/docs/mcp-server) for the full tool catalog. ### What happens when the user folds their glasses mid-flow? When the glasses fold, capture calls fail with `CaptureError.HingesClosed`, which carries an actionable message ("The glasses are folded. Open them to use the camera."). Your handler pattern-matches the capture result and decides what to do (typically pause active streams via `scope.cancel()` / `task.cancel()` and wait for the user to unfold). ## Help and resources ### Where do I file a bug or feature request? Email [hello@extentos.com](mailto:hello@extentos.com) with the `[bug]` prefix. Include your install version (`npx @extentos/mcp-server@latest whoami`), the platform (Android / iOS), the host (Claude Code / Cursor / Windsurf / Cline), and a minimal reproduction. ### Where's the source code and license? The MCP server (`@extentos/mcp-server`) is MIT-licensed on [npm](https://www.npmjs.com/package/@extentos/mcp-server). The native libraries are distributed as public packages — Android on [Maven Central](https://central.sonatype.com/artifact/com.extentos/glasses), iOS as a Swift package from [swift-glasses on GitHub](https://github.com/extentos/swift-glasses) — that you build against; their source is proprietary. See [resources/license](/docs/resources/license) for current declarations. ### Where do I find release notes? The [changelog](/docs/resources/changelog) tracks notable releases. Pre-1.0 means APIs may shift between minor versions until the hardware test loop closes — pin to an exact version (see the [npm package](https://www.npmjs.com/package/@extentos/mcp-server) for the current one) if you need cross-session reproducibility. ## Related - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install + first prompt + verify - **[Transport vs app simulation](/docs/concepts/transport-vs-app)** — the differentiator framing - **[Vendors: Meta Ray-Ban](/docs/vendors/meta)** — the production target's full capability matrix - **[The managed AI gateway](/docs/concepts/ai-gateway)** — how the assistant's AI runs, and what's metered - **[Pricing](/docs/resources/pricing)** — what's free, and the one metered surface - **[Security](/docs/resources/security)** — data collection, opt-out, end-user privacy --- source: https://extentos.com/docs/resources/license --- --- title: License description: Extentos license terms. The @extentos/mcp-server npm package is MIT-licensed (declared in its package.json, surfaced on the npm package page). The Android library (com.extentos:glasses) and the iOS library (Extentos) are distributed as public packages with proprietary source. This page states the current, verifiable licensing position per component. type: resource platform: all related: - /docs/resources/changelog - /docs/resources/support - /docs/mcp-server/install --- This page states the **current, verifiable** licensing position for each Extentos component. Extentos is pre-1.0; where a component's formal license terms aren't yet finalized, this page says so plainly rather than implying more than is committed. ## MCP server — MIT The **`@extentos/mcp-server`** npm package is **MIT-licensed**. This is declared in the package's `package.json` (`"license": "MIT"`) and is shown on the [npm package page](https://www.npmjs.com/package/@extentos/mcp-server). MIT permits commercial use, modification, distribution, and private use, with the standard attribution and no-warranty terms. This covers the MCP tooling you run via `npx -y @extentos/mcp-server@latest` — discovery, code generation, validation, and simulator orchestration. The code it *generates into your app* is yours to license however you choose; the generator is MIT. ## Native libraries — public packages, proprietary source The **Android library** (`com.extentos:glasses` + `com.extentos:glasses-ui`, package `com.extentos.glasses.core`) ships as a public artifact on [Maven Central](https://central.sonatype.com/artifact/com.extentos/glasses); the **iOS library** (`Extentos`) ships as a Swift package from [swift-glasses on GitHub](https://github.com/extentos/swift-glasses). You depend on the published packages and build against their public API — the **library source is proprietary** and not distributed. There is **no open-source grant for the libraries** today, and no separate `LICENSE` file. If you need a specific license commitment — commercial use, redistribution — email [hello@extentos.com](mailto:hello@extentos.com) and ask. ## License position by component | Component | Distribution | License position | |---|---|---| | `@extentos/mcp-server` | [npm](https://www.npmjs.com/package/@extentos/mcp-server) | **MIT** (declared in `package.json`) | | Android library (`com.extentos:glasses`, `:glasses-ui`) | [Maven Central](https://central.sonatype.com/artifact/com.extentos/glasses) (binary) | Proprietary; no open-source grant | | iOS library (`Extentos`) | Swift package ([swift-glasses on GitHub](https://github.com/extentos/swift-glasses)) | Proprietary; no open-source grant | | Web app, backend, docs | Not distributed | Proprietary | ## Third-party dependencies Extentos builds on third-party SDKs and libraries that carry their own licenses — most notably Meta's Device Access Toolkit (governed by Meta's developer terms, **not** Extentos's license) and the usual npm / Gradle / SPM dependency trees. Your obligations toward those are independent of Extentos's MIT grant on the MCP server. In particular, shipping to real Meta Ray-Ban hardware requires accepting Meta's developer terms via your own [Meta Developer Center](https://developers.meta.com) registration. ## Trademarks Extentos names the platforms it supports so developers can find the right integration. Those names belong to their owners, and using them descriptively is not a claim of any relationship. - **Android**, **Android XR**, **Jetpack**, **Glimmer** and **Google Play** are trademarks of Google LLC. - **Meta** is a trademark of Meta Platforms, Inc. - **Ray-Ban** is a registered trademark of Luxottica Group S.p.A. - **Brilliant Labs**, **Halo** and **Frame** are trademarks of Brilliant Labs. Extentos is an independent product. It is **not affiliated with, endorsed by, sponsored by, or certified by** Google, Meta, Luxottica, or Brilliant Labs. Where this documentation describes a vendor's platform, capabilities, or release plans, it describes that vendor's own published position and links the source; it does not speak for them. ## Questions If you need a clarification on licensing — commercial use, redistribution of the libraries, or a written license commitment ahead of the pending `LICENSE` file — email [hello@extentos.com](mailto:hello@extentos.com) with the `[license]` prefix. The maintainer treats licensing questions as high priority because they block downstream adoption. ## Related - **[Changelog](/docs/resources/changelog)** — release history per component - **[Support](/docs/resources/support)** — how to file issues, including licensing questions - **[Install](/docs/mcp-server/install)** — getting the MIT-licensed MCP server running --- source: https://extentos.com/docs/resources/pricing --- --- title: Pricing description: Extentos is free to start — discovery, validation, the on-device simulator, and real-hardware DAT testing need no account. A free account unlocks the browser simulator, project scaffolding, and the managed AI gateway. The gateway is the one metered surface; every account gets $2 of free credit, then prepaid credits at provider list price. type: resource platform: all related: - /docs/getting-started/with-agent - /docs/concepts/ai-gateway - /docs/concepts/transport-vs-app - /docs/resources/security - /docs/resources/changelog --- Extentos is free to start, and a good deal of it needs no account at all. Two lines matter, and they fall in different places: the **account line** — a free account unlocks the browser simulator, project scaffolding, and the managed AI gateway; everything else is anonymous — and the **money line** — the managed AI gateway is the one surface that meters real provider spend. Even the money line starts with a free credit grant. Here's the whole picture, and exactly where each line falls. ## Free, no account at all These work with no signup, no email, no payment — anonymously, keyed only to an install id: - **Discovery & reference MCP tools** — `getPlatformInfo`, `getCapabilityGuide`, `getCodeExample`, `searchDocs`. Everything an agent reads to learn the platform. - **Validation & guidance MCP tools** — `inspectIntegration`, `validateIntegration`, `getPermissions`, `getProductionChecklist`, `getCredentialGuide`. All the correctness gates and setup guidance. The full list is the [MCP tools reference](/docs/reference/mcp-tools). - **`LocalSimTransport`** — Extentos's on-phone in-memory simulator (no DAT SDK, no network, no backend). - **Real-hardware testing through `RealMetaTransport`** — you bring your own Meta Developer Center registration (a one-time, free, ~15–30 min setup with your personal Meta credentials). Testing the DAT-native primitives on glasses touches no Extentos account. These never expire, never charge, and require no Extentos account. (Two things that *look* like they'd be here aren't: minting a **browser-simulator** session and running **`generateConnectionModule`** both need a free account — see below — because each provisions an account-bound backend resource.) ## Free, with a free account Three things need an Extentos account — and the account is free (no card, no payment). The first time your agent hits one, the backend returns an `auth_required` response with a verification URL, your agent surfaces it, you sign in once, and from then on it just works: - **The browser simulator** at [extentos.com/s](https://extentos.com/s) — `createSimulatorSession` mints an account-scoped session. It runs on Extentos's backend (WebSocket hub, voice STT/TTS proxy, event-log persistence), the one piece that costs per-session infrastructure. - **Project scaffolding** — `generateConnectionModule` mints your **account-bound project key** (baked into the build so production gateway usage attributes to your account), so it needs the account too. - **The account-scoped project tools** — assistant config, credentials, connection-page writes, analytics. These read and write your dashboard project, so they need to know whose it is. You can skip them entirely: everything they set has a code-side equivalent, including the assistant model (`AssistantProvider.Managed(model = …)` wins over the dashboard). Details: - **Free email-only account.** Google OAuth (one click, ~5 seconds) or email + password. Sign-in is the entire commitment. - **Signing in also opens your managed-gateway free credit** — every account starts with **$2** of free AI-gateway credit (see the next section). - **Simulator session minting is unlimited and free.** No session quota, no cap on *creating* sessions. (What a session *runs* — managed AI usage inside it — draws down your gateway credit like anywhere else; see below.) - **Everything anonymous still works.** The account gates only these three surfaces and the managed AI gateway — discovery, validation, guidance, the on-device sim, and real-hardware DAT testing stay anonymous. The flow happens through your agent without manual token-paste — the device-code mechanic ([RFC 8628](https://datatracker.ietf.org/doc/html/rfc8628)) handles the handoff, the same pattern `gh auth login` and `vercel login` use. ## Metered: the managed AI gateway If your app uses the [Phase 4 assistant runtime](/docs/concepts/assistant), its voice AI runs through Extentos's [managed gateway](/docs/concepts/ai-gateway) on Extentos's provider key by default — no API key in your app. That convenience is the one thing Extentos meters and bills, because it's real provider spend Extentos fronts. **Billing is live.** It's a prepaid **credit** model — you're never surprised by an invoice: - **Every account starts with $2 of free managed-gateway credit.** Sign in and it's already there. - **After the free credit, usage draws down a prepaid balance** you top up in the dashboard's **Billing** hub (buy any amount; optional auto-reload). When the balance reaches zero, managed-gateway sessions **stop** until you add credits — prepaid means Extentos never fronts more than you've paid, and you never get a bill you didn't expect. - **Priced at the provider's list price, no markup** — token counts for token-billed models, or connected minutes for per-minute voice models (xAI Grok voice). Extentos's margin is the provider's volume discount, not a surcharge on you. - **All managed usage counts toward your balance — simulator, dev, and real hardware alike.** There's no free-in-the-simulator loophole; the $2 is one unified allowance across every environment, and the dashboard shows it deplete in real time. **There is no bring-your-own-key option.** The assistant runs on the Extentos managed gateway, always. Extentos holds the provider account; you draw down credit. This is not a roadmap item — it is the design. **Not metered at all:** apps that don't use the managed gateway. A DAT-native app (photo capture + TTS, no AI), or one that calls its own AI provider from handler code (handler-code BYOK), touches no billed Extentos surface — its Extentos runtime cost is zero. ## Real-hardware and production cost Running your app on actual Meta Ray-Ban glasses uses Meta's Device Access Toolkit and your own Meta Developer Center registration — free, one-time, ~15–30 minutes. The DAT path (`RealMetaTransport`) doesn't bill anything to Extentos. The **only** runtime cost from Extentos on a shipped app is managed-gateway AI usage, if you use it. Ship a DAT-native app, or one that calls its own AI from handler code, and Extentos's runtime cost to you is zero. Ship a managed-gateway voice app and the AI usage draws down your credit balance (the $2 free credit, then prepaid credits above). There's no per-seat, per-app, or subscription fee to ship either way. ## Frequently asked questions ### Can I try Extentos without signing up? Yes — a good deal of Extentos works with no account. Install the MCP server, run discovery and validation, do on-device simulation through the local in-memory sim, and test on real Meta Ray-Ban glasses, all anonymously. A free account links in only for the browser simulator, the `generateConnectionModule` scaffold step (it mints your account-bound project key), and the managed AI gateway — the signup ask fires via the same device-code flow the first time one of those is used. ### Do I need to pay to ship to production? Not for a DAT-native app (camera, audio, sensors, voice triggers, display) or one that calls its own AI provider — those bill nothing to Extentos. If you ship the **managed-gateway voice assistant**, its AI usage is billed: the $2 free credit per account, then prepaid credits. It's prepaid, so there's no surprise invoice — and no per-seat or subscription fee to ship. ### How much is free before I pay for the gateway? **$2 of managed-gateway credit per account** — granted the moment you sign in, shared across the simulator, dev, and production. That's roughly a few days of active voice-assistant dogfooding. You watch it deplete live in the dashboard; when it runs out, managed-gateway sessions pause until you add credits. Everything outside the managed gateway stays free regardless. ### Is the on-device `LocalSimTransport` ever gated? No. `LocalSimTransport` runs entirely in-memory on the developer's phone or emulator and never reaches an Extentos backend. Always free, no account. ### What exactly counts toward managed-gateway usage? Only calls made through the assistant runtime on Extentos's key — measured per the provider's billing unit: tokens (audio + text) for token-billed models, connected minutes for duration-billed voice models like xAI Grok — priced at the provider's list price. **Every environment counts** — simulator, dev, and real hardware all draw down the same balance (the $2 free credit is one unified allowance, not per-environment). See [the gateway page](/docs/concepts/ai-gateway#metering-and-billing). ### What if my development environment has no browser? The device-code flow is designed for exactly that case. The CLI prints a URL and a short user-code (e.g. `ABCD-1234`); open the URL on your phone or another signed-in machine, approve the code, and the CLI on your headless machine picks up the link — the same pattern `gh auth login` and `vercel login` use. ### What if I already have an account from a previous machine? Same flow, simpler — the `/activate` page shows "Approve this terminal for [your email]?" with one click. No second signup; it just tells your existing account about the new install. ## Related - **[The managed AI gateway](/docs/concepts/ai-gateway)** — how AI billing and metering work - **[Security](/docs/resources/security)** — what's collected, what never is - **[Quickstart with an agent](/docs/getting-started/with-agent)** — install and first run - **[Changelog](/docs/resources/changelog)** — when credit billing and other features shipped --- source: https://extentos.com/docs/resources/roadmap --- --- title: Roadmap description: The Extentos public roadmap — shipping today (Meta Ray-Ban, both native SDKs, managed AI gateway, the voice assistant, display), in development, and tracked further out. type: resource platform: all related: - /docs/vendors - /docs/concepts/ai-gateway - /docs/resources/changelog - /docs/resources/pricing - /docs/resources/security - /docs/resources/support --- This page is the **public roadmap** — strategic direction for Extentos, dated for honesty about how recent the picture is. It is not a commitment to timelines and not a substitute for the changelog (which tracks what actually shipped). For the **live state** of any specific capability — what's GA today vs preview vs unsupported on a target vendor — your installed AI agent calls `getPlatformInfo` against the MCP server, which returns the authoritative tier marker per primitive. This page covers direction; the agent gets reality. > **At runtime, your installed agent has the live capability state.** Once Extentos is registered with your agent, calling [`getPlatformInfo({ sections: ["capabilities"], glasses: "meta_rayban" })`](/docs/mcp-server/tools/discovery) returns the current capability catalog with per-primitive `tier` markers (`dat_native`, `phone_side_workaround`, etc.) and `confidence` levels (`high`, `degraded`) — that's the source of truth for what's actually shippable today. The roadmap below describes *direction*; the live MCP response describes *current reality*. Always trust the live response when building a real handler; this page exists for evaluation, planning, and search retrieval. ## Status — as of 2026-07 | Tier | Examples | |---|---| | **Currently shipping** | Full Meta Ray-Ban support across all variants (Gen 1 frame styles, Gen 2, Ray-Ban Meta Optics, Oakley Meta HSTN, Oakley Meta Vanguard, Meta Glasses, Display); the MCP tool surface (see the [tools reference](/docs/reference/mcp-tools)); the browser simulator at extentos.com/s; all four transports (SystemAudio / RealMeta / BrowserSim / LocalSim); the [managed AI gateway](/docs/concepts/ai-gateway) with live prepaid billing; the [Phase-4 voice assistant runtime](/docs/concepts/assistant) on both platforms; both native libraries — Android on Maven Central, iOS as a Swift package (lockstep releases); the [on-device model tier](/docs/concepts/local-models) with per-device Automatic selection and on-device neural voices; anonymous-first auth | | **In active development** | **Brilliant Labs** — the BLE transport, on-device Lua bundle, display translation, still capture and microphone are built and ship **inside the SDK on both platforms** (nothing to add on Android, `TransportChoice.brilliant` in `GlassesCore` on iOS), with both devices selectable in the simulator; video is unsupported permanently (no video primitive or codec on either device); no hardware has run it yet and there is no emulator either, so it is preview until a Halo verifies it. iOS on-glasses display delivery (the `glasses.display.*` API and simulator support ship on both platforms; Android renders on hardware today); **Android XR** — the projected transport is built and emulator-proven, the module stays unpublished while Google's platform is in developer preview ([status](/docs/vendors/android-xr)); continuing work on the on-device model tier — new rungs, better quantisations and runtime upgrades as they land; developer analytics dashboard; post-1.0 stability hardening | | **Tracked roadmap (multi-quarter)** | Apple smart-glasses transport implementation as that SDK and hardware mature; Android XR production support, which follows Google shipping hardware; SOC 2 Type II certification | | **Depends on third parties** | Meta DAT capability expansions (richer display media, custom gestures, "Hey Meta" wake word access), Apple smart-glasses SDK availability, additional MCP-host integrations (Gemini in Android Studio, Xcode AI features, JetBrains AI Assistant) | | **Out of scope (intentionally)** | Web app builders, server-side smart-glasses experiences, end-user-facing apps published by Extentos itself | ## Currently shipping What's GA today (install the current `@extentos/mcp-server@latest` — see the [npm package](https://www.npmjs.com/package/@extentos/mcp-server) for the published version): - **Vendor**: Meta Ray-Ban (every variant in market — Gen 1 frame styles, Gen 2, Ray-Ban Meta Optics, Oakley Meta HSTN, Oakley Meta Vanguard, Meta Glasses, and Ray-Ban Meta Display). Camera, voice, audio, sensors, hardware events all work uniformly across variants. - **MCP surface**: deterministic tools grouped by job — discovery, code-example reference, scaffold generation, implementation guidance, validation, simulator orchestration, production, and search. The full catalog with input schemas is the [MCP tools reference](/docs/reference/mcp-tools). - **Transports**: `SystemAudioTransport` (the vendorless baseline — mic + speaker over OS Bluetooth routing, no vendor SDK), `RealMetaTransport` (production hardware), `BrowserSimTransport` (extentos.com/s simulator), `LocalSimTransport` (Extentos's own in-memory sim on phone/emulator — no DAT SDK). All four behind the same `glasses.*` API. - **Managed AI gateway**: the [Phase-4 voice assistant](/docs/concepts/assistant) routes through Extentos's [managed gateway](/docs/concepts/ai-gateway) on Extentos's provider key — zero AI config, content relayed but never stored, usage metered per project. (The assistant always runs on Extentos's provider key; supplying your own is not offered and not planned.) - **Native libraries**: Android `com.extentos:glasses` + `:glasses-ui` (Kotlin), published to Maven Central; iOS (Swift), published as a Swift package from [`github.com/extentos/swift-glasses`](https://github.com/extentos/swift-glasses) — products `GlassesCore` + `GlassesUI`, version-locked to the Android release line. - **On-device model tier**: `com.extentos:glasses-local` + `:glasses-local-voice` (Android) and `GlassesLocal` + `GlassesLocalVoice` (iOS) run the assistant's brain and voice on the phone — a ladder of Qwen open-weights rungs, **Automatic** per-device selection with honest cloud fallback, and eleven on-device neural speakers. No per-token cost, no conversation content leaving the device. See [on-device models](/docs/concepts/local-models). - **Anonymous-first auth**: discovery, guidance, validation, real-hardware testing, and on-device simulation work without an account. A free email-only account (Google or email + password; no payment) links for browser-simulator sessions, the `generateConnectionModule` scaffold step, and the account-scoped project tools. Once linked, simulator sessions are unlimited. The managed gateway is metered and billed today (prepaid credit; every account starts with $2 of free managed-gateway credit) — see [pricing](/docs/resources/pricing). - **Telemetry pipeline + privacy posture**: structured event log, opt-out at three independent surfaces, end-user content never collected. Live state: see [/docs/vendors/meta](/docs/vendors/meta) for the Meta capability matrix and [/docs/concepts/architecture](/docs/concepts/architecture) for the system overview. ## In active development These items have meaningful work in progress as of mid-2026. Specific timelines aren't published — pre-1.0 means surprises happen. ### Display capability — confirming it on Display hardware Meta opened heads-up display content to third parties, and Extentos exposes it as `glasses.display.*` (a UI tree plus full-surface video). The API, the simulator support and the on-glasses delivery path via Meta DAT 0.8.0 now ship **in both SDKs**. What is outstanding is a confirmation run on real Meta Ray-Ban Display hardware — the tree, the translation and the input routing are exercised end to end in simulation and in unit tests on both platforms, and the on-glasses render is the one step nobody has watched yet. Also tracking Meta's display API expansions (richer media, nested video) and extending the capability as they land. See [vendors/meta](/docs/vendors/meta) for the per-variant display status. ### On-device models — a standing program, not a shipped feature The [local tier](/docs/concepts/local-models) is published on both platforms and works today, and it is also the surface with the most headroom left. Open-weights models, the runtimes that execute them, and phone memory budgets are all improving on their own schedules, so keeping the ladder current is ongoing work rather than something that completed at 1.11.0. What that work looks like in practice: adding rungs as models that call tools reliably at small parameter counts appear; re-quantising and re-sourcing existing rungs (moving Automatic's floor rung to 4-bit cut its peak memory from 2250 MB to 1584 MB on a Galaxy A25 with no loss in generation speed, which moved mid-range Android phones off cloud fallback entirely); measuring `requiredMb` on real device classes rather than estimating it; and improving the shared conversation state machine in the Rust core, where turn-taking, barge-in and tool-call handling live for both platforms at once. Two things are open rather than scheduled. **Should Automatic be able to refuse instead of falling back to the cloud** — for apps where zero cost or on-device-only audio is a hard constraint and not a preference? That one is a genuine question we are asking developers, not a decision already made ([the reasoning, and where to answer](/blog/local-models-zero-cost-voice)). And **English-only coverage** across the on-device recogniser and the bundled voices is the tier's largest limitation today; a multilingual product still belongs on the gateway. The point of the seam is that this improves under your app. Model ids keep their meaning, `local-auto` re-resolves at every session start, and core improvements land on iOS and Android in the same release — so a phone that fell back to the cloud last quarter can be running locally after an SDK bump, with no code change. ### Next vendor beyond Meta Ray-Ban A second smart-glasses vendor is the active focus. The architectural seams are already in place — the SDK surface is vendor-agnostic, the `GlassesTransport` interface is the only piece that's vendor-aware — so adding a vendor is a bounded engineering project rather than a rewrite. When it lands, existing Extentos apps targeting Meta Ray-Ban will run on the new vendor with a config change, not a rewrite. See [vendors](/docs/vendors) for the structural picture. ### Developer-facing analytics dashboard Mentioned in `docs/mcp/TELEMETRY.md` as the developer-side cut of the existing telemetry pipeline. Telemetry events are already collected from MVP day-one (`category: "runtime"` events tagged with `appId` + `accountId`); the dashboard is a backend query layer over data that's already being captured. Devs will see their app's voice-trigger usage, AI-call latency, error rates, device-model distribution, and weekly active users at `extentos.com/apps/`. No new library version required — the data is there as soon as the dashboard ships. ### Post-1.0 hardening Several pre-GA items are tracked: - **API stability commitment**: pre-1.0 means minor versions can break APIs. The 1.0 release locks the public library and MCP tool APIs. Until then, pin to an exact MCP version (see the [npm package](https://www.npmjs.com/package/@extentos/mcp-server)) for cross-session reproducibility. - **Telemetry ingest hardening**: the current scheme trusts self-reported `installId` / `appId + anonymousDeviceId`. Pre-GA hardening adds per-request signing or rate-limit-based outlier detection. No wire-protocol breaking change required. ## Tracked roadmap (multi-quarter) Direction without specific timeline commitments. ### Additional vendors Smart-glasses ecosystem expansion is structurally easy for Extentos but bounded by what each vendor ships: - **Android XR** — Google's XR platform with Samsung partner devices. **The transport is built**, not merely tracked: `:glasses-xr` implements Google's projected-activity model behind the same transport seam Meta uses, and it connects end to end on the Android XR emulator. What's missing is a platform out of developer preview and glasses somebody can buy, so the module stays unpublished and nothing is hardware-verified. Simulator sessions already accept an Android XR device identity, so app behaviour under it is testable today ([status](/docs/vendors/android-xr)). - **Brilliant Labs (Halo, Frame)** — an open-source BLE platform, and the inverse of Android XR: the hardware ships and you can buy it, but nothing has run our transport. The wire protocol is verified byte-for-byte against Brilliant's own implementation and the display translation is mutation-tested; it ships on Android and iOS alike, so anyone with a Halo can be the first to try it. See the [vendor page](/docs/vendors/brilliant). - **Apple smart glasses** — pending Apple's release of a third-party developer SDK for whatever smart-glasses hardware Apple ships. Apple has not released such an SDK as of 2026-05; Extentos will add a transport when one becomes available. For both: the SDK surface is vendor-agnostic by design. Adding a vendor means implementing one `GlassesTransport` against that vendor's SDK, and existing apps reach it without handler-code changes. Android XR and Brilliant Labs are the proof rather than the promise — the transport landed without changing a single app-facing API, and Android XR is Android-only by construction (a projected activity *is* an Android activity), so it needs no iOS counterpart. ### SOC 2 Type II certification Tracked as part of the pre-launch / post-launch hardening pass. The architecture is designed to be SOC 2-compatible (audit logging, access controls, vendor management, retention policies) but the formal Type II report is post-GA work. Today's compliance posture: GDPR-friendly by construction (no end-user PII stored), 30-day deletion SLA. See [security](/docs/resources/security#compliance-posture). ## Depends on third parties Some roadmap items aren't Extentos's to ship — they require movement from a third party first. Honest framing on each: ### Meta DAT toolkit expansions | Capability | Status | Extentos's role | |---|---|---| | Heads-up display rendering on Ray-Ban Display | **Shipped** — Meta opened display content to third parties; Extentos exposes it as `glasses.display.*` (a UI tree plus full-surface video) in both SDKs; on-glasses rendering via Meta DAT 0.8.0 ships on both platforms, not yet confirmed on Display hardware | Track Meta's display API expansions (nested video, richer media) and extend `glasses.display.*` as they land | | Custom gestures (multi-finger tap/swipe) on Ray-Ban Meta | Not in DAT preview as of 2026-04 | Will add when Meta opens; new gesture events on `glasses.runtime.events` | | "Hey Meta" wake-word access for third parties | System-level; not exposed to third-party apps | Unlikely to open — wake word is core to Meta AI's product positioning. Extentos's wake-phrase pattern (`glasses.audio.transcriptions()` + handler-side string match) uses phone-side STT, which is the realistic path | | Public Meta App Store submission flow | Preview-limited as of 2026-04, Meta-curated partners only | Will document distribution paths as Meta opens the store | These all show up in `docs/mcp/research/` and `vendors/meta` as items we track but cannot drive directly. Watch [Meta Connect](https://www.meta.com/connect/) (held annually in September) for major announcements; Extentos's public-toolkit support trails Meta's releases. ### Apple smart-glasses developer toolkit Apple has signaled smart-glasses development without releasing a third-party developer SDK or hardware as of 2026-05. Extentos will add an Apple transport when one exists; until then, [Apple in the ecosystem](/docs/ecosystem/platforms/apple) is the neutral platform view (roadmap-only for Extentos). ### MCP host adoption The Model Context Protocol's adoption across IDEs and AI coding agents drives where Extentos works: - **Verified end-to-end**: Claude Code, Cursor, Windsurf, Cline (see [supported agents](/docs/mcp-server/agents)) - **Likely to add support**: Gemini in Android Studio, Xcode AI features, JetBrains AI Assistant, GitHub Copilot. Extentos works with any MCP-compatible host without changes — the question is whether each IDE's built-in agent ships MCP support - **Out of MCP territory by design**: Replit Agent, Lovable, v0.dev, Bolt.new, etc. — these target web apps, not native mobile. See [agents § FAQ](/docs/mcp-server/agents#frequently-asked-questions) ## What's not on the roadmap (intentionally) To save you time waiting for things that aren't coming: - **Web app builder integration**. Extentos targets native mobile apps that connect to smart glasses over Bluetooth. Web apps can't talk to Bluetooth glasses through public toolkits. Adding "Extentos for v0.dev" makes no sense at the artifact layer. - **Server-side / cloud-only smart-glasses experiences**. The architecture assumes a phone app paired with the glasses over Bluetooth. Headless / no-phone setups aren't supported and aren't planned. - **Extentos-published consumer apps**. Extentos is infrastructure for developers, not a consumer app vendor. We won't ship Extentos-branded apps on the Meta App Store or App Store. - **Meta App Store submission as an Extentos service**. Even when Meta opens the public store, Extentos won't submit apps on developers' behalf — that's a developer-relationship boundary we won't cross. - **Multi-seat team workspaces in MVP**. Single-account-per-developer at launch. Multi-seat may come post-1.0 as the developer base grows. > **Note — the managed AI gateway is now shipped, not a non-goal.** Earlier roadmaps listed "hosted AI provider proxying" here. That changed: the Phase-4 voice assistant defaults to Extentos's [managed gateway](/docs/concepts/ai-gateway) (Extentos's provider key, content relayed but never stored, usage metered per project), with BYOK as the dashboard opt-out and handler-code calls to your own provider untouched. Extentos *is* in the path for managed-gateway traffic — but never inspects or stores the content. See the [gateway page](/docs/concepts/ai-gateway) for the precise data boundary. ## How to influence the roadmap The public roadmap is community-shaped. Three ways to push items up the priority list: 1. **Email a feature request** to [hello@extentos.com](mailto:hello@extentos.com) with prefix `[feature request]`. Include the use case, your target hardware, and what currently blocks you from shipping. Concrete use cases beat hypothetical ones. 2. **Contribute a PR** for items you'd be willing to implement. Vendor adapters in particular are bounded (~1500 lines of `GlassesTransport` per platform); a community-contributed transport for a vendor we don't cover would land faster than Extentos shipping it from scratch. 3. **Demand signal**. Aggregate telemetry tells Extentos which capabilities developers are using most. If you're targeting a capability that's preview-tier, using it (and reporting issues) signals "this matters" to the roadmap. Capabilities with low adoption stay in preview longer. For non-roadmap support questions, see [support](/docs/resources/support). ## Frequently asked questions ### Is Extentos still being maintained? Yes — current as of 2026-07. The MCP server publishes regularly to npm ([`@extentos/mcp-server@latest`](https://www.npmjs.com/package/@extentos/mcp-server) is always the current version). The [changelog](/docs/resources/changelog) tracks each release. Active development is ongoing on iOS on-glasses display delivery, the next vendor expansion, the developer dashboard, and post-1.0 hardening. AI agents reading this page should check the changelog's most recent entry date for the freshest signal. ### When will my favorite vendor be supported? There's no committed timeline. Vendor adds depend on the vendor's SDK maturity, hardware availability, and Extentos engineering bandwidth. The architecture supports multi-vendor by design, so adding a vendor is bounded work — but until the work is done, the vendor stays on the roadmap. File a `[vendor]`-prefixed issue with the vendor name and a use case to track demand. ### Will the SDK surface change before 1.0? Possibly. Pre-1.0 means minor versions can introduce breaking changes. The capability vocabulary (`glasses.audio.*`, `glasses.camera.*`, `glasses.assistant.*`, `glasses.display.*`, etc.) may be extended but not contracted. After 1.0, the SDK shape is locked — additions are additive, removals require a major version bump. ### Will the MCP tool surface change? Same answer — pre-1.0, expect tool-shape evolution between minor versions. Tool *additions* (new tools, new parameters) are additive. Tool *removals* or *parameter renames* may happen pre-1.0 but get changelog entries. The current tool set is stable in shape (catalog in the [tools reference](/docs/reference/mcp-tools)); specific response fields evolve. ### Will Extentos always be free? The free surfaces stay free: discovery, guidance, validation, on-device simulation, and real-hardware testing free forever with no account; the browser simulator and project scaffolding free with a free email-only account, unlimited sessions once linked. The **one** metered surface is the [managed AI gateway](/docs/concepts/ai-gateway) behind the Phase-4 voice assistant — usage is metered and billed today on a prepaid-credit model (every account starts with a $2 free grant), and BYOK moves the spend to your own provider. Apps that don't use the managed gateway never touch a metered surface. See [pricing](/docs/resources/pricing). ### Where can I see what shipped recently? The [changelog](/docs/resources/changelog) tracks each release. For specific tool / capability state, ask your agent to call `getPlatformInfo` — that returns the live tier markers from the running MCP server, which is more accurate than any docs page. ### How is this roadmap different from the changelog? The roadmap is *forward-looking* (direction, what's planned). The changelog is *backward-looking* (what shipped, with dates). They complement each other — the roadmap tells you where Extentos is going; the changelog tells you what's already arrived. ### Who decides what goes on the roadmap? Pre-1.0, roadmap priorities are set by the maintainer based on (a) demand signal from issues and aggregate telemetry, (b) blocking dependencies (a new vendor's SDK becoming available, etc.), and (c) post-1.0 hardening priorities. There's no formal RFC process at MVP scale — that may change as the project grows. ## Related - **[Vendors](/docs/vendors)** — the multi-vendor strategic story; per-vendor capability matrices - **[Pricing](/docs/resources/pricing)** — current free model and the paid-tier framing - **[Security](/docs/resources/security)** — compliance posture, SOC 2 trajectory, data-handling commitments - **[Changelog](/docs/resources/changelog)** — what's actually shipped, dated - **[Support](/docs/resources/support)** — file feature requests, get help, see source --- source: https://extentos.com/docs/resources/security --- --- title: Security and data handling description: What Extentos collects (aggregate dev + runtime metadata), what it never collects (transcripts, AI prompts, PII), and the one path that puts end-user media on our infrastructure — display hosting. Anonymous-first, GDPR-friendly. type: resource platform: all related: - /docs/mcp-server/auth - /docs/resources/pricing - /docs/concepts/architecture - /docs/getting-started/with-agent - /docs/resources/changelog --- Extentos collects **aggregate metadata** about how developers use the platform — which capabilities are popular, which vendors get adopted, how flows perform, where errors surface. This data powers three things: the developer-facing analytics dashboard (coming soon — you'll see your own app's usage in one view), the Extentos team's product roadmap (which vendors to add next, which capabilities to harden), and the ecosystem insights Extentos shares with glasses vendors as aggregate market data (10+ account minimum). What Extentos **never collects as telemetry** is end-user content (speech transcripts, photo or video bytes, audio samples, AI prompts and responses), end-user PII (names, emails, phone numbers), toggle values, or stack traces. **Captured media runs locally.** Photos, video, and sensor frames flow from the glasses to your app on the user's phone and never touch an Extentos server — the production transport is Bluetooth, and we are not in the path. There is exactly **one** exception, and it is not telemetry: rendering a *captured* photo or clip **on a Ray-Ban Display**. The glasses can only load media from an `http(s)` URL they fetch themselves, so rather than make every developer stand up storage, we host a copy for you. It's a convenience you can decline — host the media yourself and pass a URL, and nothing is uploaded. If you do use it, read [Hosted display media](#hosted-display-media--the-one-content-path); it changes what your own privacy policy has to say. This page is the honest, complete picture of what crosses the network from your install and your shipped apps to Extentos's backend. ## TL;DR | | What | |---|---| | **What we collect** | Aggregate metadata — tool calls, install events, vendor adoption, capability usage, error rates, library/MCP versions, device models, country-from-IP, anonymous device identifiers, plus managed-gateway AI usage (token counts + cost, no content). Tagged with anonymous IDs that can never be reversed back to a person. | | **What we never collect** | Speech transcripts, photo bytes, video frames, audio samples, AI prompt and response text, handler-code input/output payloads, toggle values, IP addresses (dropped after country derivation), stack traces, end-user PII. The managed AI gateway *relays* assistant audio and text but never stores it (see [AI calls](#ai-calls-the-managed-gateway-and-byok)). | | **The one content exception** | Rendering a captured photo or clip **on a Ray-Ban Display** uploads a copy to Extentos's media host, because the glasses fetch media only from an `http(s)` URL. It persists until your app releases it. Capture never uploads; no other capability does. See [Hosted display media](#hosted-display-media--the-one-content-path). | | **What end users see** | Nothing about Extentos. End users have no relationship with us. The production transport (`RealMetaTransport`) never talks to our backend — it speaks only to Meta DAT and the glasses over Bluetooth. A shipped app has up to three network paths to us: runtime telemetry (metadata only, on by default, `telemetryConsent = false` disables it), the managed AI gateway if you use the assistant (relayed, never stored), and the display media host if you render captured media (see above). | | **Opt-out** | Three independent surfaces — `EXTENTOS_TELEMETRY=0` env var, `extentos-mcp decline-privacy` CLI, or `ExtentosConfig.telemetryConsent = false` in your shipped app. Any one disables telemetry upload; on-device debug logs still work. | | **Compliance** | GDPR-friendly by construction — no end-user PII ever stored. Aggregate metadata retained indefinitely under Art. 6(1)(f) legitimate-interest basis to power both the developer dashboard and (multi-vendor-dependent) the vendor data product; right-to-erasure honored within 30 days regardless of retention policy. CCPA: planned cross-vendor data product qualifies as a "sale"; developer-level Do-Not-Sell opt-out is `ExtentosConfig.dataSharingConsent = false` (a shipped SDK field — the vendor aggregate product it gates is still forthcoming). SOC 2 Type II on the post-launch roadmap. | ## Why we collect telemetry — the honest answer Extentos's business model includes monetizing **aggregate ecosystem insights**, and we're upfront about it. Three concrete uses: 1. **Your own analytics dashboard.** When the developer dashboard ships, you'll see your app's usage at `extentos.com/apps/` — which voice triggers your users fire, which AI calls are slowest, which device models your users have, error rates, session lengths, weekly active users. Same data Extentos collects centrally, scoped to your `accountId`. This is genuinely valuable to developers and is why the underlying collection is broad rather than narrow. 2. **Extentos's product roadmap.** Aggregate signal across all developers tells us which vendors to add next (if 30% of new MCP installs ask about a vendor we don't support, we prioritize that transport), which capabilities are over-used relative to expectations, where the SDK has rough edges. Without this, every roadmap call is a guess. 3. **Cross-vendor ecosystem signals to glasses vendors (planned, multi-vendor only).** Once Extentos supports ≥2 vendors with material installed bases, aggregate **comparative** market data — capability mix across the ecosystem, vendor market share, session-hour totals across vendors — becomes uniquely valuable to glasses vendors who only see their own SDK telemetry. We plan to sell this as market research, only as aggregates over **at least 10 distinct accounts**, never as individual-developer or individual-end-user data. This is a planned future revenue line that activates at multi-vendor scale; a single-vendor version isn't a meaningful product because each vendor already collects their own platform's usage data. We chose a transparent collect-and-aggregate model rather than a "we collect nothing" claim that vendors quietly walk back. The discipline is in the firm content/PII boundary, not in collecting less data overall. ## What we collect All telemetry events conform to a single envelope: ```json { "eventId": "tev_a1b2c3d4", "timestamp": "2026-04-18T14:22:10.432Z", "category": "runtime", "name": "connection.opened", "accountId": "acct_xyz", "installId": "inst_abc123", "appId": "com.example.app", "anonymousDeviceId": "dev_789", "libVersion": "2.8.0", "mcpVersion": null, "vendor": "meta_rayban", "platform": "android", "osVersion": "14", "deviceModel": "pixel_8", "country": "US", "userSegment": null, "properties": { "transportType": "RealMeta", "timeToConnectMs": 1820 } } ``` The events split into two categories: ### Dev-time events (MCP server, `category: "mcp"`) Emitted from your machine while your AI agent operates Extentos — so they're tagged with your `installId` (anonymous) and, after linking, your `accountId`. | Event | What it carries | |---|---| | `mcp.installed` | First run on this machine. npm version, OS, detected agent host. | | `mcp.consent_accepted` / `mcp.consent_declined` | Privacy policy version. | | `tool.called` | Every MCP tool invocation. `toolName`, `durationMs`, `outcome`. | | `session.created` | `createSimulatorSession` invoked. Vendor, recording flag. | | `account.login_initiated` / `_completed` / `linked` / `logged_out` | Auth flow lifecycle. Trigger reason (cli_explicit, account_required), outcome, duration. | ### Runtime events (your shipped app, `category: "runtime"`) Emitted by the Extentos library *if* the developer enables runtime telemetry (`ExtentosConfig.telemetryConsent = true`, default). Tagged with `appId` and `anonymousDeviceId` — never with end-user identity. | Event | What it carries | |---|---| | `app.initialized` | Library version, vendor, deviceModel, osVersion. | | `connection.opened` / `closed` | Transport type (`SystemAudio`/`RealMeta`/`BrowserSim`/`LocalSim`), `timeToConnectMs` / `durationMs`, disconnect `cause` (typed enum). | | `stream.started` / `stopped` | `streamType` (video_frames, audio_chunks, transcription_incremental, etc.), `durationMs`. Safe subset of negotiated config (resolution, frame rate, codec, bitrate). **No peer addresses, URIs, destination identifiers.** | | `toggle.changed` | `key` (which of the 8 runtime toggles flipped), `source` (`ui` / `voice` / `automation`). **No old/new values** — the values can reveal product behavior. | Notice the bold disclaimers — every event explicitly carries metadata only, never content. **Retired (deprecated) runtime events:** pre-pivot the library also emitted `trigger.fired`, `flow.completed`, `block.executed`, `callback.invoked`, and `spec.loaded` to describe spec-DSL execution. Those events were retired with the pure-SDK pivot — the library no longer interprets a spec, so the events that described its execution no longer fire. Their schemas remain in the event registry marked `deprecated: true` for historical event-log validation; no new code emits them. ## What we never collect Verbatim list from the canonical telemetry doc (this list is published in the privacy policy). It describes the **telemetry pipeline** — the events your shipped app uploads to `api.extentos.com/api/telemetry/events`. Two features sit outside the telemetry pipeline and have their own data stories: the [managed AI gateway](#ai-calls-the-managed-gateway-and-your-own-calls) (relays, never stores) and [hosted display media](#hosted-display-media--the-one-content-path) (stores, until you release it). - Speech transcripts (the words your user said) - Photo bytes, video frames, audio samples — *as telemetry*. The display media host is the documented exception; see below. - Handler-code input/output payloads (the LLM prompts your handler builds, the responses it receives, anything your business logic passes around between SDK calls) - AI provider responses, prompt text, tool-use traces - Toggle values, variable values - End-user PII of any kind (names, emails, phone numbers, account identifiers in your app) - IP addresses (derived to country at ingest, then dropped — we never store the raw IP) - Stack traces (may contain arbitrary values; stay local in your on-device event log) - Anything tied to a specific end-user identity beyond the per-device anonymous `anonymousDeviceId` The library architecture enforces this — events are payload-stripped *before* they enter the upload queue, not in transit. No end-user content can reach the telemetry pipeline even if the network were intercepted, because the content never enters the queue in the first place. That guarantee is about telemetry. It is not a claim that no end-user content ever reaches Extentos by any route — the gateway and the display media host are two routes that do, both opt-in by the capability you choose to use, both documented below. ## Hosted display media — the one content path If your app renders a **captured** photo or clip on a Ray-Ban Display, the SDK uploads a copy to Extentos before the glasses can show it. This is the only feature in the SDK that puts end-user media at rest on our infrastructure, and it exists because of a hardware constraint, not a product preference: the display fetches media only from an `http(s)` URL it can reach over its own Wi-Fi. A `data:` or `file:` URL is rejected on hardware, and the phone does not proxy the bytes. So a locally captured asset cannot reach the lens without becoming a fetchable URL first. Rather than make every developer stand up hosting, the SDK does it. **Exactly what happens.** | | | |---|---| | **Trigger** | `display.show { image(photo) }` or `{ video(clip) }` with a **local** capture. Passing `image(url)` / `video(url)` uploads nothing. | | **Not triggered by** | Capture itself. `capturePhoto()`, `captureVideo()`, frame streams and audio never upload. Media only leaves the device if you display it. | | **Where it goes** | An Extentos-operated storage bucket, reached at an unguessable random UUID path. | | **Who can read it** | Anyone with the exact URL — the glasses fetch it unauthenticated, so it cannot be access-gated. The URL is not enumerable and is never published, but treat it as a secret-by-obscurity link, not a private object. | | **How long** | Until your app deletes it. There is **no expiry** — deliberately, so an asset can't vanish mid-life while a user is looking at it. | | **How to delete it** | `forgetHostedImage(photo)` / `forgetHostedVideo(clip)`. Build the `Photo` / `VideoClip` the same way you built it for `show`, so both resolve to the same hosted copy. | | **Scope today** | Android only. The iOS display client has no local-media hosting path yet — on iOS you pass an `http(s)` URL you host. | | **Tagging** | The upload carries the app's `anonymousDeviceId` for attribution. No end-user identity is attached. | **What this means for you as the developer.** Two obligations, both small: 1. **Wire `forgetHosted*` into your delete path.** If a user deletes a photo in your app and you don't call it, the hosted copy stays. That is the single most likely way this becomes a real privacy problem, and it is entirely in your code. 2. **Disclose it in your own privacy policy** if you use the display with captured media. Your end user's relationship is with your app, not with us; "media is uploaded to our SDK vendor's host to render on the glasses, and deleted when you delete the item" is the honest line. **How to avoid it entirely.** Host the media yourself and pass `image(url)` / `video(url)`. The SDK uploads nothing on that path. If your app never uses the display, or only renders text and layout nodes, this section doesn't apply to you at all. A developer-supplied storage option (bring your own bucket) is on the roadmap so this stops being a choice between our host and hand-rolling your own. ## End-user privacy — your shipped apps The clearest line: **on real Ray-Ban Meta glasses, the production transport never talks to Extentos.** - `RealMetaTransport` (the production transport) speaks to Meta DAT and the glasses over Bluetooth. It does not connect to `api.extentos.com`. End users on real hardware have no network relationship with Extentos at all. - `BrowserSimTransport` (the simulator transport) does talk to the Extentos backend over WebSocket — but `BrowserSimTransport` only runs in `config.debug == true` builds during development. Shipped release builds resolve to `RealMetaTransport` when a vendor module claims bonded glasses, otherwise `SystemAudioTransport`, which doesn't connect. - `LocalSimTransport` (in-memory simulation — no DAT SDK) is fully on-device. No network at all. Runtime library telemetry is **on by default** (`ExtentosConfig.telemetryConsent = true`): the library emits the runtime events above to `https://api.extentos.com/api/telemetry/events`. It carries metadata, never content. The developer can turn it off per-app via `telemetryConsent = false`. Telemetry is the only network path that is on by *default*. Two more open only if you use the capability behind them: | Path | When it opens | What crosses | Turn it off by | |---|---|---|---| | Runtime telemetry | On by default | Metadata only, never content | `telemetryConsent = false` | | [Managed AI gateway](#ai-calls-the-managed-gateway-and-your-own-calls) | You use the Phase 4 assistant runtime | Audio, transcripts, responses — relayed verbatim, never stored | Don't use the managed assistant; call your own provider from handler code | | [Display media host](#hosted-display-media--the-one-content-path) | You render a *captured* photo/clip on a Ray-Ban Display | The media file itself, stored until released | Pass `image(url)` / `video(url)` with your own hosting | A shipped app that leaves telemetry on, uses the managed assistant, and renders captured media on the display touches all three. An app that does none of those emits nothing to us at runtime. End users never see Extentos branding, never get prompted by us, never need to consent to anything from us. Their privacy relationship is entirely with the developer's app — Extentos is invisible. ## The 5 ID model Five identifiers, each with a distinct scope. **None of them is end-user PII**, and Extentos cannot reverse any ID back to a person. | ID | Scope | When created | Where stored | |---|---|---|---| | `installId` | Per-machine MCP install | First `npx @extentos/mcp-server` run | `~/.extentos/install.json` | | `accountId` | Per-Extentos-account | When developer signs up via device-code flow | Backend; cached in `~/.extentos/auth.json` | | `appId` | Per-Extentos-app | Set by developer in `ExtentosConfig.appId` at library init | Compiled into the app | | `anonymousDeviceId` | Per-end-user-device | First library init on the user's device | App-private storage on the user's device | | `userSegment` | Developer-defined cohort label | Optional; `glasses.telemetry.setUserSegment("...")` | App memory; flushes with events | The `anonymousDeviceId` lets Extentos count distinct devices and sessions without ever learning end-user identity. The `userSegment` is set by the developer — Extentos never sets or interprets it. It exists so the developer can cut their own dashboard by cohorts they define (beta tiers, geography, app version) without exposing any personal information. ## AI calls: the managed gateway, and your own calls Extentos touches AI two different ways, with two different data stories. The full mechanics are on [the managed AI gateway](/docs/concepts/ai-gateway) page; here's the privacy-relevant summary. **Managed gateway (the Phase 4 assistant runtime, default).** Voice-AI traffic routes through Extentos's gateway on Extentos's provider key. Extentos **is in the path** — but as a transient relay: audio, transcripts, and model responses are forwarded verbatim and **never stored**. The only thing recorded is the provider's per-response **usage** object — token counts (input/output, text/audio, cached) plus the computed cost — tagged with your `appId` and `accountId` for per-project metering. **No prompt, transcript, photo, or response content is stored.** The content boundary in the "what we never collect" list holds for the gateway: relayed, never inspected, never persisted. **Your own AI calls.** Your handler calls a *non-assistant* provider (a vision model on a captured photo, a translation API) directly, with a key you wire into your own code. **Extentos has no hook into that path** — it never sees the key, the request, or the response. The `getCredentialGuide` MCP tool generates setup instructions for you to follow with your provider; Extentos never receives the credentials. So "Extentos never proxies your AI" is true for **AI you call yourself from handler code** — and that was the only model the earlier docs described. For the **managed gateway** it isn't: Extentos relays the call, but never stores its content. This page states both honestly. ## Three opt-out surfaces Telemetry is opt-out at three independent levels, each disabling a different layer: | Surface | Where | What it disables | |---|---|---| | `EXTENTOS_TELEMETRY=0` env var | Developer's shell | MCP-server-side telemetry for the current shell session. Doesn't persist across reboots. | | `extentos-mcp decline-privacy` CLI command | Writes `~/.extentos/consent` | MCP-server-side telemetry permanently for this install. Survives reboots. Run `accept-privacy` to re-enable. | | `ExtentosConfig.telemetryConsent = false` | Developer's app code | Library-runtime telemetry from this app. Per-app — a developer with multiple apps can opt one out and leave others on. | Any combination works. Telemetry never affects functionality — opting out doesn't disable any feature, just stops uploads. The on-device event log (the 512-entry ring buffer your agent queries via `getEventLog`) keeps working regardless of telemetry consent — in production that data never crosses the network (in simulator mode it is mirrored to the backend for agent debugging). See [event log vs telemetry boundary](/docs/concepts/architecture#state-and-persistence--what-lives-where). ## Retention and deletion | Data class | Retention | |---|---| | Raw telemetry events | **Indefinite.** Historical trend data is the asset behind both the developer dashboard and the (multi-vendor-dependent) vendor data product — cutting retention kills year-over-year comparisons. Lawful basis under GDPR is Art. 6(1)(f) legitimate interest, with a documented Legitimate Interest Assessment (aggregate metadata, no end-user PII, clearly disclosed purposes, full right-to-erasure preserved). | | Aggregated rollups (daily / weekly / monthly per account, plus global) | Indefinite | | `anonymousDeviceId` records | Retained while the device sends events. No automatic expiration on inactivity — that destroys cohort retention analysis. | | Backend session event log (dev-time simulator only) | **48 hours** after each event is recorded, then purged automatically. | | [Hosted display media](#hosted-display-media--the-one-content-path) | **Until the app deletes it** via `forgetHostedImage` / `forgetHostedVideo`. No time-based expiry by design — an asset disappearing while a user is viewing it is worse than one that persists. This is the one retention window that is controlled by *your* code rather than by us; wire it to your delete path. | **Right-to-erasure (GDPR Art. 17 / CCPA right to delete).** The indefinite-retention default does not block right-to-erasure. Developers can email support to delete their account's events; Extentos runs `DELETE WHERE accountId = ...` and acknowledges within **30 days** (GDPR-compatible SLA). Note that this covers telemetry events — **hosted display media is not reachable by an account-scoped delete**, because uploads are not account-attributed. Delete those from your app with `forgetHostedImage` / `forgetHostedVideo`; if you need a bulk purge, contact support. This deletes raw events tied to the account; aggregated rollups derived before the deletion request stay (those have no `accountId` attribution by construction). Anonymous events without an `accountId` (emitted before account linking and never retroactively re-tagged) cannot be associated to a person and are kept as aggregate-only data. **CCPA "sale" classification.** The planned cross-vendor data product qualifies as a "sale" under CCPA when it ships. The developer-level Do-Not-Sell mechanism is `ExtentosConfig.dataSharingConsent = false` (a shipped SDK field) — disables aggregate inclusion only, while the developer's own analytics dashboard keeps working. The current `telemetryConsent = false` is all-or-nothing. ## Transport and auth security - **TLS everywhere.** All Extentos backend connections (`api.extentos.com` for telemetry and simulator-session WebSockets) use TLS 1.2+. No plaintext channels. - **Auth tokens at file mode `0600`.** When you link an account, `~/.extentos/auth.json` is written with owner read/write only. Other users on the machine can't read your token. - **Identity on telemetry calls.** Telemetry uploads carry the `installId` (MCP-side) or `appId + anonymousDeviceId` (library-side). When the developer has linked an account, the MCP server also attaches its account bearer so events attribute to the account — no *end-user* credentials are ever sent. - **Local bridge bound to `127.0.0.1` only.** The MCP server's auto-bind listener (port `31337`) refuses non-loopback connections. The `installId` it serves is not a secret — it's the same value sent to `api.extentos.com` on every tool call — so loopback exposure is intentional. - **Pre-GA hardening note.** The current telemetry ingest scheme trusts self-reported IDs (per `docs/mcp/TELEMETRY.md`). For MVP this is acceptable — spoofed events skew aggregate fleet data, not security-sensitive. Before external GA, this hardens to one of: ingest-signed install tokens, per-request HMAC keyed on a secret baked at library init, or rate-limit-based outlier detection. ## How to verify what we collect You don't have to take this page's word for it — **inspect the wire.** Proxy your install's and your app's traffic to `api.extentos.com` and you'll see exactly what's sent: aggregate metadata, never content or PII. The `@extentos/mcp-server` package is **MIT-licensed** on [npm](https://www.npmjs.com/package/@extentos/mcp-server), so its telemetry pipeline (`src/telemetry/`) is open to read directly; the native libraries are proprietary, but everything they send crosses that same observable network boundary. Documentation can drift; the bytes leaving your app can't. Watch the wire and hold us to this page. ## Compliance posture | Standard | Status | |---|---| | GDPR | Compatible. Lawful basis for analytics is Art. 6(1)(f) legitimate interest, documented in our LIA. No end-user PII stored. Right-to-erasure honored within 30 days regardless of retention policy. | | CCPA | Compatible with explicit "sale" disclosure. The planned cross-vendor data product qualifies as a "sale" under CCPA when it ships. Developer-level Do-Not-Sell mechanism is `ExtentosConfig.dataSharingConsent = false` (a shipped SDK field) — separate from telemetry collection. Right to know honored at request. | | HIPAA | Not certified at MVP. Production runtime emits no Extentos traffic from `RealMetaTransport`, so a HIPAA-bound app shipping with telemetry-off is technically usable; a formal BAA isn't available pre-GA | | SOC 2 | Type II on the post-launch roadmap. The architecture is designed SOC 2-compatible (audit logging, access controls, vendor management, retention policies). Once the 12-month observation period kicks off, the report becomes available under NDA to vendor data-product customers. | | ISO 27001 | Not pursued at MVP | For compliance-sensitive apps today: ship with `ExtentosConfig.telemetryConsent = false` to disable all library telemetry. Combined with the fact that `RealMetaTransport` doesn't talk to Extentos's backend, your shipped app emits zero traffic to us at runtime — **provided you also skip the two opt-in paths**: use your own AI provider from handler code rather than the managed assistant, and pass pre-hosted `image(url)` / `video(url)` rather than local captures if you use the display. With those three settings, the privacy boundary is enforced architecturally, not just by policy. For developers who want telemetry on (for the developer dashboard) but want to opt their app's data out of vendor aggregates: set `ExtentosConfig.dataSharingConsent = false`. Your own dashboard analytics keep working; your data is excluded from cross-account aggregates we may sell to glasses vendors once the multi-vendor product ships. (The `dataSharingConsent` field is available in the SDK; the cross-vendor product it would exclude you from is still forthcoming, so today it changes nothing in practice.) ## Frequently asked questions ### Wait — Extentos collects telemetry from my shipped app? By default, yes — runtime metadata events (no content, no PII) flow to `api.extentos.com/api/telemetry/events` from your shipped app *if* `ExtentosConfig.telemetryConsent = true` (the default). Set it to `false` and your app emits zero Extentos network traffic. Many apps will leave it on because the developer dashboard is built on this data — you're trading runtime metadata for an analytics surface for your own app. ### Will my users' voice transcripts ever leave their device? It depends which AI path you use. Transcripts your own handler processes (via `glasses.audio.transcriptions()`) are never read, logged, or transmitted by Extentos. If you use the **managed-gateway assistant**, the audio and transcripts are relayed through Extentos's gateway to the AI provider — forwarded verbatim and **never stored**; only token-usage metadata is recorded. If your handler calls your own provider directly (handler-code BYOK), that's between your app and your provider — Extentos isn't in the path. See [the gateway data path](/docs/concepts/ai-gateway#what-the-gateway-does-and-doesnt-do-with-your-data). ### Can Extentos see what my user captured in a photo? No — Extentos never inspects photo content. Capture bytes flow from the glasses to your handler and Extentos doesn't read, store, or upload them. But "can Extentos *see* it" and "does a copy ever sit on Extentos infrastructure" are different questions, and the honest answer to the second is: once, in one case. If you render a captured photo or clip **on a Ray-Ban Display**, the SDK uploads a copy to our media host so the glasses can fetch it — it stays until your app calls `forgetHostedImage` / `forgetHostedVideo`, and nobody at Extentos looks at it, but it is stored. [Full details, including how to avoid it.](#hosted-display-media--the-one-content-path) Two smaller nuances: if you pass an image to the **managed-gateway assistant** for vision, it's relayed to the AI provider (never stored by us); and during *development*, simulator sessions mirror captured frames to the backend debug log, truncated, for 48 hours. Outside those paths, photo bytes only reach an external service if your own handler code sends them. ### Will Extentos sell data about my app to my competitors? When Extentos supports multiple vendors, we plan to sell *aggregate cross-vendor ecosystem signals* (vendor market share, capability popularity across platforms, session-hour totals across the ecosystem) to glasses vendors as market research, only as aggregates over at least 10 distinct accounts. We additionally suppress vendor-level data when fewer than 25 developers ship to that vendor — small-vendor ecosystems are protected from inadvertent disclosure. Individual-account data, your specific app's usage patterns, and anything that could identify your app or your users is never shared with third parties. These two floors are the structural guarantees against competitive leakage. This product is multi-vendor-dependent — at single-vendor scale today, no such sale takes place because each vendor already has their own platform's data via their SDK. Developers can also opt their app out of inclusion in future vendor aggregates via `ExtentosConfig.dataSharingConsent = false` (a shipped SDK field) — separate from disabling telemetry collection. Your own developer-dashboard analytics keep working; your data is excluded from cross-account aggregates. ### Can my users opt out of Extentos telemetry? End users have no direct relationship with Extentos and don't see our branding. Their opt-out is via the developer's app — if you want to give your users a toggle, gate `ExtentosConfig.telemetryConsent` behind a setting in your app. Or simply ship with telemetry off if your privacy posture requires it. ### Is my Extentos auth token shared across machines? No. `~/.extentos/auth.json` is per-machine. To use the same account on another machine, run `extentos-mcp login` there. The token is stored at file mode `0600` (owner read/write only). ### Does Extentos work in air-gapped or no-network environments? The MCP-driven dev loop (code generation, validation, on-device `LocalSimTransport` simulation) works fully offline. The browser simulator (`BrowserSimTransport`) and the simulator-session features need network access to `api.extentos.com`. Your shipped app on real glasses (`RealMetaTransport`) doesn't need network access to Extentos — only between the phone and the glasses, plus whatever AI providers your app calls. ### When does the developer dashboard ship? It's on the roadmap; the data pipeline is already collecting events from MVP day-one so the dashboard is purely a backend query layer over data that already exists. Watch the [changelog](/docs/resources/changelog) for actual ship dates. ### What about developer source code? Your generated code, your spec, your handler implementations, your test fixtures — none of it ever leaves your machine. The MCP server reads your project to inspect state (`inspectIntegration`, `validateIntegration`) and writes generated files, but it does not upload your code, contents, or git history anywhere. ### How do I request deletion of my data under GDPR? Email Extentos support from the email associated with your account. We run `DELETE WHERE accountId = ...` and acknowledge within 30 days. The indefinite-retention default does not block right-to-erasure — deletion requests are honored independent of retention policy. Aggregated rollups derived before the deletion request stay (those have no `accountId` attribution by construction); raw events tied to your account are removed. Anonymous events without an `accountId` (emitted before account linking) cannot be associated to you and are retained as aggregate-only data. ## Related - **[Pricing](/docs/resources/pricing)** — what the free tier covers, the account-required browser simulator - **[Auth](/docs/mcp-server/auth)** — anonymous-first identity model, device-code flow, account linking - **[Architecture](/docs/concepts/architecture)** — how telemetry fits into the broader system, what the on-device event log captures - **[Changelog](/docs/resources/changelog)** — track when the developer dashboard and other features ship --- source: https://extentos.com/docs/resources/status --- --- title: Status description: Service status for Extentos. There is no public status page or uptime instrumentation yet — Extentos is pre-1.0. The backend that powers the browser simulator and the managed AI gateway runs at api.extentos.com. For what shipped, see the changelog; to report an outage or incident, email hello@extentos.com with the [incident] prefix. type: resource platform: all related: - /docs/resources/changelog - /docs/resources/support - /docs/concepts/ai-gateway --- **Honest status: there is no public status page or uptime dashboard yet.** Extentos is pre-1.0, and formal uptime instrumentation (a hosted status page, historical SLAs) hasn't been stood up. This page will link a dedicated status page once one exists. ## What runs where - **Backend** — `api.extentos.com`. This is the one piece of Extentos infrastructure with runtime dependencies: it powers the [browser simulator](/docs/concepts/transport-vs-app) session hub (WebSocket, voice STT/TTS proxy, event-log persistence) and the [managed AI gateway](/docs/concepts/ai-gateway) behind the Phase-4 voice assistant. - **What does *not* depend on Extentos uptime** — the MCP server runs on your machine (or via `npx`), on-device simulation (`LocalSimTransport`) never reaches a backend, and production apps on real Ray-Ban Meta hardware (`RealMetaTransport`) talk to the glasses over Bluetooth, not to Extentos. An Extentos backend outage does **not** take down a shipped app unless that app uses the managed gateway. ## Reporting an incident If something on the Extentos backend looks down (simulator sessions won't mint, the gateway returns errors), email [hello@extentos.com](mailto:hello@extentos.com) with the `[incident]` prefix. Include the timestamp, what failed (session mint / gateway / event log), and any error surfaced by your agent or the simulator. There's no SLA at launch — see [support](/docs/resources/support#response-expectations) for honest response expectations. ## Where to look instead - **[Changelog](/docs/resources/changelog)** — what shipped and when (the closest thing to a "what changed" feed today). - **[Support](/docs/resources/support)** — diagnostics, bug reports, and the security disclosure path. ## Related - **[Changelog](/docs/resources/changelog)** — release history - **[Support](/docs/resources/support)** — file issues, self-diagnose - **[The managed AI gateway](/docs/concepts/ai-gateway)** — the backend-dependent assistant path --- source: https://extentos.com/docs/resources/support --- --- title: Support description: How to get help with Extentos — which GitHub issue tracker owns which component (android-glasses, swift-glasses, mcp-server), self-service diagnostics through the extentos-mcp CLI (whoami, status, getEventLog), what to include in a bug report so it's actionable across the four components (MCP server, native libraries, backend, simulator UI), the private email path for security and billing, and pre-1.0 response expectations. No paid support tiers exist at launch. type: resource platform: all related: - /docs/resources/faq - /docs/resources/security - /docs/resources/changelog - /docs/mcp-server/auth --- Extentos is pre-1.0. Bugs and feature requests go in the public GitHub issue trackers, anything private goes to email, and there's no paid support tier and no SLA. What there is: a structured-event-log debugging surface that makes most issues self-diagnosable from your own machine, and a maintainer who reads every report. This page is where to start when something goes wrong. ## Where to send it **Bugs, feature requests, and documentation errors go in the GitHub issue tracker** for the component you hit. Public issues are searchable, so the next developer to hit the same thing finds your report instead of filing it again. | What broke | Where to file | |---|---| | Android SDK (`com.extentos:glasses`) | [extentos/android-glasses](https://github.com/extentos/android-glasses/issues) | | iOS SDK (the `Extentos` Swift package) | [extentos/swift-glasses](https://github.com/extentos/swift-glasses/issues) | | MCP server, agent scaffolding, browser simulator, these docs | [extentos/mcp-server](https://github.com/extentos/mcp-server/issues) | Not sure which one owns it? File on any of the three and it gets moved. The component triage table below is the rule of thumb if you'd rather route it yourself. The bug template asks for the same six things listed in [what to include](#what-to-include-in-a-bug-report). **Some things don't belong in a public tracker.** For those, use the **[contact form](/contact)**, which lands in the inbox already tagged and with your reply address attached, or email **[hello@extentos.com](mailto:hello@extentos.com)** with a subject prefix: | Issue type | Subject prefix | |---|---| | Security issues | **Use the private path**, see [security disclosure](#security-disclosure) below | | Billing or account | `[billing]` | | Sales or partnership | `[sales]` | | Vendor support requests (any smart-glasses maker) | `[vendor]`, include the public SDK link | | Anything you'd rather not raise in public | `[question]`, and see the [FAQ](/docs/resources/faq) for common answers | No GitHub account, and no wish to make one? Email the bug with the `[bug]` prefix and the checklist below, and it gets filed for you. ## Self-service before filing Most issues are diagnosable from the developer's machine without escalation. Before opening an issue, run through this: ### 1. Capture install state with `whoami` ```bash npx @extentos/mcp-server@latest whoami ``` Prints the install ID, account ID (if linked), tier, meter remaining, auth expiry, and MCP / library versions. **Always include this output in a bug report** — it's the single line that tells maintainers what your install actually looks like. ### 2. Check consent and config state with `status` ```bash npx @extentos/mcp-server@latest status ``` Prints consent state (privacy notice accepted / declined), telemetry on/off, install ID, linked account, and effective config dir (`~/.extentos/` or whatever `EXTENTOS_CONFIG_DIR` points at). Useful when an issue might be config-related. ### 3. Ask the agent to query the structured event log If the bug is "something happened at runtime," the structured event log is the primary debugging surface. Tell your AI agent: ```text Run getEventLog with filter "errors" and limit 50 on session ``` The agent calls the MCP tool; the response is a structured trace of what happened in the simulator session — every capability call (record, capture, speak, …) emits a request + result event with a stable id, and any failure routes to the `errors` chip. Filter by chip (`errors` / `voice` / `camera` / `display` / `ai` / `lifecycle` / `custom`) to narrow to one modality. ### 4. Run `validateIntegration` for integration issues If the symptom is "agent says everything is fine but the app doesn't run," ask your agent: ```text Run validateIntegration on this project ``` `validateIntegration` checks the generated connection module against the manifest, the dependency list, handler wiring, required permissions, and bootstrap setup. It catches most class-of-issue mismatches before they become runtime bugs. ### 5. Reproduce against the published packages If a behavior contradicts what's documented, pin the exact published versions and capture a minimal repro: - MCP server — [`@extentos/mcp-server`](https://www.npmjs.com/package/@extentos/mcp-server) (npm, MIT) - Android library — [`com.extentos:glasses`](https://central.sonatype.com/artifact/com.extentos/glasses) (Maven Central) - iOS library — [`Extentos`](https://github.com/extentos/swift-glasses) (Swift package, swift-glasses on GitHub) — pin the version as shown in the [iOS install guide](/docs/sdk/ios/install) Reports that include the exact versions (`whoami` output) and a `getEventLog` trace are easiest to triage. ## What to include in a bug report A useful bug report has six things. Maintainers can act on a bug with this; they often can't without it. The GitHub bug template asks for exactly these, so filing there gets you the checklist for free. ```text 1. Extentos version — output of `whoami` (mcpVersion + libVersion) 2. Platform — Android API level, or iOS version + device model 3. Agent host — Claude Code / Cursor / Windsurf / Cline / other 4. Steps to reproduce — exact agent prompts + commands run 5. Expected vs actual — what should have happened, what did 6. Logs — getEventLog output if applicable, otherwise Logcat (Android) / os_log (iOS) / MCP stderr ``` If the bug involves a simulator session, also include: - `sessionId` (if active or recent) - The capability call that failed (e.g. `capture_photo`, `record_discrete`, `speak`) and its event id from `getEventLog` - Whether the simulator browser tab was open - `autoOpenAttempted` and `autoOpenPlatform` from the `createSimulatorSession` response If the bug involves real hardware: - Ray-Ban Meta variant (Gen 1 / Gen 2 / Display) - Frame style (Wayfarer / Headliner / Skyler / etc.) if relevant - Bluetooth pairing status before the bug - Whether the bug repros on `LocalSimTransport` (the in-memory local sim) too If the bug involves cross-platform behavior, repro on both iOS and Android if you have access — or note which platform you couldn't test. ## Component triage — figuring out which component owns the bug Extentos has four main components, and one common reason bug reports go cold is mis-routing. Here's the rule of thumb: | Symptom | Likely component | |---|---| | MCP tool call returns an error or hangs | **MCP server** (`mcp-server/src/`) | | Agent generates wrong / broken code into the app | **MCP server** — code generators in `mcp-server/src/tools/handlers/` | | Speech recognized but your wake-phrase handler doesn't react | **Your handler code** first (the phrase match runs in your handler over `glasses.audio.transcriptions()`), then the **native library** if the transcription events themselves are wrong | | Simulator browser tab shows wrong frames or doesn't accept input | **Simulator browser UI** (`src/components/simulator/`) | | Simulator session expires immediately, fails to provision, or returns 5xx | **Backend** (`backend/src/`) | | `RealMetaTransport` BLE link drops, photo capture fails on hardware | **Native library** + maybe Meta DAT itself | | Permission request denied on Android 14, capability silently no-ops | **Native library** — permission derivation in `permissions.ts` and runtime handling | | `getPermissions` returns the wrong permission set for your handler's capabilities | **MCP server** — `mcp-server/src/tools/util/permissions.ts` | Most bugs are at the boundary between two components — e.g., the MCP server generates a manifest, the library reads it. Including `getEventLog` output usually disambiguates which side actually misbehaved. ## Security disclosure **Do not report security issues publicly.** Email **[hello@extentos.com](mailto:hello@extentos.com)** with subject prefix `[SECURITY]`. Include: - Affected component (MCP server / library / backend / simulator UI) - Affected version - Reproduction steps - Impact assessment (data exposure, privilege escalation, etc.) - Whether you've shared this with anyone else Acknowledgment within 7 days. Patch timeline depends on severity: - **Critical** (RCE, auth bypass, mass data exposure): patch within 14 days, coordinated disclosure thereafter - **High** (PII exposure, single-user data leak, persistent XSS): patch within 30 days - **Medium / Low**: patched in the next minor release The Extentos library architecture is designed to minimize blast radius — `RealMetaTransport` doesn't connect to the Extentos backend, and handler-code input/output payloads (which may contain end-user data) never reach Extentos servers. AI keys for handler-code calls live entirely in your app; for the [managed AI gateway](/docs/concepts/ai-gateway), Extentos relays the call on its own key — there is no bring-your-own-key option — but the conversation content is never stored — only token-usage metadata. Most security-relevant code paths are in the MCP server and the simulator/gateway backend; these are the areas most worth careful review by external researchers. See [security and data handling](/docs/resources/security) for the full architectural privacy story. ## Response expectations Pre-1.0 means honest expectations: - **Bug acknowledgment**: typically 1–3 business days. Sometimes longer when the maintainer is heads-down on the next release. - **Bug fix timeline**: triaged by severity. Crashes, data corruption, or security issues land in a same-week patch release. Feature gaps, minor inconsistencies, or polish items land in the next minor (`@0.0.x` → `@0.0.x+1`). - **Feature requests**: read and considered, but no commitment to a timeline. The roadmap is community-shaped — popular requests with clear use cases land sooner. - **Vendor-add requests** (e.g., "add Brilliant Labs"): tracked but require the vendor's SDK to be public + sufficiently mature, and hardware people can actually buy. See [vendors](/docs/vendors) for current support state. There is no paid support tier with a faster SLA at launch — this may change post-1.0. ## When the agent itself can debug A frequently-overlooked debugging path: **ask your AI agent to triage**. Modern AI coding agents reading a structured `getEventLog` response can often pinpoint the failing layer faster than a human scanning the same trace. Try: ```text My "describe the scene" feature failed. Run getEventLog with filter "errors" on the active session, read the trace, and tell me where it broke — the capability call (camera/audio), my handler logic, or the transport. ``` The agent has direct access to the MCP debug surface; it can correlate a capability request with its result/completed events and any error in between. Often this surfaces the actual bug without needing to file an issue. If the agent can't figure it out either, *that* trace becomes excellent evidence in the bug report — paste both the agent's analysis and the raw event log. ## Frequently asked questions ### What's the difference between filing an issue and asking my agent for help? Your agent has the structured event log, the project state via `inspectIntegration`, and your full project source. It can fix many issues without escalation — config problems, missing handlers, manifest mismatches, version drift between the generated module and the library. File once the agent has tried, the trace is conclusive, and the bug is in Extentos itself rather than your project. A "verified Extentos bug" report is far more actionable than an "agent and I are confused" thread. ### How do I send long log files? In an issue, put the log inside a collapsed `
` block or link a [GitHub Gist](https://gist.github.com). Over email, attach it as a `.txt` file. Don't paste 5000-line logs inline — they bury the summary. Redact tokens and credentials before posting anything publicly: a Meta Client Token or an Extentos auth token in a public issue is a live secret. ### Is there a Discord, Slack, or community forum? No Discord or Slack at launch. The closest thing to a community surface is the GitHub issue trackers listed in [where to send it](#where-to-send-it): they're public and searchable, so a filed bug is visible to everyone else building on the same SDK. A chat channel may open as the user base grows. ### Can I get paid support / consulting? Not at launch. Extentos is fully self-service today. If a paid support tier ships post-1.0, it will be announced via the [changelog](/docs/resources/changelog). ### What if I want to contribute a fix instead of just reporting? The development monorepo is private during pre-1.0, so there's no public PR flow yet. If you've got a fix, or a strong opinion on the shape of one, open an issue on the component's tracker and describe it there, or email **[hello@extentos.com](mailto:hello@extentos.com)** with `[contribution]`. Either way it gets applied in the monorepo with attribution in the changelog. External PRs may open up as the project approaches 1.0. ### How do I know if a bug is already known? Search the tracker for the component that owns it, then check the [changelog](/docs/resources/changelog) and [status](/docs/resources/status) for recent fixes and known issues. If it's in none of the three, file it. A duplicate is cheap to close, so err toward filing. ### What if the docs say one thing but the code does another? The code wins. File on [extentos/mcp-server](https://github.com/extentos/mcp-server/issues) with the contradicting doc URL and the source-code path that proves the actual behavior. The maintainers care about doc accuracy because misaligned docs poison AI-agent recommendations downstream. ### Can I email about non-security issues? Yes. Email reaches a human for anything, and it's the right channel for anything you'd rather keep private: billing, account, security, or a report you don't want public. For a plain SDK bug, a GitHub issue is the faster path, because it's searchable and gets closed against the release that fixes it. Use [hello@extentos.com](mailto:hello@extentos.com) with the right subject prefix (see [Where to send it](#where-to-send-it)). Security uses the same address with the `[SECURITY]` prefix, handled privately. ## Related - **[FAQ](/docs/resources/faq)** — most-asked questions, often resolved without filing - **[Security](/docs/resources/security)** — what we collect, what we don't, end-user privacy guarantees - **[Changelog](/docs/resources/changelog)** — release history, what's been fixed - **[Auth](/docs/mcp-server/auth)** — `whoami`, `logout`, install state and identity - **Packages** — [`@extentos/mcp-server`](https://www.npmjs.com/package/@extentos/mcp-server) (npm) · [`com.extentos:glasses`](https://central.sonatype.com/artifact/com.extentos/glasses) (Maven Central) · [swift-glasses](https://github.com/extentos/swift-glasses) (GitHub — Swift package) --- source: https://extentos.com/docs/resources/terms --- --- title: Terms of Service description: The terms that govern your use of Extentos, the SDKs, MCP server, simulator, and managed AI gateway. Plain-language, early-stage, Danish and EU law. type: resource platform: all related: - /docs/resources/pricing - /docs/resources/security --- *Last updated: 26 July 2026* Extentos (based in Denmark) provides a development platform for building smart-glasses features into apps, including the SDKs, MCP server, simulator, and an optional managed AI gateway. By using Extentos, you agree to these terms. If you do not, please do not use it. ## Using Extentos Most of Extentos is free. Some features need a free account, and you are responsible for what happens under it. The managed AI gateway is the one paid part. It is billed at the AI provider's list price with no Extentos markup, on prepaid credits, as described on our [pricing page](/docs/resources/pricing). ## What you cannot do Do not use Extentos for anything illegal or harmful, do not try to reverse-engineer or resell the SDK, and do not abuse or overload the service. We can suspend accounts that break these rules. ## Your work is yours You own the apps and code you build with Extentos. Extentos and its SDKs stay ours, and you get a license to use them to build your apps, not to copy or resell them. How we handle data is described in our [privacy policy](/privacy). ## No guarantees Extentos is early software provided "as is." We cannot promise it is bug-free, that it works on every device, or that AI output is correct, so please check anything important before relying on it. To the fullest extent the law allows, Extentos is not liable for damages arising from your use of it. If you are a consumer in the EU, you keep your mandatory legal rights. ## Changes Extentos is evolving, so we may change or discontinue features and update these terms. We will post updates here. ## Governing law and contact These terms are governed by the laws of Denmark. If you are a consumer in the EU, you keep the mandatory protections of your home country. Questions about these terms: [hello@extentos.com](mailto:hello@extentos.com). --- source: https://extentos.com/docs/sdk --- --- title: SDK description: Native Android and iOS libraries you link into your app to expose Extentos smart-glasses capabilities over a shared Rust core. Android and iOS both published. type: overview platform: all related: - /docs/sdk/android - /docs/sdk/ios - /docs/concepts/architecture --- The SDK is what your app links against. It is a thin Kotlin/Swift shell over a shared Rust core, so both platforms run the same runtime, the same protocol handling, and the same event vocabulary — only the platform shell differs. Pick the platform you're building on: - [Android SDK](/docs/sdk/android) — Gradle artifact on Maven Central, Kotlin + coroutines, Android 12+ (API 31) - [iOS SDK](/docs/sdk/ios) — Swift Package from `github.com/extentos/swift-glasses`, iOS 17.0+ ## Source - **Android:** [`com.extentos:glasses` on Maven Central](https://central.sonatype.com/artifact/com.extentos/glasses) - **iOS:** Swift package from [`github.com/extentos/swift-glasses`](https://github.com/extentos/swift-glasses) (see [install](/docs/sdk/ios/install)) --- source: https://extentos.com/docs/sdk/android --- --- title: Android SDK description: Add Extentos to your Android app via Gradle. Smart-glasses capabilities for Meta Ray-Ban, Android 12+ (API 31), Kotlin, coroutines. type: overview platform: android vendor: meta related: - /docs/sdk/android/install - /docs/sdk/android/initialization - /docs/getting-started/android - /docs/sdk/ios --- The Extentos Android SDK is a Gradle artifact that adds smart-glasses capabilities to your Android app. It is a thin Kotlin shell over a shared Rust core, talking to Meta Ray-Ban glasses through Meta's Wearables Device Access Toolkit (DAT). Published to Maven Central as [`com.extentos:glasses`](https://central.sonatype.com/artifact/com.extentos/glasses) and `com.extentos:glasses-ui`. Requires **Android 12+ (API 31)** — Meta DAT mandates the floor. - [Install](/docs/sdk/android/install) — Gradle coordinates, SDK floors, the assistant runtime - [Manifest setup](/docs/sdk/android/manifest) — what the library auto-provides vs. what you add - [Initialization](/docs/sdk/android/initialization) — `ExtentosGlasses.create`, config, the sub-client surface - [Lifecycle](/docs/sdk/android/lifecycle) — create/shutdown, the foreground service, process death - [Threading model](/docs/sdk/android/threading) — coroutines, `Flow`, `StateFlow`, dispatchers - [Runtime internals](/docs/sdk/android/runtime-internals) — how the Kotlin shell wraps the cross-platform core Building on iOS as well? See the [iOS SDK](/docs/sdk/ios) — the same surface as a Swift package. --- source: https://extentos.com/docs/sdk/android/initialization --- --- title: Initialize the Android SDK description: Create ExtentosGlasses with ExtentosGlasses.create, configure ExtentosConfig (environment, applicationContext, transport), reach the sub-clients, and pattern-match ExtentosResult. type: guide platform: android vendor: meta related: - /docs/sdk/android/lifecycle - /docs/sdk/android/threading - /docs/reference/errors - /docs/sdk/android/install --- **Camera and display need the Meta vendor module.** `com.extentos:glasses` carries no vendor SDK, so add `implementation("com.extentos:glasses-meta")` alongside it — see [install](/docs/sdk/android/install). Without it the build still succeeds and voice still works, but `capabilities.camera` is `false` and captures return errors. The SDK logs a warning at startup when it spots that combination. You create one `ExtentosGlasses` instance and hold it for the life of your app's glasses session. There is **no required `Application` subclass** and no static singleton to register — you call a factory and keep the handle. ## Create the instance ```kotlin import com.extentos.glasses.core.ExtentosGlasses import com.extentos.glasses.core.ExtentosConfig import com.extentos.glasses.core.ExtentosEnvironment val glasses = ExtentosGlasses.create( ExtentosConfig( applicationContext = applicationContext, environment = if (BuildConfig.DEBUG) { ExtentosEnvironment.DEVELOPMENT } else { ExtentosEnvironment.PRODUCTION }, ) ) ``` `ExtentosGlasses.create(config: ExtentosConfig = ExtentosConfig()): ExtentosGlasses` is the only entry point — the constructor is internal, so always go through `create()`. Calling it with no argument (`ExtentosGlasses.create()`) is valid and uses all defaults. **Camera or display apps need two more things, and the order matters.** The snippet above is the voice-app shape. `TransportChoice.Auto` picks the transport *inside* `create(...)` and keeps it for the life of the handle, so anything that influences the choice has to be true before you call it: 1. **`BLUETOOTH_CONNECT` must already be granted.** The vendor arm claims by reading the bonded-device list, which that permission gates. Without it the check can't answer, `Auto` falls through to the audio baseline, and `capabilities.camera` reports `false` for the rest of the process — voice works, capture doesn't, and nothing throws. 2. **Pass an `activityProvider`.** Meta's registration handoff needs a live Activity, and it defaults to `null`. So a camera app requests `RECORD_AUDIO` + `CAMERA` + `BLUETOOTH_CONNECT` from its Activity, *then* creates the handle. The worked bootstrap is in the [voice-assistant guide](/docs/guides/voice-assistant#if-your-app-also-uses-the-camera). Assert the result once at startup — `glasses.transportChosen` should read `REAL_META`, not `SYSTEM_AUDIO`. A natural home for the handle is a `ViewModel` or a small holder scoped to your app's lifetime; create it once and reuse it. When you're done, call `glasses.shutdown()` (a `suspend` function) — see [Lifecycle](/docs/sdk/android/lifecycle). ## ExtentosConfig Every field is defaulted, so the config is as small as you need it. | Field | Default | Notes | |---|---|---| | `appId` | `null` | Your project id. Falls back to the host app's package name when null — that package is the project identity everywhere else (MCP, dashboard, telemetry), so leaving it null is usually fine. | | `accountId` | `null` | Optional account attribution. | | `transport` | `TransportChoice.Auto` | How to reach the glasses — see below. | | `logLevel` | `LogLevel.WARN` | `VERBOSE` / `DEBUG` / `INFO` / `WARN` / `ERROR`. | | `debug` | `false` | Enables the dev-loop pairing path under `Auto`. | | `telemetryConsent` | `true` | When false, no events are sent at all. | | `dataSharingConsent` | `true` | When false, your own dashboard still works but events are excluded from cross-account aggregates. | | `environment` | `ExtentosEnvironment.DEVELOPMENT` | **Production apps must set `PRODUCTION`** — see the callout below. Use `BETA` for Play Console / TestFlight channels. | | `telemetryEndpoint` | `null` | Override the default telemetry ingest endpoint. | | `applicationContext` | `null` | **Required whenever the transport resolves to RealMeta or SystemAudio — i.e. on any real device** (Meta DAT needs a `Context`). Leave null only for headless JVM tests; on a real phone a null Context silently resolves to the in-memory `LocalSim` mock, which has no audio at all. | | `activityProvider` | `null` | A `() -> Activity?` the SDK calls to get a foreground Activity for Meta registration. | | `hasBondedMetaDevice` | `null` | A `() -> Boolean` the `Auto` chain uses to decide RealMeta vs. the audio baseline vs. simulator. The SDK has a built-in default, so you usually omit it. | | `usedCapabilities` | `emptySet()` | The SDK capabilities your app uses (`Set`). Values: `CapabilityKind.Camera`, `.Microphone`, `.Speaker`, `.Display`, `.Location`, `.Notifications`, and `CapabilityKind.Other(name)`. Drives the connection page's capability tiles — one tile per declared capability, lit when the connected glasses provide it, dimmed when they don't. The generated bootstrap sets this from your `generateConnectionModule(capabilities = [...])`. **On Android this is display-only** — the transport is picked from the bonded-device check, not from this set. On iOS the same field also decides the transport; if you ship both platforms, declare it on both. | > **Set `environment = PRODUCTION` for production builds.** It defaults to `DEVELOPMENT` so that `installDebug` builds never pollute production analytics. If you ship a release build still on `DEVELOPMENT`, its events land in the dev backend, not your production bucket. The `BuildConfig.DEBUG` pattern above is the canonical way to wire it. (As a safety net the SDK also reconciles the declared environment against an on-device classifier and downgrades a mismatched `PRODUCTION` debug build with a warning — but declare it correctly rather than relying on that.) ### Transport choice ```kotlin sealed interface TransportChoice { data object Auto : TransportChoice // default — resolves at runtime data object SystemAudio : TransportChoice // mic + speaker over OS Bluetooth; no vendor SDK data object RealMeta : TransportChoice // force real glasses via Meta DAT sealed interface Simulated : TransportChoice { data object Local : Simulated // in-process mock, no network data class Browser(val url: String? = null) : Simulated // browser simulator } } ``` `Auto` is almost always right. In order: the browser simulator when a session URL is baked in or set in the environment; then real glasses when a vendor module claims a bonded device; then — only when you set `ExtentosConfig.debug = true`, which is a config field defaulting to `false`, **not** your `BuildConfig.DEBUG` build type — the dev pairing screen; and otherwise `SystemAudio`, mic and speaker over whatever the OS has routed, which keeps a voice app working with no vendor SDK at all. A vendor claim beats the pairing screen, so a debug build with bonded glasses still goes to hardware. See [Runtime internals](/docs/sdk/android/runtime-internals) for the full resolution order and its caveats. ## Reading which transport you got `ExtentosGlasses` exposes the resolution outcome directly — useful after an upgrade, or to confirm a release build reached real hardware rather than the audio baseline: | Property | Type | What it tells you | |---|---|---| | `transportChosen` | `TransportChosen` | Which transport resolved: `REAL_META`, `SYSTEM_AUDIO`, `BROWSER_SIM`, `LOCAL_SIM` | | `selectionSource` | `TransportSelectionSource` | Why it won. All seven: `EXPLICIT_CONFIG`, `BUILD_CONFIG`, `ENV_VAR`, `VENDOR_REGISTRY`, `BONDED_DEVICES`, `PAIRING`, `FALLBACK_DEFAULT` | | `capabilities` | `GlassesCapabilities` | Flat yes/no facts of the connected device: `camera`, `microphone`, `speaker`, `display` | ```kotlin Log.i("app", "transport=${glasses.transportChosen} (${glasses.selectionSource})") ``` A camera app that reads `SYSTEM_AUDIO` here got there for one of three reasons: `com.extentos:glasses-meta` isn't on the classpath, no Meta glasses are bonded to the phone, or `BLUETOOTH_CONNECT` wasn't granted before `create(...)`. Check them in that order. ## The sub-client surface `ExtentosGlasses` exposes typed sub-clients as properties. You call capabilities through them: | Property | What it does | |---|---| | `connection` | Connection state (`StateFlow`), registration. | | `camera` | `capturePhoto()`, `captureVideo()`, `videoFrames()`. | | `audio` | `transcriptions()`, `audioChunks()`, `recordDiscrete(...)`, `speak(...)`. | | `display` | Render UI on display-capable glasses; guard with `display.isAvailable`. | | `runtime` | The on-device event log as `runtime.events: Flow`. | | `toggles` | The runtime capability gates. | | `voice` | Wake/voice-command hints and matching. | | `telemetry` | App-level telemetry events. | | `observability` | Diagnostics surface. | | `assistant` | The end-to-end voice assistant _(`1.4.0+`)_. | | `connectionConfig` | The dashboard-managed connection-page config. | | `device` | Identity of the connected glasses (`device.type`, `device.vendor`). | | `capabilities` | The flat capability facts of the connected device (camera/microphone/speaker/display). | ```kotlin val photo = glasses.camera.capturePhoto() glasses.audio.transcriptions().collect { transcript -> /* ... */ } ``` ## Handle results with ExtentosResult Lifecycle and capability operations return `ExtentosResult` instead of throwing. It is a sealed interface with two cases: ```kotlin import com.extentos.glasses.core.ExtentosResult when (val r = glasses.camera.capturePhoto()) { is ExtentosResult.Ok -> usePhoto(r.value) is ExtentosResult.Err -> handleFailure(r.error) } ``` The failure side carries concrete, per-operation variants — `capturePhoto()` / `captureVideo()` return a `CaptureError` (`NotConnected`, `PermissionDenied`, `ThermalCritical`, `HingesClosed`, `CoexistenceBlocked`, `StreamPaused`, `DisabledByUser`, `PlatformError` — all eight, or your `when` won't be exhaustive); `connect()` returns a `ConnectError` (`NoDeviceAvailable`, `PermissionDenied`, `PairingFailed`, `RegistrationRequired`, `TransportFailure`); `recordDiscrete()` / `speak()` return an `AudioError`. See [Errors](/docs/reference/errors) for the full set. > **Do not wrap these in `runCatching`.** They never throw, so `runCatching` would only catch unrelated exceptions and silently hide the typed failure. Pattern-match the result instead. Helpers if you don't want a full `when`: ```kotlin val value = glasses.camera.capturePhoto().valueOrNull() // S? — null on failure val error = result.errorOrNull() // F? — null on success val mapped = result.map { it.uri } // transform the success val chained = result.flatMap { upload(it) } // chain another result ``` Next: [Lifecycle](/docs/sdk/android/lifecycle) covers shutdown, the foreground service, and process death; [Threading](/docs/sdk/android/threading) covers the coroutine/`Flow` model. --- source: https://extentos.com/docs/sdk/android/install --- --- title: Install the Android SDK description: Add the Extentos Android SDK to your Gradle project from Maven Central. Coordinates, minSdk 31, Kotlin/AGP/Gradle floors, and the assistant runtime. type: guide platform: android vendor: meta related: - /docs/sdk/android/manifest - /docs/sdk/android/initialization - /docs/getting-started/android - /docs/sdk/android/runtime-internals --- The Extentos Android SDK ships five artifacts on **Maven Central**. Most apps need the first two: - `com.extentos:glasses` — the core runtime (`ExtentosGlasses`, capability clients, transports) - `com.extentos:glasses-ui` — the prebuilt connection page (`ExtentosConnectionPage`). Optional on a voice-only app, which renders no connection page. - `com.extentos:glasses-meta` — **optional**; the Meta Ray-Ban transport (camera, display, the DAT device session). Only apps that use camera or display need it. - `com.extentos:glasses-local` — **optional**; on-device text models for the [local tier](/docs/concepts/local-models). - `com.extentos:glasses-local-voice` — **optional**; on-device speech for the local tier. The Extentos artifacts resolve from the default `mavenCentral()` repository. `com.extentos:glasses` and `glasses-ui` have **no vendor dependency**, so a voice-only app needs nothing beyond a normal Gradle dependency — no extra repository, no token. The Meta DAT dependencies (`com.meta.wearable:mwdat-*`) come in through **`com.extentos:glasses-meta`** only. They live on **Meta's GitHub Packages**, which needs one extra repo declaration plus a GitHub token (step 1). Skip that step entirely if your app doesn't use camera or display. ## 1. Declare the repositories **Voice-only apps can skip this whole step.** Extentos resolves from Maven Central, which is already in the default Android template. If you add `com.extentos:glasses-meta` for camera or display, its **Meta DAT** artifacts resolve from **Meta's GitHub Packages**, which requires a GitHub personal access token with the `read:packages` scope. Add both to your `settings.gradle.kts`: ```kotlin title="settings.gradle.kts" // pluginManagement MUST be the first block in a settings script — Gradle // aborts configuration otherwise. Without it, `id("com.android.application")` // can't resolve: AGP lives on google(), and Gradle's default plugin repo is // gradlePluginPortal() only. pluginManagement { repositories { google(); mavenCentral(); gradlePluginPortal() } } dependencyResolutionManagement { repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS) repositories { google() mavenCentral() // Transitive Meta DAT artifacts (com.meta.wearable:mwdat-*) live here: maven { url = uri("https://maven.pkg.github.com/facebook/meta-wearables-dat-android") credentials { username = "token" password = (providers.gradleProperty("github_token").orNull ?: System.getenv("GITHUB_TOKEN")) ?: "" } } } } ``` …followed by your module declarations: ```kotlin title="settings.gradle.kts (continued)" rootProject.name = "myapp" include(":app") ``` `FAIL_ON_PROJECT_REPOS` means a `repositories { }` block in `app/build.gradle.kts` becomes a hard build failure — declare every repository here instead. Create the token at **github.com → Settings → Developer settings → Personal access tokens** with the `read:packages` scope, then export it as `GITHUB_TOKEN`. That is the one location every surface agrees on — the block above reads it, the scaffold's variant reads it, and `extentos-mcp setup` checks it. (`~/.gradle/gradle.properties` as `github_token=...` also satisfies the block above, but the scaffold's variant and the diagnostic don't look there, so you can end up with a working build and a diagnostic that reports it missing — or the reverse.) To check your setup, run **`npx -p @extentos/mcp-server extentos-mcp setup`** from the project root — a read-only diagnostic that validates the token against GitHub Packages and prints exact remediation; it reads `GITHUB_TOKEN` or the project's `local.properties` and does **not** store the token for you. (`generateConnectionModule` emits a variant of this repo block that reads `local.properties`/env instead of the Gradle property — same URL, same key names.) Without the token the build fails at first sync with `Could not resolve com.meta.wearable:mwdat-core`. ## 2. Add the dependency ```kotlin title="app/build.gradle.kts" dependencies { implementation("com.extentos:glasses:2.8.0") implementation("com.extentos:glasses-ui:2.8.0") // ONLY if your app uses the camera or the display. This is what brings the // Meta DAT artifacts (and therefore the repository + token in step 1). // Voice-only apps omit it entirely. implementation("com.extentos:glasses-meta:2.8.0") } ``` You don't declare the Meta DAT SDK (`com.meta.wearable`) yourself — it comes in transitively through `com.extentos:glasses-meta`, and *only* through that module. It resolves from the GitHub Packages repo added in step 1, so the `read:packages` token must be in place for a camera/display build to succeed. A voice-only app never pulls those artifacts and needs no token. `glasses-meta` registers itself at app start, so adding the dependency is the whole integration — `TransportChoice.Auto` resolves to real glasses with no extra code. ## 3. Set the build floors The library is built against these versions. Match or exceed them. | Requirement | Value | Why | |---|---|---| | `minSdk` | **31** (Android 12) | Meta DAT mandates it. The Bluetooth runtime-permission model the SDK uses is Android 12+. | | `compileSdk` | **35** | Transitive deps require it. | | Kotlin | **2.1.20** | The AAR is compiled with 2.1.20; older Kotlin emits `Class file was compiled with a newer version` metadata warnings. | | Android Gradle Plugin | **8.6.0+** | Compose deps in `glasses-ui` declare an AGP 8.6 floor. | | Gradle | **8.7+** | Required by AGP 8.6. | | Java/Kotlin target | **17** | The library targets JVM 17. | The library does **not** declare a `targetSdk` — your app's `targetSdk` is inherited as-is. `targetSdk 34` or `35` both work for most apps. (`getPermissions` reports `34` as the value it derived its permission advice against; that's the floor it reasoned from, not a requirement on you.) You also need two build features that a fresh AGP 8 project leaves **off** by default: ```kotlin title="app/build.gradle.kts" plugins { id("com.android.application") version "8.6.0" id("org.jetbrains.kotlin.android") version "2.1.20" // Required if you use ExtentosConnectionPage or any Compose UI. From Kotlin // 2.0 the Compose compiler is a separate plugin — same version as Kotlin. id("org.jetbrains.kotlin.plugin.compose") version "2.1.20" } android { // AGP 8 removed the manifest-package fallback: without a namespace the build // fails with "Namespace not specified." namespace = "com.example.myapp" compileSdk = 35 defaultConfig { minSdk = 31 // targetSdk = 35 // your choice; the library doesn't impose one } buildFeatures { // AGP 8 defaults this to false. The SDK's own initialization snippet // reads BuildConfig.DEBUG, and the MCP scaffold emits buildConfigField // entries (EXTENTOS_SESSION_URL), so this must be on. buildConfig = true compose = true // only if you use Compose / ExtentosConnectionPage } compileOptions { sourceCompatibility = JavaVersion.VERSION_17 targetCompatibility = JavaVersion.VERSION_17 } kotlinOptions { jvmTarget = "17" } } ``` **`buildConfig = true` is not optional.** Every canonical initialization snippet branches on `BuildConfig.DEBUG` to pick the environment, and the browser-simulator loop reads `BuildConfig.EXTENTOS_SESSION_URL`. On AGP 8 with the feature off, `BuildConfig` does not exist and those lines don't resolve. ### The project key (production gateway attribution) If your app uses the assistant, add your account-bound project key as a `buildConfigField` so production gateway usage attributes to your Extentos account: ```kotlin title="app/build.gradle.kts" android { defaultConfig { buildConfigField("String", "EXTENTOS_PROJECT_KEY", "\"pk_your_project_key\"") } } ``` The SDK reads it reflectively and sends it as the `x-extentos-project-key` header. (Meta's App ID and Client Token go through `resValue` / `R.string` instead, not `buildConfigField` — those rotate, and `BuildConfig` string constants are compile-time-inlined so a stale one can survive a rotation. The project key is regenerated by re-running the scaffold, which rewrites this line, so the same hazard doesn't apply.) `generateConnectionModule` mints the key and emits this exact line for you (it appears in the response's `buildConfigFields[]`); if you are wiring the project by hand, get the key from your dashboard project. In the **simulator** it is unused — attribution there comes from the session token. On a **sideloaded development build on a real phone** it is load-bearing: such a build can't attest (Play Integrity doesn't recognise it), so the key is how the gateway identifies the caller. Without it, and without a simulator-session binding, `assistant.start(...)` fails with `AssistantError.NoApiKey`. It never causes a *build* error, so the only symptom is at runtime — see [gateway identity](/docs/concepts/ai-gateway#identity-how-the-gateway-knows-whos-calling). Don't commit it to a public repo; re-run `generateConnectionModule` to rotate. Some dependencies are yours, not the SDK's. `glasses-ui` brings the Compose runtime *it* needs, not the ones your UI code uses: ```kotlin title="app/build.gradle.kts" // The SDK's suspend surface needs a dispatcher from your side implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.9.0") // Only if you follow the docs' Compose idiom (collectAsStateWithLifecycle) implementation("androidx.lifecycle:lifecycle-runtime-compose:2.8.6") // Your own Compose stack, if you have UI of your own implementation(platform("androidx.compose:compose-bom:2024.09.03")) // any recent BOM works — declare only one implementation("androidx.compose.ui:ui") implementation("androidx.compose.material3:material3") implementation("androidx.activity:activity-compose:1.9.2") ``` **If you started from the Android Studio template**, its root `build.gradle.kts` already declares these plugins with `apply false` and a version. Declaring a version in both places fails with *"plugin request must not include a version"* — keep one. Either drop the `version` from the block above, or delete the root declarations. A `libs.versions.toml` catalog works too; the SDK doesn't care which you pick. ## 4. Sync and verify Run a Gradle sync (or `./gradlew :app:dependencies`) and confirm `com.extentos:glasses` resolves. Next, wire it up: - [Manifest setup](/docs/sdk/android/manifest) — add `CAMERA` / `RECORD_AUDIO` / `INTERNET` - [Initialization](/docs/sdk/android/initialization) — `ExtentosGlasses.create(...)` ## The assistant runtime The end-to-end voice **assistant** (`glasses.assistant.*` — wake/sleep state machine, barge-in, `session.say(...)`) ships in the stable **`1.4.0`** release on Maven Central — the standard dependency above includes it. No snapshot, no extra repository, no special setup. See [the assistant page](/docs/concepts/assistant) for the API surface and [the voice-assistant guide](/docs/guides/voice-assistant) for an end-to-end build. ## Source - **Artifact:** [`com.extentos:glasses` on Maven Central](https://central.sonatype.com/artifact/com.extentos/glasses) --- source: https://extentos.com/docs/sdk/android/lifecycle --- --- title: Android SDK lifecycle description: When to create and shut down ExtentosGlasses, how the bundled foreground service keeps the glasses mic alive in the background, and how Extentos behaves across Activity lifecycle and process death. type: guide platform: android vendor: meta related: - /docs/sdk/android/initialization - /docs/sdk/android/manifest - /docs/sdk/android/threading - /docs/reference/errors --- `ExtentosGlasses` is a long-lived object with an explicit start and stop. You own its lifetime — it doesn't auto-register with the Android lifecycle, so where you create and release it is your decision. ## Create once, shut down once ```kotlin // Start: hold the handle for the life of the glasses session. val glasses = ExtentosGlasses.create(config) // Stop: release the transport and cancel the SDK's coroutine scope. suspend fun tearDown() { glasses.shutdown() } ``` `shutdown()` is a `suspend` function — call it from a coroutine. It shuts down the active transport (closing the DAT/Bluetooth or WebSocket connection) and cancels the internal `SupervisorJob` scope that drives every `Flow` the SDK emits. After `shutdown()`, the instance is spent; create a new one if you need to reconnect. Where to anchor it: - **App-scoped session** (the common case — glasses connection persists across screens): create it in a holder tied to your application or a shared `ViewModel`, and call `shutdown()` when the user explicitly disconnects or the app is being torn down. - **Screen-scoped session**: create it in a screen-level `ViewModel` and call `shutdown()` from `onCleared()`. The rule is **one live instance at a time**, not one per process — this is safe only if exactly one screen owns it and the old one is shut down before the next is created. If two screens can be alive at once, use the app-scoped holder instead; two concurrent instances each run their own auto-bind probe and telemetry registration and will fight over the transport. ```kotlin class GlassesViewModel : ViewModel() { val glasses = ExtentosGlasses.create(config) override fun onCleared() { viewModelScope.launch { glasses.shutdown() } } } ``` Don't create a fresh `ExtentosGlasses` per `Activity.onCreate` if the same glasses session should outlive configuration changes — a `ViewModel` survives rotation; the Activity does not. ## Activity and Fragment wiring The SDK needs an Activity in two situations, both supplied through `ExtentosConfig`: - **`applicationContext`** — required when the transport resolves to RealMeta (Meta DAT initializes against a `Context`). Pass `applicationContext`, not an Activity, so the reference is safe to hold. - **`activityProvider: () -> Activity?`** — Meta registration (the pairing handoff) needs a *foreground* Activity. Provide a closure that returns your current top Activity (e.g. from an `ActivityLifecycleCallbacks`-tracked reference), so the SDK can launch registration when the user connects. ```kotlin ExtentosConfig( applicationContext = applicationContext, activityProvider = { topActivityRef.get() }, // your current foreground Activity ) ``` Observe connection state from your UI through `connection.state` (a `StateFlow`) — collect it lifecycle-aware with `repeatOnLifecycle` so you stop collecting when the screen is stopped. See [Threading](/docs/sdk/android/threading). ## Background capture and the foreground service Capability subscriptions (`glasses.audio.*`, the assistant) run on the SDK's coroutine scope, which is bound to your process — not to any Activity. While your app is **foreground**, that's enough. To keep the glasses mic alive while your app is **backgrounded** (a voice assistant or live transcription that must keep listening), start the library's bundled foreground service: ```kotlin import com.extentos.glasses.core.service.GlassesForegroundService // AFTER RECORD_AUDIO is granted at runtime — typically in the grant callback: GlassesForegroundService.start(this) // When background capture is no longer needed: GlassesForegroundService.stop(this) ``` The service is declared for you in the library manifest (`foregroundServiceType="microphone"`), so you only start/stop it — see [Manifest setup](/docs/sdk/android/manifest). Key rules: - **Permission first.** `RECORD_AUDIO` must already be granted. On Android 14+, starting the service without the grant throws `SecurityException` at service-create time and kills the app before the first frame draws. - **Not from `Application.onCreate`.** The grant can't exist that early. Start it from an Activity (or a `ViewModel` scoped to one) inside the permission-grant callback, or after a `checkSelfPermission(...) == PERMISSION_GRANTED` check. - **Idempotent.** Calling `start` again while running just updates the notification. - Camera-only or strictly foreground apps don't need it. ## Process death Extentos holds no implicit cross-process state to restore: an `ExtentosGlasses` instance and its transport connection do not survive the process being killed. After process death and relaunch, you re-run `ExtentosGlasses.create(...)` and reconnect — exactly as on first launch. The dev-loop pairing identity is persisted to `filesDir`, so an `Auto`-resolved dev session can re-bind without re-pairing, but the live glasses connection itself is re-established from scratch. Because operations return [`ExtentosResult`](/docs/reference/errors) rather than throwing, a connection lost across a lifecycle gap surfaces as a typed failure on the next call — `CaptureError`/`AudioError.NotConnected` from a capability call, or a `ConnectError` (`NoDeviceAvailable` / `TransportFailure`) from `connect()`/`reconnect()` — pattern-match and re-establish rather than guarding with try/catch. --- source: https://extentos.com/docs/sdk/android/manifest --- --- title: Android manifest setup description: Which permissions the Extentos Android SDK auto-merges for you and which capability permissions (CAMERA, RECORD_AUDIO, INTERNET) your app must declare, plus the bundled foreground service. type: reference platform: android vendor: meta related: - /docs/concepts/permissions - /docs/sdk/android/install - /docs/sdk/android/lifecycle - /docs/sdk/android/initialization --- **Camera and display need the Meta vendor module.** `com.extentos:glasses` carries no vendor SDK, so add `implementation("com.extentos:glasses-meta")` alongside it — see [install](/docs/sdk/android/install). Without it the build still succeeds and voice still works, but `capabilities.camera` is `false` and captures return errors. The SDK logs a warning at startup when it spots that combination. The Extentos Android SDK ships an `AndroidManifest.xml` that the Android manifest merger folds into your app at build time. Some entries are zero-config (the library declares them for you); the **capability** permissions stay app-owned because they gate features only your app knows it needs. ## What the library declares for you These are merged in automatically — you do **not** add them: | Entry | Kind | Purpose | |---|---|---| | `BLUETOOTH_CONNECT` | permission | The universal connection permission — every glasses app needs it to discover and talk to the glasses. The SDK requests it at runtime once the Meta transport is running — **but a camera app must have it granted before `ExtentosGlasses.create(...)`**, because the transport choice depends on it and is made inside that call. See [permissions](/docs/concepts/permissions#always-required). | | `BLUETOOTH_SCAN` | permission | Discovering nearby glasses. | | `MODIFY_AUDIO_SETTINGS` | permission | Routing the glasses mic + speaker over Bluetooth SCO/HFP. Without it, SCO never starts and an awake assistant hears nothing. Normal-protection, so it auto-grants. | | `GlassesForegroundService` | `` | Bundled foreground service for background-surviving mic capture (see below). | | `DAM_ENABLED` meta-data | `` | Meta's Device Access Model flag, required by the **display** capability. Defaults to **`false`**: on DAT 0.8, DAM makes the SDK demand an on-glasses companion app that current non-display glasses don't yet have, which drops the session — so camera/audio apps run the non-DAM path and stay connected. If your declared footprint includes Display, `generateConnectionModule` re-enables it for you (`DAM_ENABLED=true` via `tools:replace`). | | `networkSecurityConfig` | `` attr | A narrow cleartext exception for localhost-equivalent hosts only (`10.0.2.2`, `127.0.0.1`, `localhost`) so the dev-loop auto-bind probe can reach your local MCP server. A no-op in real-world traffic. | > **Conflict on `networkSecurityConfig`?** If your app already declares its own `android:networkSecurityConfig`, the merger reports a conflict. Resolve it by adding `tools:replace="android:networkSecurityConfig"` to your app manifest's `` tag (and add `xmlns:tools="http://schemas.android.com/tools"` to the `` root if it isn't there). ## What you add The library leaves **capability** permissions to you, so your manifest declares only what your app actually uses. Add the ones matching the capabilities you call: ```xml title="app/src/main/AndroidManifest.xml" ``` | Permission | Add it when you use… | Runtime grant? | |---|---|---| | `CAMERA` | `glasses.camera.capturePhoto()`, `captureVideo()`, `videoFrames()` | Yes — request at runtime | | `RECORD_AUDIO` | `glasses.audio.*`, `glasses.voice.*`, `glasses.assistant.*` | Yes — request at runtime | | `INTERNET` | Always (gateway, simulator, telemetry) | No — normal permission | | `FOREGROUND_SERVICE` + `FOREGROUND_SERVICE_MICROPHONE` | You start the bundled `GlassesForegroundService` (background mic for voice / transcription / the assistant). `_MICROPHONE` is required on Android 14+. | No — normal permissions | ### Camera or display apps also add the Meta credential entries Permissions aren't the whole manifest. An app using the vendor transport must also carry the two Meta DAT `` elements inside ``, or MWDAT can't initialize: ```xml title="app/src/main/AndroidManifest.xml" ``` The string resources come from `app/src/main/res/values/extentos_meta_credentials.xml`, which `generateConnectionModule` writes from the credentials you stored in your dashboard project — there is no runtime injection path through `ExtentosConfig`. See [credentials](/docs/mcp-server/credentials-checklist). **Voice-only apps skip this** — they have no DAT dependency. For the runtime-request flow and which capability needs which permission, see the [`getPermissions` MCP tool](/docs/reference/mcp-tools) and [Permissions](/docs/concepts/permissions). ## The bundled foreground service For background-surviving microphone capture (a voice assistant or live transcription that must keep listening while your app is backgrounded), the library declares a foreground service for you: ```xml ``` You don't declare it — you **start** it from an Activity, **after** `RECORD_AUDIO` has been granted at runtime: ```kotlin import com.extentos.glasses.core.service.GlassesForegroundService // In your permission-grant callback, once RECORD_AUDIO is PERMISSION_GRANTED: GlassesForegroundService.start(this) // 'this' is any Context // Later, when you no longer need background capture: GlassesForegroundService.stop(this) ``` `start` is idempotent. Do **not** call it from `Application.onCreate` — Android 14+ requires the `RECORD_AUDIO` grant to be in place before `startForeground(...)` runs, or the app crashes at service-create time. See [Lifecycle](/docs/sdk/android/lifecycle) for the full wiring. > Camera-only or strictly foreground apps don't need the service at all — the declaration is harmless if unused. A camera-type foreground service (rare; for surviving `videoFrames` in the background) requires you to subclass `GlassesForegroundService` and declare your own ``. > Building on iOS too? Permissions there are Info.plist keys, not manifest entries — see the [iOS SDK](/docs/sdk/ios) pages. --- source: https://extentos.com/docs/sdk/android/runtime-internals --- --- title: Android runtime internals description: How the Extentos Android SDK is a thin Kotlin shell over a shared Rust core (uniffi + jniLibs), the four transports and how TransportChoice.Auto resolves, and the Meta DAT bridge. type: concept platform: android vendor: meta related: - /docs/concepts/architecture - /docs/concepts/transport-vs-app - /docs/sdk/android/initialization - /docs/sdk/android/threading --- You don't need this page to build an app — the [capability surface](/docs/sdk/android/initialization) is the contract. It's here for when you want to understand what's running underneath, debug a transport selection, or reason about cross-platform parity. ## Kotlin shell over a shared Rust core The Android SDK is deliberately thin. Vendor-agnostic logic — state machines, data shapes, event vocabularies, protocol handling, display strings — lives **once** in a shared Rust core (`extentos-core`), not in the Kotlin code. The Kotlin layer holds only what must be platform-bound: Meta DAT calls, Bluetooth SCO/HFP audio, Compose UI, and the Android Context plumbing. The core reaches Kotlin through [uniffi](https://mozilla.github.io/uniffi-rs/): - **`extentos_core.kt`** — uniffi-generated Kotlin bindings for the core's types and functions. This file is **codegen output** — never hand-edit it; the source of truth is the Rust crate, and the bindings are regenerated from it. - **`jniLibs/*.so`** — the compiled Rust core for each Android ABI, loaded at runtime. The bindings load the native library through JNA. A bug fixed in the Rust core lands on Android and iOS at once; that's the whole point of keeping the shells thin. The same core compiles to an `xcframework` for the iOS SDK. (See [Architecture](/docs/concepts/architecture) for the cross-platform picture.) ## Four transports behind one abstraction Everything above the transport — every capability client — talks to a single `GlassesTransport` interface. Four implementations swap in behind it: | Transport | What it is | Network | |---|---|---| | `SystemAudioTransport` | The vendorless baseline: mic + speaker over the OS's own Bluetooth routing. No vendor SDK, so no camera and no display. | Bluetooth HFP (via the OS) | | `RealMetaTransport` | Production glasses via Meta DAT (`com.meta.wearable`, 0.8.0, through `com.extentos:glasses-meta`). | Bluetooth / DAT | | `BrowserSimTransport` | WebSocket to the Extentos backend; the hardware surrogate runs in a browser tab. | WebSocket | | `LocalSimTransport` | In-process mock (no network), for a fast inner loop and JVM unit tests. | None | Because the capability clients only see `GlassesTransport`, the **simulator runs the same library code as production** — only the transport and I/O substrate differ. A failure in the simulator is a real failure. (See [Transport vs. app behavior](/docs/concepts/transport-vs-app).) ## How `TransportChoice.Auto` resolves `Auto` (the default) picks a transport at `create()` time by walking these rules in order, stopping at the first match: 1. **Baked session URL** — a `BuildConfig.EXTENTOS_SESSION_URL` injected at compile time → `BrowserSim`. 2. **`EXTENTOS_SESSION_URL` env var** — a runtime dev override (handy for emulator runs) → `BrowserSim`. 3. **A registered vendor claims** — vendor modules (e.g. `com.extentos:glasses-meta`) register themselves at app start and are asked, in order, whether their hardware is present. Meta claims when a Meta/Ray-Ban/Oakley device is bonded over Bluetooth → `RealMeta`. (That check needs `BLUETOOTH_CONNECT`; without the grant it returns false and falls through, then self-heals on the next launch once granted.) With no vendor module on the classpath the registry is empty and resolution continues unchanged. 4. **Dev pairing** — `debug = true` with a `Context` but no URL and no vendor claim → `BrowserSim` in pairing mode (the connection page shows a short code to type into the simulator). Debug-only: a release build never reaches this rung. 5. **System audio** — a real device with a `Context` and no vendor claim → `SystemAudio`, the vendorless baseline. Mic and speaker over whatever the OS has routed; the full agent runtime works, camera and display do not. 6. **Fallback** — no `Context` at all (headless JVM tests, CI) → `LocalSim`. The net effect with zero host config: a phone with paired glasses resolves to real hardware; an emulator resolves to the simulator; a real phone with none resolves to `SystemAudio`, the vendorless audio baseline. You can always force a specific transport by setting `ExtentosConfig.transport` explicitly (`SystemAudio`, `RealMeta`, `Simulated.Local`, or `Simulated.Browser(url)`). The resolved choice is recorded on `runtime.events` as a `transport.selected` log line. **This differs from iOS on purpose.** Android can enumerate bonded Bluetooth devices, so rung 3 has a direct signal for "are the glasses actually here?" and rung 5 can safely catch everything else. iOS has no equivalent API, so its `.auto` chain keys the audio-baseline rung off the app's *declared* capability footprint instead. The practical difference: an app that declares camera, on a phone with no glasses paired, lands on `SystemAudio` here and on `RealMeta` on iOS (where `connect()` then reports no eligible device). See [iOS runtime internals](/docs/sdk/ios/runtime-internals). ## The Meta DAT bridge `RealMetaTransport` is backed by `MetaHardwareBridge`, the Kotlin glue to Meta's Device Access Toolkit. It owns the platform-bound work the core can't do: - DAT session init / registration (needs a `Context` and a foreground `Activity` — supplied via `ExtentosConfig.applicationContext` and `activityProvider`). - Bluetooth SCO/HFP routing for the glasses mic and speaker (wideband mSBC, 16 kHz duplex), which is why the library declares `MODIFY_AUDIO_SETTINGS`. - Surfacing hardware events (lifecycle, audio route, call state) up into the core's event vocabulary. The Meta DAT SDK arrives ONLY through `com.extentos:glasses-meta`, and only if you add that dependency. `com.extentos:glasses` carries no vendor SDK, so an app that never uses camera or display ships none of it and needs no credentialed repository. (This differs from iOS, where the DAT modules are still unconditional dependencies of `GlassesCore` — the module split is Android-only today.) You never declare `com.meta.wearable` yourself either way; it comes in transitively through `glasses-meta`. > Generated bindings (`extentos_core.kt`, the `uniffi/` output, `jniLibs/`) are build artifacts. If something looks wrong in them, the fix is in the Rust source, regenerated — not an edit to the generated file. --- source: https://extentos.com/docs/sdk/android/threading --- --- title: Android SDK threading model description: How the Extentos Android SDK uses Kotlin coroutines and Flow — suspend functions for one-shot ops, Flow/StateFlow for streams, and which dispatcher to collect on. type: guide platform: android vendor: meta related: - /docs/sdk/android/initialization - /docs/sdk/android/lifecycle - /docs/sdk/android/runtime-internals - /docs/reference/errors --- **Camera and display need the Meta vendor module.** `com.extentos:glasses` carries no vendor SDK, so add `implementation("com.extentos:glasses-meta")` alongside it — see [install](/docs/sdk/android/install). Without it the build still succeeds and voice still works, but `capabilities.camera` is `false` and captures return errors. The SDK logs a warning at startup when it spots that combination. The Extentos Android SDK is coroutine-native. Its public surface is split cleanly into two shapes: - **`suspend` functions** for one-shot operations that complete with a result. - **`Flow` / `StateFlow`** for continuous streams and observable state. There are no callbacks to register and no threads to manage — you `collect` flows and `await` suspend functions from your own coroutine scopes. ## One-shot operations are `suspend` Discrete capability calls suspend until they finish and return an [`ExtentosResult`](/docs/reference/errors): ```kotlin suspend fun takeShot() { when (val r = glasses.camera.capturePhoto()) { is ExtentosResult.Ok -> show(r.value) is ExtentosResult.Err -> report(r.error) } glasses.audio.speak("Got it.") // also suspend } ``` Examples: `camera.capturePhoto()`, `camera.captureVideo()`, `audio.recordDiscrete(...)`, `audio.speak(...)`, `audio.cancelSpeak()`, and `glasses.shutdown()`. Call them from a coroutine (`lifecycleScope`, `viewModelScope`, or any scope you control). They are cooperative with cancellation — cancelling the calling coroutine cancels the operation. ## Streams are `Flow` Continuous capabilities return a cold `Flow` you collect for as long as you want the stream: ```kotlin // Live transcription glasses.audio.transcriptions().collect { transcript -> updateCaption(transcript.text) } // Raw mic chunks / camera frames glasses.audio.audioChunks().collect { chunk -> /* ... */ } glasses.camera.videoFrames().collect { frame -> /* ... */ } ``` These are cold — the underlying capture starts when collection starts and stops when the collecting coroutine is cancelled. Run them on a scope whose lifetime matches the subscription (and start the [foreground service](/docs/sdk/android/lifecycle) if the stream must survive backgrounding). ## Observable state is `StateFlow` / event `Flow` Two always-available streams describe the SDK's own state: | Stream | Type | What it carries | |---|---|---| | `glasses.connection.state` | `StateFlow` | The live connection state — always has a current value. | | `glasses.runtime.events` | `Flow` | The on-device event log (toggles, transport, logs, warnings). | Collect connection state lifecycle-aware so the UI tracks it without leaking: ```kotlin lifecycleScope.launch { repeatOnLifecycle(Lifecycle.State.STARTED) { glasses.connection.state.collect { state -> render(state) } } } ``` In Compose, `collectAsStateWithLifecycle()` is the idiomatic equivalent: ```kotlin val connState by glasses.connection.state.collectAsStateWithLifecycle() ``` Because `state` is a `StateFlow`, a new collector immediately receives the current value — you don't miss the connection state by subscribing late. The event log additionally offers a one-shot snapshot if you don't want to collect: `glasses.runtime.snapshotEvents()` (a `suspend` function returning `List`). ## Dispatcher guidance - **You don't pick a dispatcher for the SDK's internals.** It runs its own work on a `SupervisorJob` + `Dispatchers.Default` scope created at `create()` and cancelled at `shutdown()`. Flow emissions and suspend-function bodies execute there. - **Collect on a UI-aware scope** (`lifecycleScope`, `viewModelScope`) and touch UI from the main dispatcher. Use `flowOn(...)` only if *your own* downstream operators are heavy — the SDK already does its capture work off the main thread. - **Don't block.** Never call a `suspend` operation from `runBlocking` on the main thread; launch it on a coroutine scope. ## Don't wrap results in `runCatching` Capability and lifecycle operations return `ExtentosResult` and **do not throw** for expected failures. Pattern-match the result; `runCatching` would only ever catch unrelated exceptions and would hide the typed failure. See [Errors](/docs/reference/errors) for the failure variants and the `valueOrNull()` / `map` / `flatMap` helpers. --- source: https://extentos.com/docs/sdk/ios --- --- title: iOS SDK description: The Extentos iOS SDK for Meta Ray-Ban smart glasses — iOS 17+, Swift 6 toolchain, async/await. Published as a Swift package. type: overview platform: ios vendor: meta related: - /docs/getting-started/with-agent - /docs/getting-started/ios - /docs/sdk/android - /docs/sdk/ios/install - /docs/sdk/ios/initialization --- The Extentos iOS SDK is a Swift package — `Extentos` — that adds Meta Ray-Ban smart-glasses capabilities to your iOS app. It shares one Rust core with the Android SDK (exposed to Swift through uniffi-generated bindings), so capability primitives, transport selection, and the connection flow behave the same on both platforms. > The iOS SDK installs from [`github.com/extentos/swift-glasses`](https://github.com/extentos/swift-glasses) as a standard SPM dependency — see [Install](/docs/sdk/ios/install). The fastest start on either platform is the [agent-driven flow](/docs/getting-started/with-agent). ## The pages These pages are accurate against the current iOS source: - [Install](/docs/sdk/ios/install) — adding the Swift package (Xcode, `Package.swift`, or xcodegen) and the agent-driven route - [Info.plist setup](/docs/sdk/ios/info-plist) — the required keys for the Meta DAT auth callback, accessory protocol, and privacy strings - [Initialization](/docs/sdk/ios/initialization) — `Extentos.create(config:)`, the `.onOpenURL` auth handoff, and the sub-client surface - [Threading model](/docs/sdk/ios/threading) — `Sendable`, `async`/`await`, and `MainActor` guidance - [Lifecycle](/docs/sdk/ios/lifecycle) — startup and `shutdown()` - [Runtime internals](/docs/sdk/ios/runtime-internals) — how the iOS shell wraps the cross-platform core ## Platform requirements | Requirement | Value | |---|---| | Deployment target | iOS 17.0+ ([why](/docs/sdk/ios/install#requirements)) | | Swift toolchain | Swift 6 (`swift-tools-version: 6.0`) | | Language mode | Swift 5 (`.swiftLanguageMode(.v5)`) — see [Threading](/docs/sdk/ios/threading) | | Meta DAT (iOS) floor | `meta-wearables-dat-ios` from `0.8.0` | For the cross-platform model — what the Meta DAT toolkit provides vs what Extentos adds — see [transport vs app simulation](/docs/concepts/transport-vs-app). For the same surface in Kotlin, see [Android](/docs/sdk/android). --- source: https://extentos.com/docs/sdk/ios/info-plist --- --- title: Info.plist setup (iOS) description: Required Info.plist keys for the Extentos iOS SDK — the Meta DAT auth callback URL scheme, external accessory protocol, background modes, and Bluetooth / camera / microphone / speech privacy strings. generateConnectionModule emits them all. type: reference platform: ios vendor: meta related: - /docs/concepts/permissions - /docs/sdk/ios/initialization - /docs/getting-started/with-agent --- The Extentos iOS SDK rides on the Meta Wearables Device Access Toolkit (DAT). DAT pairs your phone app with Meta Ray-Ban glasses over an external-accessory channel and hands the auth callback back to your app through a **custom URL scheme**. iOS gates every one of those mechanisms behind an `Info.plist` declaration, so the keys below are required for the SDK to reach the glasses through DAT. **Voice apps need almost none of this.** A voice-only app reaches its microphone and speaker through the phone's own Bluetooth audio routing, never through DAT — so it needs no `MWDAT` dict, no Meta App ID, no URL scheme, and none of the accessory keys. It needs the privacy strings its capabilities imply (`NSMicrophoneUsageDescription`, `NSSpeechRecognitionUsageDescription`) and, for background listening, `UIBackgroundModes: ["audio"]`. It also needs the two Extentos keys below — `EXTENTOS_APP_ID` and `EXTENTOSProjectKey` — which are not Meta keys: the project key authenticates your app to the managed gateway in **every** environment, App Store builds included, and `generateConnectionModule` is the only thing that mints one. An assistant app without it fails with `AssistantError.NoApiKey`. Everything else on this page is the **camera/display** set. See [choose your path](/docs/getting-started/choose-your-path). > **You don't have to add these by hand.** `generateConnectionModule(platform: "ios")` ([agent-driven path](/docs/getting-started/with-agent)) emits a **complete, ready-to-use `Info.plist` as a generated file** — you apply it verbatim (like the bootstrap), and the *same* file works on the simulator and on real glasses. (iOS has no manifest merge, so the SDK package can't supply these keys itself — the scaffold owns the file instead.) This page documents what it writes and why, so you can audit the result or wire an app manually. > > The `Info.plist` requirements below are stable and verified against the current source. Package install: [Install](/docs/sdk/ios/install). ## Required keys | Key | Purpose | |---|---| | `MWDAT` (dict) | Meta Wearables DAT configuration. Holds the DAT integration settings the toolkit reads at startup. | | `MWDAT` → `DAMEnabled` | **Set explicitly in every app** — `true` only if your capability footprint includes Display. Meta's default when the key is *missing* is `true`, and DAM runs an on-glasses DWA readiness probe on DAT 0.8 that **reaps the device session ~10 s after start** on glasses without the DWA app (every current non-display model) — so a camera/audio app with the key missing or `true` loses its session on real hardware. `generateConnectionModule` emits the correct value from your declared capabilities. | | `CFBundleURLTypes` | Registers your app's **custom URL scheme** so DAT can deliver the auth callback after the user authorizes the glasses. This is a custom scheme, **not** a universal link. | | `LSApplicationQueriesSchemes` | Must include `fb-viewapp` so the app can hand off to the Meta View / AI app during pairing. | | `UISupportedExternalAccessoryProtocols` | Must include `com.meta.ar.wearable` — the External Accessory protocol the glasses speak. Without it iOS won't surface the device to the app. | | `UIBackgroundModes` | `bluetooth-central`, `bluetooth-peripheral`, and `external-accessory` — keeps the accessory session alive when the app isn't foregrounded. **Add `audio` as well if your assistant must keep listening while the app is backgrounded** — the scaffold does not emit it, because it depends on your app's behaviour rather than its capabilities. A voice-only app needs `audio` and none of the other three. | | `NSCameraUsageDescription` | Camera privacy string — required for photo / video / frame capture. | | `NSMicrophoneUsageDescription` | Microphone privacy string — required for audio capture and voice. | | `NSBluetoothAlwaysUsageDescription` | Bluetooth privacy string — the glasses connect over Bluetooth. | | `NSSpeechRecognitionUsageDescription` | Speech-recognition privacy string — required for on-device transcription via `SFSpeechRecognizer`. See [`requestSpeechRecognitionAuthorization`](/docs/sdk/ios/initialization). | | `EXTENTOS_APP_ID` | The Extentos project identity — the generated bootstrap reads it into `config.appId`. Decoupled from the store bundle id, so Android and iOS apps whose store ids differ can share one Extentos project. | | `EXTENTOSProjectKey` | The account-bound project key — the SDK's gateway bearer whenever there is no simulator session, in every environment including App Store builds. Minted only by `generateConnectionModule`. | | `NSAppTransportSecurity` | A loopback-only exception (`localhost` / `127.0.0.1`) so the debug auto-bind probe can reach the MCP localhost bridge over HTTP. Release builds never hit the probe. | ## Why each one **`MWDAT` and the custom URL scheme (`CFBundleURLTypes`).** Pairing leaves your app, authorizes in Meta's app, and returns. DAT delivers that return through your registered custom scheme — your app catches it in `.onOpenURL` and forwards it via [`glasses.handleUrl(_:)`](/docs/sdk/ios/initialization). A universal link will **not** work here; DAT expects a custom scheme. **`LSApplicationQueriesSchemes: fb-viewapp`.** iOS blocks `canOpenURL` / cross-app handoff to schemes you haven't declared. Pairing needs to reach the Meta app, so `fb-viewapp` must be listed. **`UISupportedExternalAccessoryProtocols: com.meta.ar.wearable`.** The glasses are an MFi external accessory. iOS only exposes accessories whose protocol string the app has declared — omit this and the device is invisible to the SDK. **`UIBackgroundModes`.** A live glasses session (an open accessory channel over Bluetooth) has to survive the app backgrounding. The scaffold declares `bluetooth-central`, `bluetooth-peripheral`, and `external-accessory` so the connection doesn't drop when the user looks away from the phone. **Privacy strings.** iOS hard-crashes an app that touches the camera, microphone, Bluetooth, or speech recognition without the matching `NS…UsageDescription`. Each Extentos capability maps to one: - Camera capture / video / frames → `NSCameraUsageDescription` - Audio capture, `speak`, voice → `NSMicrophoneUsageDescription` - Any glasses connection → `NSBluetoothAlwaysUsageDescription` - `glasses.audio.transcriptions()` (on-device STT) → `NSSpeechRecognitionUsageDescription` Write copy that names *your* feature, e.g. *"Used to capture photos from your Meta Ray-Ban glasses."* The reviewer and the user both read these. ## See also - [Permissions model](/docs/concepts/permissions) — the cross-platform view of what each capability needs. - [Initialization](/docs/sdk/ios/initialization) — wiring the `.onOpenURL` auth handoff and requesting speech authorization. - [Meta Ray-Ban vendor notes](/docs/vendors/meta) — the toolkit's real capability boundaries. --- source: https://extentos.com/docs/sdk/ios/initialization --- --- title: Initialization (iOS) description: Initialize the Extentos iOS SDK with Extentos.create(config:), forward the Meta DAT auth callback via .onOpenURL and handleUrl, request speech authorization, and reach the typed sub-clients. type: guide platform: ios vendor: meta related: - /docs/sdk/ios/info-plist - /docs/sdk/ios/threading - /docs/sdk/ios/lifecycle - /docs/getting-started/with-agent --- > The symbols below are verified against the current iOS source. The [agent-driven path](/docs/getting-started/with-agent) — `generateConnectionModule(platform: "ios")` — emits this wiring for you; package install: [Install](/docs/sdk/ios/install). ## Create the instance The entry point is the `Extentos` enum. Call `Extentos.create(config:)` with an [`ExtentosConfig`](/docs/reference/ios-api), or `Extentos.default()` for defaults. Both return an `ExtentosGlasses` — the root object that owns every capability sub-client. ```swift import GlassesCore let glasses = Extentos.create(config: ExtentosConfig( appId: "com.example.myapp", transport: .auto, usedCapabilities: [.microphone, .speaker], )) ``` **Where `appId` comes from.** The SDK reads only `config.appId`; it never reads Info.plist itself. The resolution order is: 1. `config.appId`, if you pass it — as above. 2. Otherwise whatever your bootstrap supplies. The **generated** `ExtentosBootstrap` reads the `EXTENTOS_APP_ID` Info.plist key into `config.appId` for you, which is why that key is listed on the [Info.plist page](/docs/sdk/ios/info-plist). If you write your own bootstrap, that read is yours to make. 3. Otherwise `Bundle.main.bundleIdentifier`. Step 3 is the one to watch: leaving `appId` unset **does not fail**, it silently adopts your store bundle id. That is harmless when the two match and quietly wrong when they don't — dashboard Agent settings, sounds and analytics are keyed to the id you minted against, so they simply never reach the app, and a cross-platform app whose store ids differ splits into two dashboard projects. `usedCapabilities` is not decoration on iOS — it picks the transport (see the callout below). Declare your real footprint: `[.microphone, .speaker]` for a voice app, plus `.camera` / `.display` if you use them. Leave `debug` at its default unless you are driving the browser-simulator dev loop; `debug: true` opens a pending simulator socket and waits for the MCP bridge to bind it. `transport: .auto` walks these rules in order, stopping at the first match: | # | Condition | Result | |---|---|---| | 1 | `config.debug` **and** a session URL baked in by the scaffold | `.browserSim` | | 2 | `config.debug` **and** `EXTENTOS_SESSION_URL` in the environment | `.browserSim` | | 3 | `hasBondedMetaDevice` returns true | `.realMeta` | | 4 | `config.debug` (no URL, no bonded device) | `.browserSim` in pending mode — waits for the MCP localhost bridge to bind it | | 5a | `usedCapabilities` is non-empty and contains neither `.camera` nor `.display` | **`.systemAudio`** — the vendorless audio baseline | | 5b | anything else (including an empty `usedCapabilities`) | `.realMeta`; `connect()` surfaces `NoEligibleDevice` when none are paired | **Every `debug` above is the `ExtentosConfig.debug` field, not your Xcode build configuration.** It defaults to `false`, so a normal Debug-configuration run of a voice app skips rungs 1, 2 and 4 and lands on **5a** — the audio baseline — exactly like a release build. Set `debug: true` only when you are driving the browser-simulator dev loop; on a phone with no bridge running it parks on rung 4 and waits. It never falls back to a simulator on its own. **On iOS, `usedCapabilities` decides your transport — it is not just connection-page decoration.** iOS has no API for enumerating bonded Bluetooth devices, so unlike Android there is no way to detect "are the glasses actually here?"; the declared footprint is the only honest signal. **An empty `usedCapabilities` resolves to real glasses**, which is right for a camera app but wrong for a voice app — on a phone with no glasses paired, `connect()` returns `NoEligibleDevice` instead of falling back to the phone's own mic and speaker. A voice-only app must declare its footprint explicitly: ```swift ExtentosConfig(usedCapabilities: [.microphone, .speaker]) ``` The empty default is deliberate — silently downgrading an app that declared nothing would strip camera support from existing integrations. See [runtime internals](/docs/sdk/ios/runtime-internals). The simulator runs the *same* SDK code as hardware; see [transport vs app simulation](/docs/concepts/transport-vs-app). Hold a single instance for the app's lifetime. > `Extentos.create(config:)` is the customer entry point. `Wearables.configure(...)` is an internal Meta DAT type — you never call it directly. ## Forward the auth callback (`.onOpenURL`) Pairing with Meta Ray-Ban hands control to Meta's app and returns through your app's [custom URL scheme](/docs/sdk/ios/info-plist). Catch that callback in SwiftUI's `.onOpenURL` and forward it to `glasses.handleUrl(_:)`, which returns `true` when the SDK consumed the URL: ```swift import SwiftUI @main struct MyApp: App { let glasses = Extentos.default() var body: some Scene { WindowGroup { ContentView() .onOpenURL { url in Task { _ = await glasses.handleUrl(url) } } } } } ``` `handleUrl(_:)` is `async` and returns a `Bool`. The simulator transports return `false` unconditionally — only the real-Meta transport consumes the registration callback. The matching `Info.plist` `CFBundleURLTypes` entry must register the scheme, or iOS never delivers the URL. ## Request speech authorization (optional) On-device transcription (`glasses.audio.transcriptions()`) uses `SFSpeechRecognizer`, which needs `NSSpeechRecognitionUsageDescription` in the [`Info.plist`](/docs/sdk/ios/info-plist) and a one-time user grant. The SDK requests it on first use, but you can surface the prompt at a controlled moment — e.g. during onboarding: ```swift let granted = await Extentos.requestSpeechRecognitionAuthorization() ``` It returns `true` when authorized. **The microphone is a separate grant, and it's yours.** Speech-recognition authorization does not cover audio capture. Request both before starting the assistant — iOS will otherwise prompt mid-session, or capture silence: ```swift let mic = await AVAudioApplication.requestRecordPermission() let speech = await Extentos.requestSpeechRecognitionAuthorization() guard mic, speech else { return } // surface your own explanation try await glasses.assistant.start(provider: .managed()) { … } ``` Both need their Info.plist strings (`NSMicrophoneUsageDescription`, `NSSpeechRecognitionUsageDescription`) or the app is terminated on first use — see [Info.plist](/docs/sdk/ios/info-plist). This is the iOS counterpart of the Android sequencing in the [voice-assistant guide](/docs/guides/voice-assistant#1-bootstrap); iOS needs no foreground service, only the `audio` background mode. ## The sub-client surface `ExtentosGlasses` exposes typed capability sub-clients. Reach a primitive through its client — e.g. `glasses.camera.capturePhoto()`, `glasses.audio.transcriptions()`, `glasses.audio.speak(...)`. | Sub-client | Always present | What it covers | |---|---|---| | `connection` | yes | Open / close the glasses connection; connection-state stream. | | `camera` | yes | Photo capture, video capture, frame streams. | | `audio` | yes | Transcription, audio capture, `speak`, audio chunks. | | `runtime` | yes | The runtime event stream — `toggleChanged`, `coexistenceWarning`, `log`, `unrecognizedUtterance`, and `assistant` lifecycle events. | | `toggles` | yes | The runtime capability gates. | | `voice` | yes | Convenience surface over `audio.transcriptions()` that also announces your wake phrase to the simulator and connection page. | | `telemetry` | yes | Telemetry events. | | `observability` | yes | Wraps BYOK AI calls so they appear under the event log's `ai` chip. | | `assistant` | yes | The voice-AI assistant runtime — see below. | | `display` | yes | Declarative display trees for display-capable glasses (`show` / `clear`); `display.isAvailable` gates, and calls on no-display devices are silent no-ops. | | `capabilities` | yes | `DeviceCapabilitySet` — flat yes/no hardware facts of the connected device; `display` flips live with the device. A value property, not a method client. | | `conversation` | removed | Phase-3 conversation runtime — **removed** (Android in 1.4.0; never shipped in the published iOS package). Historical only. | | `ai` | removed | Phase-3 BYOK LLM completion surface — **removed** alongside `conversation`. Historical only. | The root object also exposes `usedCapabilities` (the declared capability footprint from `ExtentosConfig`) and a `connect()` convenience — what the generated bootstrap calls, equivalent to `connection.connect()`. ### Assistant The `assistant` client is the voice-AI runtime. It routes through the Extentos managed gateway on both platforms — no API key in code. Start a session with the sugar form (builds the config inline), or use the raw form `createSession(config:)` + `session.start()`: ```swift let session = try await glasses.assistant.start(provider: .managed()) { $0.instructions = "You are a helpful assistant on smart glasses." } ``` An `AssistantSession` has a wake/sleep cycle: `start()` lands it Dormant, `wake()` opens the realtime connection, `sleep()` closes it but keeps the session (history, tools, hooks) ready for the next wake, `stop()` tears it down for good. See the [iOS API reference](/docs/reference/ios-api) for the full session surface and the [agent code examples](/docs/getting-started/with-agent) for canonical flows. ## Next - [Info.plist setup](/docs/sdk/ios/info-plist) — the keys this wiring depends on. - [Threading model](/docs/sdk/ios/threading) — how to call these `async` clients safely. - [Lifecycle](/docs/sdk/ios/lifecycle) — tearing the instance down with `shutdown()`. --- source: https://extentos.com/docs/sdk/ios/install --- --- title: Install (iOS) description: Add the Extentos iOS SDK with Swift Package Manager from github.com/extentos/swift-glasses — Xcode, Package.swift, or xcodegen. type: guide platform: ios related: - /docs/getting-started/ios - /docs/getting-started/with-agent - /docs/sdk/ios/initialization - /docs/sdk/ios/info-plist - /docs/sdk/android/install --- The Extentos iOS SDK is distributed as a Swift package from [`github.com/extentos/swift-glasses`](https://github.com/extentos/swift-glasses). The current version is `2.0.0` — the Android and iOS SDKs version in lockstep on the same shared core. ## Add the package **Xcode** — *File → Add Package Dependencies…*, paste `https://github.com/extentos/swift-glasses`, and set the dependency rule to **Up to Next Major Version** starting at `2.0.0`. **`Package.swift`:** ```swift dependencies: [ .package(url: "https://github.com/extentos/swift-glasses", from: "2.8.0"), ], targets: [ .target( name: "MyApp", dependencies: [ .product(name: "GlassesCore", package: "swift-glasses"), .product(name: "GlassesUI", package: "swift-glasses"), ] ), ] ``` **xcodegen (`project.yml`):** ```yaml packages: Extentos: url: https://github.com/extentos/swift-glasses from: 2.8.0 ``` Then link the products your target uses: | Product | What it is | |---|---| | `GlassesCore` | The SDK — connection, camera, audio, voice, assistant, toggles, telemetry | | `GlassesUI` | The drop-in SwiftUI connection page (`ExtentosConnectionPage`) and theming | | `GlassesLocal` | On-device text models for the [local tier](/docs/concepts/local-models). Optional — but its MLX dependency is what sets the package's **iOS 17** floor for every product | | `GlassesLocalVoice` | On-device speech (Kokoro) for the local tier. Optional | | `GlassesDebug` | Scaffolding today — the debug-console API surface is reserved; the current panel is a placeholder | | `GlassesLifecycle` | Scaffolding today — the `extentosListening` scene-phase modifier is a passthrough; API surface reserved | | `GlassesTesting` | Scaffolding today — `FakeExtentosGlasses` doesn't implement the sub-clients yet; stub the protocols yourself for unit tests | The compiled Rust core (`extentos_coreFFI.xcframework`) is a binary target that SPM resolves automatically from the GitHub Release by URL and checksum — device arm64, a universal arm64/x86_64 simulator slice, and macOS arm64. There is nothing to download or verify by hand. ## Agent-driven install The [agent-driven flow](/docs/getting-started/with-agent) remains the fastest path: install [`@extentos/mcp-server`](https://www.npmjs.com/package/@extentos/mcp-server) in your AI coding agent and ask it to add smart-glasses capabilities to your iOS app — `generateConnectionModule(platform: "ios")` emits the bootstrap, the [`Info.plist` keys](/docs/sdk/ios/info-plist), and the `ExtentosConnectionPage` placement. The scaffold's dependency step emits the published package coordinates above. ## Requirements | Requirement | Value | |---|---| | Deployment target | **iOS 17.0+**, whichever products you link. SwiftPM applies a package's platform floor package-wide, so `GlassesLocal`'s MLX dependency raises it for `GlassesCore` too. An iOS 16 target fails to resolve with *"the package product 'GlassesCore' requires minimum platform version 17.0"*. | | Swift toolchain | Swift 6 (`swift-tools-version: 6.0`) | | Language mode | Swift 5 (`.swiftLanguageMode(.v5)` on `GlassesCore`) | | Meta DAT (iOS) | `meta-wearables-dat-ios` from `0.8.0` (resolved transitively). Unlike Android, iOS does **not** split the vendor SDK into a separate product — DAT is a dependency of `GlassesCore` on every iOS app. A voice app links it but never initializes it, so it still needs no Meta account, no credentials, and none of the DAT Info.plist keys. | After install, continue with the [`Info.plist` setup](/docs/sdk/ios/info-plist) and [initialization](/docs/sdk/ios/initialization) — or follow the [iOS quickstart](/docs/getting-started/ios) end to end. --- source: https://extentos.com/docs/sdk/ios/lifecycle --- --- title: Lifecycle (iOS) description: The Extentos iOS SDK lifecycle — create one instance for the app's lifetime, forward the auth callback, and tear down with the async shutdown(). type: guide platform: ios related: - /docs/sdk/ios/initialization - /docs/sdk/ios/threading - /docs/sdk/ios/runtime-internals --- > The lifecycle shape below is verified against the current iOS source. Package install: [Install](/docs/sdk/ios/install). ## The shape The Extentos iOS instance has a simple lifecycle, owned by your app — there's no separate framework lifecycle to register with: 1. **Create once.** Call `Extentos.create(config:)` (or `Extentos.default()`) at app start and hold the `ExtentosGlasses` for the app's lifetime — typically a `let` on your SwiftUI `App` or a long-lived view model. See [Initialization](/docs/sdk/ios/initialization). 2. **Forward the auth callback.** Route the Meta DAT custom-scheme callback through `.onOpenURL` → `glasses.handleUrl(_:)` so pairing can complete. 3. **Open the connection.** Use the `connection` sub-client (or the prebuilt `ExtentosConnectionPage` from `GlassesUI`) to connect to the glasses, and observe its connection-state stream. 4. **Tear down.** Call `await glasses.shutdown()` when you're done. It's `async` — `await` it so cleanup completes before you release the instance. ```swift func tearDown() async { await glasses.shutdown() } ``` ## Backgrounding A live glasses session — an open accessory channel and routed audio — must survive the app moving to the background. That's why the [`Info.plist`](/docs/sdk/ios/info-plist) declares `UIBackgroundModes`: without it iOS would suspend the app and drop the connection the moment the user looks away from the phone. You don't manually pause and resume the SDK around backgrounding; keep the instance alive and let the connection-state stream report any drop. There are no additional app-state hooks to wire — keep the instance alive across scene phases and observe the connection-state stream. For the behavioral model that both platforms share, see [transport vs app simulation](/docs/concepts/transport-vs-app); the Android counterpart is [Android lifecycle](/docs/sdk/android/lifecycle). --- source: https://extentos.com/docs/sdk/ios/runtime-internals --- --- title: Runtime internals (iOS) description: How the Extentos iOS Swift shell wraps the shared Rust core via uniffi, where the DAT-iOS transport bridge sits, and what's platform-bound vs shared. type: concept platform: ios related: - /docs/concepts/architecture - /docs/sdk/ios/initialization - /docs/concepts/transport-vs-app --- > The structure below is verified against the current iOS source. Package install: [Install](/docs/sdk/ios/install). ## Shell over a shared core The Extentos iOS SDK is a **thin Swift shell over a shared Rust core**. The same `extentos-core` powers both platforms: vendor-agnostic logic — transport selection, state machines, data shapes, event vocabularies, protocol handling, display strings — lives once in the core and is exposed to Swift through **uniffi**-generated bindings. The Swift shell keeps only the platform-bound parts: iOS OS APIs, the SwiftUI connection page, audio device wiring, and the Meta DAT bridge. Concretely, `GlassesCore` compiles the uniffi-generated Swift bindings together with the compiled Rust, shipped as a binary `xcframework` (`extentos_coreFFI`). That's why a fix in the core lands on iOS and Android at once, and why the generated bindings are never hand-edited — they're regenerated from the Rust source. ## The transport bridge `Extentos.create(config:)` resolves one of four transports behind a single abstraction, all sharing the core's selection logic: - **System audio** — the vendorless baseline: microphone and speaker over the phone's own Bluetooth routing (`AVAudioSession` `.playAndRecord` / `.voiceChat`), with no vendor SDK in the path. Serves the complete agent runtime on any hands-free audio route — smart glasses, ordinary earbuds, or the phone itself. No camera, no display. - **Real Meta** — production glasses via the Meta DAT iOS toolkit (`meta-wearables-dat-ios`, floor `0.8.0`). This is the iOS-specific bridge: it adapts DAT's accessory channel and the custom-URL-scheme auth callback (forwarded through [`handleUrl(_:)`](/docs/sdk/ios/initialization)) to the core's transport interface. - **Browser simulator** — a WebSocket to the Extentos backend; the hardware surrogate runs in a browser tab. This is the dev loop. - **Local simulator** — Extentos's own in-memory deterministic transport (no DAT SDK, no network), for a fast inner loop. `transport: .auto` picks among them: the injected build-time / env-var session URL (debug builds) → a bonded Meta device → a pending socket the MCP localhost bridge binds (debug builds without a URL or bonded device) → `.systemAudio` when the app declared capabilities and none of them are camera or display (a voice app) → otherwise real Meta glasses. A release build always targets real hardware or the audio baseline — it never silently falls back to a simulator. An app that declares NO capabilities keeps the real-Meta default rather than being downgraded out of camera support. The simulator paths run the **same** SDK code as hardware with only the transport and I/O swapped — a simulator failure is a real failure. See [transport vs app simulation](/docs/concepts/transport-vs-app). **This differs from Android on purpose.** Android resolves the audio baseline for any real device with no vendor claim, because it can enumerate bonded Bluetooth devices and therefore knows whether glasses are present. iOS exposes no equivalent API, so the honest signal here is what the app *declared*: a non-empty `usedCapabilities` with no camera and no display is a voice app and takes the baseline. An app that declares **nothing** keeps the real-Meta default rather than being silently downgraded out of camera support. See [Android runtime internals](/docs/sdk/android/runtime-internals). ## Where the shell stops The boundary is deliberate: anything cross-platform belongs in the Rust core, not duplicated in Swift, so the two platforms can't drift. The iOS shell exists for the things only iOS can do — `SFSpeechRecognizer` for on-device STT, `AVFoundation` audio routing, the External Accessory session, SwiftUI views, and the `.onOpenURL` auth handoff. There are no customer-facing extension points beyond the public sub-client API; customize behavior through [`ExtentosConfig`](/docs/sdk/ios/initialization) and your own handler code, not by reaching into the runtime. For the cross-platform architecture in full, see [architecture](/docs/concepts/architecture). --- source: https://extentos.com/docs/sdk/ios/threading --- --- title: Threading model (iOS) description: How the Extentos iOS SDK concurrency works — ExtentosGlasses is Sendable, every sub-client call is async, MainActor guidance for UI, and why the package builds in Swift 5 language mode. type: guide platform: ios related: - /docs/sdk/ios/initialization - /docs/sdk/ios/lifecycle - /docs/sdk/android/threading --- > The concurrency contract below is verified against the current iOS source. Package install: [Install](/docs/sdk/ios/install). ## `Sendable` instance, `async` calls `ExtentosGlasses` is `Sendable` — you can hold one instance and call it from any task or actor without a data race. There's no requirement to confine it to a particular thread. Create it once (see [Initialization](/docs/sdk/ios/initialization)) and share it. Every capability call is `async`. Reach a primitive through its sub-client and `await` it — lifecycle and capture calls return an `ExtentosResult` to pattern-match (or discard explicitly): ```swift let result = await glasses.camera.capturePhoto() // ExtentosResult if case .success(let photo) = result { /* use photo */ } _ = await glasses.audio.speak("Done.") // ExtentosResult ``` `handleUrl(_:)` and `Extentos.requestSpeechRecognitionAuthorization()` are likewise `async`. The SDK runs its work off the main thread; you don't manually hop queues to call it. ## MainActor and UI Because the SDK is `Sendable` and its methods are `async`, the idiomatic pattern is to call it from a `Task` and apply results to your UI on the `MainActor`: ```swift @MainActor final class CaptureViewModel: ObservableObject { @Published var lastPhoto: Photo? let glasses: ExtentosGlasses func capture() { Task { // capturePhoto() returns ExtentosResult — pattern-match it, don't assign directly if case .success(let photo) = await glasses.camera.capturePhoto() { // back on the MainActor — safe to mutate @Published state self.lastPhoto = photo } } } } ``` SwiftUI view modifiers like `.onOpenURL` already run on the main actor; spawning a `Task { await glasses.handleUrl(url) }` inside one (as in [Initialization](/docs/sdk/ios/initialization)) moves the SDK work off the main thread while keeping the modifier's contract. ## Streams and cancellation Continuous primitives (transcription, frame streams, connection-state updates) are surfaced as async sequences. Iterate them inside a `Task` and cancel by cancelling that task — structured concurrency tears the iteration down. Store long-lived `Task` handles on your view model and cancel them in `deinit` or when the view disappears, so a stream doesn't outlive the screen that started it. ## Swift 6 toolchain, Swift 5 language mode The package declares `swift-tools-version: 6.0`, so it builds with a **Swift 6 toolchain** — but it compiles in the **Swift 5 language mode** (`.swiftLanguageMode(.v5)` on the `GlassesCore` target). The reason: the uniffi-generated Swift bindings the core ships use a global `var` that Swift 6's *strict concurrency checking* rejects. Until those bindings are isolated or regenerated clean, the language mode stays at 5. What this means for you: - "Swift 6" unqualified overstates it — say **Swift 6 toolchain, Swift 5 language mode**. - The SDK's public API is unaffected — the `Sendable` / `async` contract above holds either way. - Your own app can adopt Swift 6 strict concurrency independently; the language-mode setting is scoped to the SDK's own target, not yours. This is a tracked SDK-internal follow-up, not something you need to work around. See [Android threading](/docs/sdk/android/threading) for the Kotlin-coroutines counterpart. --- source: https://extentos.com/docs/troubleshooting --- --- title: Troubleshooting description: Symptom-shaped fixes. Find your symptom, get the cause and the fix, with cross-links to error reference. type: troubleshooting platform: all related: - /docs/reference/errors --- Symptom-shaped pages. Each one starts with what you observed, then walks through likely causes. - [Glasses won't connect](/docs/troubleshooting/glasses-wont-connect) - [Voice trigger not firing](/docs/troubleshooting/voice-trigger-not-firing) - [Photo capture fails](/docs/troubleshooting/photo-capture-fails) - [Background session killed](/docs/troubleshooting/background-session-killed) Looking for a specific error code? See [Reference → Error codes](/docs/reference/errors). --- source: https://extentos.com/docs/troubleshooting/background-session-killed --- --- title: Background session killed description: Fix a Meta Ray-Ban session that dies when your app backgrounds — a missing Android foreground service, missing iOS background modes, or an OS low-memory kill. type: troubleshooting platform: all related: - /docs/reference/errors - /docs/guides/background-sessions - /docs/troubleshooting/glasses-wont-connect - /docs/guides/handle-disconnects --- Your app works in the foreground, but the glasses session dies the moment the app is backgrounded — transcripts stop, the assistant goes quiet, capture fails. On a phone, a glasses session is a long-running Bluetooth + capture workload, and both Android and iOS aggressively suspend backgrounded apps. Keeping a session alive in the background is **opt-in** and platform-specific. ## Symptoms - Voice / transcription / the assistant stops the instant you switch apps or the screen locks - A backgrounded capture returns `NotConnected` or silently produces nothing - Android logcat shows a `SecurityException` at `startForeground(...)` - The session comes back when you foreground the app, then dies again on background ## Android — you need a foreground service A backgrounded Android app can't hold the mic open without a **foreground service**. Extentos ships one (`GlassesForegroundService`, `foregroundServiceType="microphone"`), but you have to wire three things: 1. **Declare the permissions** (app-owned): `FOREGROUND_SERVICE` and, on Android 14+, `FOREGROUND_SERVICE_MICROPHONE`. `getPermissions` derives these from your continuous-capture capabilities. A missing `FOREGROUND_SERVICE_MICROPHONE` on 14+ is the most common cause of the `SecurityException`. 2. **Grant `RECORD_AUDIO` at runtime *before* starting the service.** Starting a `microphone`-type FGS without the grant crashes at service-create time on Android 14+. 3. **Start it from an Activity, not `Application.onCreate`** — `GlassesForegroundService.start(context)` from your permission-grant callback; stop it with `.stop(context)` when done. Camera-in-the-background needs a **`camera`-type** service you declare yourself (subclass `GlassesForegroundService` + a ``) — see [Background sessions](/docs/guides/background-sessions). ## iOS — you need the right background modes iOS keeps a backgrounded app alive only within declared **`UIBackgroundModes`**. The glasses connection needs `bluetooth-central`, `bluetooth-peripheral`, and `external-accessory`; add `audio` when you stream the mic. `generateConnectionModule(platform: "ios")` emits these and `validateIntegration` checks them. iOS background execution is more constrained than an Android foreground service — validate on a real device before relying on it. ## OS-level kills (the honest limit) Even wired correctly, the OS can reclaim a backgrounded app under memory pressure or aggressive battery optimization (some Android OEMs are especially aggressive). This isn't an Extentos bug — it's the platform. Design for it: - Treat a background disconnect as expected: branch on `connection.state` → `Disconnected` and re-establish on foreground (see [Handle disconnects](/docs/guides/handle-disconnects)). - Keep the foreground-service notification meaningful so the user — and the OS — can see the session is active. - Don't assume indefinite background life; persist anything important as you go. ## Confirm the fix - `validateIntegration` flags a missing iOS `UIBackgroundModes` entry or Android FGS permission before runtime. - In a simulator session, `getEventLog({ sessionId, filter: "lifecycle" })` shows the `Disconnected` cause when the session drops. ## Related - [Background sessions](/docs/guides/background-sessions) — the full setup guide - [Handle disconnects](/docs/guides/handle-disconnects) — reconnect cleanly when a session drops --- source: https://extentos.com/docs/troubleshooting/glasses-wont-connect --- --- title: Glasses won't connect description: Fix common first-connection failures on Meta Ray-Ban — no pairing dialog, session won't start, the DAT auth callback never fires, or an immediate disconnect. type: troubleshooting platform: all vendor: meta related: - /docs/reference/errors - /docs/concepts/sessions - /docs/troubleshooting - /docs/sdk/android/manifest - /docs/guides/handle-disconnects --- Your app calls `glasses.connection.connect()` (or the user taps connect on `ExtentosConnectionPage`) and nothing usable happens — no pairing prompt, a session that never reaches `Active`, an auth callback that never returns, or a connection that drops the instant it opens. Connection failures surface as a typed [`ConnectError`](/docs/reference/errors#connecterror) from the `connect()` / `reconnect()` result, or as a `Disconnected` state carrying a [`DisconnectCause`](/docs/reference/errors) — pattern-match those rather than guessing. This page maps each symptom to its cause and fix. > Kotlin leads the examples and is the source of truth; iOS exposes the same `ConnectError` / `GlassesState` types. ## Quick checklist (verify these first) 1. **Glasses are powered, charged, and in range.** A dead or out-of-range device surfaces as `ConnectError.NoDeviceAvailable`. Confirm the glasses are paired in the Meta AI app first — Extentos connects to an already-bonded device, it doesn't replace Meta's own pairing. 2. **Bluetooth permissions are granted at runtime.** Missing `BLUETOOTH_CONNECT` / `BLUETOOTH_SCAN` surfaces as `ConnectError.PermissionDenied`. The SDK requests `BLUETOOTH_CONNECT` at runtime, but your app must not have permanently denied it. 3. **A foreground Activity is reachable.** Meta registration (the pairing handoff) needs a live Activity — confirm your `activityProvider` closure returns the current top Activity, not `null`. See [Android lifecycle § Activity wiring](/docs/sdk/android/lifecycle). 4. **The Meta DAT auth callback URL scheme is declared.** The DAT auth round-trip returns through a **custom URL scheme**, not a universal/app link. If it's missing or misspelled, the callback never fires and connection hangs (see below). 5. **Run `validateIntegration`.** It catches the structural causes — missing Bluetooth permissions, an unwired bootstrap, an absent DAT configuration — before you ever hit the runtime failure. 6. **Read the event log.** `getEventLog(filter: 'lifecycle')` shows the connection state walk (`Registered` → `DeviceDiscovered` → `Connecting` → `Active`) so you can see exactly where it stalls; `getEventLog(filter: 'errors')` surfaces the typed failure. ## Pairing dialog never appears The Meta registration / pairing UI is launched against a **foreground Activity**. If `connect()` resolves immediately to `ConnectError.RegistrationRequired` or nothing visibly happens: - Your `activityProvider: () -> Activity?` returned `null` — the SDK had no Activity to launch registration from. Wire it to an `ActivityLifecycleCallbacks`-tracked reference to the current top Activity. See [Android lifecycle](/docs/sdk/android/lifecycle). - The install hasn't registered yet. **There is no registration API for you to call** — the SDK drives Meta's registration handoff itself, from inside `connect()` (and therefore inside `assistant.start(...)`), using your `activityProvider` to present the companion-app consent screen. So `RegistrationRequired` almost always means the handoff couldn't run rather than that you forgot to start it: check that `activityProvider` returns a live Activity, that the glasses are awake and connected in the Meta AI app, and that your package name + signing SHA-256 are registered with Meta. The packaged `ExtentosConnectionPage` supplies the Activity for you; an app without one supplies it through `activityProvider`. - **You aren't enrolled in your own release channel.** Meta only lets an account register your integration if it's a member of a channel you've assigned a version to — and that includes the account on your own test glasses. Correct App ID, correct signing SHA, and a still-failing registration usually means this. Check it before chasing `activityProvider`. Needs firmware v125+; see [release channels](/docs/vendors/meta). - You're on the **browser simulator**, where there is no Bluetooth pairing at all — the surrogate runs in a browser tab. A missing pairing dialog in sim is expected; you attach via the session URL or auto-bind, not a pairing prompt. ## Pairing succeeds but the session won't start The connection state reaches `Connecting` (or `DeviceDiscovered`) and stalls, never flipping to `Active`: - **`ConnectError.PairingFailed`** — the handshake with the device failed. Power-cycle the glasses, confirm they're bonded in the Meta AI app, and retry. This is usually a transient DAT handshake failure. - **`TransportError.HardwareUnavailable`** (carried inside `DisconnectCause.TransportFailure`) — the transport could not reach the hardware. On real glasses this is a Bluetooth/DAT-layer problem; in the simulator it means the browser surrogate denied webcam/mic permission or closed its tab. - A `SchemaMismatch` transport error means the SDK and backend protocol versions diverged — update to a matching SDK build. > **Sim ≠ hardware here.** A session that starts cleanly in the browser simulator does **not** guarantee real-hardware pairing succeeds — the simulator swaps the transport for a WebSocket and never exercises Bluetooth bonding or the DAT handshake. Treat real-hardware connection as its own verification pass. ## DAT auth callback never fires Meta DAT completes its auth/registration round-trip by redirecting back into your app through a **custom URL scheme** (DAT expects a custom scheme, *not* a universal link). When it's misconfigured, the system browser/Meta app opens, the user authorizes, and control never returns — the connection sits forever in `Registered`/`Connecting`. - **iOS:** confirm `CFBundleURLTypes` declares your custom scheme and `LSApplicationQueriesSchemes` includes `fb-viewapp`, alongside the `MWDAT` dict. `generateConnectionModule(platform: "ios")` emits the full set — re-run it if you hand-edited Info.plist and the callback stopped working. - **Android:** confirm the callback intent-filter scheme matches what you configured at DAT setup time, and that no other app claims the same scheme. - The scheme registered in Meta Developer Center must match the one in your app exactly — a mismatch silently drops the callback. ## Glasses connect but disconnect immediately The state reaches `Active` and then drops straight back to `Disconnected`. Branch on the `DisconnectCause`: - **`DeviceDroppedConnection`** — Bluetooth fell over right after connecting (interference, the glasses going back to sleep, range). Retryable; reconnect with backoff. See [handle disconnects](/docs/guides/handle-disconnects). - **A fold or overheat** — surfaces today as `DeviceDroppedConnection` (above); the dedicated `ThermalCritical` / `HingesClosed` disconnect causes are declared but not yet produced. Wait for the condition to clear, then reconnect. - **`TransportFailure`** — inspect the carried `TransportError` (`WebsocketClosed`, `Timeout`, `HardwareUnavailable`) for specifics. If a connection drops mid-operation, the in-flight call also returns its own typed error (e.g. `CaptureError.NotConnected`) — handle both the state observer and the per-call result, as covered in [handle disconnects](/docs/guides/handle-disconnects#in-flight-operations). ## Related - [Error reference](/docs/reference/errors) — every `ConnectError`, `CaptureError`, `AudioError`, and `TransportError` payload. - [Sessions](/docs/concepts/sessions) — the connection-state machine and what reaching `Active` requires. - [Android manifest setup](/docs/sdk/android/manifest) — the Bluetooth permissions and DAT entries the connection needs. - [Handle disconnects](/docs/guides/handle-disconnects) — recovering once a connection drops. - [Troubleshooting](/docs/troubleshooting) — other symptom-shaped fixes. --- source: https://extentos.com/docs/troubleshooting/photo-capture-fails --- --- title: Photo capture fails description: Fix capturePhoto() returning an error or no image on Meta Ray-Ban — CAMERA permission, thermal-critical, folded hinges, coexistence, and the camera toggle. type: troubleshooting platform: all vendor: meta related: - /docs/reference/errors - /docs/guides/photo-capture - /docs/troubleshooting/glasses-wont-connect - /docs/concepts/capabilities --- You call `glasses.camera.capturePhoto()` and get back `ExtentosResult.Err` instead of a `Photo` — or a result you can't render. Capture failures are a typed [`CaptureError`](/docs/reference/errors#captureerror); pattern-match the result (don't wrap it in `runCatching` — it won't catch a returned error). This page maps each `CaptureError` variant to its cause and fix. > Kotlin leads the examples and is the source of truth; iOS has the same `capturePhoto` surface with fewer bundled helpers (only `photo.loadImage()`). See [Capture a photo](/docs/guides/photo-capture). ## Quick checklist (verify these first) 1. **The Meta vendor module is on your classpath.** Camera lives in `com.extentos:glasses-meta`, not in `com.extentos:glasses`. If capture worked before an SDK upgrade and now fails, this is the first thing to check — the build succeeds without it and only capture breaks. Confirm with `glasses.transportChosen`. `REAL_META` means the vendor arm claimed; `SYSTEM_AUDIO` means it didn't — and the missing module is only one of three causes. The other two are that no glasses are bonded to the phone, and that `BLUETOOTH_CONNECT` wasn't granted before `ExtentosGlasses.create(...)` (the vendor arm reads the bonded list, so an ungranted permission looks identical to no glasses). If the dependency is present, check those two before re-adding it. The SDK also logs a warning at startup (`adb logcat -s Extentos:W`). 2. **`CAMERA` is declared and granted.** The library does **not** declare `CAMERA` for you — it's a capability permission your app owns. Add `` and request it at runtime. A missing grant surfaces as a capture failure (`PlatformError`) — the capture path has no dedicated `PermissionDenied`. See [Android manifest setup](/docs/sdk/android/manifest#what-you-add). 3. **The glasses are connected (`Active`) *and* report a camera.** A capture before the connection reaches `Active`, or after it drops, returns `CaptureError.NotConnected`. Gate on `glasses.connection.state` **together with** `glasses.capabilities.camera` — on the vendorless audio baseline the state is `Active` (the phone always has audio) while there is no camera at all, and capture there returns `CaptureError.PlatformError(code = "system_audio_no_camera")` rather than `NotConnected`. 4. **No camera toggle is off.** If the user (or your code) closed `camera_streaming_enabled`, capture returns `CaptureError.DisabledByUser(toggle = "camera_streaming_enabled")`. The same applies under `privacy_mode`, which kills all capture. Read `glasses.toggles.state` and surface the gate to the user. 5. **Nothing is racing the audio profile.** Another capture already in progress — most often an open `videoFrames()` stream, or a rapid second capture — can surface `CaptureError.CoexistenceBlocked`. See below. 6. **Read the event log.** `getEventLog(filter: 'camera')` shows the capture request and camera-layer events (the result event `capture_photo_returned` lands on the `custom` chip, or `errors` on failure); `getEventLog(filter: 'errors')` surfaces the typed failure (severity≥warn lands on the `errors` chip regardless of modality). ## Capture call returns an error Branch on the `CaptureError` variant — each has a distinct fix: | Variant | Cause | Fix | |---|---|---| | `PermissionDenied` | Declared, but the capture path doesn't emit it today — a missing `CAMERA` grant surfaces as `PlatformError` | Still declare `CAMERA` + request it at runtime; handle a missing grant via the `PlatformError` branch. | | `NotConnected` | The glasses connection isn't `Active` | Gate capture on `glasses.connection.state`; reconnect first. See [glasses won't connect](/docs/troubleshooting/glasses-wont-connect). | | `ThermalCritical` | Declared, not currently returned on the capture path — a thermal drop surfaces as `NotConnected` | Handle for forward-compat; treat like `NotConnected` today. | | `HingesClosed` | Declared, not currently returned on the capture path — a fold surfaces as `NotConnected` | Handle for forward-compat; treat like `NotConnected` today. | | `CoexistenceBlocked` | Another capture is already in progress (e.g. an open `videoFrames()` stream, or a rapid second capture) | See the section below. | | `DisabledByUser` | A user-facing toggle gates capture (`e.toggle` names which — e.g. `camera_streaming_enabled`, or `privacy_mode`) | Explain the toggle to the user; ask them to re-enable it via the connection page. | | `PlatformError` | A native-origin capture failure — structured `code` + `message` | Log `code`/`message`; inspect the event log entry at the catch site for the underlying native exception. | ```kotlin when (val r = glasses.camera.capturePhoto()) { is ExtentosResult.Ok -> render(r.value) is ExtentosResult.Err -> when (val e = r.error) { is CaptureError.PermissionDenied -> requestCameraPermission() is CaptureError.NotConnected -> promptReconnect() is CaptureError.ThermalCritical -> toast("Glasses too hot — let them cool") is CaptureError.HingesClosed -> toast("Open the glasses to take a photo") is CaptureError.CoexistenceBlocked -> toast("Busy — try again in a moment") is CaptureError.StreamPaused -> toast("Tap the right temple to resume the camera") is CaptureError.DisabledByUser -> explainToggle(e.toggle) is CaptureError.PlatformError -> log(e.code, e.message) } } ``` ## CoexistenceBlocked — another capture is in progress The DAT camera path returns `CaptureError.CoexistenceBlocked` when another capture is already in progress — most commonly a `capturePhoto()` fired while a `videoFrames()` stream is open, or two captures racing. (Meta Ray-Ban's Bluetooth audio path — HFP/SCO for capture, A2DP for playback — also coexists fragilely with the camera, so sequencing audio and capture remains good practice.) Fixes: - Don't hold an open `videoFrames()` stream while doing a one-shot photo unless you need both — stop the stream first. - Sequence audio and capture rather than overlapping them: finish `speak()` / `recordDiscrete()`, then capture. - The `audio_video_coexistence_policy` toggle (`prefer_video` by default) is a declared preference the library does **not** yet enforce — sequence audio and capture in your handler yourself. See the [capabilities](/docs/concepts/capabilities#toggles--runtime-gates-the-user-controls) toggle table and [audio streaming](/docs/guides/audio-streaming) for the full coexistence story. ## Capture succeeds but the image is empty or won't load You got an `Ok` `Photo`, but the bytes are missing or the image won't decode. This is almost always a URI-scheme mismatch — the photo URI differs across transports (a `data:` URI in the simulator vs a `file:` URI on hardware): - Use the **Photos URI helpers** rather than reading the URI yourself: `Photos.loadBytes(uri)`, `Photos.loadBase64(uri)`, `Photos.loadBitmap(uri)` on Android (`photo.loadImage()` on iOS). They normalize the scheme variance so the same handler runs on BrowserSim, LocalSim, and real Meta Ray-Ban. See [capabilities § Camera primitives](/docs/concepts/capabilities#camera-primitives). - Don't hardcode an assumption about the URI scheme — what works in the simulator may not match what real hardware returns. > **Sim ≠ hardware.** A photo that captures and renders perfectly in the browser simulator (webcam-sourced) does **not** guarantee the same on real glasses — camera fidelity, framing, and the URI scheme differ. Verify capture on real hardware before relying on it. ## Related - [Error reference](/docs/reference/errors#captureerror) — the full `CaptureError` payload shapes. - [Capture a photo](/docs/guides/photo-capture) — the call shape and the photo → vision-LLM flow. - [Capabilities](/docs/concepts/capabilities) — camera primitives, the Photos helpers, and the toggle gates. - [Glasses won't connect](/docs/troubleshooting/glasses-wont-connect) — fixing the `NotConnected` root cause. --- source: https://extentos.com/docs/troubleshooting/voice-trigger-not-firing --- --- title: Wake phrase not matching description: You subscribed to glasses.audio.transcriptions() and matched a wake phrase, but the handler doesn't run when you say it. Common causes and fixes. type: troubleshooting platform: all related: - /docs/reference/errors - /docs/guides/voice-triggers - /docs/concepts/assistant --- Your handler subscribes to `glasses.audio.transcriptions()` and matches strings against the Final transcripts, but the `if` branch that fires your business logic doesn't run. Most-common causes: ## Symptoms - Wake phrase matches in the browser simulator but not on real glasses (or vice-versa) - Wake phrase matches intermittently — sometimes fires, sometimes silent - Wrong wake phrase wins — the user says phrase A and the handler reacts as if phrase B was said - Final transcripts never arrive at all - Partial transcripts arrive but no Final - Transcripts arrive in the wrong case / with punctuation / with apostrophes that break string matching ## Quick checklist (verify these first) 1. **`glasses.toggles.state` snapshot.** Three gates must be open for transcripts to flow: `audio_capture_enabled = true`, `privacy_mode = false`, `listening_mode != "off"`. Any one closed and you'll see no Final transcripts. (`transcription_enabled` is declared in the catalog but **not consumed by the library yet** — don't chase it. All three real gates default to the transcript-friendly value, so a fresh install streams by default.) 2. **You're matching on the Final transcript, not partial.** Partial transcripts emit 5–20 per second and partial text can match a wake phrase mid-utterance. Pattern-match the `Transcript.Final` variant (Swift: `.final`) before string comparison. 3. **Casing + punctuation normalization on the read side.** The library does not normalize — `transcript.text` is what the recognizer returned, verbatim. Match against `transcript.text.lowercased().replace(...)` to cover variance. See `searchDocs(topic: 'voice_ux_guide')` for the canonical normalizer. 4. **You called `getVoiceCommandGuidance` on the phrases.** The tool catches homophone collisions, ambiguity against existing phrases, hard-to-recognize words, and Meta wake-word conflicts — issues the runtime can't surface back as "didn't match." 5. **The phrase is registered `WhenDormant` while an assistant session is active (Android).** On Android, a `glasses.voice.onPhrase` registration defaults to `VoiceScope.WhenDormant` — it won't fire during an active `glasses.assistant` conversation. Use `VoiceScope.Always` if the phrase must fire mid-conversation. (iOS `onPhrase` has no `VoiceScope` gating.) See [the assistant runtime](/docs/concepts/assistant). 6. **`getEventLog(filter: 'voice')`** in the simulator session. If you see Final transcripts in the log but your handler's `if` doesn't run, the string-match logic is the culprit. If you don't see Final transcripts at all, the toggle state or the connection is the culprit. ## Related - `searchDocs(topic: 'voice_ux_guide')` — phrase design rules + the canonical normalize() helper - `searchDocs(topic: 'toggles')` — full toggle gating model - `getCapabilityGuide(feature: 'transcription_incremental')` — call shape + gotchas - `getCodeExample(pattern: 'voice_qa_assistant')` — full working composition to peel from --- source: https://extentos.com/docs/vendors --- --- title: Vendors description: Smart-glasses vendors Extentos supports. Meta smart glasses are the production target as of 2026-08; Android XR is in preview (transport built and emulator-proven, module unpublished, no shipping hardware); Brilliant Labs is in preview (transport built and published, but no hardware has run it and there is no emulator); HTC VIVE Eagle is in preview (Android transport built, published, and proven against HTC's own simulator, with app registration granted off-document by HTC); Apple is a tracked roadmap vendor. type: overview platform: all related: - /docs/vendors/meta - /docs/vendors/android-xr - /docs/vendors/brilliant - /docs/vendors/htc - /docs/ecosystem - /docs/ecosystem/platforms/android-xr - /docs/ecosystem/platforms/apple - /docs/concepts/capabilities - /docs/concepts/architecture - /docs/concepts/transport-vs-app - /docs/concepts/display - /docs/getting-started/with-agent --- This whole section is the **device path** — camera, display and vendor device sessions. If you're building a voice app, you need none of it: the microphone and speaker are reached through the phone's own Bluetooth audio routing with no vendor SDK. See [choose your path](/docs/getting-started/choose-your-path). Extentos is a smart-glasses development layer that abstracts vendor-specific SDKs behind one capability surface. Today, **Meta is the only production vendor** — every Meta smart-glasses model (each Ray-Ban Meta and Oakley Meta variant) is supported through the Meta Wearables Device Access Toolkit (DAT, a public developer preview). **Android XR is the second vendor** — its projected transport is built and proven end-to-end on Google's Android XR emulator, but the module is deliberately unpublished while the platform is in developer preview and no Android XR glasses are buyable yet. **Brilliant Labs is the third** — its BLE transport is built and published, but nothing has run it on a device and, unlike Android XR, no emulator stands in either. **HTC VIVE Eagle is the fourth** — its Android transport is built, published as `com.extentos:glasses-htc`, and proven end to end against HTC's *own* first-party simulator, which no other non-Meta vendor offers; hardware has still never run it, and HTC must register your app before real glasses answer. Future Apple smart glasses remain a **tracked roadmap vendor** — supported in the architecture (vendor-agnostic Kotlin/Swift SDK, swappable `GlassesTransport` interface) but not implemented as a transport. This page is the consolidated status, the vendor model, and what "Extentos supports vendor X" actually means at any given point. ## Status — as of 2026-08 | Vendor | Status | Hardware | What it means today | |---|---|---|---| | **[Meta smart glasses](/docs/vendors/meta)** | ✅ **Production** | Ray-Ban Meta (Gen 1 / Gen 2), Ray-Ban Meta Optics, Ray-Ban Display, Oakley Meta HSTN, Oakley Meta Vanguard, Meta Glasses (2026 line) | Build and ship today. Four transports live (`SystemAudioTransport` for the vendorless audio baseline, `RealMetaTransport` for hardware, `BrowserSimTransport` for the browser simulator, `LocalSimTransport` — Extentos's own in-memory deterministic local sim, no DAT SDK, no network). | | **[Android XR](/docs/vendors/android-xr)** | 🟡 **Preview** | None buyable yet (audio-first devices expected first) | Projected transport built (`:glasses-xr`) and proven on Google's Android XR emulator. Module deliberately **unpublished** — you cannot ship against it today. Simulator sessions accept an Android XR device identity, so app behaviour under it is testable now. | | **[Brilliant Labs](/docs/vendors/brilliant)** | 🟡 **Preview** | Halo orderable ($399; Brilliant says first units ship early August 2026) — the only Brilliant product sold; Frame is the earlier model, no longer sold but still supported | BLE transport, on-device Lua bundle and display translation built and **published on both platforms** — shipping inside `com.extentos:glasses` on Android and `GlassesCore` on iOS — nothing to add on either platform, because Brilliant charges nothing for access. Nothing resolves into it automatically: it claims a session only when a Brilliant device is already bonded. Wire protocol verified byte-for-byte against the vendor's own implementation, but **no hardware has run it and there is no emulator either**, so you can build against it without yet knowing it works. Simulator sessions accept both devices, so a round 256×256 panel, a display-with-no-speaker, and hardware that cannot traverse focus are all testable now. | | **[HTC VIVE Eagle](/docs/vendors/htc)** | 🟡 **Preview** | Sold in Taiwan, Hong Kong, Singapore and Japan; HTC has announced US availability from September 2026 at $499 | Android transport built and **published** as `com.extentos:glasses-htc` — our code only, since HTC publishes no licence and their SDK is installed from HTC into your own mavenLocal. Proven end to end against **HTC's own simulator**, which implements the same client interface as the real kit; still never run on hardware. Two gates are HTC's: they must register your app (no self-serve route, and an unregistered app fails to connect *silently*), and VIVE Connect owns pairing. iOS is not built yet. Simulator sessions accept `vive_eagle`, so a second camera-and-audio vendor is testable now. | | **[Apple smart glasses](/docs/ecosystem/platforms/apple)** | 🟡 Roadmap | Pending Apple platform availability | No third-party smart-glasses SDK yet; a transport follows once Apple ships one — [ecosystem view](/docs/ecosystem/platforms/apple). | The status above is verifiable rather than asserted: `glassesEnum` in `mcp-server/src/tools/definitions.ts` is DERIVED from the device registry, and accepts `"meta"`, its `"meta_rayban"` legacy alias, `"android_xr"`, `"brilliant"` and `"htc"` — the last three marked preview because simulator sessions can be minted for them before any of them has run on hardware. Meta remains the only vendor you can ship against with confidence. ## The vendor model Extentos's vendor-portability is a **structural property of the architecture**, not a marketing promise. Three layers cooperate: | Layer | What it does | Vendor-specific? | |---|---|---| | **Customer handler code** | Your app's behaviour — wake-phrase matching, photo capture, AI calls, TTS response | No — written against vendor-agnostic SDK primitives (`glasses.audio.transcriptions()`, `glasses.camera.capturePhoto()`, `glasses.audio.speak()`, etc.) | | **Capability vocabulary** | The set of abstract primitives the SDK exposes | No — same vocabulary across every vendor | | **`GlassesTransport`** interface | The library's internal contract — `connect`, `capturePhoto`, `videoFrames`, `speak`, etc. | Each vendor has one transport implementation | Result: a handler written today against `meta_rayban` runs unchanged on a future `android_xr` target by switching the transport. No code rewrite, no migration. The transport is the only piece that's vendor-aware. This is verified in code: - `android-library/glasses-core/src/main/java/com/extentos/glasses/core/GlassesTransport.kt` — the interface - `SystemAudioTransport.kt`, `RealMetaTransport.kt`, `BrowserSimTransport.kt`, `LocalSimTransport.kt` — four implementations - `ios-library/Sources/GlassesCore/Transport/GlassesTransport.swift` — iOS parity A future Android XR vendor adds `RealAndroidXrTransport.kt` and `RealAndroidXrTransport.swift` that implement the same interface against that platform's SDK. Existing apps targeting `meta_rayban` are unaffected. For the deep dive on the abstraction, see **[concepts/capabilities](/docs/concepts/capabilities)** — the vocabulary itself — and **[concepts/architecture](/docs/concepts/architecture)** — how the layers fit together. ## What "supports vendor X" means at any point This is worth pinning down explicitly because the word "supports" is doing a lot of work: | Stage | What's true | |---|---| | Architecture-supported | The capability vocabulary and transport interface accommodate the vendor. Meta, Brilliant Labs, Android XR, HTC VIVE and Apple are all architecture-supported today. | | Transport-implemented | A concrete transport exists that translates abstract calls to the vendor's SDK or protocol. **Meta, Brilliant Labs and Android XR** are transport-implemented on both platforms; **HTC VIVE Eagle** is transport-implemented on Android only. | | Published | The transport is reachable from a real build. **Meta** needs `com.extentos:glasses-meta` (its artifacts sit behind Meta's credentialed repo, so that step is the vendor's toll, not ours). **Brilliant** needs nothing — it ships inside the SDK on both platforms. **HTC** needs `com.extentos:glasses-htc` *and* HTC's own SDK installed into your mavenLocal, because HTC publishes no licence permitting us to redistribute their binary. `:glasses-xr` is deliberately withheld: alpha `androidx.xr` dependencies would force `compileSdk 36` on every consumer. | | Simulator-coverage | The browser simulator reproduces the vendor's runtime behavior. All four vendors' devices are selectable (`setSimDevice`) with their real panel geometry and input hardware; a vendor with genuinely novel capabilities (say gaze tracking or spatial anchors) would need simulator extensions. | | Production | Transport-implemented, simulator-covered, validated end-to-end on real hardware. **Only Meta** is production today — that last step is what Brilliant, Android XR and HTC are missing, and for all three it is gated on hardware that does not yet exist in our hands. | Saying "Extentos supports Android XR" today means stages 1, 2 and 4 — built and simulatable, but not installable and not hardware-verified. It does not mean a developer can ship against it. The page for each non-Meta vendor opens with the same eleven-row status table so you can see exactly which stage applies, row by row. ## Why vendor portability matters for developers The strategic case for building on Extentos rather than directly on a vendor SDK: 1. **One code base, future hardware coverage.** When Android XR ships broadly, your existing Extentos handler code runs on it without a rewrite. Apps written directly against Meta DAT need a separate Android XR implementation. 2. **One simulator, every vendor.** The Extentos browser simulator (`BrowserSimTransport`) is vendor-agnostic at the protocol layer. Adding a vendor adds support to the same simulator the developer already uses. Apps written directly against vendor SDKs need a different test harness per vendor. 3. **One agent integration.** The MCP server's deterministic tool surface generalizes across vendors. The agent uses the same `getPlatformInfo`, `getCapabilityGuide`, `validateIntegration`, `createSimulatorSession` whether the target is `meta_rayban` or `android_xr`. No re-learning the tooling per vendor. 4. **One SDK surface.** The capability vocabulary is shared. A developer who has built one Extentos app for Meta knows the SDK for every future vendor. No starting over. 5. **Bounded vendor lock-in.** If Extentos ever doesn't fit, your handler classes are normal Kotlin/Swift code you own. Switching off Extentos is a manageable migration — rewrite the handler calls against the vendor SDK directly. The cost: vendor-specific features that aren't in the shared vocabulary (e.g., Meta's reserved "Hey Meta" wake word, or custom tap/swipe gestures that aren't in DAT's public surface) aren't accessible through Extentos. Apps that need those features need to use vendor SDKs directly. Most apps don't — most apps use camera, microphone, voice triggers, audio output, and display rendering, all of which are in the shared vocabulary. ## How a new vendor gets added This is bounded engineering work, not a rewrite, because the architecture defines the seams: 1. **Implement `GlassesTransport`** for the new vendor against its SDK — `RealXxxTransport.kt` on Android, `RealXxxTransport.swift` on iOS. Estimated scope: similar to `RealMetaTransport` (~1500 lines per platform). 2. **Declare the capability manifest** — which abstract capabilities the vendor exposes, with tier markers (`stable`, `preview`, etc.). 3. **Extend the `glassesEnum`** in `mcp-server/src/tools/definitions.ts`. 4. **Extend the simulator backend** — add the vendor's hardware-event injection UI if it has novel capabilities. 5. **Validate end-to-end** on real hardware. The SDK surface doesn't change. The MCP tools don't change. Existing developer handler code doesn't change. The new vendor is a drop-in addition. The reason the roadmap exists rather than the implementations: each vendor's SDK has its own maturity, its own developer-program access requirements, and its own hardware availability. Extentos adds a vendor when adding it makes a real product (transport + simulator + tested hardware), not when the architecture *could* accept it. ## Platforms with a build guide Extentos ships a production transport for **Meta**, and a preview transport for **Android XR**: - **[Meta smart glasses →](/docs/vendors/meta)** — the Extentos *build guide*: capability matrix, hardware tiers, development setup, distribution state, the Meta DAT toolkit, and the launch-partner picture. (For Meta's *neutral* platform view alongside every other vendor, see [Meta in the ecosystem](/docs/ecosystem/platforms/meta).) - **[Android XR →](/docs/vendors/android-xr)** — the preview status page: what is built, what is deliberately withheld, what you can test today, and why vendor support is opt-in. (Neutral platform view: [Android XR in the ecosystem](/docs/ecosystem/platforms/android-xr).) **Apple** remains on the roadmap and is not buildable with Extentos, so it has no build guide — its neutral coverage lives in the landscape: [Apple](/docs/ecosystem/platforms/apple). Every third-party smart-glasses platform (Snap, Rokid, RayNeo, Vuzix, and more) is mapped in the [ecosystem section](/docs/ecosystem). ## Cross-vendor capability matrix Rows are capabilities, columns are vendors, cells are what the transport actually does today. Read it against the [stages](#what-supports-vendor-x-means-at-any-point) above: only Meta's column is hardware-validated. | | Meta | Android XR | Brilliant | HTC VIVE Eagle | |---|---|---|---|---| | **Photo** | ✅ Production | 🟡 Typed refusal (emulator exposes no camera) | 🟡 Built, unverified | 🟡 Built, simulator-proven | | **Video / frames** | ✅ Production | 🟡 Typed refusal | ⛔ Unsupported permanently (no codec at BLE speed) | 🟡 Built, simulator-proven — Extentos muxes the MP4 itself | | **Microphone** | ✅ Production | 🟡 Built, emulator-proven | 🟡 Built, unverified | 🟡 Built, simulator-proven — four-mic array | | **Transcription** | ✅ Production | 🟡 Built | 🟡 Built | 🟡 Built, **final-only** (no partials) | | **Speak** | ✅ Production | 🟡 Built, emulator-proven | 🟡 Halo only — Frame has no speaker | 🟡 Built, **no cancel** (barge-in is a no-op) | | **Display** | ✅ Ray-Ban Display only | 🟡 Display model, 450×394 | 🟡 Halo 256×256 circle, Frame 640×400 | ⛔ No display, permanently | | **Simulator device** | ✅ 6 models | ✅ 2 models | ✅ 2 models | ✅ `vive_eagle` | | **Ships from** | `com.extentos:glasses-meta` | unpublished (`:glasses-xr`) | inside `com.extentos:glasses` | `com.extentos:glasses-htc` + HTC's own SDK | | **iOS** | ✅ | ⛔ Structurally impossible | ✅ | ⛔ Not built yet | The pattern worth noticing: no column required a change to the app-facing API. Each vendor is one `GlassesTransport` implementation, and the four disagree substantially — one has no display, one has no video, one has no speaker on one model, one cannot cancel speech — yet the handler code above them is identical. That is what "vendor-agnostic" is supposed to mean, and the disagreements are where it gets tested. ## Frequently asked questions ### Which smart glasses can I build for with Extentos today? For production, only Meta smart glasses — every Ray-Ban Meta variant (Gen 1, Gen 2, Optics, Display) plus Oakley Meta HSTN, Oakley Meta Vanguard, and the 2026 Meta Glasses line. Three more vendors are **buildable but not hardware-verified**: [Android XR](/docs/vendors/android-xr), [Brilliant Labs](/docs/vendors/brilliant) and [HTC VIVE Eagle](/docs/vendors/htc). You can write and run code against all three today, and test every one of them in the simulator; what you cannot yet do is know it works on the glasses. On Meta Ray-Ban Display you can also render UI to the heads-up display via `glasses.display.*` (see the [display capability](/docs/concepts/display)) — the wearer drives it with captouch + the Neural Band, delivered to your app as abstracted tap events. Meta keeps its first-party system experiences (navigation, WhatsApp, captions) and raw Neural Band access to itself — not third-party rendering. See [vendors/meta](/docs/vendors/meta) for the full picture. ### Will my Extentos app run on Android XR when it ships? Architecturally yes — your handler code is vendor-agnostic and the abstraction is built for portability. The Android XR transport now exists and is proven on Google's emulator, so the remaining gaps are publishing the module and Google shipping hardware. No code changes are required on your side; the transport is selected at app start based on which hardware is present. See [Android XR](/docs/vendors/android-xr) for the exact status. ### What about Vuzix, RealWear, Even Realities, or other vendors? Not on the current Extentos *support* roadmap — but we map the whole third-party smart-glasses landscape (Snap Spectacles, Brilliant Labs, Rokid, RayNeo, Vuzix, Even Realities, INMO, Solos and more) in the **[ecosystem section](/docs/ecosystem)**, with each platform's SDK, app model, distribution, and AI story. Roadmap inclusion *here* (a shipped Extentos transport) requires the vendor to have a public SDK Extentos can target, sufficient developer-program access, and meaningful hardware availability. Suggest a vendor by emailing [hello@extentos.com](mailto:hello@extentos.com) with the `[vendor]` prefix. ### Can Extentos add display-rendering support for the Ray-Ban Meta Display variant? Yes — it already exists. Meta's DAT opened a third-party display path, and Extentos exposes it as `glasses.display.*` (a declarative node tree on the Ray-Ban Display, shipped in both SDKs and fully sim-testable; on-glasses rendering via DAT 0.8.0 ships on both platforms, not yet confirmed on Display hardware). See the [display capability](/docs/concepts/display). What stays Meta-only is the **first-party full-screen system experiences** (navigation, WhatsApp, captions — Meta's own apps), not third-party rendering. ### Is Extentos vendor-locked to Meta? No. The architecture is vendor-agnostic by design, and that is now demonstrated rather than asserted: four vendors have transports — Meta, Android XR, Brilliant Labs and HTC VIVE Eagle — and none of them changed the app-facing API. In practice Meta is the only vendor validated on hardware, so most developers are targeting Meta today. The lock-in is bounded: your handler classes are normal Kotlin/Swift you own. Switching off Extentos is a manageable migration — rewrite the SDK calls against the vendor SDK directly — if it ever becomes necessary. ### When will Android XR / Apple be production-supported? No committed timelines. Vendors are added when adding them makes a real product (transport implementation + simulator coverage + tested hardware), which depends on each vendor's SDK maturity and hardware availability. Android XR has cleared the transport and simulator steps but not the hardware one — nothing is hardware-verified because no Android XR glasses are buyable. HTC VIVE Eagle is closest to clearing it: the transport is proven against HTC's own simulator, and HTC has announced US availability from September 2026, so hardware validation becomes possible for the first time on a non-Meta vendor. See the [changelog](/docs/resources/changelog) for actual release activity. ### Does my Extentos app target the Meta companion app or run on the glasses? Extentos apps are *phone* apps that talk to the glasses over Bluetooth. The phone runs your Android or iOS app; the glasses run Meta's firmware. This is the same model Meta DAT uses — see [vendors/meta § Audio architecture](/docs/vendors/meta#audio-architecture-developer-relevant) for the audio-routing details and [concepts/architecture](/docs/concepts/architecture) for the system overview. ## Related - **[Capabilities](/docs/concepts/capabilities)** — the vendor-agnostic vocabulary every vendor's transport implements - **[Architecture](/docs/concepts/architecture)** — how the SDK, library, transports, and backend fit together - **[Transport vs app simulation](/docs/concepts/transport-vs-app)** — what each transport actually does, why the layering enables vendor portability - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install Extentos and target Meta today --- source: https://extentos.com/docs/vendors/android-xr --- --- title: Android XR (preview) seoTitle: Android XR smart glasses SDK — Extentos support status description: "Android XR support in Extentos: the projected transport is built and proven on the Android XR emulator, the module is deliberately unpublished while the platform is in developer preview, and no Android XR glasses are buyable yet. What you can test today, and what you can't." type: reference platform: android vendor: android-xr related: - /docs/vendors - /docs/vendors/meta - /docs/ecosystem/platforms/android-xr - /docs/concepts/transport-vs-app - /docs/concepts/capabilities - /docs/concepts/architecture --- Android XR is the second vendor Extentos targets. The distinction that matters: **the transport is built, and the hardware does not exist yet.** Google ships the Android XR SDK and emulator in developer preview, no Android XR glasses are on sale, and Extentos deliberately does not publish its Android XR module while that remains true. This page is the honest status — what runs today, what is deliberately withheld, and what you can actually test right now. ## TL;DR - **Built and proven**: the projected transport (`:glasses-xr`), the vendor registry seam, and end-to-end connection on the Android XR emulator. - **Deliberately unpublished**: the `:glasses-xr` module is not on Maven Central. You cannot add it to an app today. - **No hardware**: no Android XR glasses are buyable. Audio-first devices are the expected first wave. - **Testable today**: you can mint a simulator session with an Android XR device identity and check how your app behaves under it — without any Android XR dependency. - **Meta is unaffected**: Android XR support costs Meta-only apps nothing, by design. See [dependency isolation](#vendor-support-is-opt-in) below. ## Extentos support The same eleven rows appear on every vendor page, in the same order, so vendors can be compared directly rather than by reading three differently-shaped tables. Depth specific to this vendor follows below. | | State | Detail | |---|---|---| | **Hardware you can buy** | ⛔ None | No Android XR glasses are purchasable. Google committed on the record to audio glasses "this fall" (2026), display glasses later. | | **Distribution** | ⛔ No channel | There is **no public distribution channel for this device class at all**. Play distribution today covers XR headsets and wired XR glasses, explicitly not audio or display glasses. Extentos's `:glasses-xr` is unpublished too, so there would be nothing to depend on even if a channel opened tomorrow. | | **Transport** | ✅ Built, Android only | `:glasses-xr` over Google's projected-activity model. Android-only **structurally** — a projected activity *is* an Android activity — so the missing iOS half is not drift. | | **Published** | ⛔ Withheld | Deliberately not on Maven Central: it carries alpha `androidx.xr` dependencies that Meta-only consumers must never inherit. | | **Verified on hardware** | ⛔ None | Proven end to end on Google's emulator instead, which is the strongest claim available while no device exists. | | **Camera** | ⛔ Refused | The device reports a camera and the simulator surfaces it, but the projected transport returns a typed refusal for capture. | | **Display** | ✅ Built | Renders through Google's Glimmer toolkit on Display Glasses — a 450x394 panel driven by the temple touchpad. Emulator-rendered, not hardware-verified. | | **Microphone** | ✅ Emulator-proven | Works end to end, including real per-virtual-device consent. | | **Speaker** | ✅ Emulator-proven | Raw PCM reaches the glasses over A2DP from the projected context. | | **Simulator** | ✅ Both devices | Audio and Display identities are both mintable. | | **Integration** | 🟡 One dependency | `generateConnectionModule` **refuses** a non-Meta vendor rather than emitting something wrong. Scaffold with `glasses: "meta"` for the shared bootstrap, then add `:glasses-xr` — it self-registers through `androidx.startup`, so no code line is needed. (`ExtentosXr.register()` remains public and idempotent for manual control.) Handler code is unaffected. | ## What you can do today You can test how your app behaves when the connected device reports as Android XR, without adding any Android XR dependency, because the simulator carries the device identity independently of the transport. Ask your agent to mint a simulator session targeting Android XR, then run your app against it as usual. Your handler code does not change — that is the point of the capability model. What changes is the device identity your app sees, and therefore which capabilities report as available. Two Android XR devices exist in the device registry: - **Android XR Audio Glasses** — supported. Camera, microphone and speaker. - **Android XR Display Glasses** — supported. Everything above plus the display. Your `glasses.display.show { … }` tree renders through Google's Glimmer toolkit, and the simulator draws the real panel: 450x394, not Meta Ray-Ban Display's square. The wearer drives it with the temple touchpad rather than a wristband, which is a different gesture for the same three actions. ## What you cannot do today - **Ship an Android XR app.** The module is unpublished; there is no version to depend on. - **Run on real hardware.** None exists to run on. - **Rely on any of this being final.** Preview platform, alpha dependencies, no hardware verification. ## Vendor support is opt-in A second vendor should cost nothing to developers who do not use it. In Extentos that is enforced structurally rather than promised. `:glasses-xr` is **not a dependency of `:glasses-core`**. It carries alpha `androidx.xr.projected` libraries, and a Meta-only app never inherits them. A vendor module plugs itself in with a single line in the host app, before the SDK is created: ```kotlin // Adding the :glasses-xr dependency is the whole integration — it self-registers. // ExtentosXr.register() is still available if you removed the startup entry. val glasses = ExtentosGlasses.create( ExtentosConfig(applicationContext = applicationContext) ) ``` With no registration the vendor registry is empty and transport resolution behaves exactly as it did before the registry existed — a release build that never calls `register()` cannot resolve an Android XR transport at all. Opting in is a deliberate act; opting out is the default and costs nothing. This is also why adding vendors does not gradually degrade the SDK for everyone: each vendor's dependencies stay inside that vendor's module. ## Building for the Android XR display The display surface is real code on this vendor: your `glasses.display.show { … }` tree is translated to **Glimmer**, Google's display toolkit, and rendered by a projected activity the SDK ships. You write the same tree you write for Meta. Five things are worth knowing before you build against it. **One manifest entry, and the SDK owns the activity.** A projected app *is* an Android activity, so something has to be running on the glasses before anything can appear. Rather than making you write it, the SDK ships that activity — you declare it once: ```xml ``` Without that entry a display show is a logged no-op. Nothing crashes, and nothing appears. **Design for the smaller panel.** Android XR display glasses are 450x394; Meta Ray-Ban Display is 600x600. Neither scrolls an over-tall tree — it clips. A layout that fits 450x394 fits both. The simulator draws each device's real panel, so you can see it rather than guess. **Stay inside the shared icon set.** `icon(name)` resolves against Meta's 116-glyph enum on one vendor and a set of shared concepts on the other (`check`, `close`, `place`, `search`, `settings`, `person`, `phone`, `star`, `play`, `arrow_back`, and about twenty more). A name a vendor doesn't know renders nothing rather than a wrong glyph — the same degradation on both, so an unmapped name is a silent gap, not a crash. **Images need a reachable URL, and the phone fetches them.** That is already true on Meta, where the glasses fetch; here the phone does, because the phone is what renders. Locally-captured photos and clips are handled for you — the SDK uploads them and rewrites the node to the hosted copy. Your app needs the `INTERNET` permission it almost certainly already has. **Read `isAvailable` when you need it.** On Android XR the SDK asks the connected device whether it has a screen and reports `false` until the answer arrives, because claiming an unconfirmed screen would make an app draw into nothing. Check it at the point of use rather than caching it at startup. ### Input: the same actions, a different gesture Your `onClick` and `onBack` handlers are unchanged across vendors, and that is not a coincidence. Google's gesture API for the temple touchpad is exactly three callbacks: ```kotlin Modifier.onIndirectPointerGesture( onSwipeForward = { … }, onSwipeBackward = { … }, onClick = { … }, ) ``` Focus next, focus previous, select. Those are the same three actions Meta's Neural Band delivers as a thumb-slide either way and an index-pinch. Two head-worn devices with no cursor and a small panel converged on one interaction model, which is why a single display API serves both. **Back is the one difference, and it is a real one.** Meta's mid-pinch is a back gesture that reaches your `onBack`. Google's touchpad API has no back callback — it is absent from the modifier, not merely undocumented. On Android XR, back can only arrive through the system back path. So build screens the wearer can leave by selecting something, and treat back as a bonus rather than the exit. One implementation difference that does not reach your code: on Meta the platform reads the band and tells the SDK which node was selected, while on Android XR the SDK tracks focus itself and draws the focus ring. Same behaviour, different side of the boundary. ## How this fits the capability model Extentos does not model "an Android XR app" or "a Meta app" — it models capabilities. Your code asks whether a capability is available and responds accordingly, and the same handler runs on any vendor that provides it. That is why the transport can land before the hardware does: when Android XR glasses ship, apps written against the capability surface do not need rewriting. See [capabilities](/docs/concepts/capabilities) and [transport vs app](/docs/concepts/transport-vs-app). ## Frequently asked questions ### Can I build an Android XR app with Extentos today? No. The transport exists and works on the emulator, but the module is unpublished, so there is nothing to depend on — and there is no hardware to run it on. ### Why build a transport for hardware nobody can buy? Because the architecture claim needed proving. "Vendor-agnostic" is easy to assert and hard to verify with one vendor. Building a second, genuinely different transport — Android XR uses a projected-activity model, not Bluetooth — is what tests whether the seam actually holds. ### Does supporting Android XR slow down or complicate my Meta app? No. The Android XR module is not a dependency of the core, so a Meta-only app never pulls its libraries, and transport resolution is unchanged unless you explicitly register it. ### When will Android XR be production-supported? No committed timeline. It depends on Google shipping hardware and moving the platform out of developer preview. See the [neutral platform view](/docs/ecosystem/platforms/android-xr) for the wider Android XR picture. ### What about the display variant? Supported, as of 2026-07-27. The display path is real code: your display tree is translated to Glimmer, Google's display toolkit, and rendered by a projected activity the SDK owns. The simulator offers the device with its own panel geometry and its own input hardware, so a layout you build there is a layout the glasses can hold. Two honest limits. Nothing is hardware-verified, because no Android XR glasses of either class are on sale, and the display class is expected later than the audio one. And `video` inside a display tree has no Glimmer equivalent; the SDK plays it through the Android framework instead, which is built but not yet proven on a device. --- source: https://extentos.com/docs/vendors/brilliant --- --- title: Brilliant Labs (preview) seoTitle: Brilliant Labs Halo and Frame — Extentos support status description: "Brilliant Labs support in Extentos: the BLE transport, on-device Lua bundle, display translation, still capture and microphone ship inside the SDK on both platforms — nothing to add on Android, TransportChoice.brilliant on iOS — but no hardware has run any of it and there is no emulator either. Video is unsupported permanently. Halo and Frame are selectable in the simulator today with their real panel geometry and input." type: reference platform: all vendor: brilliant related: - /docs/vendors - /docs/vendors/android-xr - /docs/ecosystem/platforms/brilliant-labs - /docs/concepts/capabilities - /docs/concepts/display - /docs/getting-started/choose-your-path --- Brilliant Labs is the third vendor Extentos targets, and it is the **inverse of Android XR**. Where Android XR has an emulator and no purchasable glasses, Brilliant has purchasable glasses and no emulator — so the transport is shippable, and nothing has run it against a device, real or simulated. This page is the honest status: what runs today, what is unverified, and what Brilliant breaks that a Meta-shaped app quietly assumes. ## TL;DR - **Built**: the BLE transport, the on-device Lua bundle, the connect handshake, the display translation, **still capture** and the microphone (`audioChunks`, `transcriptions`, `recordDiscrete`) — the protocol half in the shared Rust core, so both platforms run the same code. - **Verified as far as it can be without hardware**: the wire protocol is checked **byte-for-byte against Brilliant's own implementation**, and the layout is mutation-tested. - **Ships with the SDK on both platforms**: inside `com.extentos:glasses` on Android and `GlassesCore` on iOS — **no vendor dependency to add on either.** Brilliant charges nothing for access (no SDK, no account, no credentials, just BLE), so Extentos charges nothing either. You *can* build against it — you just cannot yet know it works. - **No hardware verification, and no emulator either.** This is a weaker claim than Android XR's, and it is stated separately for that reason. - **Testable today**: both devices are selectable in the simulator with their real panel geometry and input hardware. - **Meta is unaffected.** A Meta-only app never inherits any of it. ## Extentos support The same eleven rows appear on every vendor page, in the same order, so vendors can be compared directly rather than by reading three differently-shaped tables. Depth specific to this vendor follows below. | | State | Detail | |---|---|---| | **Hardware you can buy** | 🟡 Orderable | Halo, $399 — but Brilliant says the first units ship **early August 2026**, so ordering today is a pre-order rather than a device in hand. Frame is no longer sold. | | **Distribution** | ✅ Open | The most open in the category: no Brilliant store, no review, no allowlist, no invite waitlist. Your app is an ordinary phone app that talks BLE, so it ships through the App Store and Play on their usual terms and Brilliant gates nothing. The constraint is the other end of the funnel — hardware ships early August 2026, so there is nobody to ship it to yet. | | **Transport** | ✅ Both platforms | `BrilliantCore` in the shared Rust core; each platform shell is a socket and nothing else. | | **Published** | ✅ Ships with the SDK | Inside `com.extentos:glasses` on Android and `GlassesCore` on iOS — nothing to add on either platform. The former `com.extentos:glasses-brilliant` artifact still resolves as an empty deprecated stub so an existing build does not break on a version bump; drop the line at your convenience. | | **Verified on hardware** | ⛔ None on Brilliant hardware | No Brilliant device has run any of this, and unlike Android XR there is no emulator standing in either. The iOS *audio* path was separately verified on a real iPhone — see below. | | **Camera** | 🟡 Stills built | `capturePhoto()` is implemented end to end — the request goes out, the device streams the JPEG back in chunks, and you get a `Photo`. Unverified on hardware like everything else here. **Video is a separate answer** — see the row below, and note the resolution split: Frame takes any even 100–720, while Halo has exactly ONE legal resolution, so on a Halo the requested `Resolution` is advisory and `Photo.width` reports what actually arrived. | | **Display** | ✅ Built | Halo's **256×256 circle** and Frame's 640×400 rectangle. Your `glasses.display.show { … }` tree lays out for whichever panel is connected. | | **Microphone** | ✅ Works | `audioChunks()`, `transcriptions()` and `recordDiscrete()`. Brilliant streams microphone audio over its own BLE link rather than pairing as a Bluetooth headset, so reaching it is the transport's job rather than the phone's. `recordDiscrete` is bounded by the recogniser's own endpointing — the same division of labour the Meta path uses, where silence detection belongs to the platform rather than the core — so `silenceTimeoutSeconds` selects silence-bounded behaviour rather than setting an exact threshold. | | **Speaker** | 🟡 Halo only | Halo has bone-conduction speakers and the audio path to them is built; phone-side synthesis is not wired yet, so `speak()` returns a typed refusal. **Frame has no speaker at all** — guard on the speaker capability, not on whether a display exists. | | **Simulator** | ✅ Both devices | Real panel geometry and input, including the circle. | | **Integration** | ✅ Nothing to add | Not scaffolded, and does not need to be. Android: nothing — it ships in the SDK and self-registers. iOS: one transport choice plus `NSBluetoothAlwaysUsageDescription`. Handler code is identical to Meta's either way. | ### Beyond the spine Detail that has no counterpart on other vendors, kept separate so the table above stays comparable. | Thing | State | What it means | |---|---|---| | Wire protocol | ✅ Built + verified | Framing, chunking and reassembly, checked byte-for-byte against the vendor's own BSD-3 implementation rather than against documentation. | | On-device Lua bundle | ✅ Built | Uploaded on connect. Until it runs, a connected device understands nothing — which is why `Connected` and `Ready` are separate states. | | Video | ⛔ Unsupported, permanently | No video primitive and no codec on either device, and frame-by-frame bitmaps are not viable at BLE speed — so `captureVideo()` and `videoFrames()` both refuse with that reason, and `videoFrames` **throws** it rather than ending in silence. This is the one vendor where stills work and streaming cannot, which is why the SDK's transport seam is named `videoStreamUnavailable` rather than `cameraUnavailable`. Unlike the camera row, this is not pending work: no effort changes it. The **simulator reproduces it** — select a Brilliant device and `videoFrames` refuses there too, so an app cannot be built against a capability the hardware lacks. | | iOS audio path, on a real iPhone | ✅ Verified | Everything downstream of "bytes arrived" ran on an iPhone (iOS 26.5.2): transport resolution, and synthesised speech through the same seam a Halo feeds — `BleAudioInput` → the recogniser → a transcript, 6/6 words. Resolution also never triggers a Bluetooth permission prompt. The CoreBluetooth **socket** — scan, connect, MTU — still needs a Halo. See `ios-library/DeviceVerify/`. | ## The two devices **Brilliant Halo** — a **256×256 round** display, camera, dual mics, bone-conduction speakers, a physical button and an IMU tap. **$399, orderable now — the only product Brilliant currently sells.** Brilliant states the first units ship in **early August 2026** (checked 2026-07-29), so ordering today is a pre-order rather than a device in hand. **Brilliant Frame** — the earlier model: a 640×400 display, camera, one microphone, one tap gesture, and **no speaker**. **No longer sold** (its product page 404s and it is absent from the store, checked 2026-07-29), but the vendor still documents it, people still own them, and the transport detects and supports it. If you are buying hardware to develop against, buy a Halo. Frame is worth pausing on: it is the first shipping device with a **screen and no audio out**. If your app renders a display and falls back to `audio.speak` when something goes wrong, that fallback is silence on Frame. Branch on the speaker capability, not on whether a display exists. ## What Brilliant breaks that Meta does not Four assumptions a Meta-shaped app makes quietly. All four are visible in the simulator, so you can design around them without owning either device. **The panel is round, and it is the smallest in the catalog.** Halo's addressable canvas is 256×256 and circular — a full-width row laid out to the panel width loses both of its ends. The SDK handles this for you: content lays out inside the inscribed square automatically, so you write one tree and it is correct on a circle. What it cannot do is make your tree shorter. A layout that fits Meta's 600×600 shows roughly a third of its content here, and trees clip rather than scroll on every vendor. **There is one font size.** Neither device's text primitive takes a size argument, so `HEADING`, `BODY` and `CAPTION` render identically. Colour still differentiates, which is why secondary text carries its own colour on the wire. If your hierarchy depends on type size, it will flatten. **The hardware cannot move focus.** Halo has a button and an IMU tap but **no swipe of any kind**; Frame has only the tap. This is the first vendor where the four display actions do not all have a gesture. Build single-action screens, or screens where the one thing you can act on is the obvious one. **Audio is not Bluetooth audio.** Brilliant's mic and speaker run over a **custom GATT characteristic**, not the phone's Bluetooth audio routing. A Halo never appears as a system headset — so the [vendorless voice path](/docs/getting-started/choose-your-path), which reaches most glasses with no vendor SDK at all, does not reach Brilliant. Voice on Brilliant needs this transport. ## Adding it **Android** — nothing to add. The transport ships inside `com.extentos:glasses` and registers itself at app start, so if you already have the SDK you already have Brilliant. Your handler code does not change either: the same `glasses.display.show { … }` you write for Meta is what renders on a Halo. Why nothing, when Meta needs `glasses-meta`? Because a vendor should cost your app what the **vendor** charges. Meta's artifacts live on a credentialed repo, so reaching them means accepting Meta's terms and minting a token — that toll is theirs and we cannot remove it. Brilliant charges nothing: no SDK, no account, no credentials, just BLE. Charging you a dependency line anyway would have been our own invention. If your build still carries `implementation("com.extentos:glasses-brilliant:…")` it keeps working — that coordinate now publishes as an empty stub that depends on the SDK, so a version bump cannot break you. Remove the line whenever convenient. **iOS** — already in `GlassesCore`, so there is nothing to add. Pick the transport: ```swift let glasses = Extentos.create(config: ExtentosConfig(transport: .brilliant)) ``` Add `NSBluetoothAlwaysUsageDescription` to your Info.plist. Halo streams over its own BLE link rather than pairing as a headset, so iOS asks for the Bluetooth string even though what you are building is, from the wearer's side, a microphone. Brilliant lives in the core on iOS and in its own artifact on Android. That is deliberate rather than drift: Android's `:glasses-core` is kept vendor-free, so a vendor there must be a separate module; `GlassesCore` on iOS already depends on Meta's Device Access Toolkit, so there is no such isolation to protect. On Android, `TransportChoice.Auto` resolves to Brilliant only when a Brilliant device is already **bonded** to the phone. On iOS, `.auto` consults the `hasBrilliantDevice` closure you supply — the same shape as the existing `hasBondedMetaDevice`, because iOS exposes no bonded-device list an app may read. `ExtentosBrilliant.hasConnectedDevice` is a ready-made one; omit it and `.auto` behaves exactly as it did before this transport existed. That gate matters — the alternative would be scanning on every connect, which costs seconds of radio for every app whether or not the hardware is anywhere nearby. The practical consequence: **first pairing is an explicit act in your app** (scan, pick, connect), and every connect after that resolves automatically. Meta works the same way, except bonding happens in Meta's own app first. ## What you can do today Test how your app behaves when the connected device reports as Halo or Frame, without any hardware — the simulator carries the device identity independently of the transport. ```ts await setSimDevice({ sessionId, device: "brilliant_halo" }); ``` The simulator draws the real 256×256 circle, not a square, and its control pad shows the real input: a button with press / double-press / long-press, and **disabled** controls for the gestures the hardware does not have. A flow you can drive in the simulator is a flow the device could perform. Switch to `brilliant_frame` to exercise the display-but-no-speaker path, which no other device in the catalog can give you. ## What you cannot do today - **Know that it works.** Built and unverified — and unlike Android XR, there is no emulator standing in for it either. If you have a Halo, you would be the first person to run this. - **Capture photos, record a clip, or use the speaker.** Each refused with a typed error naming what is missing. Live microphone audio and transcription *do* work — that is the part a voice app needs. ## Why publish something unverified? Because withholding it would not have made it safer, only slower to find out about. Adding it is **additive** for anyone already integrated: on Android one dependency line, on iOS one transport choice, with no change to handler code — and the core already carries the Brilliant vocabulary on both platforms. Nobody who built against the simulator has anything to redo. Meanwhile an unpublished module can only ever be tested by us. Halo shipped into a developer-heavy community; if you own one, you can find out in an afternoon what would otherwise wait on our hardware arriving. What that costs you is honesty about the state, which is the point of this page. The protocol underneath is verified about as well as protocol code can be without a device — checked against the vendor's own working implementation, the connect handshake driven against a fake socket, the layout mutation-tested. That is enough to believe the design is right. It is not enough to claim it works. The first things likely to move on real hardware are the **font metrics** — Brilliant publishes none, so every text measurement is currently an educated guess — and possibly the **input mapping**, since press-to-select and long-press-for-back is a judgement about ergonomics that wearing the device would settle in a minute. Both are cosmetic rather than structural. ## How this fits the capability model Extentos does not model "a Brilliant app". Your code asks whether a capability is available and responds; the same handler runs on any vendor that provides it. That is why the transport could be built before the hardware arrived, and why Frame's missing speaker needs no special case — it is an absent capability, handled by the same policy as any other. See [capabilities](/docs/concepts/capabilities) and [the display capability](/docs/concepts/display). ## Frequently asked questions ### Can I build a Brilliant app with Extentos today? You can build one. Whether it runs is genuinely unknown — no device has executed this code. The display path, the microphone (including `recordDiscrete`) and still capture are all real; only video is refused, and it is refused permanently rather than pending. The simulator will show you the layout consequences — and the video refusal — without any hardware. ### Is this the same status as Android XR? No, and the difference is worth stating. Android XR is **emulator-proven** — its transport has run end to end against Google's emulator, but the module is unpublished (alpha dependencies) and no hardware exists to buy. Brilliant is the mirror image: the module is published and the hardware is on sale, but the transport has run against nothing but tests. One has been executed against something device-like; the other you can actually install. ### Does supporting Brilliant affect my Meta app? No. The transport now ships inside the SDK on both platforms, but shipping is not claiming: it only *claims* a session when a Brilliant device is already bonded to the phone, it reads the bonded list without ever scanning (a BLE scan on every connect would cost power for apps that will never see one), and `Auto` ranks it behind Meta. A Meta app resolves exactly as it did before the fold. ### Brilliant's SDK is open source. Why not use it directly? You can — it is BSD-3 and genuinely good. What Extentos adds is that the same app code also runs on Meta smart glasses and Android XR, and that the display tree you write is not Brilliant-specific. See the [neutral ecosystem view](/docs/ecosystem/platforms/brilliant-labs) for the platform on its own terms. --- source: https://extentos.com/docs/vendors/htc --- --- title: HTC VIVE Eagle (preview) seoTitle: HTC VIVE Eagle — Extentos support status description: "HTC VIVE Eagle support in Extentos: the Android transport is built and proven end to end against HTC's own first-party simulator, and ships as com.extentos:glasses-htc. No hardware has run it, HTC must register your app before real glasses answer, and iOS is not built yet. Eagle is selectable in the simulator today." type: reference platform: all vendor: htc related: - /docs/vendors - /docs/vendors/brilliant - /docs/vendors/android-xr - /docs/ecosystem/platforms/htc-vive-eagle - /docs/concepts/capabilities - /docs/getting-started/choose-your-path --- HTC VIVE Eagle is the fourth vendor Extentos targets, and it is the one where **the vendor ships a simulator**. HTC's `ViveGlassSimulator` implements the very same `ViveGlassClient` interface as the real kit, so the Extentos transport is driven end to end by HTC's own binary with only the substrate simulated. That is a stronger position than a vendor with no hardware to buy and no emulator to stand in for it — and it is still not hardware verification. This page is the honest status: what runs, what has never run, and the two things HTC controls that no amount of code on our side can resolve. ## TL;DR - **Built and simulator-proven on Android**: connect, photo, video frames, a muxed MP4 clip, the microphone stream, transcription and speech — 8/8 against HTC's own simulator binary. - **Publishes as `com.extentos:glasses-htc`** — the module and its publishing config are done, and the artifact goes to Maven Central with the next release cut, alongside the rest of the SDK. It is **our transport code only**: HTC's SDK is not inside it and never will be, because HTC publishes no licence permitting redistribution, so you install theirs from HTC. - **Never run on hardware.** Eagle was Asia-only until HTC announced US availability from September 2026, so nobody on our side could buy one. - **HTC must register your app.** There is no self-serve route, and an unregistered app does not error — it silently fails to connect. Extentos turns that silence into a typed `ConnectError.RegistrationRequired`. - **Both platforms are simulator-proven** — 8/8 against HTC's own simulator on each. iOS ships as a separate `swift-glasses-htc` package so existing `swift-glasses` consumers are untouched, and that isolation is verified: `swift-glasses` still builds with no HTC framework present anywhere. - **Testable today**: `vive_eagle` is selectable in the simulator, so an app's behaviour under a second camera-and-audio vendor is verifiable now. - **Meta is unaffected.** A Meta-only app never inherits any of this — HTC's binary must never reach `com.extentos:glasses`. ## Extentos support The same eleven rows appear on every vendor page, in the same order, so vendors can be compared directly rather than by reading differently-shaped tables. Depth specific to this vendor follows below. | | State | Detail | |---|---|---| | **Hardware you can buy** | 🟡 Expanding | Sold in Taiwan, Hong Kong, Singapore and Japan. HTC has announced **US availability from September 2026 at $499**; the UK and EU have no dated purchase path. This is the row that changed — Eagle was unbuyable in the West for the whole time this transport was being written. | | **Distribution** | 🟡 Gated by HTC | Your app is an ordinary phone app and ships through the App Store and Play on their usual terms — there is no HTC store. But the **glasses half** is gated: HTC must register your app before real glasses will answer it, and the registration route is not documented. This is the same first-party coupling Meta has, without Meta's published process. | | **Transport** | ✅ Both platforms | `HtcEagleTransport` over the shared Rust core through `HardwareBridge` — `:glasses-htc` on Android, `GlassesHtc` (the `swift-glasses-htc` package) on iOS. The two differ only where HTC's own two SDKs do: iOS collapses Android's two refusal connection-states into a thrown error, delivers media as `CMSampleBuffer`, and returns permissions `async`. | | **Published** | 🟡 Ships next release | The module and its publish config are done and the POM is verified; `com.extentos:glasses-htc` reaches Maven Central with the next release cut. HTC's AAR is **not** in it and is not redistributed by us. Their coordinates are named in our POM, so a build without their SDK fails at dependency resolution with the coordinate in the error rather than at runtime on first connect. | | **Verified on hardware** | ⛔ None | Both platforms are proven against HTC's own simulator — 8/8 each, driving their real client interface — which is a genuinely different claim from unverified. It is still not hardware-green, and that rule does not bend for this vendor. | | **Camera** | ✅ Built | 12 MP stills via `capturePhoto()` (HEIC, reported honestly rather than re-encoded behind your back), live frames via `videoFrames()`, and `captureVideo()`. HTC has **no record-to-file API at all**, so the SDK muxes their live H.264 + AAC stream into an MP4 itself and honours `maxDurationSeconds`; HTC caps a clip at ten minutes regardless of what you ask for. | | **Display** | ⛔ None, permanently | Eagle has no display and the SDK has no render or overlay surface anywhere in it. `glasses.display.isAvailable` is false and display calls no-op per the degradation policy — the same shape as every Meta model except Ray-Ban Display. | | **Microphone** | ✅ Built | `audioChunks()`, `transcriptions()` and `recordDiscrete()` from the **four-mic beamforming array** (1 directional + 3 omnidirectional), not the phone's microphone. HTC delivers it AAC-encoded and Extentos decodes to i16 PCM at 16 kHz, so what reaches your handler is the shape every other vendor emits. Transcription is **final-only** — HTC's API has no partial result, so interim UI has nothing to render here. | | **Speaker** | 🟡 Built, two limits | Open-ear stereo speakers via HTC's `speakText(text, locale)`, which is the **entire** outbound audio surface of their SDK. There is no raw audio-out API, so the assistant's streamed PCM rides the OS's Bluetooth route through `SystemAudioBridge` — the same substrate Ray-Ban audio uses. And there is **no cancel**: `cancelSpeak` is a no-op on this vendor, so barge-in cannot stop an utterance mid-sentence. | | **Simulator** | ✅ `vive_eagle` | Selectable in the browser simulator with its real capability set — camera, microphone, speaker, no display. | | **Integration** | 🟡 Manual, both platforms | Not scaffolded, and could not be: the steps that matter are HTC's to grant. Android adds a Gradle module; iOS adds a second SwiftPM package and one `register()` call. See [Adding it](#adding-it). | ### Beyond the spine Detail with no counterpart on other vendors, kept separate so the table above stays comparable. | Thing | State | What it means | |---|---|---| | HTC's own simulator | ✅ The reason this vendor was picked | `ViveGlassSimulator` implements the same `ViveGlassClient` interface as the real kit, so the transport is written once and the substrate is a constructor argument. It is what turned "we cannot buy one" into "we can prove the whole path anyway". | | Licence | ⛔ None published | HTC publishes no licence or EULA — not on the download page, and not inside either archive. Commercial terms are simply unstated. This is why their binary is never redistributed by us, and why you fetch it yourself. | | App registration | ⛔ Off-document | No self-serve form. How it is keyed — package id, signing certificate or publisher — is undocumented: HTC's iOS `AppInfo` carries `teamId`, `bundleId`, a certificate common name and a sha256, while the Android handshake sends no explicit app identity at all. | | VIVE Connect | ⚠️ Hard runtime dependency | It owns pairing, the auth handshake and the user's language setting, exactly as the Meta AI app does for Ray-Bans. Not optional infrastructure. | | Encoded media | ✅ Handled in the bridge | HTC hands us AAC audio and H.264 video where every other transport hands us decoded data. Extentos decodes both, so your handler sees the same PCM and the same frames it sees everywhere else. | | Device identity | 🟡 Static | Battery level, wear state and the connected model sit below HTC's public API and are unreachable from it. `glasses.device.type` reports `vive_eagle` statically rather than resolving it from the glasses. | | Physical inputs | ⛔ One of four | The SDK exposes only the AI button. Touchpad, capture button and voice control are reserved to HTC's own assistant. | | Simulator artifact in release builds | ⚠️ Known wart | `com.extentos:glasses-htc` names HTC's simulator as a runtime dependency, so it resolves into release builds too, even though HTC's simulator refuses to run outside a debuggable build. Splitting it out is deliberate future work, not a blocker for a preview vendor. | ## What Eagle breaks that Meta does not Three assumptions a Meta-shaped app makes quietly. **Failure is silent, not loud.** An app HTC has not registered does not throw and does not crash — the connection simply never completes. This is the single most likely thing you will hit, and it is why the transport turns `ERROR_UNREGISTER_APP` into a typed `ConnectError.RegistrationRequired` rather than letting it read as a timeout. **Barge-in does not work.** `cancelSpeak` is the primitive an assistant uses to stop talking when the wearer starts. HTC exposes no way to stop an utterance, so on Eagle it is a no-op. If your voice loop depends on interrupting playback, it degrades here — design the turn-taking so that waiting out a long utterance is not the only escape. **Two privacy behaviours are enforced in hardware, with no API and no reason code.** The camera auto-disables when the glasses are not worn or the capture LED is covered, and connecting throws `GlassesNotWornException` if they are not on the wearer. Neither is something your app can detect in advance or work around; both are states to handle, not bugs to fix. ## Adding it Neither platform is scaffolded by `generateConnectionModule`, and that is deliberate: the steps that decide whether this works at all are HTC's to grant, not ours to generate. ### Android Three steps. **1. Install HTC's SDK into your `mavenLocal`.** HTC ships it as a `.7z` containing a file-based Maven repository, from [their download page](https://developer.vive.com/resources/vive-ai-glasses-sdk/) — free, no sign-in, no approval. Extract it and install `com.htc.viveglass.sdk:viveglass_client` and `:viveglass_simulator` into `~/.m2/repository`. There is no remote repository to point Gradle at, and we cannot mirror one for you. **2. Let Gradle see it, scoped to HTC's group alone** — in `settings.gradle.kts`: ```kotlin dependencyResolutionManagement { repositories { mavenLocal { content { includeGroup("com.htc.viveglass.sdk") } } google() mavenCentral() } } ``` **3. Add the module.** ```kotlin dependencies { implementation("com.extentos:glasses-htc:2.8.0") } ``` This coordinate **ships next release** — it is not on Maven Central yet. Until it lands, build `:glasses-htc` from the repo as a local module. Everything else on this page is accurate today. It self-registers through `androidx.startup`, so adding the module *is* the integration. `TransportChoice.Auto` resolves to it once Eagle is the connected device, and your handler code does not change — the same `glasses.camera.capturePhoto()` and `glasses.audio.transcriptions()` you write for Meta. Then get HTC to register your app. Until they do, everything above builds and runs and the glasses never answer, silently. There is no self-serve form; HTC grants it on request. ### iOS Two steps, plus the same HTC registration. **1. Install HTC's SDK.** Same archive, different half — run `scripts/install-viveglass-sdk-ios.sh` from the `swift-glasses-htc` package, which fetches HTC's `.7z` and stages `ViveGlassKit.xcframework` and `ViveGlassSimulator.xcframework` into `Artifacts/`. HTC nests `.7z` inside `.7z` on iOS, so you need a 7-Zip tool (`brew install sevenzip`). **2. Add the package and register it.** ```swift // Package.swift .package(url: "https://github.com/extentos/swift-glasses-htc", from: "2.8.0"), ``` ```swift // in your App's init, before Extentos.create(...) import GlassesHtc ExtentosHtc.register() ``` Android's module self-registers through `androidx.startup`, so adding it *is* the integration there. iOS has no equivalent — nothing runs a library's code before the app's — so registration is one explicit call. Handler code is identical to Meta's on both platforms. `swift-glasses-htc` is a **second package**, not a product of `swift-glasses`. SwiftPM validates every binary target when it loads a manifest, so declaring HTC's frameworks inside the main package would break `swift build` for every existing consumer the moment they resolved without those frameworks staged. This split is what keeps your build unaffected if you never touch Eagle. It is not published yet — it ships alongside the next release cut. ## Why the module is separate Meta's transport lives in `com.extentos:glasses-meta` because Meta's artifacts sit behind a credentialed repository. HTC's lives in `com.extentos:glasses-htc` for a sharper reason: **their binary has no stated licence**, so it must never become something a Meta customer inherits by depending on `com.extentos:glasses`. A voice-only app, or a Meta app, pulls none of it and needs nothing from HTC. The same boundary is why the artifact we publish contains our code and not theirs. --- source: https://extentos.com/docs/vendors/meta --- --- title: Meta smart glasses (Meta DAT) seoTitle: Meta smart glasses SDK — Ray-Ban & Oakley (Meta DAT) description: "Meta smart glasses developer guide: Wearables Device Access Toolkit (DAT 0.8.0) capabilities, supported models (Ray-Ban Meta, Oakley Meta, Ray-Ban Display), 2026 distribution state, and how Extentos abstracts the toolkit." type: reference platform: all vendor: meta related: - /docs/vendors - /docs/concepts/transport-vs-app - /docs/concepts/vs-meta-dat - /docs/concepts/capabilities - /docs/concepts/display - /docs/concepts/permissions - /docs/getting-started/with-agent - /docs/mcp-server - /docs/resources/pricing - /docs/getting-started/ios - /docs/getting-started/android --- Meta's smart glasses are the primary production target for Extentos. "Ray-Ban Meta" is one model line, not the platform — the hardware family as of 2026 spans five camera-and-audio models — Ray-Ban Meta (Gen 1 and Gen 2), Ray-Ban Meta Optics, Oakley Meta HSTN, Oakley Meta Vanguard, and Meta Glasses (the 2026 Meta-branded EssilorLuxottica line) — plus Meta Ray-Ban Display, the tier with a monocular heads-up display and the Meta Neural Band wrist controller. Third-party apps reach the glasses through Meta's **Wearables Device Access Toolkit (DAT)** — an Android and iOS SDK, currently in **public developer preview**, that exposes camera, microphone, and audio capabilities over Bluetooth, adds a display-rendering surface on Meta Ray-Ban Display, and ships a Mock Device Kit for testing without hardware. This page is the consolidated capability matrix, the distribution-and-development state, and the practical answer to "what can I actually build and ship for Meta smart glasses as a third-party developer in 2026?" > This is the **Extentos build guide** for Meta — how to build and ship with the Extentos SDK. For the *neutral*, cross-platform view of Meta as one vendor among many (comparable to Snap, Rokid, and the rest of the landscape), see **[Meta in the ecosystem](/docs/ecosystem/platforms/meta)**. ## TL;DR | What you can build today | Status | |---|---| | Photo / video / camera-stream apps via DAT | ✅ Works today (public developer preview) | | Voice-triggered apps (custom phrases, phone-side recognition) | ✅ Works today | | Always-listening apps (continuous mic while your app runs) | ✅ Works today — battery and audio-quality trade-offs apply | | TTS-driven audio apps (phone TTS played over Bluetooth) | ✅ Works today | | Apps that render UI on the **Meta Ray-Ban Display** (`glasses.display.*`) | ✅ Shipped in both SDKs, fully sim-testable; on-glasses rendering via DAT 0.8.0 ships on both, not yet confirmed on Display hardware — [display capability](/docs/concepts/display) | | Apps that hook the **"Hey Meta"** wake-word engine | ❌ Reserved for Meta AI | | Raw **Neural Band** gestures / custom gesture recognition | ❌ Not exposed — Band input reaches your display UI only as abstracted tap events | | Motion-sensor (IMU) consumers | ❌ No public API in DAT 0.8.0 | | Distribution to invited testers via **release channels** | ✅ Works today, no content review | | Public distribution (to anyone, store-style) | ⚠️ Partner-gated during the preview; Meta has said broader publishing arrives in 2026. iOS App Store publishing is not yet supported | The short version: a third-party developer can build rich photo/video/voice/audio apps for Meta smart glasses today using the public DAT preview — and, on the Display tier, render their own UI via `glasses.display.*`. Distribution today runs through Meta's release channels (invited testers), because public publishing is still limited to select Meta partners while the toolkit is in preview. Wake-word-engine access and Neural Band raw input stay Meta-only. ## Extentos support The same eleven rows appear on every vendor page, in the same order, so vendors can be compared directly rather than by reading three differently-shaped tables. Depth specific to this vendor follows below. | | State | Detail | |---|---|---| | **Hardware you can buy** | ✅ On sale | Six models in market: Ray-Ban Meta (Gen 1 and 2), Ray-Ban Meta Optics, Oakley Meta HSTN, Oakley Meta Vanguard, Meta Glasses, and Ray-Ban Display. | | **Distribution** | 🟡 Preview-gated | Two paths. Today: create versions and **release channels** in the Wearables Developer Center and invite testers by email — channel membership is what lets a user register your integration, and there is **no Meta content review**. Publishing so that *anyone* can register is limited to select Meta partners, with no public submission flow yet. The phone app itself ships through your normal channels, with one hard exception: **iOS App Store publishing is not supported**, because DAT uses `ExternalAccessory` and draws rejection under MFi and privacy-manifest rules. Meta has said it plans to support it. | | **Transport** | ✅ Both platforms | `RealMetaTransport`, wrapping Meta's Device Access Toolkit. | | **Published** | ✅ Yes | `com.extentos:glasses-meta:2.8.0` on Android; ships inside `swift-glasses` 2.0.2 on iOS. | | **Verified on hardware** | ✅ Yes | The GA vendor — camera, voice and audio paths run on real glasses. Ray-Ban Display's on-glasses rendering is in its verification phase. | | **Camera** | ✅ Works | Photo, video and frame streaming through DAT. | | **Display** | ✅ Works | `glasses.display.*` on **Ray-Ban Display** only. The other five models have no panel, and the call is a safe no-op there — branch on `glasses.display.isAvailable`, never on the model name. | | **Microphone** | ✅ Works | Continuous capture, transcription, and custom wake phrases. The glasses pair as a Bluetooth headset, so they become the phone's microphone. | | **Speaker** | ✅ Works | Phone-side TTS routed over Bluetooth. | | **Simulator** | ✅ All six models | Every model is selectable, so display-vs-no-display branching is testable without hardware. | | **Integration** | ✅ Scaffolded | `generateConnectionModule` emits the whole bootstrap, including the credentialed DAT repository. | ## Compatible models **Extentos runs on every Meta smart-glasses model in market.** Compatibility is determined by Meta's Wearables Device Access Toolkit — and DAT exposes the same camera, microphone, and speaker surface uniformly across every model, with Meta Ray-Ban Display adding the display surface on top. Frame styles within a model line don't change what Extentos can do; a Wayfarer-style and a Headliner-style Ray-Ban Meta have identical capabilities to Extentos. DAT 0.8.0's device-type enum covers every model in market; Extentos surfaces them with these ids — `rayban_meta`, `oakley_meta_hstn`, `oakley_meta_vanguard`, `rayban_meta_optics`, `meta_glasses`, and `rayban_display` — and every one of them is simulatable in the browser simulator. Ray-Ban Meta Gen 1 and Gen 2 share the one `rayban_meta` profile — the generation changes camera quality and battery, not the capability surface — and frame styles within a line (including the Meta Glasses celebrity co-designed editions) don't carry their own identity. | Model (Extentos id) | Variants | Camera / Voice / Audio | Display rendering | |---|---|---|---| | **Ray-Ban Meta** (`rayban_meta`) — Gen 1 2023, Gen 2 2025 | Wayfarer, Headliner, Skyler frames | ✅ Fully supported | n/a (no display) | | **Ray-Ban Meta Optics** (`rayban_meta_optics`) | Prescription-first frames | ✅ Fully supported | n/a (no display) | | **Oakley Meta HSTN** (`oakley_meta_hstn`, 2025) | Sport-focused variant | ✅ Fully supported | n/a (no display) | | **Oakley Meta Vanguard** (`oakley_meta_vanguard`, 2025) | Performance/sport variant | ✅ Fully supported | n/a (no display) | | **Meta Glasses** (`meta_glasses`, 2026) | Meta-branded EssilorLuxottica line, incl. celebrity co-designed frames | ✅ Fully supported | n/a (no display) | | **Meta Ray-Ban Display** (`rayban_display`, 2025) | Monocular HUD + Meta Neural Band | ✅ Fully supported | ✅ `glasses.display.*` | ### What this means for the Display model This is the most-asked compatibility question. Extentos runs *on* the Meta Ray-Ban Display the same as it runs on any other Meta smart-glasses model — your camera capture, voice triggers, audio playback, and hardware events all work identically. The wearer can use a Display-tier device with your Extentos-built app and get the full app experience. On the Display tier you **can** render UI to the heads-up display through `glasses.display.*` — Meta's DAT opened a third-party display path, and Extentos exposes it as a declarative node tree in both SDKs, fully sim-testable on both; on-glasses rendering via DAT 0.8.0 ships on both platforms, not yet confirmed on Display hardware. (The local-media roots — `image(photo:)`, `video(clip:)` — ship on both.) Input works like this: the wearer controls your display session with **captouch gestures on the temple and EMG gestures on the Meta Neural Band**, and DAT delivers those interactions to your app as **abstracted tap events on your components** (a button press, a selection). What you do *not* get is raw Neural Band access — no EMG stream, no gesture recognition API, no Band input outside a display session. See the [display capability](/docs/concepts/display) page. One more distinction worth knowing: the polished full-screen experiences on Meta Ray-Ban Display — pedestrian navigation, WhatsApp, live captions — are **Meta's own first-party system apps**, not a partner slot that third-party apps compete for. Your app's surface is the display session, rendered through `glasses.display.*`. See [distribution state](#distribution-state-2026-07) for how partner publishing differs from what the public preview allows. ### Compatibility approach Extentos doesn't maintain a model-by-model verification matrix because the underlying SDK (Meta DAT) is the same across every Ray-Ban Meta and Oakley Meta variant. A capability that works on one model works on all of them, modulo hardware tier. New Meta-branded smart-glasses models should work without code changes as long as Meta keeps shipping them behind DAT — the consistent pattern since the toolkit launched in late 2025. > **As of 2026-07** — model lineup and toolkit boundaries move with Meta's release cadence. Current state lives at Meta's [Wearables Developer Center](https://wearables.developer.meta.com/), the [DAT GitHub repository](https://github.com/facebook/meta-wearables-dat-android), and the [Meta smart-glasses product page](https://www.meta.com/smart-glasses/). ## What's in the Wearables Device Access Toolkit (DAT) Meta DAT is the official public SDK for talking to Meta smart glasses from third-party Android and iOS apps. It is a **developer preview** (0.x versions — 0.8.0 as of 2026-06), open to any developer. The modules are `mwdat-core` / `MWDATCore` (registration, device discovery, sessions, permissions), `mwdat-camera` / `MWDATCamera`, `mwdat-display` / `MWDATDisplay`, and `mwdat-mockdevice` / `MWDATMockDevice`. As of 0.8.0 the toolkit plus the standard Bluetooth profiles expose: ### Capabilities (public developer preview) | Capability | API surface | Notes | |---|---|---| | **Photo capture** | `stream.capturePhoto()` during an active stream; photos arrive as `PhotoData` (iOS: `photoDataPublisher`) | JPEG stills from the glasses' 12 MP camera. Short-lived stream pattern — Extentos manages the stream lifecycle for you. | | **Video frame stream** | `session.addStream(StreamConfiguration)` | Frame rates 2 / 7 / 15 / 24 / 30 fps; resolutions LOW 360×640, MEDIUM 504×896, HIGH 720×1280. Quality adapts to Bluetooth bandwidth automatically. | | **Video recording (clip)** | Extentos records the DAT frame stream **phone-side** (`glasses.camera.captureVideo`) | DAT has no clip API — nothing is recorded to glasses storage. Extentos encodes the stream on the phone and hands you a file. | | **Microphone (mic stream)** | Bluetooth **HFP** — standard platform audio (`AudioRecord` on Android, `AVAudioEngine` on iOS), not a DAT API | Wideband speech (mSBC, 16 kHz mono), beamformed to the wearer's voice. Speech recognition runs on the phone. Continuous capture is supported while your app runs. | | **Audio playback (TTS / earcon)** | Phone-side TTS + Bluetooth **A2DP** | Audio is synthesized on the phone and played out the glasses speakers. | | **Session & hardware events** | DAT session/device state + platform APIs | Closing the hinges drops Bluetooth and stops the session (hinge position itself is not exposed); Extentos surfaces it as `connection.state → Disconnected(cause)` — today the cause arrives as `DeviceDroppedConnection` (dedicated `HingesClosed`/`ThermalCritical` causes are declared, not yet wired). Thermal events come from the phone's own thermal APIs. | | **Display rendering** (Display tier) | `MWDATDisplay` — Text, Image, Button, Icon, FlexBox nodes + MP4 playback | Full-layout sends (no partial updates). Wearer interactions (captouch + Neural Band) arrive as abstracted tap events. Extentos wraps this as `glasses.display.*`. | | **Mock Device Kit** | `mwdat-mockdevice` artifact / `MWDATMockDevice` SPM module | Simulates the device layer; test on a phone or emulator without paired glasses, including simulated camera feeds. | ### Capabilities NOT in the public toolkit These are real Meta smart-glasses features but not third-party-accessible: - **The "Hey Meta" wake-word engine.** The glasses run a dedicated low-power engine that is always listening for Meta AI's reserved phrases ("Hey Meta"). In DAT's model those are system-owned *transactions*, and the engine is closed to third parties — you can't register your own phrase with it, and you don't receive its triggers. A custom wake phrase is therefore a *phone* job: stream the normal glasses microphone over Bluetooth and match the phrase with phone-side speech recognition. That's exactly what Extentos's `wake_word` primitive and the `listening_mode` toggle implement for you. - **Raw Neural Band input.** There is no EMG stream and no gesture-recognition API. On Meta Ray-Ban Display the Band is how the wearer drives your display UI, but DAT hands your app only the abstracted result (tap/selection events on your components) — never the gesture itself. - **Custom gesture recognition.** Captouch behavior is system-defined: taps pause/resume a session, tap-and-hold stops it, and display apps receive tap events on components. You can observe all of that, but you can't map arbitrary tap/swipe patterns to your own semantics. - **Motion sensors (IMU).** DAT 0.8.0 exposes no accelerometer, gyroscope, or magnetometer API. When Meta's docs say "sensors" they mean camera, microphone, and speakers. - **Listening while your app isn't running.** Continuous mic capture works for as long as your integration is active (see [audio architecture](#audio-architecture-developer-relevant)), but there's no way to install an always-on background listener that outlives your app — that privilege belongs to Meta's own wake-word engine. > **Why these matter for app design:** voice triggers in your app must be custom phrases recognized on the *phone* over Bluetooth audio — not "Hey Meta, my app, do X". The glasses are the *microphone*; the phone is the *recognizer*. ## Distribution state (2026-07) This is the part that surprises new Meta smart-glasses developers. DAT is in developer preview, and distribution reflects that: there are two paths, and the fully public one isn't open to everyone yet. ### Path 1 — Release-channel distribution (works today) During the preview, an end user can only *register* your glasses integration if they're enrolled as a tester. The flow: - In the [Meta Wearables Developer Center](https://wearables.developer.meta.com/), you create **versions** of your integration and assign them to **release channels**. - You invite testers by email. Each invitee needs a Meta account and accepts the invitation (testers manage theirs at [wearables.meta.com/invites](https://wearables.meta.com/invites)). Channel membership is what allows a user to register your app with their glasses through the Meta AI app. - The **phone app itself** ships however you normally distribute: sideload/internal/enterprise tracks or Google Play on Android; TestFlight, enterprise, or Ad Hoc on iOS. - **iOS App Store publishing is not yet supported.** Meta states this explicitly: the SDK currently uses Apple's `ExternalAccessory` framework, which leads to App Store rejection under the MFi program and privacy-manifest requirements. Meta plans to support App Store publishing in the future; until then, TestFlight plus release channels is the iOS path. - There's no Meta *content review* of your app — the gate is tester enrollment, not approval. Distribution features require recent glasses firmware (v125+). This is the path every non-partner Meta smart-glasses app uses today, and the one Extentos optimizes for. ### Path 2 — Public publishing (partner-gated during the preview) Publishing an integration so that *any* user can register it — no tester invitation — is limited to select Meta partners while the toolkit is in developer preview. There is no public submission flow yet; Meta has said the ability to publish to the broader community is expected in **2026**. There's also no store-style storefront for glasses integrations today — users connect and manage integrations through the Meta AI app. > **As of 2026-07** — distribution policy is the area most likely to evolve. Meta's annual Connect conference (typically September) is historically when developer-program expansions are announced. ### Partner integrations (the "how did they get that" question) A small set of launch partners built with DAT ahead of and during the preview, and they're the visible demos of what the toolkit can do: - **Twitch** and **Streamlabs (Logitech)** — hands-free POV livestreaming from the glasses camera - **18Birdies** — real-time golf yardages and club recommendations by voice - **Be My Eyes** and **Microsoft Seeing AI** — camera-based accessibility for blind and low-vision users - **L+R** and **Pixel and Texel** — early design/engineering partners Two things people often conflate with DAT partnerships, which are different programs entirely: **Meta AI voice integrations** (Spotify, Audible, and similar "Hey Meta, play…" experiences are Meta AI feature partnerships, not DAT apps), and **Meta's first-party Display experiences** (pedestrian navigation, WhatsApp, live captions on Meta Ray-Ban Display are Meta's own system apps — navigation, for instance, is Meta's, not a Google Maps integration). The partners' distribution privilege is publishing reach — the *capability surface* they build on is the same public DAT you have. A new third-party developer should plan for the public preview toolkit plus release channels as of 2026-07. ## Development state — what you can test with today Building and testing an app for Meta smart glasses in 2026 is more accessible than the distribution story suggests. The development tools are open to everyone: | Tool | What it does | Cost | |---|---|---| | **Meta DAT SDK** | The real toolkit — Android (`mwdat-core`, `mwdat-camera`, `mwdat-display`) and iOS (`MWDATCore`, `MWDATCamera`, `MWDATDisplay`) | Free | | **Meta Mock Device Kit** | Simulates the device layer on a phone or emulator, including simulated camera feeds | Free, ships with DAT | | **Meta Wearables Developer Center registration** | One-time signup with your personal Meta credentials; gives you `MetaAppID`, `ClientToken`, `TeamID`, plus versions/release channels for testers | Free, ~15–30 min setup | | **Real Meta smart-glasses hardware** | Pair to your phone over Bluetooth using the Meta AI app, then your DAT app registers against it | Cost of the glasses | | **Extentos browser simulator** | App-layer simulator at [extentos.com/s](https://extentos.com/s) — tests the wearer experience without real hardware | Free email-only account (Google or email + password); unlimited sessions once linked | | **Extentos `LocalSimTransport`** | Extentos's own in-memory local sim; same `glasses.*` API works against simulator and real glasses | Free forever, no account | You don't need permission from Meta to start developing. Sign up at the Wearables Developer Center, drop the DAT SDK into your Android or iOS app (or use Extentos, which wraps it), and you can test today against Mock Device Kit, Extentos's browser simulator, or your own paired glasses. The *public-distribution* gate is the hard one. *Development* is open. ## Required setup for development Concrete checklist for a third-party Meta smart-glasses app, regardless of whether you use Extentos: ### Both platforms - Meta Wearables Developer Center account (one-time, free) - Meta App ID and Client Token from the Developer Center - Paired glasses to test on real hardware (or Mock Device Kit / Extentos simulator instead) ### Android - Min SDK: 31 (the Extentos library floor; `com.extentos:glasses` requires API 31+ — DAT itself supports lower) - Maven artifacts under the `com.meta.wearable` group (singular): `com.meta.wearable:mwdat-core`, `com.meta.wearable:mwdat-camera`, `com.meta.wearable:mwdat-display`, optional `com.meta.wearable:mwdat-mockdevice` for testing. Pinned to `v0.8.0` in current Extentos builds. - Distributed via Meta's GitHub Packages with token-based access setup - AndroidManifest permissions: `BLUETOOTH_CONNECT`, `BLUETOOTH_SCAN` (the library declares these), plus `RECORD_AUDIO`, `CAMERA` (app-declared, depending on capabilities), `INTERNET` - Signature registered with the Meta App ID for production ### iOS - Min iOS: 16.0 — the Extentos package floor. (Meta's DAT itself tracks the Meta AI app's OS minimums, which are lower, and supports Xcode 14+; the Extentos package builds with the Swift 6 toolchain.) - Swift Package Manager dependency on `https://github.com/facebook/meta-wearables-dat-ios` - SPM modules: `MWDATCore`, `MWDATCamera`. (Meta also ships an optional `MWDATMockDevice` module for testing; Extentos builds don't consume it.) Pinned to `from: "0.8.0"` in current Extentos builds. - `Info.plist` keys: - `MWDAT` dictionary with `MetaAppID`, `ClientToken`, `TeamID`, `AppLinkURLScheme` (a custom URL scheme — not a universal link) - `CFBundleURLTypes` matching the `AppLinkURLScheme` - `LSApplicationQueriesSchemes` includes `fb-viewapp` (to query the Meta AI app) - `UISupportedExternalAccessoryProtocols` includes `com.meta.ar.wearable` - `UIBackgroundModes` includes `bluetooth-central`, `bluetooth-peripheral`, `external-accessory` - Privacy strings: `NSCameraUsageDescription`, `NSMicrophoneUsageDescription`, `NSSpeechRecognitionUsageDescription`, `NSBluetoothAlwaysUsageDescription` - The companion-app registration callback returns to your app via the configured URL scheme — `Wearables.shared.handleUrl(_:)` must be wired in `onOpenURL` for the registration flow to complete Extentos generates the boilerplate for all of this through the MCP server's `generateConnectionModule` tool — see [Quickstart with an AI agent](/docs/getting-started/with-agent). ## Audio architecture (developer-relevant) This is a recurring source of confusion that's worth pinning down explicitly: - **Microphone capture path:** the glasses microphones physically capture audio (beamformed to the wearer's voice). That audio streams to the phone over Bluetooth via the **HFP/SCO** profile, which negotiates wideband speech (mSBC, 16 kHz mono). Speech recognition runs on the **phone**, not on the glasses. - **Speaker playback path:** TTS audio is synthesized on the **phone**, then routed to the glasses speakers via Bluetooth **A2DP**. Synthesis is a phone job; playback is glasses output. - **Continuous listening is supported.** Your app can hold the HFP mic open for as long as it's running — this is how always-listening assistants and Extentos's continuous transcription work, and it's what the `listening_mode` toggle (`off` / `wake_word` / `on_demand` / `always_on`) governs — see [capabilities](/docs/concepts/capabilities). The honest trade-offs: HFP and A2DP are mutually exclusive, so playback drops from hi-fi A2DP to voice-band HFP mono while the mic is open; battery drain is real; background capture needs a foreground service on Android and Bluetooth background modes on iOS; and the wearer can always pause or stop the session by folding, removing, or tapping the glasses. - **Coexistence constraint:** A2DP (high-quality stereo) and HFP/SCO (low-latency bidirectional) on the same Bluetooth connection compete, and camera streams add pressure. Extentos handles the routing; a blocked combination surfaces as a `CoexistenceBlocked` error on the affected call. - **Zero Extentos runtime cost.** Both platforms handle the hard parts (recognition, synthesis) natively. The Extentos library wires the Bluetooth audio session so the right mic and speaker are in the path. Your shipped app pays nothing to Extentos for voice or audio at runtime. ## Roadmap and what to expect > **As of 2026-07** — this section is forward-looking. Update after each Meta Connect. Based on the toolkit's trajectory since its late-2025 developer-preview launch (versions 0.3 through 0.8 in under a year): - **The release cadence is steady.** Meta has shipped display rendering, video-stream quality controls, Mock Device Kit upgrades, and session-lifecycle improvements across the 0.x line. Expect capability adds to keep landing between Connects. - **Third-party display rendering already opened** — Extentos surfaces it as `glasses.display.*` in both SDKs, fully sim-testable; on-glasses rendering via DAT 0.8.0 ships on both platforms, not yet confirmed on Display hardware. - **Public publishing is the one to watch.** Meta has said broader publishing (beyond select partners) is expected in 2026, and that App Store publishing support is planned. Watch Meta Connect (September). - **Voice invocation may be coming.** The Wearables Developer Center already lists a "voice invocation" permission alongside camera and microphone, with no public API documented yet — a hint that launching integrations by voice is on Meta's radar. - **Custom gestures** are plausible future-toolkit work — captouch hardware exists and display apps already receive abstracted taps; only a general-purpose API surface is missing. - **"Hey Meta" wake-word-engine access is unlikely to open** — the engine is core to Meta AI's product positioning. Plan on phone-side recognition for custom phrases. Plan for the public DAT preview as it stands. Treat release-channel distribution as the production path until broader publishing opens. ## How Extentos relates Extentos sits on top of Meta DAT and abstracts it: - **`RealMetaTransport`** wraps the real DAT SDK (`mwdat-core` / `MWDATCore` + `mwdat-camera` / `MWDATCamera`, plus `mwdat-display` on Android — the iOS package links `MWDATCore` + `MWDATCamera`). Your app code talks to `glasses.camera.capturePhoto()`; Extentos translates to DAT calls. - **`LocalSimTransport`** is Extentos's own in-memory deterministic transport. Same `glasses.*` API, no real glasses needed, no network, no DAT SDK. - **`BrowserSimTransport`** is Extentos-original — a browser-based simulator that tests the *wearing experience* (voice triggers, TTS, hardware alerts) at the app layer rather than the Bluetooth layer. See [transport vs app simulation](/docs/concepts/transport-vs-app) for the framing. When Meta opens new public-toolkit capabilities, Extentos adds them to the SDK surface. Display rendering already landed this way (`glasses.display.*`); when Meta opens custom gestures or broader publishing, Extentos's capability vocabulary and the simulator extend to cover them. Your handler code doesn't change shape. When other vendors enter the market, Extentos ships additional transports and the capability primitives stay vendor-agnostic. That is demonstrated rather than promised: the [Android XR](/docs/vendors/android-xr) transport landed in 2026-07 without changing a single app-facing API, so existing Meta handler code already runs under an Android XR device identity in the simulator. ## Frequently asked questions ### Can I distribute my Meta smart-glasses app to the public today? Not store-style, yet. During the DAT developer preview, public publishing is limited to select Meta partners; Meta has said broader publishing is expected in 2026. What works today for everyone: distribute your phone app through your normal channels and enroll users as testers in a release channel in the Meta Wearables Developer Center — that enrollment is what lets them register your integration with their glasses. There's no Meta content review involved. ### How do users install my Meta smart-glasses app? Two halves. The *phone app* installs like any app — sideload, internal track, or Google Play on Android; TestFlight, enterprise, or Ad Hoc on iOS (the iOS App Store isn't supported yet — Meta's SDK currently triggers App Store rejection under Apple's MFi and privacy-manifest rules, and Meta plans to fix this). The *glasses link* additionally requires the user to be an invited tester in one of your release channels, after which your app pairs with their glasses via the DAT registration flow through the Meta AI app. ### Why do Meta's own Display apps look richer than what I can build? The full-screen experiences on Meta Ray-Ban Display — pedestrian navigation, WhatsApp, live captions — are Meta's first-party system apps (navigation is Meta's own, not a Google Maps integration). Third-party apps get the display-session surface instead: your own UI rendered through `glasses.display.*` (see the [display capability](/docs/concepts/display)), with wearer input arriving as abstracted tap events. That's a real rendering path — it's just not the system-app tier. ### Can my third-party app trigger on "Hey Meta, [my app], do X"? No. "Hey Meta" is detected by a dedicated always-listening wake-word engine on the glasses, and that engine is system-owned — third parties can't register phrases with it or receive its triggers. Use a custom wake phrase instead: stream the normal glasses microphone over Bluetooth and recognize the phrase on the phone. Extentos's `wake_word` primitive does this for you. ### Can my app listen continuously? Yes, while it's running. The glasses mic streams over Bluetooth HFP for as long as your app holds the session, which is how always-ready voice assistants work — Extentos exposes this through continuous transcription and the `listening_mode` toggle (`always_on`). The trade-offs are battery, playback dropping from hi-fi A2DP to voice-band HFP mono while the mic is open, and platform background-execution rules. What you can't do is listen when your app *isn't* running — that's the wake-word engine, which is Meta-only. ### Does my app work on all Ray-Ban Meta and Oakley Meta variants? Yes. Extentos runs on every model in market — Ray-Ban Meta Gen 1 and Gen 2 (Wayfarer, Headliner, Skyler, and other frame styles), Ray-Ban Meta Optics, Oakley Meta HSTN, Oakley Meta Vanguard, Meta Glasses (the 2026 Meta-branded line), and Meta Ray-Ban Display. Frame style doesn't affect capabilities. Camera, microphone, and speaker support is identical across every model. ### Does Extentos work with Meta Ray-Ban Display? Yes. Your Extentos-built app runs on a Meta Ray-Ban Display the same way it runs on Gen 1 or Gen 2 — full camera, voice trigger, and audio support. You can also render UI to the heads-up display via `glasses.display.*` when `glasses.display.isAvailable` reports the surface is present — guard on availability and fall back to voice on display-less devices. On-glasses rendering via DAT 0.8.0 ships on both platforms, not yet confirmed on Display hardware. The wearer drives your display UI with captouch and the Neural Band; your app receives the interactions as tap events. Raw Neural Band gesture access isn't part of the public toolkit. See the [display capability](/docs/concepts/display). ### What Meta smart-glasses models does Extentos support? All of them. Compatibility is determined by Meta's Wearables Device Access Toolkit, which is uniform across every Ray-Ban Meta and Oakley Meta variant — see the [model table](#compatible-models) above. Newer Meta-branded models that ship after this writing should work without code changes as long as Meta keeps shipping them behind DAT. ### Do I need a paid Meta developer account to ship a Meta smart-glasses app? No. The Meta Wearables Developer Center account is free. The DAT SDK is free. Mock Device Kit is free. Meta charges nothing for using the public preview toolkit. ### What happens to my app when the user folds their glasses? Closing the hinges drops the Bluetooth link and stops the device session (DAT doesn't expose hinge position directly — the fold shows up as the session ending). Extentos surfaces it through `glasses.connection.state` as a `Disconnected` transition (today with cause `DeviceDroppedConnection`; the dedicated `HingesClosed` cause is declared but not yet wired) — branch on the disconnect state and decide what to do. Reopening the hinges restores Bluetooth but does not auto-restart the session; Extentos's reconnect flow handles starting a fresh one. Most apps cancel active work (`scope.cancel()` / `task.cancel()`) and wait for reconnection. ### Is real-hardware testing required to develop a Meta smart-glasses app? No. Extentos's on-device `LocalSimTransport` (in-memory local sim) and the Extentos browser simulator (`BrowserSimTransport` at [extentos.com/s](https://extentos.com/s)) cover the bulk of the dev loop. Real hardware verifies the final-mile fit — fidelity of the camera frames, latency of the voice loop, real-world A2DP/HFP coexistence — but is not required for daily iteration. ### What about Apple smart glasses? Apple has signaled smart-glasses development but has not shipped a third-party SDK as of 2026-07. Extentos plans to add an Apple transport once Apple opens a developer toolkit; your handler code won't need to change. ### What about Android XR? Android XR is in preview: the projected transport is built and proven on Google's emulator, and the simulator carries an Android XR device identity, so you can already check how your app behaves under it. The module is unpublished and no Android XR glasses are buyable, so nothing there is production-supported. See [Android XR](/docs/vendors/android-xr) for the exact status. ## Related - **[Transport vs app simulation](/docs/concepts/transport-vs-app)** — how Meta DAT's Mock Device Kit and Extentos's browser simulator complement each other - **[Capabilities](/docs/concepts/capabilities)** — Extentos's vendor-agnostic capability primitives - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install the MCP server and start building - **[iOS quickstart](/docs/getting-started/ios)** · **[Android quickstart](/docs/getting-started/android)** - **[Pricing](/docs/resources/pricing)** — Extentos is free; Meta DAT is free; only the glasses cost money