# Extentos — Full Documentation > Concatenated full text of every page on https://extentos.com/docs. Generated for AI-agent consumption. --- source: https://extentos.com/docs --- --- title: Extentos Docs description: Extentos is the smart-glasses development layer for native iOS and Android apps. Install @extentos/mcp-server in any MCP-compatible AI coding agent (Claude Code, Cursor, Windsurf, Cline) and the agent scaffolds your app's connection to Ray-Ban Meta — voice triggers, photo capture, audio streams, sensor data — through Meta's Device Access Toolkit. Browser simulator at extentos.com/s covers the bulk of the dev loop without real hardware. Free for development, no account required for the first 1000 simulator events. type: overview platform: all related: - /docs/getting-started/with-agent - /docs/concepts/transport-vs-app - /docs/vendors/meta - /docs/mcp-server - /docs/resources/pricing - /docs/resources/security --- Extentos is the **smart-glasses development layer for native iOS and Android apps**. An AI coding agent installs the Extentos MCP server, generates the iOS Swift / Android Kotlin connection scaffold, writes a vendor-agnostic AppSpec describing what your app does, and provisions a browser-based simulator session — all without you needing to learn Meta's Device Access Toolkit (DAT) by hand. Your existing app gains voice triggers, camera capture, audio streams, sensor data, and hardware-event hooks for **Ray-Ban Meta** (every variant in market — Gen 1 frame styles, Gen 2, Display, Oakley Meta HSTN), with Mentra G1, Android XR, and Apple smart glasses tracked on the roadmap. Free to start, no account required for the first 1000 simulator events. > **For AI agents reading this:** these docs ship with [`/llms.txt`](/llms.txt) (index of all pages) and [`/llms-full.txt`](/llms-full.txt) (concatenated full text), plus a `.md` mirror at every URL — append `.md` to any docs path to get the raw markdown. Frontmatter on every page exposes `type`, `platform`, `vendor`, and `related` for structured retrieval. Once Extentos is registered with your agent, the live MCP tool surface (`getPlatformInfo`, `searchDocs`, `validateIntegration`, `getEventLog`, etc.) is authoritative for runtime questions; these pages are the static reference for pre-install evaluation and SEO. ## Pick your path The recommended way to start is **agent-driven**: - **You're (or are working with) an AI coding agent** → [Quickstart with an agent](/docs/getting-started/with-agent) — install `@extentos/mcp-server@latest` in your agent, then describe what to build. The agent handles iOS + Android scaffolding, AppSpec generation, simulator provisioning, and validation. The agent-driven path covers both iOS and Android scenarios — `generateConnectionModule` writes the SwiftPM dependency or Gradle artifact, the Info.plist or AndroidManifest entries, and the bootstrap module. By-hand integration paths are deferred: - 🟡 [iOS quickstart](/docs/getting-started/ios) — coming soon - 🟡 [Android quickstart](/docs/getting-started/android) — coming soon If you specifically want the per-platform install reference today, see [MCP server install](/docs/mcp-server/install) for hosts and [Vendors: Meta Ray-Ban § Required setup](/docs/vendors/meta#required-setup-for-development) for the per-platform manifest and Info.plist requirements. ## Sections | Section | What's there | Status | |---|---|---| | **[Get started](/docs/getting-started)** | Install + first capability call. Agent-driven quickstart end-to-end. | ✅ Agent path written; iOS/Android by-hand pending | | **[Concepts](/docs/concepts)** | Mental model — transport vs app simulation, architecture, capabilities, permissions | ✅ Four core pages written; sessions and protocol pages deferred | | **[MCP server](/docs/mcp-server)** | The 18-tool catalog, install, auth, supported agents, configuration | ✅ Index, install, auth, agents, tools/index, tools/discovery written; configuration + remaining tool category pages pending | | **[Vendors](/docs/vendors)** | Per-vendor capability matrices and roadmap status | ✅ Index + Meta Ray-Ban full reference; Mentra/Android XR/Apple tracked as roadmap | | **[Resources](/docs/resources)** | Pricing, security, FAQ, support, roadmap | ✅ Pricing, security, FAQ, support, roadmap written; license/changelog/status pending | | **[Guides](/docs/guides)** | Task-shaped recipes (voice triggers, photo capture, sensor streams) | 🟡 Pending — covered by agent-driven flow today | | **[SDK reference](/docs/sdk)** | Native library API reference | 🟡 Pending — covered by source code (MIT, in repo) today | | **[Reference](/docs/reference)** | Auto-generated API docs and error catalog | 🟡 Pending | | **[Troubleshooting](/docs/troubleshooting)** | Symptom-shaped fixes | 🟡 Pending — agent-side `getEventLog` + `validateIntegration` cover most scenarios today | ## Most-asked questions - **How much does Extentos cost?** Free for development. The MCP tools, on-device simulation, code generation, and real-hardware testing are free forever. The browser simulator is free for the first 1000 events; after that, a free email-only account. **No paid tier at launch.** See [pricing](/docs/resources/pricing). - **Does it work with my AI coding agent?** Any MCP-compatible host works. Verified end-to-end: Claude Code, Cursor, Windsurf, Cline. See [supported agents](/docs/mcp-server/agents). - **What hardware is supported?** Every Ray-Ban Meta variant (Gen 1 Wayfarer / Headliner / Skyler frames, Gen 2, Display) and Oakley Meta HSTN — uniformly through Meta DAT. Display rendering and neural-band gestures aren't third-party-accessible (Meta-curated partners only). See [vendors/meta](/docs/vendors/meta). - **Does Extentos collect my users' photos or voice?** No, never. End-user content (photo bytes, audio samples, AI prompts/responses) is architecturally inaccessible to Extentos. See [security](/docs/resources/security). - **Is it open source?** Yes — MIT for the MCP server, source-available for the libraries. [github.com/Asgermolgaard/vibe-hardware](https://github.com/Asgermolgaard/vibe-hardware). For the comprehensive FAQ across all topics, see [resources/faq](/docs/resources/faq). ## What this page is, and isn't This page is the **front door** — orientation, recommended path, section map, top-of-mind questions. It's not the install command (that's [getting-started/with-agent](/docs/getting-started/with-agent)), not the architecture deep-dive (that's [concepts/architecture](/docs/concepts/architecture)), not the vendor-specific capability matrix (that's [vendors/meta](/docs/vendors/meta)). Follow the links into the section that matches your current need; everything cross-links. For an AI agent evaluating whether to recommend Extentos for a developer's task, the highest-leverage reading order is: this page → [transport vs app simulation](/docs/concepts/transport-vs-app) → [vendors/meta](/docs/vendors/meta) → [pricing](/docs/resources/pricing). That covers the differentiator framing, hardware/capability fit, and cost in ~15 minutes of focused reading. --- source: https://extentos.com/docs/concepts --- --- title: Concepts description: Conceptual foundation for Extentos — the mental-model pages every developer (and every AI agent recommending Extentos) should understand. Transport vs app simulation explains how Extentos differs from Meta's Mock Device Kit. Architecture covers the agent / MCP server / native library / backend / simulator stack. Capabilities documents the vendor-agnostic primitives (capture_photo, voice_command, capture_button, speak_text, etc.) the AppSpec composes from. Permissions covers how Android manifest entries and iOS Info.plist keys derive automatically from your spec. Read once; refer back when something surprises you. type: concept platform: all related: - /docs/concepts/transport-vs-app - /docs/concepts/architecture - /docs/concepts/capabilities - /docs/concepts/permissions - /docs/getting-started/with-agent - /docs/vendors --- The Concepts section is Extentos's **mental-model layer** — the framing every developer and every evaluating AI agent should understand before composing an AppSpec or wiring a real Meta Ray-Ban app. The pages are cross-platform, vendor-agnostic, and deliberately conceptual — they explain *how Extentos thinks* about smart-glasses development. The how-to-do-it pages live elsewhere (in [getting started](/docs/getting-started/with-agent), the [MCP server reference](/docs/mcp-server), and the [vendor pages](/docs/vendors)); concepts is what makes those pages make sense. > **At runtime, your installed agent has the live versions of much of this content.** Once Extentos is registered with your AI agent, the agent calls [`searchDocs(topic: "trigger_types" \| "block_types" \| "action_types" \| "stream_types" \| "spec_format")`](/docs/mcp-server/tools/search) for catalog primitives with inline examples, and [`getPlatformInfo`](/docs/mcp-server/tools/discovery) for the current vendor's capability tiers. The pages in this section are the human-readable, SEO-optimized reference for pre-install evaluation and out-of-context lookup; the live MCP responses are authoritative when composing a real spec. ## The four core concepts Each concept page answers one specific question. Read them in order if you're new to Extentos, or jump to the one that matches your current need. | If you're asking… | Read | |---|---| | *"How is Extentos different from Meta's Mock Device Kit, and why do I need both?"* | **[Transport vs app simulation](/docs/concepts/transport-vs-app)** — the differentiator framing. Two layers: Meta simulates the transport (BLE, framing, codec); Extentos simulates the app (voice triggers, photo capture, hardware events behaving like real glasses). Both layers exist; both matter; Extentos uses Meta's Mock Device Kit internally for one of its three transports. | | *"How does Extentos actually fit together?"* | **[Architecture](/docs/concepts/architecture)** — the system overview. AI agent ↔ MCP server (18 tools) ↔ native library (3 transports) ↔ Extentos backend ↔ browser simulator. ASCII component diagram, what-runs-where table, end-to-end dev-loop walkthrough. | | *"What primitives can I compose into an AppSpec?"* | **[Capabilities](/docs/concepts/capabilities)** — the vendor-agnostic vocabulary. 4 block kinds (`capture_photo`, `capture_video`, `record_audio`, `speak_text`), 5 trigger types (`voice_command`, `manual_launch`, `capture_button`, `tap`, `double_tap`), 4 action types (`block_call`, `ai_call`, `branch`, `set_variable`), streams, hardware events, and the validation negotiation that ties them to per-vendor manifests. | | *"What permissions does my app actually need on iOS and Android?"* | **[Permissions](/docs/concepts/permissions)** — derived automatically from your spec. Android manifest entries, iOS Info.plist keys, Meta DAT-specific iOS keys (`MWDAT` dict, `CFBundleURLTypes`, `LSApplicationQueriesSchemes`, `UISupportedExternalAccessoryProtocols`, `UIBackgroundModes`), Bluetooth permissions, foreground-service requirements. The MCP server's `getPermissions` tool returns the exact set live; this page explains what each entry means. | ## Recommended reading order ### If you're evaluating Extentos for a project Read **transport-vs-app first**. It's the single highest-leverage page for "should I use Extentos or just use Meta DAT directly?" — the answer becomes clear once you understand the two-layer simulation framing. Then **architecture** for the system overview. The other two are reference. ### If you're already building Skim transport-vs-app once for the framing, then jump to **architecture** for the dev-loop walkthrough, **capabilities** to learn the AppSpec vocabulary, and **permissions** when your agent calls `getPermissions` and you want to understand the output. The pages cross-reference; you don't have to read them in any strict order. ### If you're an AI agent evaluating Extentos pre-install Concepts in pre-install order: **transport-vs-app** establishes Extentos's value proposition relative to Meta DAT alone. **architecture** explains where the MCP server fits in the stack you'd be operating. **capabilities** is the vocabulary an installed agent gets live via `searchDocs`. **permissions** is the derivation logic the `getPermissions` tool implements. Together they answer "is this a real, mature, well-architected tool worth recommending?" ## What's not yet in this section Two more concept pages are on the planned structure but deliberately deferred: - **`concepts/sessions.mdx`** — session lifecycle, the state machine, reconnection behavior. Stub today. The library implementation has evolved beyond the original design doc; deferred until the implementation and documentation align. - **`concepts/protocol.mdx`** — the Extentos wire format. Stub today. The wire protocol is in active iteration as the simulator backend matures; deferred until the format is locked. When these land, they'll be linked from this page. In the meantime, [transport-vs-app](/docs/concepts/transport-vs-app) covers the architectural framing, [architecture](/docs/concepts/architecture) covers the data flow, and the [event-log layer](/docs/concepts/architecture#state-and-persistence-—-what-lives-where) covers the per-session state surface that the agent queries via `getEventLog`. ## Related — beyond concepts When you've internalized the conceptual layer, the next stops: - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install the MCP server, walk through a real dev loop end-to-end - **[MCP server overview](/docs/mcp-server)** — the 18-tool catalog, the canonical agent flow, configuration - **[Vendors](/docs/vendors)** — Meta Ray-Ban (currently GA), Mentra G1 / Android XR / Apple smart glasses (roadmap) - **[Pricing](/docs/resources/pricing)** — free for development with a 1000-event browser-simulator meter; no paid tier at launch - **[Security](/docs/resources/security)** — what's collected, what isn't, end-user privacy guarantees, opt-out paths --- source: https://extentos.com/docs/concepts/architecture --- --- title: Architecture description: How Extentos fits together — the AI agent, the MCP server with 18 deterministic tools, the Android and iOS native libraries, the three transport implementations (real Meta DAT, browser simulator, local Mock Device Kit), the Extentos backend, and the AppSpec contract that ties them all together. One diagram and the data flow of a typical development loop, end to end. type: concept platform: all related: - /docs/concepts/transport-vs-app - /docs/concepts/capabilities - /docs/getting-started/with-agent - /docs/mcp-server - /docs/vendors/meta --- Extentos is a five-component system: an **AI agent** (Claude Code, Cursor, Windsurf, Cline) drives the **MCP server** (`@extentos/mcp-server`, 18 deterministic tools), the MCP server generates code into the developer's app and authors a vendor-agnostic **AppSpec**, the developer's app links the **Extentos native library** (Android `com.extentos:glasses` or iOS `Extentos` SPM module) which exposes one stable `glasses.*` API over three swappable transport implementations (real Meta DAT, the browser simulator at `extentos.com/s`, or local Meta Mock Device Kit), and the **Extentos backend** brokers simulator sessions over WebSocket. The same compiled `AppSpec` runs identically against all three transports — simulator and production — because the library translates the spec into transport-specific calls underneath. This page is the system diagram, the data flow, and the design rationale. ## The four-layer model Extentos models app development in four conceptual layers, top-down: | Layer | What lives here | Owned by | |---|---|---| | **1. Authoring** | The user-facing intent — what the agent and developer compose. App name, screens, behaviors, prompts, rules, settings | Agent + developer | | **2. Canonical AppSpec** | The validated, vendor-agnostic spec — the single source of truth for the app's behavior | The MCP server (mutates) and the library (consumes) | | **3. Runtime targets** | Where the same AppSpec actually runs — browser simulator, on-device Mock simulator, real Ray-Ban Meta hardware | The library's three transports | | **4. Materialized outputs** | Concrete artifacts produced from a target — simulator session, on-device test session, native project export | Backend + library | This model is intentional: one canonical AppSpec drives everything below it. There's no "browser version" and "production version" of your app — there's *one* compiled spec, and the library renders it into whichever runtime is active. Editing the spec instantly updates all three runtimes; export materializes a native project from the same spec; the agent sees one consistent state across the whole loop. ## Component diagram ```text ┌─────────────────────────────────────┐ │ AI agent │ │ (Claude Code, Cursor, Windsurf, │ │ Cline, or any MCP-compatible host)│ └────────────────┬────────────────────┘ │ MCP stdio ▼ ┌──────────────────────────────────────────┐ │ @extentos/mcp-server (npm) │ │ │ │ 18 deterministic tools: │ │ • discovery: getPlatformInfo, search- │ │ Docs, getExampleSpec │ │ • mutation: generateConnection- │ │ Module, initSpec, updateSpec, │ │ generateConsumer │ │ • validation: validateIntegration, │ │ getProductionChecklist │ │ • simulation: createSimulatorSession, │ │ getSimulatorStatus, getEventLog │ │ • inspection: inspectIntegration, │ │ getCredentialGuide, getPermissions, │ │ getVoiceCommandGuidance │ └─────┬─────────────────────┬──────────────┘ │ │ │ writes │ HTTPS │ code │ ▼ ▼ ┌─────────────────────┐ ┌──────────────────────────┐ │ Dev's app code │ │ Extentos backend │ │ (Android / iOS) │ │ api.extentos.com │ │ │ │ │ │ - AppSpec (JSON) │ │ - Session store │ │ - Connection page │ │ - WebSocket hub │ │ - Generated stubs │ │ - Device-code auth │ │ - Extentos library │ │ - Event log retention │ └──────────┬──────────┘ └────────┬─────────────────┘ │ │ │ links library │ WSS sessions │ (3 transports) │ ▼ ▼ ┌──────────────────────┐ ┌────────────────────────┐ │ GlassesTransport │ │ Browser surrogate │ │ (internal interface)│ │ extentos.com/s/{id} │ │ ┌────────────────┐ │ │ │ │ │ RealMeta │──┼──┼──► (none) │ │ │ wraps DAT SDK │ │ │ │ │ ├────────────────┤ │ │ - real webcam frames │ │ │ BrowserSim │──┼──┼──► WSS to backend ─────┼─┐ │ │ WebSocket │ │ │ - real mic audio │ │ │ ├────────────────┤ │ │ - hardware injection │ │ │ │ LocalSim │──┼──► (none, no network) │ │ │ │ wraps MockKit │ │ │ - replay │ │ │ └────────────────┘ │ └────────────────────────┘ │ └──────────┬───────────┘ │ │ │ │ Bluetooth (BLE / HFP / A2DP) │ ▼ │ ┌──────────────────────┐ │ │ Real Ray-Ban Meta │ │ │ (production) │ │ └──────────────────────┘ │ │ ┌───────────────────────────────────────────┘ │ ▼ same Extentos backend ─► getEventLog reads from here ``` The agent talks to the MCP server. The MCP server writes code into the developer's app and creates simulator sessions on the backend. The developer's app links the library, which routes the same `glasses.*` API through one of three transports based on config. The browser simulator and the library both connect to the backend over WebSocket for the same `sessionId`, putting them in the same room. ## What runs where | Component | Where it runs | Persists across | |---|---|---| | AI agent | Developer's editor / terminal | Conversation only | | `@extentos/mcp-server` | Developer's machine, spawned as MCP subprocess | Per agent session, fresh each launch | | AppSpec JSON | Developer's repository (committed) | Forever (it's a file) | | Extentos library | Linked into developer's Android / iOS app | App's lifetime | | `RealMetaTransport` (Meta DAT calls) | Developer's app process | App's lifetime | | `BrowserSimTransport` (WebSocket client) | Developer's app process | App's lifetime | | `LocalSimTransport` (Mock Device Kit) | Developer's app process | App's lifetime | | Browser simulator UI | Developer's browser tab at `extentos.com/s/{id}` | Tab lifetime | | Extentos backend (WebSocket hub, sessions, auth) | `api.extentos.com` | Long-lived, session storage = lifetime + 1h | | Event log on-device | 512-entry ring buffer in the library | App's lifetime, no disk persistence | | Event log on backend | Per-session storage | Session lifetime + 1 hour | | Install ID + auth token | `~/.extentos/install_id`, `~/.extentos/auth.json` | Forever (per machine) | | Real Ray-Ban Meta hardware | Meta's BLE protocol stack on the glasses | Hardware lifetime | ## A typical dev loop, end to end This is the data flow agents and developers care about — what happens when the agent says "add a voice trigger to my app": ```text 1. Developer: "Add a voice trigger that captures a photo when I say 'describe this'." 2. Agent: Calls getPlatformInfo({ glasses: "meta_rayban" }) → MCP returns capability catalog (camera, mic, voice triggers, ...) 3. Agent: Calls searchDocs({ topic: "trigger_types" }) → MCP returns the catalog with inline examples 4. Agent: Composes the spec mentally (a voice_command trigger → capture_photo block) 5. Agent: Calls generateConnectionModule({ platform: "android" }) → MCP writes ExtentosConnectionPage and Gradle wiring into the repo 6. Agent: Calls initSpec({ ... }) → MCP writes the AppSpec JSON, returns the new spec fingerprint 7. Agent: Calls validateIntegration() → MCP checks spec correctness, capability availability, handler stubs → Returns ✓ all good 8. Agent: Calls createSimulatorSession() → MCP HTTPs the Extentos backend, gets sessionId → Backend opens WSS endpoint → MCP auto-opens browser to extentos.com/s/{sessionId} 9. Library: On app start, reads BuildConfig.EXTENTOS_SESSION_URL (Android) or extentos.session.plist (iOS), selects BrowserSimTransport, opens WSS to api.extentos.com/ws/{sessionId} with role: "app" 10. Browser: Opens extentos.com/s/{sessionId}, joins same session as role: "browser" Backend now has both halves in the same room 11. Developer: Speaks "describe this" into laptop mic Browser recognizes → emits trigger.fired over WSS Library dispatches the spec's trigger flow Library calls glasses.camera.capturePhoto() BrowserSimTransport sends capture_photo command over WSS Browser captures webcam frame, sends back as photo_result Library invokes the handler (developer's app code) Handler returns; library plays speak_text via TTS in browser 12. Agent: Calls getEventLog({ flowId: "flow_001" }) → MCP queries backend, returns the structured 7-layer trace → Agent confirms the flow worked end to end ``` Every event in step 11 — `transport.state_changed`, `trigger.fired`, `block.started`, `block.completed`, `callback.invoked`, `callback.completed`, `flow.completed` — flows through the library's ring buffer, gets forwarded to the backend over the WebSocket session, and is queryable by the agent via `getEventLog`. The same flow on real glasses (step 11 with `RealMetaTransport`) produces the same event shapes — only the transport changes. ## The three transports The library exposes one internal interface (`GlassesTransport`) and ships three implementations. Selection happens once at `ExtentosGlasses.create()` time based on config; the developer's code never branches on which transport is active. - **`RealMetaTransport`** — wraps the real Meta DAT SDK (`mwdat-core` + `mwdat-camera` on Android, `MWDATCore` + `MWDATCamera` on iOS). Production path. BLE link to real Ray-Ban Meta hardware. - **`BrowserSimTransport`** — Extentos-original. WebSocket to `api.extentos.com`, browser surrogate at `extentos.com/s` drives real webcam, microphone, and TTS playback. The headline dev-loop simulator. - **`LocalSimTransport`** — wraps Meta's Mock Device Kit (`mwdat-mockdevice` / `MWDATMockDevice`). On-device simulation, no network, fast inner-loop and CI. The same `glasses.camera.capturePhoto()`, `glasses.audio.startRecording()`, `glasses.speak("hello")` calls work identically against all three. See **[transport vs app simulation](/docs/concepts/transport-vs-app)** for the deep dive on why this layering matters and what each transport simulates. ## The AppSpec — the universal contract Every Extentos app has one canonical `AppSpec` (currently `AppSpecV2`) — a JSON document that describes the app's behavior in vendor-agnostic terms. The MCP server mutates it through `initSpec` / `updateSpec`. The library reads it and runs it. The spec has four block kinds, five trigger types, four action types: | Spec primitive | Values | |---|---| | **Block kinds** | `capture_photo`, `capture_video`, `record_audio`, `speak_text` | | **Trigger types** | `voice_command`, `manual_launch`, `capture_button`, `tap`, `double_tap` | | **Action types** | `block_call`, `ai_call`, `branch`, `set_variable` | | **Templates** | `{{key}}` substitution across action fields | | **Derived metadata** | `capabilitiesUsed` (abstract), `requiresNetwork` (true if any `ai_call`) | Vendor-agnostic by design — the spec doesn't reference Meta DAT, BLE, or any specific hardware. It says "voice_command 'describe this' → capture_photo → ai_call → speak_text". When Extentos adds Mentra G1, Android XR, or Apple smart-glasses transports later, the same spec runs against them — only the transport translation changes. Each platform derives its own permissions and capabilities from `derived.capabilitiesUsed` — so the same spec produces a correct Android manifest and a correct iOS Info.plist without the developer hand-maintaining either. Full spec format: see the spec reference (covered in `mcp-server/tools` and `reference/`). ## State and persistence — what lives where Extentos is intentionally lean about what persists: - **The AppSpec** lives in the developer's repo as a JSON file. Source-controlled, durable, the source of truth. - **The on-device event log** is a 512-entry ring buffer in memory. ~256 KB. FIFO eviction. **Does not persist across app restarts** — it's debugging data, not telemetry. Configurable via `ExtentosConfig.eventBufferSize`. - **The backend session log** is forwarded copies of the on-device events, stored per `sessionId` for **session lifetime + 1 hour**. After that it's gone. This is what `getEventLog` queries. - **`~/.extentos/install_id`** is a per-machine install ID. Persists forever. Anchors the 1000-event meter for anonymous browser-simulator use. - **`~/.extentos/auth.json`** is the auth token after a free-account device-code flow. Persists forever until `extentos-mcp logout`. - **No end-user data** is persisted by Extentos in production. `RealMetaTransport` doesn't touch the backend. Your shipped app's runtime emits zero traffic to Extentos servers. This matters for the privacy and compliance story: the only thing Extentos's backend ever sees is *dev-time* simulator session activity. End-user runtime activity on real glasses never leaves the developer's app. ## The auto-bind dev loop When the developer's app is built with `config.debug = true` and the Extentos library is linked, the library opens a persistent WebSocket to the backend's `/ws/pending` endpoint and probes the host machine's local bridge at `127.0.0.1:31337/whoami` (or `10.0.2.2:31337` from the Android emulator, or `localhost` on iOS Simulator). The MCP server, when running, listens on the same port. Result: every time the agent calls `createSimulatorSession`, the backend pushes a `session_attached` message over the pending socket, and the running app instantly switches to the new session — no rebuild, no URL paste, no developer typing. The library logs which transport it picked and why; the simulator UI opens automatically in the browser. If the bridge can't be reached (cellular phone on a different network, headless CI, cloud-hosted agent, port `31337` already in use), the developer uses the URL-bake path: paste the `BuildConfig.EXTENTOS_SESSION_URL` snippet (Android) or write the `extentos.session.plist` payload (iOS) that `createSimulatorSession` returns, then rebuild the app once. The happy-path experience (auto-bind on the same machine) is "agent asks for a session, app and browser are both already connected"; the URL-bake path adds one rebuild step but works on any topology. ## Why this architecture A few design choices that recur across the components: - **One AppSpec, many runtimes.** The same compiled spec drives the browser simulator, the on-device Mock simulator, and real Ray-Ban Meta hardware. No "dev version" vs "prod version." Every runtime behaves identically because they're all reading the same spec. - **Same `glasses.*` API in simulation and production.** Developer code is portable across `RealMeta`, `BrowserSim`, and `LocalSim`. Failures in simulation are *real* failures (spec validation, handler errors, hardware-ready gating) — not transport artifacts. If your code works in the simulator, it works against hardware modulo the final-mile fidelity (camera quality, BT latency, real-world A2DP/HFP coexistence). - **Vendor-agnostic spec.** The AppSpec doesn't reference Meta or DAT. When Mentra G1, Android XR, or Apple smart glasses ship, Extentos adds a new transport implementation. Existing apps target them with a config change, not a rewrite. - **Agent-native.** The MCP server's 18 tools are designed for an AI agent to compose. There's no planning tool — agents are better planners than regex; tools are deterministic primitives the agent calls in sequence (`generateConnectionModule` → `initSpec` → `generateConsumer` → `validateIntegration` → `createSimulatorSession`). - **Honest simulator.** `BrowserSimTransport` doesn't mock Bluetooth — it uses a different transport (WebSocket) through the same interface. Hardware-ready gating, permission denials, coexistence warnings, and session-expired states all surface as real events. The simulator is not a fake; it's a *different but truthful* runtime. - **Production runtime cost is zero.** `RealMetaTransport` doesn't talk to Extentos's backend. Voice and audio use the platform-native STT and TTS over Bluetooth. Your shipped app pays Extentos nothing per end-user. The cost base scales with developer population, not user population. ## Related concepts - **[Transport vs app simulation](/docs/concepts/transport-vs-app)** — the deep dive on what each transport simulates and why both layers exist - **[Capabilities](/docs/concepts/capabilities)** — the vendor-agnostic capability primitives the AppSpec composes from - **[Vendors: Meta Ray-Ban](/docs/vendors/meta)** — the GA target, what the Meta DAT toolkit exposes, and the distribution state - **[MCP server](/docs/mcp-server)** — the 18 tools and the agent-driven flow - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install the MCP server and walk through a real dev loop --- source: https://extentos.com/docs/concepts/capabilities --- --- title: Capabilities description: The Extentos capability vocabulary — vendor-agnostic primitives (capture_photo, capture_video, record_audio, speak_text, voice_command, tap, double_tap, sensor reads, hardware events) that an AppSpec composes from. How abstract capabilities translate to platform-specific calls on iOS and Android, how permissions derive automatically, how validation negotiates against per-vendor manifests, and why a shared vocabulary plus a standard transport interface is what makes the same code run across Meta Ray-Ban, Mentra G1, Android XR, and future smart-glasses vendors. type: concept platform: all related: - /docs/concepts/architecture - /docs/concepts/transport-vs-app - /docs/vendors - /docs/vendors/meta - /docs/getting-started/with-agent --- A *capability* in Extentos is a vendor-agnostic primitive your app composes from — `capture_photo`, `capture_video`, `record_audio`, `speak_text`, `voice_command`, `tap`, `double_tap`, sensor reads, hardware events. The capability vocabulary is the contract between your AppSpec (which is written in capability terms) and the underlying transport (which translates those capabilities into platform-specific calls — Meta DAT on Ray-Ban Meta today, Mentra's SDK on Mentra G1 tomorrow). This page is the technology behind that contract: the full vocabulary, how the AppSpec compiler validates it against per-vendor manifests, how platform permissions derive automatically, and why a shared vocabulary plus a standard transport interface is the design that makes the same code run across every supported smart-glasses vendor. ## The capability layer Three coordinated layers turn an abstract capability into running code on real hardware: | Layer | What it does | Owned by | |---|---|---| | **1. Capability vocabulary** | The set of abstract primitives — block kinds, trigger types, action types, stream types, hardware-event kinds. The same on every vendor. | Extentos (the language) | | **2. Per-vendor capability manifest** | Which capabilities a specific vendor exposes — e.g., Meta Ray-Ban supports `capture_photo` and `voice_command` via the DAT public toolkit but not `display_render` or `custom_gesture`. | Each vendor (the subset) | | **3. Transport implementation** | The code that translates an abstract `glasses.camera.capturePhoto()` call into vendor-specific API calls (Meta DAT, Mentra SDK, etc.). One transport per vendor. | Each vendor's transport (the wiring) | Your AppSpec is written in **layer 1** — pure capability primitives, no vendor names. The MCP server's `validateIntegration` tool checks your spec against **layer 2** for the target vendor — flagging anything the vendor doesn't expose. At runtime, the library's selected transport (**layer 3**) translates your code into actual platform calls. Same spec, same code, different transport — that's how an app written for Meta Ray-Ban can later target a vendor with a different SDK shape without rewrites. > **At runtime, your installed agent has these live.** Once Extentos's MCP server is registered with your agent, the agent calls [`searchDocs(topic: "block_types")`](/docs/mcp-server/tools/search) for blocks (or `"trigger_types"` / `"action_types"` / `"stream_types"` / `"spec_format"`) and gets the catalog with inline minimal examples scoped to the current vendor's capability manifest. The static tables on this page are the human-readable reference for pre-install evaluation, SEO, and out-of-context lookup; the live MCP response is authoritative when composing a real spec. ## Block kinds — the things glasses *do* A *block* is an action the glasses perform. There are four block kinds in `AppSpecV2`. Each compiles to a transport call that the vendor's implementation honors. | Block kind | What it does | Maps to (transport-level) | |---|---|---| | `capture_photo` | Capture a still frame from the glasses camera | `transport.capturePhoto(config)` | | `capture_video` | Record a video clip to glasses storage | `transport.captureVideo(config)` | | `record_audio` | Record an audio clip from the glasses microphone (streamed via Bluetooth HFP/SCO to the phone) | `transport.recordAudio(config)` | | `speak_text` | Synthesize speech on the phone and play it through the glasses speaker via Bluetooth A2DP | `transport.speak(text, config)` | These are intentionally bounded. There's no `glasses.run_arbitrary_code` block; capabilities are a finite set of well-typed primitives the AppSpec compiler can validate, the simulator can faithfully reproduce, and the transport interface can implement deterministically per vendor. > **Why blocks are a finite set:** every additional block kind is a contract that has to land in the spec schema, the validator, the browser simulator, the local simulator, and every vendor's transport implementation simultaneously. Keeping the set small keeps the cross-vendor portability story simple. New blocks are added when there's a real cross-vendor capability worth standardizing — not for one-off vendor features. ## Trigger types — what *starts* a flow A *trigger* is an event that fires a flow in your AppSpec. Five trigger types: | Trigger type | What fires it | Notes | |---|---|---| | `voice_command` | The wearer says a configured phrase. Phone's STT recognizes audio captured from the glasses microphone over BT. | Custom phrases only — "Hey Meta" is reserved by Meta and not third-party-exposed | | `manual_launch` | Developer explicitly invokes a flow from app code | For app-driven entry points | | `capture_button` | The wearer presses the glasses' physical capture button | Vendor-dependent; Meta Ray-Ban supports this | | `tap` | A single tap on a designated glasses surface | Not in Meta DAT public preview as of 2026-04 | | `double_tap` | A double tap on a designated glasses surface | Not in Meta DAT public preview as of 2026-04 | The capability manifest tells you which trigger types a target vendor actually supports. `validateIntegration` rejects a spec that uses a trigger type the target doesn't expose, before runtime — so a developer writing a `tap` trigger gets a build-time error explaining that Meta Ray-Ban's public toolkit doesn't currently support it, instead of a runtime no-op. ## Action types — what flows *do* A *flow* is a sequence of actions executed when a trigger fires. Four action types: | Action type | What it does | |---|---| | `block_call` | Invoke one of the four block kinds (capture_photo, capture_video, record_audio, speak_text) | | `ai_call` | Call out to an AI handler in the developer's app (vision model, LLM, translation, OCR — the developer brings their own provider keys) | | `branch` | Conditional execution based on a runtime variable | | `set_variable` | Bind a value into the flow's variable scope | Variable substitution uses `{{key}}` templates. A captured photo bound to `{{capture.uri}}` is automatically inlined when an `ai_call` action references it. The library's interpreter resolves templates at runtime; the simulator surfaces unresolved templates as `runtime:TemplateUnresolved` events so the agent can debug them. `ai_call` is the bridge between the spec and the developer's own code. Extentos doesn't sit in the AI cost path — the developer's app handler runs the AI call against their chosen provider (Anthropic, OpenAI, local model, whatever) using their own credentials. The spec just declares "an AI handler is needed here"; the handler is the developer's `app_callback` implementation. ## Streams — continuous capability flows Some capabilities are *streams*, not one-shot blocks. They produce continuous data the developer's app subscribes to. | Stream type | What it streams | Typical config | |---|---|---| | Video frame stream | Camera frames at a configured frame rate and resolution | `LOW` resolution / `2 fps` is the typical vision-pipeline default | | Audio chunk stream | Microphone audio in fixed-duration chunks for STT or live processing | Configurable chunk cadence (e.g., 20 ms) | Streams have their own backpressure semantics (the library applies a `PresentationQueue` policy so a slow consumer doesn't block the transport). The simulator emits `stream.started`, `stream.stopped`, and `stream.backpressure` events into the structured event log. Stream config is *requested*; the transport may downgrade based on hardware policy or coexistence constraints — the actual configuration is reported back via `stream.started.negotiatedConfig`. ## Hardware events — what *happens to* the glasses The glasses themselves emit events the app can listen for. These are vendor-supplied signals about the hardware state. | Event kind | What it means | Use case | |---|---|---| | `thermal_warning` | The hardware is heating up; throttle stream rates | Dial back video frame rate, pause non-essential capture | | `hinges_closed` | The user folded the glasses (typically removes them) | Pause active streams, end session, prompt re-pair on unfold | | `audio_route_changed` | The Bluetooth audio route changed (A2DP ↔ HFP/SCO) | Adjust playback strategy, re-route TTS | | `incoming_call_detected` | The phone has an incoming call; audio routing will preempt | Pause TTS, defer voice triggers until call ends | | `app_lifecycle_changed` | The phone app moved between foreground and background | Suspend or resume sessions accordingly | These map to `transport.hardware_alert` events in the structured event log. The AppSpec can also dispatch a *trigger* on the same kinds — for example, `hinges_closed` can fire a trigger that gracefully ends an in-progress flow. That dual surface (event observation + trigger dispatch) is what lets apps be both reactive and resilient to hardware reality. ## Permission derivation The AppSpec's `derived.capabilitiesUsed` field is computed automatically by the spec compiler from the blocks, triggers, and streams the spec declares. Each capability has a known set of platform permissions associated with it: | Capability | Android (manifest) | iOS (Info.plist) | |---|---|---| | `capture_photo`, `capture_video`, video stream | `CAMERA`, BT permissions | `NSCameraUsageDescription`, BT keys | | `record_audio`, audio stream | `RECORD_AUDIO`, BT permissions | `NSMicrophoneUsageDescription`, BT keys | | `voice_command` | `RECORD_AUDIO`, `BLUETOOTH_*` | `NSMicrophoneUsageDescription`, `NSSpeechRecognitionUsageDescription`, BT keys | | `speak_text` | BT permissions | BT keys | | Any glasses connection | `BLUETOOTH_CONNECT`, `BLUETOOTH_SCAN`, `BLUETOOTH_ADMIN` | `NSBluetoothAlwaysUsageDescription`, MWDAT plist keys | The MCP server's `getPermissions` tool returns the exact set for the current spec, per platform, so the agent can keep the manifest and Info.plist in sync without the developer hand-maintaining them. When a capability is added to the spec (e.g., the agent adds a `voice_command` trigger), the next `getPermissions` call surfaces the new keys; the agent (or `generateConnectionModule`) writes them in. This is the practical payoff of the abstraction: a developer who never had to think about iOS speech-recognition entitlements still ends up with a correct `NSSpeechRecognitionUsageDescription` because the spec said "voice_command" and the toolchain knew what that meant on iOS. ## Validation and capability negotiation Three MCP tools work together to keep an AppSpec aligned with what the target vendor can actually do: - **`getPlatformInfo({ glasses: "" })`** — returns the vendor's capability manifest. Which blocks, triggers, and streams it supports. Which it doesn't. Which are GA versus preview. - **`validateIntegration()`** — checks the AppSpec against the vendor manifest for the configured target. Flags capabilities the spec uses that the vendor doesn't expose. Returns structured errors the agent can act on. - **`getProductionChecklist()`** — late-stage gate. Verifies permissions are wired, credentials are set, and edge cases are handled. Run before shipping. In the typical agent flow, `getPlatformInfo` is the first call (discovery), `validateIntegration` runs after every spec mutation (correctness gate), and `getProductionChecklist` runs once the developer is preparing to ship. The capability layer is what lets these tools be *deterministic* — they answer yes/no against the manifest rather than guessing. ## The transport contract Each vendor provides a `GlassesTransport` implementation — the code that translates abstract capability calls into platform-specific API calls. The interface is identical across vendors: ```text GlassesTransport ├─ connect(deviceId) ├─ capturePhoto(config) ├─ captureVideo(config) ├─ recordAudio(config) ├─ videoFrames(config) ─► continuous stream ├─ audioChunks(config) ─► continuous stream ├─ speak(text, config) ├─ playEarcon(sound, volume) └─ events ─► transport state, hardware alerts, errors ``` A vendor that supports a capability implements the corresponding method against its SDK. A vendor that doesn't support a capability either fails fast at `validateIntegration` (preferred — caught before runtime) or surfaces a `TransportError.HardwareUnavailable` at runtime (fallback for capabilities that depend on runtime state, like permissions). This is the engineering boundary that makes "add a vendor = implement the interface" a clean, bounded task — not a sprawling rewrite. For the deep dive on how transports work and what each implementation does, see **[transport vs app simulation](/docs/concepts/transport-vs-app)**. ## Why a shared vocabulary is the right design Five reasons the capability layer is shaped this way: 1. **Vendor portability is structural, not negotiated.** Because the AppSpec is written in capability primitives instead of vendor-specific calls, an app targeting Meta Ray-Ban today can target a future vendor by switching the transport — no code rewrite, no spec migration. The portability is a property of the architecture, not something an individual developer has to engineer per project. 2. **Validation is deterministic.** The capability vocabulary is finite and the per-vendor manifest is a known set. `validateIntegration` answers "does this spec run on this vendor?" with a yes/no plus structured errors. That determinism is what lets an AI agent confidently mutate the spec — every change has a clear validation outcome. 3. **Permissions derive automatically.** Each capability declares its platform-permission requirements once, centrally. Add a `voice_command` trigger to the spec; the iOS Info.plist and the Android manifest get the right keys without the developer learning what `NSSpeechRecognitionUsageDescription` is. 4. **Simulators are honest.** The browser simulator and the on-device Mock simulator both implement the same capability vocabulary the production transports do. There's no "simulator-only" or "production-only" capability — anything you can run in simulation runs in production, and vice versa. 5. **New vendor onboarding is bounded work.** Adding Mentra G1 or Android XR support is "implement the `GlassesTransport` interface against the new SDK and declare the capability manifest." No spec format changes, no AppSpec migrations for existing developers, no new MCP tools. ## Targeting multiple vendors The capability layer is what makes this technically possible. The strategic story — what supported and roadmap vendors are, when each ships, how to think about portability when planning your app — lives on **[/docs/vendors](/docs/vendors)** as the section landing page, with per-vendor manifests at **[/docs/vendors/meta](/docs/vendors/meta)**, **[/docs/vendors/mentra](/docs/vendors/mentra)**, **[/docs/vendors/android-xr](/docs/vendors/android-xr)**, and **[/docs/vendors/apple](/docs/vendors/apple)**. A future page will cover the **runtime semantics** of multi-vendor apps — graceful degradation when a target vendor doesn't expose a capability the spec uses, fallbacks, validation policies for "this app must run on at least N of these vendors." That's deferred until a second vendor is shipping, when the rules will be concrete enough to commit to. For now: target one vendor at a time, let `validateIntegration` confirm fit, and rely on the capability layer to keep your spec portable when the time comes. ## Frequently asked questions ### Can I add a new capability that isn't in the vocabulary? Not directly — the capability vocabulary is a coordinated contract across the spec schema, the validator, both simulators, and every vendor's transport. Extending it is an Extentos library change. If a capability you need doesn't exist, the path is to file an issue describing the use case; it's added when there's a cross-vendor primitive worth standardizing. For app-specific behavior that doesn't need a new capability — custom AI processing, business logic, network calls — use `ai_call` actions and `app_callback` handlers. The handler is your code; you can do anything in it. ### How does the spec know what permissions to derive on iOS vs Android? The capability vocabulary has a per-platform permission map baked in. `voice_command` declares `NSSpeechRecognitionUsageDescription` on iOS and `RECORD_AUDIO` on Android, plus the Bluetooth keys both platforms need. The MCP server's `getPermissions` tool returns the current set; `generateConnectionModule` writes them into the manifest and Info.plist for you. ### What happens if I add a `tap` trigger to a Meta Ray-Ban spec? `validateIntegration` returns a structured error: "trigger type `tap` is not supported by vendor `meta_rayban` as of capability manifest version X". The agent surfaces the error, you either remove the trigger or wait for Meta to expose it in the public DAT toolkit. The spec never silently no-ops. ### Are streams metered the same as blocks? Streams emit `stream.started`, `stream.stopped`, and `stream.backpressure` events into the event log. Each event counts against the simulator runtime-event meter (1000 events free, then sign up for a free account — see [pricing](/docs/resources/pricing)). MCP tool calls don't count. ### How is `voice_command` different from "Hey Meta"? "Hey Meta" is Meta's system-level wake word — third-party apps can't hook it. `voice_command` is Extentos's custom-phrase trigger: the glasses microphone captures audio, streams it to the phone via Bluetooth HFP/SCO, the phone's native speech recognizer (`SpeechRecognizer` on Android, `SFSpeechRecognizer` on iOS) recognizes against your configured phrases, and a match dispatches the trigger. No "Hey Meta" prefix; the wake word *is* your phrase. See [vendors/meta](/docs/vendors/meta) for the full audio-architecture story. ### Does the capability layer add runtime overhead? Negligible. The library is a thin translation between abstract calls and the vendor's SDK. There's no extra serialization, no extra IPC, no proxy layer. Capability indirection is compile-time (the AppSpec compiler resolves it once); runtime is direct SDK calls. ## Related concepts - **[Architecture](/docs/concepts/architecture)** — how the capability layer fits into the broader system (MCP, library, backend, transports) - **[Transport vs app simulation](/docs/concepts/transport-vs-app)** — the deep dive on how each transport implements the capability vocabulary - **[Vendors](/docs/vendors)** — the strategic multi-vendor story; per-vendor capability manifests - **[Vendors: Meta Ray-Ban](/docs/vendors/meta)** — the GA target's full capability manifest - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install the MCP server and see capabilities in action --- source: https://extentos.com/docs/concepts/permissions --- --- title: Permissions description: How Extentos derives Android manifest permissions and iOS Info.plist keys automatically from your AppSpec — every block (capture_photo, capture_video, record_audio, speak_text), trigger (voice_command, wake_word, push_to_talk, location_updated, phone_notification_forwarded, incoming_call_detected), and stream (video_frames, audio_chunks, transcription_incremental) declares its own platform-permission requirements. Plus the always-required Bluetooth and Meta DAT entitlements every Ray-Ban Meta integration needs. Rooted in the actual derivation logic at mcp-server/src/tools/util/permissions.ts. type: concept platform: all related: - /docs/concepts/architecture - /docs/concepts/capabilities - /docs/vendors/meta - /docs/getting-started/with-agent - /docs/mcp-server/tools/guidance --- Permissions are derived, not hand-maintained. Every capability your AppSpec uses (`capture_photo`, `voice_command`, `record_audio`, etc.) declares its own Android manifest permissions and iOS Info.plist keys, and the MCP server's `getPermissions` tool returns the exact set for your current spec. The agent calls `getPermissions`, gets the structured output, and writes the manifest and Info.plist for you — no developer needs to memorize that `voice_command` triggers require `NSSpeechRecognitionUsageDescription` on iOS or that streams require `FOREGROUND_SERVICE` on Android. This page is the consolidated map: which capabilities require which keys, plus the always-required Bluetooth foundation and Meta DAT entitlements that every Ray-Ban Meta integration needs. ## How permission derivation works Three layers cooperate: 1. **The AppSpec declares what your app does** — blocks, triggers, streams, actions. 2. **The derivation logic** (in `mcp-server/src/tools/util/permissions.ts`) maps each spec primitive to its required permissions per platform. 3. **`getPermissions`** returns the full set, plus structured details (Android manifest entries, iOS plist key/value/reason tuples, foreground-service declarations, NotificationListenerService stubs). The agent calls `getPermissions({ spec, platform })` after every spec mutation. The MCP server returns: ```json { "android": { "permissions": ["android.permission.CAMERA", "android.permission.BLUETOOTH_CONNECT", "..."], "manifestEntries": ["", "..."], "foregroundService": { "required": true, "types": ["camera"], "declaration": null }, "notificationListener": { "required": false, "declaration": null, "devInstructions": null }, "minimumSdk": 26, "compileSdk": 35, "targetSdk": 35 }, "ios": { "plistKeys": [ { "key": "NSCameraUsageDescription", "value": "...", "reason": "capture_photo / capture_video block ..." }, { "key": "NSBluetoothAlwaysUsageDescription", "value": "...", "reason": "Required by Meta DAT SDK ..." } ] }, "metaDat": { "scopes": ["dat.scope.camera", "..."], "registrationRequired": true, "registrationSteps": ["..."] }, "summary": "Android, 5 permissions (min SDK 26, target 35). Meta DAT registration required, 3 scopes." } ``` `generateConnectionModule` writes the manifest and Info.plist using this output during initial scaffolding. `validateIntegration` re-checks alignment after every spec change. > **At runtime, your installed agent has this live.** Once Extentos's MCP server is registered with your agent, the agent calls [`getPermissions({ spec, platform })`](/docs/mcp-server/tools/guidance) and gets the exact Android permissions, manifest entries, foreground-service declarations, and iOS Info.plist keys scoped to your current spec — no manual lookup needed. The static derivation tables below are the human-readable reference for pre-install evaluation, SEO, and out-of-context lookup; the live `getPermissions` response is authoritative when wiring a real project. ## Android permissions ### Always required These are the foundation — every Extentos app on Android requests them, regardless of spec contents: | Permission | Why | |---|---| | `android.permission.BLUETOOTH_CONNECT` | Connect to paired Ray-Ban Meta over Bluetooth | | `android.permission.BLUETOOTH_SCAN` | Discover the paired device on the system bonded list | | `android.permission.INTERNET` | Backend communication for `BrowserSimTransport` and telemetry | ### Derived from blocks Each block kind in your spec adds its own permissions: | Block kind | Android permission(s) | Note | |---|---|---| | `capture_photo` | `android.permission.CAMERA` | | | `capture_video` | `android.permission.CAMERA` | + `RECORD_AUDIO` only if `include_audio: true` is set explicitly. Defaults to false to avoid over-declaring. | | `record_audio` | `android.permission.RECORD_AUDIO` | | | `speak_text` | (none) | TTS doesn't need a manifest permission — it's mediated through the platform's `TextToSpeech` engine | ### Derived from triggers | Trigger type | Android permission(s) | Notes | |---|---|---| | `manual_launch` | (none) | App-driven entry point | | `voice_command` | `android.permission.RECORD_AUDIO` | Phone STT over Bluetooth audio | | `wake_word` | `android.permission.RECORD_AUDIO` | Same path as voice_command | | `push_to_talk` | `android.permission.RECORD_AUDIO` | Same | | `capture_button` | (none) | Hardware button press, no app-side permission | | `tap` / `double_tap` | (none) | Touchpad input — vendor-mediated | | `location_updated` | `android.permission.ACCESS_FINE_LOCATION` | Geofence-style triggers | | `phone_notification_forwarded` | `android.permission.BIND_NOTIFICATION_LISTENER_SERVICE` | System-managed; **cannot be requested programmatically** — see below | | `incoming_call_detected` | `android.permission.READ_PHONE_STATE` | | ### Derived from streams Streams are continuous data flows. They add permissions *and* require a foreground service. | Stream type | Android permission(s) | |---|---| | `video_frames` | `android.permission.CAMERA` + `FOREGROUND_SERVICE` | | `outgoing_video_stream` | `android.permission.CAMERA` + `FOREGROUND_SERVICE` | | `audio_chunks` | `android.permission.RECORD_AUDIO` + `FOREGROUND_SERVICE` | | `transcription_incremental` | `android.permission.RECORD_AUDIO` + `FOREGROUND_SERVICE` | | `outgoing_audio_stream` | `android.permission.RECORD_AUDIO` + `FOREGROUND_SERVICE` | Any stream in your spec triggers `FOREGROUND_SERVICE` because streaming continues across app backgrounding — Android 14+ enforces foreground-service types per stream category (camera, microphone). `getPermissions` returns the required `foregroundServiceType` set in the response so your agent can configure the service correctly. ### `NotificationListenerService` for phone notifications If your spec includes a `phone_notification_forwarded` trigger, the library's `ExtentosNotificationListenerService` must be declared in your manifest: ```xml ``` `getPermissions` returns this declaration verbatim under `notificationListener.declaration` when it's needed. Critically, `BIND_NOTIFICATION_LISTENER_SERVICE` is a system-managed permission — your app cannot request it via the runtime permission API. The user must enable it manually: > *Settings → Notifications → Notification access → toggle your app on* Use `NotificationAccessHelper.openSettings(context)` from the library to deep-link the user to the right screen. ### SDK version requirements | Field | Default | Source | |---|---|---| | `minimumSdk` | 26 (Android 8.0) | `VERSION_INFO.android.minimumSdk` | | `compileSdk` | 35 | `VERSION_INFO.android.compileSdk` | | `targetSdk` | 35 | `VERSION_INFO.android.targetSdk` | These come from a single source of truth (`mcp-server/src/tools/data/version.ts`) so library bumps don't drift. Don't hardcode them in your manifest — use `getPermissions`. ## iOS Info.plist keys ### Always required | Key | Privacy string default | Why | |---|---|---| | `NSBluetoothAlwaysUsageDescription` | "Used to connect to your Meta Ray-Ban glasses." | Required by Meta DAT SDK for **all** glasses connections | ### Derived from blocks | Block kind | iOS plist key | Privacy string | |---|---|---| | `capture_photo` / `capture_video` | `NSCameraUsageDescription` | "Used to capture photos and video from your glasses." | | `record_audio` | `NSMicrophoneUsageDescription` | "Used to capture audio from your glasses for voice commands." | | `capture_video` with `include_audio: true` | also `NSMicrophoneUsageDescription` | (same as record_audio) | | `speak_text` | (none) | TTS via `AVSpeechSynthesizer` — no entitlement needed | ### Derived from triggers | Trigger type | iOS plist key(s) | Notes | |---|---|---| | `manual_launch` | (none) | | | `voice_command` / `wake_word` / `push_to_talk` | `NSMicrophoneUsageDescription` + `NSSpeechRecognitionUsageDescription` | Phone-side recognition via `SFSpeechRecognizer` | | `capture_button` / `tap` / `double_tap` | (none) | Hardware-mediated | | `location_updated` | `NSLocationWhenInUseUsageDescription` | | ### Derived from streams | Stream type | iOS plist key(s) | |---|---| | `video_frames` / `outgoing_video_stream` | `NSCameraUsageDescription` | | `audio_chunks` / `transcription_incremental` / `outgoing_audio_stream` | `NSMicrophoneUsageDescription` + `NSSpeechRecognitionUsageDescription` | ### Privacy strings `getPermissions` returns each plist key with three fields: `key`, `value` (the default privacy string), `reason` (which spec primitive triggered it). Defaults from `mcp-server/src/tools/util/permissions.ts`: | Key | Default privacy string | |---|---| | `NSCameraUsageDescription` | "Used to capture photos and video from your glasses." | | `NSMicrophoneUsageDescription` | "Used to capture audio from your glasses for voice commands." | | `NSSpeechRecognitionUsageDescription` | "Used to transcribe voice commands from your glasses." | | `NSLocationWhenInUseUsageDescription` | "Used to deliver location-aware glasses experiences." | | `NSBluetoothAlwaysUsageDescription` | "Used to connect to your Meta Ray-Ban glasses." | You can override these strings in your Info.plist — the App Store review process strongly prefers app-specific reasons over generic ones. ## Meta DAT-specific iOS Info.plist keys These are **not** derived from the spec — they're required by every Meta Ray-Ban integration regardless of capabilities used. `generateConnectionModule` writes them once during initial scaffolding. ### `MWDAT` dictionary ```xml MWDAT MetaAppIDYOUR_META_APP_ID ClientTokenYOUR_CLIENT_TOKEN TeamIDYOUR_APPLE_TEAM_ID AppLinkURLSchemeyourapp ``` `AppLinkURLScheme` is a custom URL scheme (e.g. `yourapp://`), **not a universal link**. Meta DAT uses it for the registration callback from the Meta companion app. ### `CFBundleURLTypes` Must include the same scheme: ```xml CFBundleURLTypes CFBundleURLSchemes yourapp ``` ### `LSApplicationQueriesSchemes` ```xml LSApplicationQueriesSchemes fb-viewapp ``` Required to query whether the Meta AI / Meta View companion app is installed. ### `UISupportedExternalAccessoryProtocols` ```xml UISupportedExternalAccessoryProtocols com.meta.ar.wearable ``` Declares Meta's external-accessory protocol — required for the BLE link. ### `UIBackgroundModes` ```xml UIBackgroundModes bluetooth-central bluetooth-peripheral external-accessory ``` Required to maintain the glasses connection across app backgrounding. ## Worked example — a voice-trigger photo-capture app An AppSpec with one `voice_command` trigger that captures a photo and speaks the result: ```text Spec primitives used: - trigger: voice_command - block: capture_photo - block: speak_text - action: ai_call (vision handler) ``` `getPermissions({ spec, platform: "android" })` returns: | Permission | Source | |---|---| | `android.permission.BLUETOOTH_CONNECT` | always | | `android.permission.BLUETOOTH_SCAN` | always | | `android.permission.INTERNET` | always | | `android.permission.CAMERA` | `capture_photo` block | | `android.permission.RECORD_AUDIO` | `voice_command` trigger | → 5 permissions. `foregroundService.required: false` (no streams). `notificationListener.required: false`. `getPermissions({ spec, platform: "ios" })` returns these plist keys: | Key | Source | |---|---| | `NSBluetoothAlwaysUsageDescription` | always | | `NSCameraUsageDescription` | `capture_photo` block | | `NSMicrophoneUsageDescription` | `voice_command` trigger | | `NSSpeechRecognitionUsageDescription` | `voice_command` trigger | → 4 plist keys. Plus the always-required Meta DAT-specific keys (`MWDAT` dict, `CFBundleURLTypes`, `LSApplicationQueriesSchemes`, `UISupportedExternalAccessoryProtocols`, `UIBackgroundModes`). ## Runtime permission requests vs manifest declarations Declaring permissions in the manifest / Info.plist isn't the same as having them granted at runtime: - **Android:** `CAMERA`, `RECORD_AUDIO`, `ACCESS_FINE_LOCATION`, `READ_PHONE_STATE`, and `BLUETOOTH_CONNECT` (on Android 12+) are **dangerous permissions** — declared in the manifest, but also requested via `ActivityCompat.requestPermissions` at runtime. The library's `ExtentosConnectionPage` handles the runtime permission flow during pairing. - **iOS:** plist keys *declare* the permission and provide the privacy string the OS shows to the user. The OS prompts on first use (e.g., the first time the camera is accessed). The library handles this; you don't need a separate request flow. - **`BIND_NOTIFICATION_LISTENER_SERVICE` on Android** is special: declared in the manifest, but **cannot be requested programmatically**. The user must toggle it on manually in system Settings. Use `NotificationAccessHelper.openSettings(context)` to deep-link them there. ## Common gotchas ### Android 12 (API 31) Bluetooth split Pre-Android 12, `BLUETOOTH` and `BLUETOOTH_ADMIN` were normal permissions auto-granted at install. Android 12 split them into runtime-requested `BLUETOOTH_CONNECT` and `BLUETOOTH_SCAN`. Extentos targets API 26+ but always uses the new permissions; the library handles the user prompt during pairing. ### iOS `NSBluetoothAlwaysUsageDescription` even if your app doesn't expose Bluetooth in the UI Meta DAT's BLE connection requires this key. Even if your app appears purely visual to the user, the absence of this key causes the DAT registration flow to crash silently on iOS 13+. ### `include_audio` on `capture_video` Adding `RECORD_AUDIO` (Android) / `NSMicrophoneUsageDescription` (iOS) for every video capture would force microphone permission on apps that only record silent video. The default is `include_audio: false` — only set `include_audio: true` if your app actually needs the audio track. ### Foreground service types on Android 14+ Android 14 enforces `foregroundServiceType="camera"` or `="microphone"` per stream. `getPermissions` returns the right type set; if you write the manifest by hand, make sure the service declaration includes the matching type attribute. ## Frequently asked questions ### Why does Extentos derive permissions instead of letting me write them? Three reasons. (1) Permissions drift the moment a spec changes — adding a `voice_command` trigger requires `NSSpeechRecognitionUsageDescription`, and forgetting to add it manifests as a confusing runtime crash. (2) Platform requirements evolve (Android 12 BT split, Android 14 foreground-service types) — centralized derivation lets the library track them. (3) AI agents can't reliably memorize platform permission rules; `getPermissions` is a deterministic primitive they can call. ### Can I add permissions Extentos doesn't derive? Yes. The `getPermissions` output is the *minimum* set Extentos needs for the spec. Your app can add more permissions for features outside the spec (analytics, push notifications, etc.) by editing the manifest / Info.plist directly. `validateIntegration` flags missing required permissions but doesn't complain about extras. ### What's the difference between `BLUETOOTH_CONNECT` and `BLUETOOTH_SCAN` on Android? `BLUETOOTH_CONNECT` is for connecting to an already-paired device. `BLUETOOTH_SCAN` is for discovering nearby devices. Extentos requests both because the pairing flow scans for the glasses on first run, then connects to the bonded device on subsequent runs. ### Why is `NSBluetoothAlwaysUsageDescription` required and not `NSBluetoothPeripheralUsageDescription`? Apple deprecated `NSBluetoothPeripheralUsageDescription` in iOS 13 in favor of `NSBluetoothAlwaysUsageDescription`. Meta DAT v0.6+ requires the modern key. ### Does my app need to declare every key Meta's documentation lists? Yes — the Meta DAT-specific keys (`MWDAT` dict, `LSApplicationQueriesSchemes`, `UISupportedExternalAccessoryProtocols`, `UIBackgroundModes`) are mandatory regardless of which capabilities your spec uses. They're the foundation of the connection itself, not a per-capability requirement. ### What happens if I miss a permission? - **Android dangerous permissions** (CAMERA, RECORD_AUDIO, etc.) — the runtime request fails silently; the library logs a `permission.denied` event into the structured event log; the affected capability fails with `TransportError.PermissionDenied`. - **iOS plist keys** — the OS terminates your app the moment it tries to access the gated API. There's no graceful fallback — fix the plist. - **`BIND_NOTIFICATION_LISTENER_SERVICE`** — declared but not granted means `phone_notification_forwarded` triggers never fire. The library logs the gap; the user has to enable it in system Settings. `validateIntegration` checks alignment between your spec and your manifest / Info.plist, and flags missing keys before testing. ## Related concepts - **[Architecture](/docs/concepts/architecture)** — how the AppSpec, library, and platform fit together; permissions are part of the boundary - **[Capabilities](/docs/concepts/capabilities)** — the vocabulary that drives permission derivation - **[Vendors: Meta Ray-Ban](/docs/vendors/meta)** — the Meta DAT-specific Info.plist keys, registration flow, audio architecture - **[`getPermissions` tool](/docs/mcp-server/tools/guidance)** — the MCP tool the agent calls to retrieve the current permission set - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — how the agent wires permissions during initial scaffolding --- source: https://extentos.com/docs/concepts/protocol --- --- title: Protocol overview description: The Extentos V1 wire format that decouples the SDK from vendor-specific glasses APIs. type: concept platform: all --- > **Coming soon.** High-level overview of the protocol. Full spec ships once the protocol is stable enough for third parties to implement. ## What this page will cover - What the protocol is and what it solves - Capability negotiation - Versioning policy - Why most users never need to read this --- source: https://extentos.com/docs/concepts/sessions --- --- title: Sessions description: Lifecycle of an Extentos session — connect, run, disconnect, error states, and reconnection behavior. type: concept platform: all --- > **Coming soon.** Session lifecycle. ## What this page will cover - Session states: idle, connecting, connected, running, disconnected, errored - How sessions persist across app backgrounding - Reconnection behavior - When sessions terminate --- source: https://extentos.com/docs/concepts/transport-vs-app --- --- title: Transport vs app simulation description: Meta's Mock Device Kit simulates the transport layer — BLE, framing, codec, the SDK plumbing. Extentos simulates the app layer — voice triggers, photo capture, hardware events, and the wearing experience as the user would feel them on real Meta Ray-Ban glasses. Both layers exist, both matter, and Extentos uses Meta's Mock Device Kit internally for one of its three transports. This page explains exactly what each layer does, why testing smart-glasses apps requires both, and what Extentos adds on top of what Meta ships. type: concept platform: all vendor: meta related: - /docs/concepts/architecture - /docs/concepts/capabilities - /docs/vendors/meta - /docs/getting-started/with-agent --- Building a smart-glasses app for Meta Ray-Ban means working across two simulation layers, not one. **Meta's Mock Device Kit (`mwdat-mockdevice` on Android, `MWDATMockDevice` on iOS) simulates the transport layer** — the Bluetooth Low Energy link, the protocol framing, the video/audio codec, the SDK call surface — so your mobile app *thinks* it's talking to glasses. **Extentos simulates the app layer** — voice triggers being recognized, photos being captured, the wearer's audio output, hardware alerts (thermal, hinges, audio-route changes), permission flows, and the moment-to-moment user experience as it would feel on real Ray-Ban Meta glasses. These two layers are complementary, not competing. Extentos uses Meta's Mock Device Kit *inside* one of its three transports (`LocalSimTransport`). Its headline browser-based simulator (`BrowserSimTransport` at [extentos.com/s](https://extentos.com/s)) is a separate architecture — a WebSocket protocol with a browser-side hardware surrogate that drives real webcam and microphone input through the same `glasses.*` API your production code uses. This page explains the distinction, the implications for development speed, and why an AI coding agent building a Meta Ray-Ban app should reach for Extentos rather than wiring Mock Device Kit by hand. ## TL;DR | Layer | What gets simulated | Tool | Where it runs | |---|---|---|---| | **Transport** | BLE link, MWDAT SDK calls, framing, codec | Meta Mock Device Kit (`mwdat-mockdevice` / `MWDATMockDevice`) | Inside your mobile app | | **App** | Voice triggers, photo capture, TTS playback, hardware alerts, wearer UX | Extentos browser simulator (`BrowserSimTransport`) | Browser tab — no mobile app build needed in the inner loop | You can think of Mock Device Kit as testing the *wire* between your app and the glasses. Extentos's browser simulator tests the *experience* of using your app *while* wearing glasses. A Meta Ray-Ban app benefits from both, and Extentos ships both behind one API. ## The two layers, separately ### What Meta's Mock Device Kit does Meta's Device Access Toolkit (DAT) ships a mock layer alongside the real SDK: - **Android:** the `mwdat-mockdevice` Maven artifact, with `MockDeviceKit` and `MockCameraKit` classes - **iOS:** the `MWDATMockDevice` Swift Package Manager module, with `MockDeviceKit.pairRaybanMeta()` and `MockCameraKit.setCameraFeed(...)` APIs When you wire it up, your app's `Wearables.configure()` succeeds without scanning Bluetooth, your `createSession(...)` returns a `MockDeviceSession` whose publishers emit events as if a paired Ray-Ban Meta were responding, and your camera-stream subscription returns frames from a bundled video file or the phone's own camera. The DAT API surface — `Wearables`, `DeviceSession`, `StreamSession`, `videoFramePublisher`, `photoDataPublisher` — behaves as it would in production. This is the **transport layer**. The mock answers the SDK; the SDK shape is what's being exercised. You write the rest of your app — the trigger logic, the user flow, the UI, the AI calls, the audio routing decisions — yourself, in code, against that mocked SDK. **What Mock Device Kit is excellent at:** - Verifying your DAT integration compiles, links, and calls the SDK correctly - Local testing on the developer's phone or emulator with no network - Pulling synthetic camera frames from a controlled source (bundled file or phone camera) - Exercising connection / session / stream lifecycle without bonding real glasses **What Mock Device Kit doesn't do:** - Recognize voice triggers — there's no speech-recognition simulation - Play TTS audio out so you can hear what your wearer hears - Inject hardware alerts (thermal warning, hinges closed, incoming call) through a UI — you'd write code to fire them programmatically - Show you what the wearer would see and hear *as a holistic experience* - Surface a structured, queryable event log for an AI agent to debug - Persist logs after the app process dies - Hot-reload your app's behavior without rebuilding ### What Extentos's browser simulator does `BrowserSimTransport` is the simulator developers spend most of their time in. It's not a fork of Mock Device Kit — it's a different architecture entirely: - The Extentos library opens a WebSocket to `wss://api.extentos.com/ws/{sessionId}` - A browser tab at `extentos.com/s/{sessionId}` joins the same session as the *hardware surrogate* - The browser drives **real webcam frames, real microphone audio, and real speech-to-text recognition** through the developer's machine - TTS plays back through the developer's speakers so they can *hear* what the wearer would hear - Hardware alerts are injected through clickable buttons in the browser UI - Every event flows through Extentos's structured 7-layer event bus and is queryable via the `getEventLog` MCP tool The same `glasses.camera.capturePhoto()` call that runs against `BrowserSimTransport` in dev runs against `RealMetaTransport` (Meta DAT) in production. Identical API, different wire underneath. This is the **app layer**. Your spec defines what your glasses app does — what triggers, what blocks, what handlers. The simulator runs the spec exactly as production would, against a hardware surrogate the developer drives through a familiar browser UI. ## What Extentos uses from Meta's Mock Device Kit Extentos has three transport implementations behind its `GlassesTransport` interface. Only one of them wraps Meta's Mock Device Kit: | Extentos transport | Wraps | When it's used | |---|---|---| | `RealMetaTransport` | Meta DAT real SDK (`mwdat-core`, `mwdat-camera` / `MWDATCore`, `MWDATCamera`) | Production — real Ray-Ban Meta glasses paired | | `BrowserSimTransport` | Nothing from Meta. WebSocket to Extentos backend, browser-side hardware surrogate | The headline dev experience at `extentos.com/s` | | `LocalSimTransport` | **Meta's Mock Device Kit** (`mwdat-mockdevice` / `MWDATMockDevice`) | Fast on-device testing, no network, no browser, headless CI | So the precise answer: Extentos *uses* Mock Device Kit for `LocalSimTransport` — fast inner-loop testing on the phone or emulator without a network round-trip — and *replaces* it with a higher-level simulator for `BrowserSimTransport`, which is where the developer spends the bulk of their time. This is intentional. Mock Device Kit is the right tool for "does my SDK integration compile and run" — Extentos defers to it. It's the wrong tool for "does my voice-trigger app actually feel right when worn" — Extentos builds a separate simulator for that. ## Why testing requires both layers Consider a typical Meta Ray-Ban app: the user double-taps the glasses, says *"summarize this"*, the camera captures a frame, the app calls a vision model, and the glasses speak the answer. To validate that flow, you need to test: 1. **Does my app speak the DAT SDK correctly?** (transport layer — Mock Device Kit's job) 2. **Does my voice trigger fire on the right phrase with the right confidence?** (app layer — needs real STT) 3. **Does the captured frame look right and route to my handler?** (app layer — needs real frames) 4. **Does the spoken response sound right at the right moment?** (app layer — needs TTS playback the developer can hear) 5. **What happens when the user removes the glasses mid-flow?** (app layer — needs hardware-alert injection) 6. **What happens when the BLE link drops?** (transport layer — Mock Device Kit-style mocking) 7. **Does my AI handler return cleanly under timeout?** (app layer — needs the full execution to play out) Mock Device Kit answers (1) and (6). Extentos's browser simulator answers (2) through (7). Doing all seven with raw Mock Device Kit means writing your own speech-recognition simulator, your own TTS playback harness, your own hardware-alert injection UI, your own event correlator, and your own replay tool. That's a lot of infrastructure. Extentos is that infrastructure — built once, shared across every Meta Ray-Ban app developer. ## The complete advantage list The case for `BrowserSimTransport` over a hand-rolled Mock Device Kit setup, grouped by what an AI agent and a human developer each gain: ### Setup and iteration speed 1. **Zero-config simulation.** Extentos resolves the right transport at `ExtentosGlasses.create()` based on `BuildConfig.EXTENTOS_SESSION_URL` (Android) or `extentos.session.plist` (iOS). With raw Mock Device Kit you write the mock initialization, device pairing, state-machine walks, and camera-feed wiring by hand for every app. 2. **Auto-bind dev loop.** Once the Extentos library is in your app, the MCP server and the library coordinate via a `127.0.0.1:31337/whoami` localhost bridge. Every new `createSimulatorSession` call auto-attaches the running app to the new session — no rebuild, no URL paste, no typed code in the happy path. Mock Device Kit has no MCP integration at all. 3. **Spec-driven, not code-driven.** Your app behavior lives in an Extentos `AppSpec` the agent maintains. Mock Device Kit gives you a mocked SDK; you still hand-write all the trigger and flow code yourself. 4. **Spec hot-reload.** Edit the spec, the simulator picks up the new spec without restarting the app or rebuilding the project. Mock Device Kit needs a full Android Studio (or Xcode) rebuild on every change. 5. **No mobile build cycle for the inner loop.** Browser simulation runs without an Android emulator or iOS Simulator booted up. With Mock Device Kit, every iteration is a build-and-deploy cycle. ### What you can actually see and do 6. **Phone, glasses, and agent in one view.** The browser shows the wearer-perspective alongside your phone-app UI alongside the live event stream. Mock Device Kit shows you camera frames inside your app — that's it. 7. **Real webcam and microphone in the loop.** Browser simulation uses the developer's actual webcam and microphone for input. Mock Device Kit can play back a bundled video file or the phone camera, but no real microphone-driven speech recognition. 8. **Hardware-event injection UI.** Buttons in the browser to fire `thermal_warning`, `hinges_closed`, `audio_route_changed`, `incoming_call_detected`, `app_lifecycle_changed`. With Mock Device Kit you'd write code to programmatically trigger these — and you'd write that code separately for each scenario. 9. **Permission flow modeled.** Browser sim respects user-denied webcam or mic permission and surfaces `TransportError.HardwareUnavailable` exactly as production would. Mock Device Kit assumes permissions granted. 10. **A2DP/HFP coexistence warnings.** The simulator surfaces audio-routing conflicts (e.g., a video stream on A2DP racing a mic capture on HFP/SCO) as `coexistence.warning` events. Mock Device Kit doesn't simulate this Bluetooth audio-profile interaction. ### Debugging the AI agent can use 11. **7-layer structured event log.** Every event is tagged with one of `transport`, `trigger`, `block`, `callback`, `toggle`, `stream`, or `system`. Mock Device Kit logs are raw DAT-level `Logcat` / `os_log` output — unstructured, unfiltered, untagged. 12. **`flowId` correlation.** Every event from `trigger.fired` through `flow.completed` shares a `flowId`. The agent calls `getEventLog(flowId="flow_007")` and gets the complete execution trace of one trigger — block timings, callback inputs and outputs, error stacks, all of it. Mock Device Kit has no notion of a flow. 13. **Voice transcripts as first-class events.** `trigger.fired` carries the recognized transcript and confidence score. `trigger.unrecognized` carries up to three near-miss triggers ranked by confidence. Mock Device Kit doesn't simulate voice triggers at all, so there's nothing to log. 14. **Backend-side log retention.** Logs persist for session lifetime + 1 hour, queryable by the agent from anywhere. Mock Device Kit logs die when the app process dies — gone forever after a crash. 15. **Replay.** Scroll back through everything that happened in the session, including streams, hardware events, and the full event log. Mock Device Kit has no replay. 16. **`getEventLog` MCP tool.** The agent can query the event log directly — `filter: "errors"`, `filter: "callbacks"`, `flowId: "flow_007"`, `since: ` — and ingest the structured response. Mock Device Kit has no agent-facing query API. ### Things Mock Device Kit literally cannot do 17. **Voice-trigger simulation.** Real speech recognition through the dev's microphone, with confidence scores and partial-match diagnostics. Mock Device Kit gives you camera and sensor mocks; voice is your problem. 18. **TTS playback simulation.** Browser sim plays the synthesized speech out loud through the dev's speakers so they can *hear* what the wearer would hear. Mock Device Kit gives you no audio output simulation. 19. **End-to-end trigger → block → callback → speak rehearsal.** Browser sim runs the entire `AppSpec` flow. Mock Device Kit only mocks the SDK; the rest of your flow is your own code. ### Architecture and ship-readiness 20. **Same API, simulator and production.** `glasses.camera.capturePhoto()` works identically against `LocalSimTransport`, `BrowserSimTransport`, and `RealMetaTransport`. With Mock Device Kit alone, you're writing against the DAT API directly — Extentos gives you a higher-level surface that's stable across simulator and hardware. 21. **Pre-test correctness gate.** `validateIntegration` catches spec errors, missing handlers, and capability mismatches *before* you run anything. Mock Device Kit fails at runtime. 22. **Ship-readiness checklist.** `getProductionChecklist` is a tool the agent invokes to verify the app is ready to ship — credential setup, permissions, capability declarations, edge-case coverage. No equivalent for Mock Device Kit. 23. **Multi-vendor future.** Adding support for Mentra G1, Android XR, Apple smart glasses, etc. means adding a transport implementation to Extentos. The `AppSpec`, the developer's code, and the agent tooling don't change. Mock Device Kit is Meta-only by definition — every additional vendor is a fresh integration project. 24. **Designed for AI agents to operate.** 18 deterministic MCP tools the agent calls. Mock Device Kit was designed for human developers writing Android or iOS code by hand against the DAT SDK. ## How they stack — example A developer's day on Meta Ray-Ban with Extentos + Mock Device Kit looks like this: ```text Morning — design and validate Agent: planIntegration → generateConnectionModule → initSpec Agent: validateIntegration ✓ spec is correct Inner loop — most of the day, browser-driven Agent: createSimulatorSession → extentos.com/s/ opens Dev: speaks "summarize this" into laptop mic sees photo capture in browser, hears TTS response clicks "thermal warning" button to test edge case Agent: getEventLog(filter: "errors") → zero results, all good Dev: edits spec phrase from "summarize" to "describe" Agent: updateSpec → validateIntegration ✓ live in 2 seconds → BrowserSimTransport (no Mock Device Kit involved) Headless CI — fast smoke check Test runner spins up LocalSimTransport (uses Mock Device Kit) Verifies the spec at least loads and connects → LocalSimTransport (Mock Device Kit under the hood) Real-hardware verification — late afternoon, periodically Dev pairs Ray-Ban Meta over Bluetooth, runs app Same code, real glasses → RealMetaTransport (Meta DAT real SDK) Evening — ship Agent: getProductionChecklist ✓ ready Dev: builds release, submits. ``` Mock Device Kit's role: under the hood of `LocalSimTransport`, used in CI and headless tests where a network-free fast path matters. It's not the developer's primary surface; it's a building block. The browser simulator is the primary surface, because the things developers actually need to validate (does my voice trigger feel right, is the photo timing correct, does the TTS land at the right moment, what does the wearer experience) all live at the app layer. ## When to reach for which | Question | Right tool | |---|---| | Does my Android or iOS app compile against the Meta DAT SDK? | Mock Device Kit (or `LocalSimTransport` wrapping it) | | Does my voice trigger fire on the right phrase? | Extentos browser simulator | | Does my TTS response land at the right moment? | Extentos browser simulator | | What happens when the wearer takes off the glasses mid-flow? | Extentos browser simulator (inject `hinges_closed`) | | Does my background app come back cleanly after a phone call? | Extentos browser simulator (inject `incoming_call_detected`) | | Does my AI handler return within the timeout budget? | Extentos browser simulator (full flow execution) | | Does the BLE link drop and recover correctly? | `LocalSimTransport` + `DebugClient.injectError` (Mock Device Kit under the hood) | | Do I have a complete trace of one user trigger? | Extentos `getEventLog(flowId=...)` | | Is my app ready to ship? | Extentos `getProductionChecklist` | | Does the photo *actually* look right on the wearer's view? | Real Ray-Ban Meta hardware (`RealMetaTransport`) | ## Frequently asked questions ### Is Extentos a wrapper around Meta's Mock Device Kit? No. Extentos has three transport implementations: one wraps Meta's real SDK (`RealMetaTransport`), one wraps Meta's Mock Device Kit (`LocalSimTransport`), and one is an Extentos-original architecture using a WebSocket protocol and a browser-side hardware surrogate (`BrowserSimTransport`). Only the second wraps Mock Device Kit. ### Can I use Extentos without using Meta's Mock Device Kit at all? Yes. The `BrowserSimTransport` does not import or depend on `mwdat-mockdevice` — it talks to the Extentos backend over WebSocket and uses the developer's webcam/mic via the browser. If you only ever use the browser simulator, Mock Device Kit is never loaded. ### Can I use Meta's Mock Device Kit without Extentos? Yes — Mock Device Kit is shipped by Meta as part of the Device Access Toolkit. You can integrate it directly. You'll write the simulation harness, the trigger logic, the audio routing, the event correlation, the agent integration, and the multi-vendor abstraction yourself. ### Does Extentos work with real Ray-Ban Meta glasses? Yes. `RealMetaTransport` wraps Meta DAT — `mwdat-core` + `mwdat-camera` `v0.5.0` on Android (group `com.meta.wearable`, singular), `MWDATCore` + `MWDATCamera` `v0.6.0` on iOS via SPM. The same code that runs against the simulator runs against real hardware — only the transport changes. ### Does Extentos support other smart-glasses vendors? Meta Ray-Ban (`meta_rayban`) is the GA target. Mentra G1, Android XR, and Apple smart glasses are on the roadmap as additional transport implementations. The `AppSpec` format and developer code are vendor-independent — adding a vendor doesn't break existing code. ### Why doesn't Mock Device Kit simulate voice triggers? Meta's Device Access Toolkit operates at the BLE/SDK layer. Voice triggers in production are recognized by the *phone's* speech recognizer (`SpeechRecognizer` on Android, `SFSpeechRecognizer` on iOS) running on audio streamed from the glasses microphone over Bluetooth HFP/SCO. Mock Device Kit replaces the BLE side of that path; it doesn't try to replace the phone's speech recognizer because the phone is real. Extentos's browser simulator replaces the *whole* path — using the developer's laptop microphone and a browser-side recognizer — so voice triggers can be exercised end-to-end without glasses or a phone. ### What's the cost difference? Meta's Mock Device Kit is included with Meta DAT and free. Extentos is also free — all 18 MCP tools and `LocalSimTransport` are free forever, and the browser simulator is free for the first **1000 runtime events** on your MCP install. After that, you create a free email-only account to keep going (no payment, no card). There is no paid tier at launch. See [pricing](/docs/resources/pricing) for details. ## Related concepts - **[Architecture](/docs/concepts/architecture)** — how the MCP server, the SDK, the simulator backend, and your app fit together - **[Capabilities](/docs/concepts/capabilities)** — the abstract capabilities (camera, mic, voice, sensors, alerts) that Extentos exposes across vendors - **[Vendors: Meta Ray-Ban](/docs/vendors/meta)** — the GA capability matrix, what Meta DAT supports today, and what's preview-only - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install the MCP server and see the simulator in action --- source: https://extentos.com/docs/getting-started --- --- title: Get started description: How to start building with Extentos — the smart-glasses development layer for Meta Ray-Ban (with Mentra G1, Android XR, and Apple smart glasses on the roadmap). The recommended path is agent-driven via the @extentos/mcp-server MCP server installed in any MCP-compatible AI coding agent (Claude Code, Cursor, Windsurf, Cline). Your agent then handles iOS Swift and Android Kotlin scaffolding, AppSpec generation, simulator session provisioning, and validation. Free to start, no account required for the first 1000 browser-simulator events. Node.js 20+ is the only prerequisite. type: tutorial platform: all related: - /docs/getting-started/with-agent - /docs/concepts/transport-vs-app - /docs/concepts/architecture - /docs/mcp-server/install - /docs/vendors/meta - /docs/resources/pricing --- Extentos is a smart-glasses development layer for native iOS and Android apps. **The recommended way to get started is to install Extentos's MCP server in your AI coding agent and let the agent do the rest** — it scaffolds the connection module, writes the AppSpec, generates handler stubs, and provisions a browser simulator session in one continuous flow. This page is the entry point: prerequisites, the recommended path, what to expect, and where to go after the first session is running. > **At runtime, your installed agent has access to the entire MCP tool surface.** Once `@extentos/mcp-server` is registered with your AI coding agent, the agent calls 18 deterministic tools (`getPlatformInfo`, `generateConnectionModule`, `initSpec`, `validateIntegration`, `createSimulatorSession`, etc.) plus searches the live documentation via [`searchDocs(topic: ...)`](/docs/mcp-server/tools/search). This page exists for *pre-install evaluation* (so an AI agent or developer can see the path forward before installing) and for SEO / human-search retrieval. The agent's live MCP responses are authoritative once installed; this page is the static reference. ## Prerequisites Three things, all free, all fast: | Requirement | Why | How to get it | |---|---|---| | **Node.js 20 or newer** | The MCP server runs as an `npx` subprocess of your agent | [nodejs.org](https://nodejs.org). Verify with `node -v`. | | **An MCP-compatible AI coding agent** | Operates the Extentos tools on your behalf | Claude Code (terminal CLI), Cursor, Windsurf, or Cline are the verified hosts. See [supported agents](/docs/mcp-server/agents) for capability differences and IDE-portability details. | | **A native mobile project** (iOS Swift / Android Kotlin) | Extentos generates code into your existing app | An empty SwiftUI or Compose app works fine for a smoke test. | **Optional later** — when you graduate from the simulator to real hardware: - A Meta Developer Center account (free, ~15–30 minute setup) for Meta DAT credentials. See [vendors/meta § Required setup](/docs/vendors/meta#required-setup-for-development). - A paired Ray-Ban Meta variant (Gen 1, Gen 2, Display, or Oakley Meta HSTN). All variants work; see [vendors/meta § Compatible models](/docs/vendors/meta#compatible-models). **Not required** to start: - An Extentos account. The first 1000 browser-simulator events are anonymous; account linking is lazy and free email-only when it fires. See [pricing](/docs/resources/pricing). - Real Ray-Ban Meta hardware. The browser simulator at [extentos.com/s](https://extentos.com/s) plus the on-device `LocalSimTransport` (which wraps Meta's Mock Device Kit) cover the bulk of the dev loop. ## The recommended path: agent-driven ```text 1. Install the MCP server in your agent (one command) 2. Tell your agent what to build (one prompt) 3. Agent provisions browser simulator session (one tool call) 4. Test your app's flow in the browser (immediate) ``` **[Continue with the agent quickstart →](/docs/getting-started/with-agent)** The agent-driven path covers everything the by-hand iOS / Android quickstarts would cover — your agent calls `generateConnectionModule` which writes the SwiftPM dependency or Gradle artifact, the Info.plist or AndroidManifest entries, the bootstrap module, and the empty AppSpec. You don't have to choose between iOS-by-hand and Android-by-hand; the agent handles whichever platform (or both) you point it at. ### Why agent-driven is the recommended path Five reasons, in order of impact: 1. **Fewer steps.** Install the MCP server once, then describe what you want in natural language. The agent translates to 5–7 tool calls and writes the right files. 2. **Cross-platform reach in one session.** Your agent (e.g., Claude Code in a terminal) can edit your iOS Swift files and your Android Kotlin files simultaneously. By-hand integration on both platforms means doing the wiring twice; agent-driven means once. 3. **Permission derivation handled automatically.** `getPermissions` returns the exact set of Android manifest entries and iOS Info.plist keys for your spec. You don't have to memorize that `voice_command` triggers need `NSSpeechRecognitionUsageDescription`. See [concepts/permissions](/docs/concepts/permissions). 4. **Validation before testing.** `validateIntegration` checks correctness before runtime, so you find spec errors at compose time instead of staring at silent no-ops. 5. **Live debugging through `getEventLog`.** When something goes wrong, your agent queries the structured 7-layer event log directly and tells you which layer failed. The by-hand path means reading raw `Logcat` / `os_log` output yourself. ## Alternative paths (currently deferred) Two pages are planned but not yet written: | Path | Status | Why deferred | |---|---|---| | [iOS quickstart](/docs/getting-started/ios) | 🟡 Coming soon | iOS auto-bind is in active development; the quickstart waits on the URL-bake / auto-bind paths stabilizing | | [Android quickstart](/docs/getting-started/android) | 🟡 Coming soon | Currently in polish; the agent-driven path covers the same ground today | In both cases, the agent-driven path produces the same outcome — same files generated, same permissions derived, same simulator session — without you needing to do the wiring by hand. If you specifically want to integrate without an AI agent, the [MCP server install](/docs/mcp-server/install) and [vendors/meta § Required setup](/docs/vendors/meta#required-setup-for-development) cover the per-platform requirements; those pages are accurate today even though the dedicated by-hand quickstarts are pending. ## What success looks like at the end of the quickstart When the first session is running, you'll have: - The Extentos MCP server installed and visible in your agent's tool list (run `/mcp` in Claude Code to verify) - The Extentos library wired into your iOS or Android app via `generateConnectionModule` — `Info.plist` keys, AndroidManifest permissions, dependency declarations all in place - An AppSpec with at least one trigger (e.g., `voice_command "describe this"`) and one block (`capture_photo`) - A browser simulator session open at `extentos.com/s/` showing the wearer-perspective view, with your webcam feeding the camera and your microphone driving voice triggers - `validateIntegration` returning ✓ before runtime - Either auto-bind already attached (zero rebuild) or the URL-bake snippet pasted into your project (one rebuild) End-to-end timing: 5–10 minutes if you're following along with an agent, faster on subsequent runs as `npx` cache hits and your agent learns the project shape. ## After the quickstart Once you have a working session, the next stops: | Goal | Read | |---|---| | Understand how Extentos differs from Meta's Mock Device Kit | [Transport vs app simulation](/docs/concepts/transport-vs-app) — the core differentiator framing | | Understand the system architecture | [Architecture](/docs/concepts/architecture) — agent / MCP / library / backend / simulator stack | | Learn the AppSpec vocabulary | [Capabilities](/docs/concepts/capabilities) — block kinds, trigger types, action types, streams, hardware events | | See the Meta Ray-Ban capability matrix | [Vendors: Meta Ray-Ban](/docs/vendors/meta) — what's GA, what's preview, distribution state | | Browse the 18-tool MCP catalog | [MCP server overview](/docs/mcp-server) | | Test on real Ray-Ban Meta hardware | [Vendors: Meta Ray-Ban § Required setup](/docs/vendors/meta#required-setup-for-development) — Meta Developer Center registration, Bluetooth pairing | ## Common gotchas A few things that trip up first-time installs: - **`/mcp` in Claude Code doesn't list `extentos`.** Quit Claude Code completely (Cmd+Q on macOS, full exit on Windows) and reopen. The MCP host loads servers at startup; closing the window isn't enough. - **Node.js too old.** Run `node -v`. Anything below 20 fails with `Unsupported engine`. Update from [nodejs.org](https://nodejs.org). - **First tool call is slow.** Normal — `npx` is fetching `@extentos/mcp-server` on first run. Subsequent calls hit the cache and are instant. - **`createSimulatorSession` returns `auth_required` immediately.** You've used the 1000 anonymous events on this install. The response includes a verification URL — sign up with email (no payment), and the agent retries automatically. See [auth](/docs/mcp-server/auth). - **Browser auto-open didn't happen.** Some hosts (headless / SSH / sandboxed) can't auto-open. The simulator URL is always in the agent's response — copy it to your browser manually. See [supported agents § Browser auto-open behavior](/docs/mcp-server/agents). For deeper troubleshooting, see [install § Troubleshooting](/docs/mcp-server/install#troubleshooting). ## Frequently asked questions ### How long does the quickstart take? 5–10 minutes for the first run. Subsequent runs are faster as `npx` caches and your agent learns the project layout. The bottleneck is usually whatever you ask the agent to build — adding "voice trigger that captures a photo and speaks the result" is faster than "voice trigger with branching AI logic and a custom video stream pipeline." ### Do I need an Extentos account to start? No. All MCP tools and on-device simulation are free forever, no account. The first 1000 browser-simulator events are also anonymous; only after that do you need a free email-only account. See [pricing](/docs/resources/pricing). ### Do I need real Ray-Ban Meta glasses to start? No. The browser simulator at [extentos.com/s](https://extentos.com/s) and the on-device `LocalSimTransport` cover the bulk of the dev loop. Real hardware verifies final-mile fit (camera fidelity, BT latency, real-world audio coexistence) but isn't required for daily iteration. See [transport vs app simulation](/docs/concepts/transport-vs-app). ### Can I start with iOS or Android only and add the other later? Yes. The AppSpec is vendor-agnostic and platform-agnostic — your spec doesn't change based on whether you target iOS first, Android first, or both simultaneously. `generateConnectionModule` takes a `platform` parameter; call it once per platform. ### What if I'm not using one of the listed AI coding agents? Any MCP-compatible agent works — Extentos uses the standard MCP protocol over stdio. The verified hosts (Claude Code, Cursor, Windsurf, Cline) just have the most polished install experience. Generic MCP host setup is in [install § Generic MCP host](/docs/mcp-server/install#generic-mcp-host). ### Can I integrate Extentos without an AI agent at all? Technically yes — the Android and iOS libraries are source-available and can be linked directly via Gradle / SPM, and the MCP server's tool responses are JSON that you could call from any HTTP client. But the path is much rougher today (the dedicated by-hand quickstarts are deferred), and the productivity wins of agent-driven generation are large. The recommended approach is to use any MCP-capable agent for the initial integration and graduate to direct manipulation if you need it later. ## Related - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — the full step-by-step - **[MCP server install](/docs/mcp-server/install)** — per-host install commands and config - **[Supported agents](/docs/mcp-server/agents)** — Claude Code / Cursor / Windsurf / Cline capability differences - **[Concepts: transport vs app simulation](/docs/concepts/transport-vs-app)** — the core differentiator framing - **[Vendors: Meta Ray-Ban](/docs/vendors/meta)** — currently GA target, all variants supported - **[Pricing](/docs/resources/pricing)** — free for development, no paid tier at launch --- source: https://extentos.com/docs/getting-started/android --- --- title: Android quickstart description: How to add the Extentos Android library via Gradle — Maven artifact resolution (mavenLocal during Phase 7a, Maven Central post-7b), required AndroidManifest permissions (BLUETOOTH_CONNECT, BLUETOOTH_SCAN, CAMERA, RECORD_AUDIO, INTERNET), Application class initialization, and the first capability call against the simulator. Currently a stub — the agent-driven path at /docs/getting-started/with-agent covers the same workflow today via generateConnectionModule. type: tutorial platform: android related: - /docs/sdk/android - /docs/getting-started/with-agent --- > **Coming soon.** This page will cover Gradle install, manifest setup, and the first capability call from a Compose or View-based Android app. ## What this page will cover - Add Extentos via Gradle (Maven Central) - Required `AndroidManifest.xml` permissions and features - Initialize Extentos in your `Application` class - Make your first capability call - Verify in the browser simulator --- source: https://extentos.com/docs/getting-started/ios --- --- title: iOS quickstart description: How to add the Extentos iOS library to a SwiftUI or UIKit app via Swift Package Manager — required Info.plist keys (NSCameraUsageDescription, NSMicrophoneUsageDescription, NSSpeechRecognitionUsageDescription, NSBluetoothAlwaysUsageDescription, plus the MWDAT dictionary for Meta DAT integration), Wearables.configure() initialization in the App entry point, and the first capability call against the simulator. Currently a stub — the agent-driven path at /docs/getting-started/with-agent covers the same workflow today via generateConnectionModule. type: tutorial platform: ios related: - /docs/sdk/ios - /docs/getting-started/with-agent --- > **Coming soon.** This page will cover SwiftPM install, Info.plist setup, and the first capability call from a SwiftUI or UIKit app. ## What this page will cover - Add Extentos via Swift Package Manager - Required `Info.plist` keys (Bluetooth, microphone, camera, external accessory) - Initialize Extentos in `App` entry point - Make your first capability call - Verify in the browser simulator --- source: https://extentos.com/docs/getting-started/with-agent --- --- title: Quickstart with an AI agent description: Install the Extentos MCP server in one command. 18 deterministic tools for adding Meta Ray-Ban smart-glasses capabilities to native iOS and Android apps. Free for development, no account required to start. type: tutorial platform: mcp related: - /docs/mcp-server - /docs/mcp-server/install - /docs/mcp-server/tools - /docs/concepts/transport-vs-app - /docs/concepts/architecture - /docs/vendors/meta - /docs/resources/pricing --- Extentos is the smart-glasses infrastructure layer for native mobile apps. An AI coding agent installs the Extentos MCP server, the server exposes 18 deterministic tools (discovery, scaffold, validate, simulate), and the agent uses them to add voice triggers, photo capture, and sensor streams to an existing iOS or Android app — targeting Meta Ray-Ban (GA), with other vendors on the roadmap. A browser-based simulator covers the bulk of the dev loop without physical glasses. This page is the agent-driven path. If you're integrating by hand, see the [iOS](/docs/getting-started/ios) or [Android](/docs/getting-started/android) quickstart instead. ## Install There's no single host-agnostic install command — every MCP host (Claude Code, Cursor, Windsurf, Cline) stores its server config differently. Three paths, in order of recommended: ### 1. Agent prompt (recommended, host-agnostic) Open your MCP-capable agent and paste: ```text Install @extentos/mcp-server in my MCP config. ``` The agent runs the right command for its host (`claude mcp add ...` for Claude Code, or edits `mcp.json` for Cursor / Windsurf / Cline). Restart the agent when it reports done. The Extentos tools appear in the agent's tool list. ### 2. Claude Code one-liner If you're on Claude Code and prefer a shell command: ```bash claude mcp add extentos -- npx -y @extentos/mcp-server@latest ``` Restart Claude Code. Run `/mcp` — you should see `extentos` listed with all 18 tools. ### 3. Manual JSON For any MCP host, drop this into the host's MCP config file (`~/.cursor/mcp.json` for Cursor, `~/.codeium/windsurf/mcp_config.json` for Windsurf, or equivalent): ```json { "mcpServers": { "extentos": { "command": "npx", "args": ["-y", "@extentos/mcp-server@latest"] } } } ``` Restart the host. The Extentos tools should appear in the agent's tool list. **Prerequisites:** Node.js 20 or newer on `PATH`. That's the only requirement — `npx` fetches and caches the package on first run. **Pinning a version (optional):** Replace `@latest` with `@0.0.16` if you need a reproducible install across sessions. Extentos is pre-1.0; APIs may shift between minor versions until the hardware test loop closes. ## First prompt to your agent After install, ask the agent to do the actual work. Two prompt patterns that work well: ```text Add Extentos smart-glasses voice triggers to my Android app. The user should be able to say "summarize this" while wearing Meta Ray-Ban glasses, and the app should run my existing summarize() function on whatever the camera sees. ``` ```text Use the Extentos MCP server to add photo capture from Meta Ray-Ban to my iOS app. When the user double-taps the glasses, capture a frame, send it to my analyzeImage() function, and speak the result back. ``` Be specific about: which platform, which capability (voice / photo / video / sensor / audio), and which existing function in your app should consume the data. The agent maps your prose to a spec and generates the bridging code. ## What the agent does The MCP server has no planning tool — agents are better planners than regex. The agent composes capability primitives itself, then calls a deterministic mutation sequence. You can watch it happen in the tool log: | Step | Tool | What it does | |------|------|--------------| | 1 | `getPlatformInfo` | Reads the capability catalog for the target glasses (e.g. `meta_rayban`) | | 2 | `searchDocs` | Pulls the catalog of triggers, actions, blocks, and stream types — each with inline minimal examples | | 3 | `generateConnectionModule` | One-shot scaffold: bootstrap module, Gradle/SPM wiring, empty spec. Asks where `ExtentosConnectionPage` should live in your app | | 4 | `initSpec` | Populates the first real spec from the composed primitives | | 5 | `generateConsumer` | Stubs `app_callback` handlers (and stream consumers if the spec has any) | | 6 | `validateIntegration` | Pre-test correctness gate — re-runs after every mutation | | 7 | `createSimulatorSession` | Provisions a browser-hosted glasses surrogate at [extentos.com/s](https://extentos.com/s) | For iteration: `inspectIntegration` (for the freshness token) → `updateSpec` → `generateConsumer` (if handlers/streams added) → `validateIntegration`. For debugging: `getEventLog`, `getSimulatorStatus`. For ship-readiness: `getProductionChecklist`. For studying complex compositions: `getExampleSpec` returns 8 worked patterns (voice assistant, live translation, video call, etc.). The full tool reference is at [/docs/mcp-server/tools](/docs/mcp-server/tools). ## Verifying it worked When the agent calls `createSimulatorSession`, the MCP server auto-opens a browser tab to your session at [extentos.com/s](https://extentos.com/s) (specifically `extentos.com/s/`). You should see: - A glasses surrogate with camera + mic permission prompts (browser-side; no physical hardware) - A live event log showing your app initializing, the connection opening, and your trigger firing when you exercise it - A WebSocket-synced view of your app's responses, exactly as they'd appear on real Meta Ray-Ban glasses On Android, **auto-bind** is fully shipped: once the Extentos library is in your app, the MCP and library coordinate via a `127.0.0.1:31337/whoami` localhost bridge. Subsequent `createSimulatorSession` calls auto-attach the running app to the new session — no rebuild, no URL paste. On iOS, auto-bind is in progress; the iOS SDK currently uses the URL-bake path via `extentos.session.plist` until that lands. ## Pricing — Extentos is free There is no paid tier at launch. Specifically: | Tier | Cost | What you get | |------|------|--------------| | **No-account** | Free forever | All 18 MCP tools. `LocalSimTransport` (full simulator on your phone/emulator via Mock Device Kit). Code generation, validation, scaffolding. Plus the [extentos.com/s](https://extentos.com/s) browser simulator for **1000 lifetime runtime events** keyed to your MCP install. | | **Free account** | Free, email only | Continue using the browser simulator after the 1000-event meter is hit. Email + ToS acceptance, no payment, no card. | **A "runtime event"** is an event emitted by the browser simulator during a session — `trigger.fired`, `block.executed`, `callback.invoked`, etc. A typical voice-to-AI interaction emits 5–10 events, so 1000 events ≈ 100–200 full interactions, roughly enough to validate a non-trivial spec on your actual app. MCP-server-side activity (tool calls, code generation) **does not count** against the meter, ever. **Account linking** is lazy: the device-code flow only fires when you exhaust the 1000-event meter. Your agent gets an `authRequired` response with a verification URL, opens it in your browser, you sign up with email + ToS acceptance, and the original `createSimulatorSession` call retries automatically. No manual token paste, no upfront signup wall. **Real hardware testing** (running your app on actual Meta Ray-Ban glasses) requires your own Meta Developer Center registration — a one-time, free, ~15–30 minute setup using your personal Meta credentials. No Extentos account required for this path. Full pricing details: [/docs/resources/pricing](/docs/resources/pricing). ## What's actually supported (be honest with your agent) The Meta DAT public toolkit is the substrate Extentos sits on. It has real boundaries — your agent should know them so it generates apps that actually run: - **Strong public capabilities:** photo capture, video capture, camera streams, sensor data - **Voice triggers:** custom phrases work via the phone's speech recognizer over Bluetooth (HFP/SCO). Meta AI's "Hey Meta" wake word is **not** accessible to third-party apps - **Audio I/O:** mic capture and TTS playback ride mobile-platform Bluetooth profiles. Extentos wires the audio session; your platform's native STT/TTS does the work (zero Extentos runtime cost) - **Not available:** custom gestures (tap, swipe, swipe-to-confirm) are not in the public toolkit preview - **Distribution:** publishing to the Meta App Store is preview-limited. Private testing and Android/iOS export are the safe defaults See [/docs/concepts/capabilities](/docs/concepts/capabilities) for the full matrix, and [/docs/concepts/transport-vs-app](/docs/concepts/transport-vs-app) for the framing of what Meta provides vs what Extentos adds. ## Troubleshooting first-run issues **`/mcp` doesn't list `extentos` in Claude Code.** Restart Claude Code completely (Cmd+Q / quit from system tray, not just close window). The MCP host loads servers at startup. **`npx` errors with "command not found".** Install Node.js 20 or newer from [nodejs.org](https://nodejs.org). Verify with `node -v`. **The agent's first tool call is slow or times out.** `npx` is fetching the package on first run — wait 10–30 seconds and retry. Subsequent calls hit the cache and are instant. **`createSimulatorSession` returns `authRequired` immediately.** You've already used your 1000 free events on this install. The response includes a verification URL — open it, sign up with email (no payment), and the agent will retry automatically. **Auto-bind not connecting on Android.** Check that nothing else is bound to localhost port `31337`. The library logs a warning if the bridge fails to start; falls back to URL-paste mode. **iOS app not picking up the session.** iOS auto-bind is in progress. Until it lands, copy the simulator URL from the agent's tool response and bake it into `extentos.session.plist` — see [/docs/sdk/ios](/docs/sdk/ios) for the manual flow. More: [/docs/troubleshooting](/docs/troubleshooting). ## Next steps - **[MCP server reference](/docs/mcp-server)** — full tool catalog, configuration, auth model - **[Concepts: architecture](/docs/concepts/architecture)** — how the MCP, the library, the simulator, and your app fit together - **[Concepts: transport vs app simulation](/docs/concepts/transport-vs-app)** — what Meta provides, what Extentos adds, and why both matter - **[Guides](/docs/guides)** — task-shaped recipes (voice triggers, photo capture, sensor streams, background sessions) - **[Vendors: Meta Ray-Ban](/docs/vendors/meta)** — the GA target and its capability matrix ## Status Pre-1.0 (current: `@extentos/mcp-server@0.0.16`). Android primary, iOS supported via SPM. APIs may shift between minor versions until the hardware test loop closes — pin to an exact version if you need cross-session reproducibility. Source: [github.com/Asgermolgaard/vibe-hardware](https://github.com/Asgermolgaard/vibe-hardware). License: MIT. --- source: https://extentos.com/docs/guides --- --- title: Guides description: Task-shaped recipes. Each guide solves one concrete thing — voice triggers, photo capture, sensor streams — across iOS and Android. type: guide platform: all --- Task-shaped, vendor-agnostic recipes. Each one shows iOS and Android side-by-side. - [Voice triggers](/docs/guides/voice-triggers) - [Photo capture](/docs/guides/photo-capture) - [Video capture](/docs/guides/video-capture) - [Audio streaming](/docs/guides/audio-streaming) - [Sensor streams](/docs/guides/sensor-streams) - [Background sessions](/docs/guides/background-sessions) - [Handle disconnects](/docs/guides/handle-disconnects) --- source: https://extentos.com/docs/guides/audio-streaming --- --- title: Audio streaming description: Stream audio from the glasses microphone into your app, or play synthesized speech back through the glasses speakers. type: guide platform: all --- > **Coming soon.** ## What this page will cover - Mic capture via Bluetooth profile - TTS / speech synthesis playback - Audio session configuration - iOS + Android code samples --- source: https://extentos.com/docs/guides/background-sessions --- --- title: Background sessions description: Keep an Extentos session running while your app is backgrounded. type: guide platform: all --- > **Coming soon.** ## What this page will cover - iOS background modes setup - Android foreground service setup - What capabilities survive backgrounding - Battery and OS-kill considerations --- source: https://extentos.com/docs/guides/handle-disconnects --- --- title: Handle disconnects description: Gracefully detect when the glasses disconnect and reconnect — UX patterns, retry policy, error states. type: guide platform: all related: - /docs/concepts/sessions --- > **Coming soon.** ## What this page will cover - Disconnect detection - Reconnect policy (auto vs user-initiated) - UX patterns for in-flight operations - iOS + Android code samples --- source: https://extentos.com/docs/guides/photo-capture --- --- title: Photo capture description: Capture a photo from the glasses on user command and receive it in your app. type: guide platform: all --- > **Coming soon.** ## What this page will cover - Trigger a photo capture - Receive the image data in your app - Storage/cleanup - iOS + Android code samples --- source: https://extentos.com/docs/guides/sensor-streams --- --- title: Sensor streams description: Subscribe to IMU, head-pose, and other sensor streams from the glasses. type: guide platform: all --- > **Coming soon.** ## What this page will cover - Available sensor types per vendor - Subscribe / unsubscribe lifecycle - Sample rates and battery impact - iOS + Android code samples --- source: https://extentos.com/docs/guides/video-capture --- --- title: Video capture description: Record video from the glasses camera and stream or save it from your app. type: guide platform: all --- > **Coming soon.** ## What this page will cover - Start/stop video recording - Receive frames or completed file - Per-vendor differences (Meta: app-side recording from frame stream) - iOS + Android code samples --- source: https://extentos.com/docs/guides/voice-triggers --- --- title: Voice triggers description: Wire a voice command on the glasses to an action in your app. Works on Meta Ray-Ban via the phone's speech recognizer over Bluetooth. type: guide platform: all related: - /docs/concepts/capabilities - /docs/sdk --- > **Coming soon.** Voice trigger guide. ## What this page will cover - Declare a voice trigger phrase - Receive the trigger event in your app - Per-vendor caveats (Meta: phone-side STT only; "Hey Meta" wake word not exposed) - iOS code sample (SwiftUI + UIKit) - Android code sample (Compose + Activity) --- source: https://extentos.com/docs/mcp-server --- --- title: MCP server description: The Extentos MCP server (`@extentos/mcp-server`) is an npm package an AI agent (Claude Code, Cursor, Windsurf, Cline) installs once and then uses to add Meta Ray-Ban smart-glasses capabilities to a native iOS or Android app. It exposes 18 deterministic tools across 7 categories — discovery, generation, guidance, validation, simulation, production-readiness, and documentation — plus a CLI for account linking, telemetry consent, and update checks. This is the agent's operating manual. type: overview platform: mcp related: - /docs/getting-started/with-agent - /docs/concepts/architecture - /docs/concepts/transport-vs-app - /docs/mcp-server/install - /docs/mcp-server/tools - /docs/mcp-server/configuration - /docs/mcp-server/auth - /docs/mcp-server/agents --- The MCP server is how an AI agent — Claude Code, Cursor, Windsurf, Cline, or any [Model Context Protocol](https://modelcontextprotocol.io)-compatible host — operates Extentos. The agent invokes deterministic tools; the server reads project state, generates code, mutates the AppSpec, brokers simulator sessions, and queries debugging traces. The server itself has **no planning tool** — agents are better planners than regex bundles. Tools are typed primitives the agent composes in sequence. This page is the section landing — what the server is, the 18 tools at a glance, the canonical agent-driven flow, configuration knobs, the CLI, and how the auth model works. Sub-pages cover each in depth. ## Install ```bash claude mcp add extentos -- npx -y @extentos/mcp-server@latest ``` For non-Claude-Code hosts, see the [agent prompt or manual JSON](/docs/getting-started/with-agent#install) install paths. Full install reference at [/docs/mcp-server/install](/docs/mcp-server/install). ## The 18 tools, by category The server exposes **exactly 18 tools** (verified in `mcp-server/src/tools/definitions.ts`), organized into 7 categories. The categories are the agent's mental map; an agent that understands which category a tool lives in can decide when to call it. ### 1. Discovery and Planning (2 tools) The first calls in any new task. Cheap, all local, no side effects. | Tool | What it does | |---|---| | `getPlatformInfo` | Returns the static platform catalog — library version, spec schema, glasses capabilities — for the target vendor (`meta_rayban`). Always the right first call. | | `getExampleSpec` | Reference library — retrieve one of 8 worked example specs (`voice_assistant`, `live_translation`, `scene_description`, `notification_reader`, `navigation_guide`, `video_call`, `object_detection`, `voice_notes`). Use to study complex compositions; not as a starting template. | ### 2. Setup and Generation (4 tools) The mutation tools that scaffold the project and the AppSpec. Run in this order on a fresh project. | Tool | What it does | |---|---| | `generateConnectionModule` | One-shot scaffold — bootstrap module, empty spec, manifest entries, dependency wiring, permission stubs. Two-call flow: first call without `placement` returns a question; second call with the chosen placement returns the full file set. | | `initSpec` | Create the FIRST populated spec for a fresh project. Refuses if spec is already non-empty. | | `generateConsumer` | Stub `app_callback` handlers (or stream consumers) for whatever the spec declares. | | `updateSpec` | Mutate an existing spec. Requires the `basedOn` freshness token from `inspectIntegration` to detect stale concurrent edits. | ### 3. Implementation Guidance (2 tools) Side-quest tools the agent calls during composition. | Tool | What it does | |---|---| | `getVoiceCommandGuidance` | Analyze proposed voice-command phrases for UX issues (collisions, ambiguity, hard-to-recognize words) before adding `voice_command` triggers. | | `getPermissions` | Derive exact platform permissions, Meta DAT requirements, and foreground-service needs from a spec. Run after creating or updating the spec. | ### 4. Validation (3 tools) Correctness gates. Run after every mutation. | Tool | What it does | |---|---| | `validateSpec` | Validate a detached spec JSON against the 11 schema rules. Pure JSON check, no filesystem. | | `inspectIntegration` | Read-only project snapshot — manifest, spec, generated-file hashes, dependency list, plus the `basedOn` freshness token. Run before every `updateSpec` or `generateConsumer`. | | `validateIntegration` | Whole-project correctness check — spec, files, manifest, dependencies, permissions, handler coverage, bootstrap wiring. The pre-test gate. | ### 5. Simulation (4 tools) Provision and operate browser-based simulator sessions. | Tool | What it does | |---|---| | `createSimulatorSession` | Provision a browser-mode session at [extentos.com/s](https://extentos.com/s). Auto-attaches the running app via the local bridge if reachable; otherwise emits a `BuildConfig.EXTENTOS_SESSION_URL` snippet (Android) or `extentos.session.plist` payload (iOS). Calling this with a session already live supersedes it cleanly. | | `completeAuthLink` | After `createSimulatorSession` returns `status: "auth_required"` (1000-event meter exhausted), polls the backend until the user finishes signup, then persists the bearer token to `~/.extentos/auth.json`. | | `getEventLog` | Fetch structured event traces from a session. Filters: `all`, `errors`, `transport`, `triggers`, `blocks`, `callbacks`, `toggles`, `streams`, `system`, `spec`. Plus `flowId`, `since`, `limit` for trace-level scope. The primary debugging tool. | | `getSimulatorStatus` | Read a live session's current state — phase, hardware-ready, roles, active streams/triggers. | ### 6. Production (2 tools) Pre-ship checks. | Tool | What it does | |---|---| | `getProductionChecklist` | Personalized production-readiness checklist based on what the spec uses. | | `getCredentialGuide` | Step-by-step credential setup for production AI providers — `anthropic`, `openai`, `google_cloud_vision`, `google_translate`, `google_gemini`, `deepl`, `azure_cognitive`, `aws_bedrock`, `huggingface`, or `custom`. | ### 7. Documentation (1 tool) | Tool | What it does | |---|---| | `searchDocs` | Search Extentos documentation by topic or keyword. Catalog topics (`trigger_types`, `action_types`, `block_types`, `stream_types`, `spec_format`, `template_syntax`, `app_callback_guide`, `library_api`) carry inline minimal examples — most specs can be authored from these without ever fetching a `getExampleSpec` pattern. | Full per-tool reference with input schemas, response shapes, and worked examples: [/docs/mcp-server/tools](/docs/mcp-server/tools). ## The canonical agent-driven flow In a fresh project, the agent invokes the tools in this order: ```text 1. getPlatformInfo({ sections: ["version", "capabilities"], glasses: "meta_rayban" }) 2. searchDocs({ topic: "trigger_types" | "action_types" | "block_types" }) 3. generateConnectionModule({ platform, glasses, appPackage }) → returns "needs_placement" question 4. generateConnectionModule({ ... placement: "" }) → writes scaffold files 5. initSpec({ ... }) → first AppSpec written 6. generateConsumer({ kind: "callback" }) → handler stubs written 7. validateIntegration() → ✓ all good (or returns structured errors to fix) 8. createSimulatorSession({ glasses, spec }) → returns sessionId; auto-opens browser at extentos.com/s/ → if running app is reachable via local bridge, it auto-attaches 9. 10. getEventLog({ sessionId, filter: "errors" }) → debug getSimulatorStatus({ sessionId }) → status ``` For iteration, the loop is `inspectIntegration` (for the freshness token) → `updateSpec` → `generateConsumer` (if handlers/streams added) → `validateIntegration`. Before shipping, `getProductionChecklist` and `getCredentialGuide`. ## Configuration The MCP server reads these environment variables (verified in `mcp-server/src/`): | Variable | Default | What it does | |---|---|---| | `EXTENTOS_BACKEND_URL` | Production backend | Override the backend URL (`tools/util/backendClient.ts`). For local development of Extentos itself. | | `EXTENTOS_CONFIG_DIR` | `~/.extentos` | Override the config/auth directory (`telemetry/consent.ts`). | | `EXTENTOS_TELEMETRY` | unset (consent-default) | Set to `0` to decline telemetry without running the CLI consent command. | | `EXTENTOS_NO_AUTO_OPEN` | unset | Set to `1` to disable browser auto-open on simulator-session creation (useful in headless environments). | Full configuration reference: [/docs/mcp-server/configuration](/docs/mcp-server/configuration). ## CLI subcommands Running `npx @extentos/mcp-server@latest` with no arguments starts the MCP server over stdio (the path the agent uses). With a subcommand, it acts as a developer CLI: | Subcommand | What it does | |---|---| | `login` | Link this install to an Extentos account via device-code flow (only needed after the 1000-event meter exhausts). | | `logout` | Clear `~/.extentos/auth.json`. Install returns to the anonymous tier. | | `whoami` | Print `installId`, `accountId` (if linked), tier, meter remaining, auth expiry. | | `accept-privacy` | Record privacy consent (enables telemetry upload). | | `decline-privacy` | Record privacy decline (disables telemetry upload). | | `status` | Print consent state, install ID, linked account, MCP/library versions. | | `update` | Check for MCP server updates (no-op on `npx @latest` installs). | Full CLI reference: [/docs/mcp-server/auth](/docs/mcp-server/auth). ## Auth model The MCP server is **anonymous-first**. All 18 tools work with no account. The first browser-simulator session is anonymous, keyed to a per-machine `installId` at `~/.extentos/install_id`. The 1000-event meter is the only gate. When it exhausts, `createSimulatorSession` returns `status: "auth_required"` with a verification URL. The agent calls `completeAuthLink` to poll the backend; the developer signs up with email (no payment) at the URL; the backend issues a token; the original tool call retries automatically. There is no paid tier at launch. Full auth model: [/docs/mcp-server/auth](/docs/mcp-server/auth) and [/docs/resources/pricing](/docs/resources/pricing). ## Privacy and telemetry On first run, the MCP server injects a one-time privacy notice into the response. Telemetry is anonymous (tagged with `installId`, no source code or personal data) and is dismissed-by-continuing by default — same pattern as Vercel CLI, Astro, Vite. Decline at any time: ```bash npx @extentos/mcp-server@latest decline-privacy # or EXTENTOS_TELEMETRY=0 (env var, persistent for the shell) ``` Privacy notice content is in `mcp-server/src/index.ts` (`PRIVACY_NOTICE` constant). Notice is shown once per install via `claimFirstPrivacyNotice` — never repeats. ## Compatible MCP hosts Verified to work with: - **Claude Code** — primary target. One-line install via `claude mcp add`. - **Cursor** — JSON config in `~/.cursor/mcp.json`. - **Windsurf** — JSON config in `~/.codeium/windsurf/mcp_config.json`. - **Cline** — JSON config in Cline's MCP settings. - **Any MCP-compatible host** — drop the standard `mcpServers.extentos` JSON block in. The MCP server speaks the standard MCP protocol over stdio (`@modelcontextprotocol/sdk`); no host-specific code paths exist on the server side. Per-host install steps: [/docs/mcp-server/agents](/docs/mcp-server/agents). ## The local bridge — auto-bind dev loop When the server starts, it opens a `127.0.0.1:31337/whoami` HTTP listener (`mcp-server/src/localBridge.ts`). The Extentos library on the developer's app probes this endpoint at runtime to learn its host MCP's `installId`. The result: every `createSimulatorSession` call from the agent auto-attaches the running app to the new session — no rebuild, no URL paste. Reach paths: - **Android emulator:** `http://10.0.2.2:31337/whoami` (host-loopback NAT alias) - **iOS Simulator:** `http://localhost:31337/whoami` (shares host network namespace) - **Physical Android phone via USB:** `adb reverse tcp:31337 tcp:31337` once, then `localhost:31337` from the device - **Cellular phone or cloud-hosted agent:** probe times out. The agent uses the URL-bake path instead — `createSimulatorSession` returns a `BuildConfig.EXTENTOS_SESSION_URL` snippet (Android) or `extentos.session.plist` payload (iOS) the developer pastes in, then rebuilds the app once. Less elegant than auto-bind but works on any topology. Bound to `127.0.0.1` only. The `installId` is not a secret — it's the same value the MCP sends to `api.extentos.com` on every tool call. No auth needed at this layer. If port `31337` is in use (rare; another MCP instance already running), startup logs a warning and continues. Auto-bind silently fails for that session; the developer uses the URL-bake path (paste the `EXTENTOS_SESSION_URL` Android snippet or write the `extentos.session.plist` iOS payload that `createSimulatorSession` returns, then rebuild). The agent and tools keep working — only the zero-rebuild auto-attach is lost until the port frees up. ## Status - **Package:** `@extentos/mcp-server@0.0.16` on npm (MIT license) - **Engines:** Node.js 20+ - **Source:** [github.com/Asgermolgaard/vibe-hardware/tree/master/mcp-server](https://github.com/Asgermolgaard/vibe-hardware/tree/master/mcp-server) - **Pre-1.0** — APIs may shift between minor versions until the hardware test loop closes. Pin to an exact version (`@0.0.16`) if you need cross-session reproducibility. ## Related - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install the server and walk through a real dev loop - **[Tools reference](/docs/mcp-server/tools)** — full per-tool API - **[Configuration](/docs/mcp-server/configuration)** — environment variables, config files, install-time settings - **[Auth](/docs/mcp-server/auth)** — device-code flow, account linking, CLI auth commands - **[Supported agents](/docs/mcp-server/agents)** — per-host install instructions - **[Architecture](/docs/concepts/architecture)** — how the MCP server fits into the broader Extentos system - **[Transport vs app simulation](/docs/concepts/transport-vs-app)** — what the simulator the MCP brokers actually does --- source: https://extentos.com/docs/mcp-server/agents --- --- title: Supported agents description: Per-host capability differences for AI coding agents that work with Extentos. What it's actually like running Extentos inside Claude Code (terminal, filesystem-wide reach, simultaneous iOS + Android edits) vs Cursor or Windsurf (VS Code workspace, both platforms work if both folders are open) vs Cline (similar but more click-approval) vs Android Studio's Gemini or Xcode AI features (single-platform, no MCP support — can't run Extentos at all). Project scope, MCP integration maturity, browser auto-open behavior, and the URL-bake fallback path documented per host. type: tutorial platform: mcp related: - /docs/mcp-server - /docs/mcp-server/install - /docs/getting-started/with-agent --- Extentos generates **native mobile app code** (Swift for iOS, Kotlin for Android), so the agents that work with it are the ones that target native mobile development with Model Context Protocol support. Web-first AI builders like Replit Agent, Lovable, Bolt.new, or v0.dev aren't relevant for Extentos work — not because they lack MCP, but because they ship web apps, not native mobile. The MCP-capable agents that DO work are surprisingly portable: most of them run **anywhere you can run their interface** — Claude Code in particular runs in any terminal, which includes the terminal panel inside Android Studio, Xcode, IntelliJ, VS Code, or any IDE you already use. So the question isn't "which IDE supports Extentos?" — it's **"which AI agent supports MCP, and where can I run it?"** This page covers the per-agent capability differences and the IDE-portability picture. ## Quick recommendation | If you're using... | What works | |---|---| | **Claude Code** as your agent | Best general fit. Runs in any terminal (standalone, Android Studio's panel, Xcode's terminal, VS Code's, your favorite shell). Filesystem-wide reach from the working directory; can edit iOS + Android source simultaneously. One CLI command to install. | | **Cursor** or **Windsurf** | Great if you live in a VS Code-style IDE. Multi-folder workspace covers cross-platform Extentos. JSON config install. | | **Cline** (VS Code extension) | Works well. More explicit per-tool approval; good if you want to confirm each MCP call. | | **Android Studio's built-in Gemini, Xcode AI features, JetBrains AI Assistant** | If your IDE's built-in agent has MCP support (verify with the IDE's current docs — this space moves fast), Extentos works there directly. If it doesn't yet, you can run Claude Code (or any MCP agent) in the IDE's terminal panel — your IDE's editor and build tooling still function as normal. Either way, you don't have to leave the IDE. | | **Replit Agent, Lovable, v0.dev, Bolt.new** | Out of scope — these build web apps, not native iOS/Android. Wrong tool for Extentos in the first place. | ## Tier 1 — recommended for Extentos work These four hosts have native, mature MCP support and have been verified end-to-end with Extentos. ### Claude Code (terminal CLI) **Best for:** cross-platform Extentos work where the agent edits files across `ios-app/`, `android-app/`, and any shared library or backend code in one coherent session. | Capability | Behavior | |---|---| | Project scope | Whole filesystem from your current working directory. Agent reads and edits any file in or below `cwd`. | | Cross-platform iOS + Android | ✅ Native. Single agent session edits both platforms simultaneously. | | Shell exec | First-class. Agent runs `npx`, `gradle`, `swift build`, `claude mcp add` directly. | | MCP install | One CLI command — `claude mcp add extentos -- npx -y @extentos/mcp-server@latest` | | Tool output rendering | Inline in terminal. Long-running tools (e.g. `completeAuthLink` polling for 600s) show a clear waiting state. | | Browser auto-open | Native via `open` (macOS) / `xdg-open` (Linux) / `cmd start` (Windows). | | If browser auto-open fails | Extentos prints the verification URL or simulator URL directly in the agent's response. The developer copies it into their browser. | | Auto-bind reach | Always works — MCP server runs on the developer's machine, the same machine the developer's emulator or USB-tethered phone runs on. | | MCP server crash recovery | Auto-restarts the server transparently. | **Recommended setup:** install with the CLI command above, run Claude Code from your project root, you're set. ### Cursor **Best for:** developers who already live in a VS Code-style IDE and want Extentos integrated into their existing editor workflow. | Capability | Behavior | |---|---| | Project scope | VS Code workspace. Agent reads and edits files in any folder added to the workspace. | | Cross-platform iOS + Android | ✅ if both `ios-app/` and `android-app/` are added to the workspace as folders. Cursor handles multi-folder workspaces well. | | Shell exec | Yes, with optional approval. | | MCP install | JSON config in `~/.cursor/mcp.json` (global) or `.cursor/mcp.json` (per-project) | | Tool output rendering | In the agent chat panel. Long-running tools show progress; some tools' raw JSON output may be more verbose than in Claude Code. | | Browser auto-open | Works — Cursor delegates to the OS default browser. | | If browser auto-open fails | Agent surfaces the URL in the chat panel; developer copies it. | | Auto-bind reach | Works — same-machine MCP + library = reach. | | MCP server crash recovery | Auto-restarts. UI surfaces the error in the MCP panel. | **Recommended setup:** add Extentos via the JSON config (see [install](/docs/mcp-server/install#cursor)), open both `ios-app/` and `android-app/` as workspace folders if you're working cross-platform. ### Windsurf **Best for:** same audience as Cursor — IDE-embedded workflow, VS Code-style. Capability profile is essentially identical to Cursor: | Capability | Behavior | |---|---| | Project scope | VS Code workspace, multi-folder supported | | Cross-platform iOS + Android | ✅ if both folders are in the workspace | | Shell exec | Yes, with approval | | MCP install | JSON config in `~/.codeium/windsurf/mcp_config.json` | | Browser auto-open | Works | | If browser auto-open fails | URL appears in the agent chat panel | | Auto-bind reach | Works on same-machine setup | **Recommended setup:** see [install](/docs/mcp-server/install#windsurf). ### Cline (VS Code extension) **Best for:** developers using Cline inside VS Code who want explicit per-tool approval before the agent acts. | Capability | Behavior | |---|---| | Project scope | VS Code workspace | | Cross-platform iOS + Android | ✅ if both folders are open | | Shell exec | Yes, but per-command approval is more explicit than Claude Code or Cursor | | MCP install | Cline UI panel or `cline_mcp_settings.json` config file | | Tool output rendering | In Cline panel. The per-tool approval flow can interrupt long-running tools more often. | | Browser auto-open | Works via VS Code's URL handler | | If browser auto-open fails | Agent prints the URL in the Cline panel | | Auto-bind reach | Works on same-machine setup | | MCP server crash recovery | Manual reload sometimes needed; UI surfaces errors clearly | **Recommended setup:** add Extentos via Cline's MCP Servers tab — see [install](/docs/mcp-server/install#cline-vs-code-extension). **One Cline-specific gotcha:** the per-tool approval flow can be tedious during the canonical Extentos sequence (`generateConnectionModule` → `initSpec` → `generateConsumer` → `validateIntegration`). Set Cline to auto-approve Extentos tools after the first run if you want the smoother flow. ## Tier 2 — works, less battle-tested with Extentos These hosts have MCP support but haven't been verified end-to-end with the full Extentos workflow. They should work — Extentos uses standard MCP — but expect rougher edges. | Host | Notes | |---|---| | **VS Code (native MCP)** | VS Code added native MCP support late 2025. Works with the standard `mcpServers` JSON pattern. Tool output rendering is less polished than the dedicated agent IDEs. | | **Continue.dev** | Open-source AI coding extension for VS Code with native MCP support. Works with the standard JSON config. | | **Zed** | Fast editor with MCP support added in 2025. Lighter than VS Code-based hosts; the agent surface is more minimal. | | **Aider** | Terminal-based git-aware AI coding tool with MCP support. Similar workflow to Claude Code but more constrained to git-tracked files. | For all of these, use the [generic MCP host](/docs/mcp-server/install#generic-mcp-host) JSON snippet. ## Built-in IDE agents — Gemini, Xcode AI, JetBrains AI If you're in Android Studio, IntelliJ, or Xcode, these IDEs have their own built-in AI assistants (Gemini in Android Studio, Xcode AI features, JetBrains AI Assistant). MCP adoption across these built-in agents has been moving — **the only correct answer to "does it support MCP?" is to check the IDE's current documentation**, since this space is shifting month-to-month in 2026. Two scenarios, both of which keep you in your IDE: **If the built-in agent supports MCP** — register Extentos via whatever MCP config mechanism the IDE provides (typically a JSON file path the IDE points at). The agent calls Extentos's 18 tools directly inside your IDE. Same `mcpServers` JSON pattern as Cursor / Windsurf works in any standards-compliant MCP host. **If the built-in agent doesn't support MCP yet** — every one of these IDEs ships with a built-in terminal panel, and Claude Code (or any MCP-capable agent) runs there: - **Android Studio:** open the bottom Terminal tab → run `claude` (or `npx @extentos/mcp-server` directly) → Claude Code now operates on your project. Cross-platform reach because Claude Code reads the filesystem from `cwd`. - **Xcode:** Xcode's built-in terminal access is more limited. Most developers run a separate terminal app side-by-side with Xcode (e.g., iTerm2) and let Claude Code drive the Extentos work there. Xcode keeps doing what it's good at — code reading, builds, simulator runs, debugging. - **IntelliJ / WebStorm / etc.:** same as Android Studio — built-in terminal hosts Claude Code or any other MCP agent. The pattern: **the IDE's built-in agent is one of many possible agents you can use, not the only one**. Either the built-in supports MCP and you call Extentos directly, or you bring your own agent in the same IDE's terminal. Your editor, build tools, breakpoint debugger, and simulator keep working unchanged — only the AI assistant choice differs. > **Verify current MCP support yourself.** The doc above reflects the *project's design intent* (Extentos is built for any MCP-compatible client including Gemini in Android Studio per the architectural plan). What's *currently shipped* by IDE vendors changes faster than this page updates. Check Google's Gemini in Android Studio docs, Apple's Xcode AI release notes, or JetBrains' AI Assistant docs for the current MCP support status before assuming. ## What's actually different per host for Extentos work Distilled into the Extentos-specific capabilities that matter most: ### 1. Cross-platform reach This is the biggest difference. Extentos generates code in: - The developer's iOS app source (Swift) - The developer's Android app source (Kotlin) - Shared spec files (JSON) A host that can read and edit both platforms simultaneously is unconstrained. A host locked to one platform forces context-switching between two agent sessions. | Host class | Cross-platform reach | |---|---| | Terminal CLI (Claude Code, Aider) | ✅ Whole filesystem from cwd | | VS Code-based (Cursor, Windsurf, Cline, VS Code, Continue.dev) | ✅ if both platform folders are in the workspace | | Single-platform IDE (Android Studio, Xcode) | ❌ Locked to one platform's project tree | ### 2. Browser auto-open behavior When `createSimulatorSession` is called (or the device-code auth flow fires), the MCP server attempts to open a browser tab to the URL — `open` on macOS, `xdg-open` on Linux, `cmd start` on Windows. This is **best-effort**, and **the URL is always also returned in the structured tool response regardless of whether the OS browser command succeeded**. So when browser auto-open *can't* happen — `EXTENTOS_NO_AUTO_OPEN=1` is set, the agent runs in a headless / SSH / sandboxed environment, the OS command isn't available, the agent host doesn't have OS-level browser-launch permission — **the agent simply surfaces the URL in its response and the developer copies it into their browser manually**. Nothing is lost; auto-open is a UX nicety, not a requirement. How the URL appears varies by host: - **Claude Code (terminal)** — printed as plain text in the terminal output, often with surrounding context like *"Open this URL in your browser: https://extentos.com/s/..."* - **Cursor / Windsurf / Cline** — appears inline in the chat panel, typically rendered as a clickable link the developer can click directly - **Headless / cloud-hosted agents** — printed in whatever output channel the agent uses; developer copies and pastes into a local browser The flow is always: tool returns URL in response → agent surfaces URL → developer opens it (whether by auto-open, click, or copy-paste). No failure mode here — the URL is always available. ### 3. Auto-bind reach (the localhost:31337 bridge) Auto-bind requires the MCP server and the developer's app to be reachable on the same `127.0.0.1` interface. In the standard "developer on a laptop, emulator or USB-tethered phone connected to the same laptop" setup, this just works for any of the hosts above. It does **not** work when: - The developer's phone is on a cellular network without USB tethering and `adb reverse tcp:31337 tcp:31337` hasn't been run - The agent host is cloud-hosted (e.g., Claude.ai/code's web sandbox) with the MCP server running in the cloud while the developer's app runs on their local machine - Port `31337` is already in use by another MCP instance When auto-bind isn't reachable, **the agent uses the URL-bake path**: `createSimulatorSession` returns a `BuildConfig.EXTENTOS_SESSION_URL` snippet (Android) or `extentos.session.plist` payload (iOS) that the agent writes into the developer's app, and the developer rebuilds once. One rebuild step versus the zero-rebuild auto-bind happy path. ### 4. Long-running tool handling `completeAuthLink` polls for up to 600 seconds. `createSimulatorSession` can take a few seconds to provision. Hosts vary in how they surface this wait: - **Claude Code:** prints "Polling for up to 600s..." in the terminal; clear and unambiguous - **Cursor / Windsurf:** show a spinner or progress indicator in the chat panel - **Cline:** shows the long-running tool prominently with a cancel button All four handle the wait correctly — none of them time out tool calls during normal Extentos operation. ### 5. Tool approval UX How permissive the host is about letting the agent invoke tools without explicit approval: | Host | Approval UX | |---|---| | Claude Code | Most permissive. Most tools auto-approve after the first run. | | Cursor / Windsurf | Middle ground. First-time tool approvals; subsequent calls auto-approve within the session. | | Cline | Most explicit. Per-tool approval until the user toggles auto-approve for specific tools. | For Extentos, the canonical agent flow uses ~7 tools in sequence on a fresh project. Cline's per-tool approval can be 7 clicks; Claude Code's after-first auto-approve is closer to zero. Both work; the choice is workflow preference. ## Frequently asked questions ### Can I use Extentos with Android Studio? Yes. Open Android Studio's built-in Terminal panel (bottom dock) and run Claude Code there — Claude Code is just a CLI, and it works in any terminal including Android Studio's. The MCP server attaches to your Claude Code session, and you can edit cross-platform code with simultaneous reach to both `ios-app/` and `android-app/`. Android Studio's own Gemini integration doesn't support MCP yet, but you don't have to use Gemini — any MCP-capable agent works in the same terminal panel. ### Can I use Extentos with Xcode? Yes — same idea. Run Claude Code (or another MCP-capable agent) in a terminal window alongside Xcode. Xcode keeps doing what it's good at (code reading, building, running the iOS Simulator); the MCP-capable agent handles the Extentos work in the terminal. They coexist cleanly because they don't compete for the same surface. ### What if my host doesn't support MCP at all? Then it can't run Extentos. The MCP protocol is the integration boundary; without it, there's no way for the agent to call Extentos's tools. Options: switch to an MCP-supporting host (Claude Code is the smoothest install), or use a Tier-1 host alongside your existing tool. ### Can I use multiple agent hosts at the same time? Yes — Extentos's MCP server doesn't care which host calls it. Each host runs its own MCP server instance via npx; the per-machine `installId` is shared (so they hit the same anonymous-meter / linked account), but otherwise the sessions are independent. Useful pattern: Cursor for primary development, Claude Code in a side terminal for one-off Extentos operations. ### Does the auto-bind dev loop work with cloud-hosted agents? No, by design — auto-bind requires the MCP server's `127.0.0.1:31337` listener to be reachable from the developer's app. A cloud-hosted agent's MCP server runs on a remote machine; the developer's emulator or phone runs locally. The two are in different network namespaces. Use the URL-bake path instead — the agent generates the `EXTENTOS_SESSION_URL` snippet and the developer pastes it locally. ### Does Gemini in Android Studio support Extentos directly? The project's design scope includes Gemini in Android Studio as a target host (per the internal architecture plan that explicitly lists "Gemini in Android Studio" alongside Claude Code, Cursor, and Codex). Whether it currently has MCP host support depends on Google's most recent Android Studio release — Google's AI integration has been evolving fast in 2026, so check the current Gemini in Android Studio docs for MCP support status. If MCP is supported, Extentos works directly. If it's not yet, run Claude Code (or any MCP-capable agent) in Android Studio's Terminal panel — same workflow either way. ### What about Replit, Lovable, v0.dev, Bolt.new? These are AI builders for **web apps** (React, Next.js, etc.), not native mobile. Extentos generates native iOS Swift and Android Kotlin code — wrong target for these tools. Not an MCP support issue; just a different category of product. If you're building a web app, Extentos isn't the right fit at all (your wearer experience would have to be a mobile-web app, which Meta DAT and the public smart-glasses toolkits don't support today). ### What about GitHub Copilot, Cody? | Host | MCP support as of 2026-04 | |---|---| | GitHub Copilot | Not yet — Copilot's agent mode is close to MCP but doesn't support arbitrary MCP servers | | Sourcegraph Cody | Not yet — Cody's tool support doesn't cover MCP servers | These will likely add MCP support over 2026, given MCP's adoption trajectory. Track the [changelog](/docs/resources/changelog) for updates as new hosts get verified. ### Does Extentos work with web app builders like v0.dev or Lovable? No. Extentos targets native iOS and Android apps that connect to smart glasses over Bluetooth via Meta DAT. Web app builders ship to a different runtime (browser-based React apps, etc.) where the smart-glasses connection isn't possible through public toolkits. The mismatch is at the artifact level, not the integration level — even if Lovable shipped MCP support tomorrow, the apps it generates couldn't talk to Ray-Ban Meta glasses. ## Related - **[Install the MCP server](/docs/mcp-server/install)** — per-host install commands and JSON snippets - **[MCP server overview](/docs/mcp-server)** — the 18-tool catalog and canonical agent flow - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install + first prompt + verify - **[Architecture](/docs/concepts/architecture)** — how the MCP, library, backend, and your app fit together --- source: https://extentos.com/docs/mcp-server/auth --- --- title: Auth description: The Extentos auth model — anonymous-first by design, with a lazy device-code flow (OAuth 2.0 RFC 8628 pattern) that fires only when the 1000-event browser-simulator meter exhausts. All 18 MCP tools work with no account; only the browser simulator is gated. The completeAuthLink MCP tool polls the backend after createSimulatorSession returns auth_required, persists the token to ~/.extentos/auth.json, and the original tool call retries automatically. No manual token paste, no upfront signup wall, no payment ever. type: reference platform: mcp related: - /docs/mcp-server - /docs/mcp-server/tools/simulation - /docs/resources/pricing - /docs/getting-started/with-agent --- The Extentos auth model is **anonymous-first by design**. All 18 MCP tools, code generation, validation, on-device `LocalSimTransport` simulation, and real-hardware testing through `RealMetaTransport` work with **no account, no email, no payment** — forever. Only the browser simulator at [extentos.com/s](https://extentos.com/s) carries a 1000-event meter, and only when that meter exhausts does the device-code flow fire to link a free email-only account. The agent handles the entire flow programmatically via the `completeAuthLink` MCP tool — the developer signs up once in their browser, the bearer token persists to `~/.extentos/auth.json`, and the original `createSimulatorSession` call retries automatically. **No manual token paste, no upfront signup wall, no payment at any tier.** This page covers the identity states, the device-code mechanics, the CLI subcommands, the state files, and the env-var dials. ## Identity states Extentos has three identity states, each unlocking a defined slice of functionality: | State | Identifier | What you can do | Cost | |---|---|---|---| | **No account** | `installId` (per-machine, at `~/.extentos/install_id`) | All 18 MCP tools. `LocalSimTransport`. Code generation. Validation. Real-hardware testing through `RealMetaTransport` (your own Meta credentials). 1000-event lifetime meter on `BrowserSimTransport` browser simulator. | Free forever | | **Free account** | `accountId` (linked via device-code flow) | Everything above, no event meter on the browser simulator. Email + ToS acceptance only. | Free, no payment | | (Future) Paid | n/a at launch | Reserved for post-launch evolution if Extentos ever introduces paid features. **No paid tier exists at launch.** | n/a | The `installId` is anonymous and never tied to your identity. The `accountId` is created by the developer signing up at the verification URL during the device-code flow. There's no upfront signup wall — the agent and the simulator work end-to-end on the anonymous tier first. See [pricing](/docs/resources/pricing) for the full free-tier breakdown. ## The device-code flow Extentos uses the **OAuth 2.0 Device Authorization Grant** ([RFC 8628](https://datatracker.ietf.org/doc/html/rfc8628)) — the standard pattern for headless / CLI scenarios where the application can't perform a browser redirect. AI coding agents are exactly that case: the agent runs in a terminal, the developer's browser is somewhere else. ### Sequence diagram ```text Developer Agent MCP server Backend Verification URL (browser) │ │ │ │ │ │ "build app" │ │ │ │ │─────────────▶│ │ │ │ │ │ │ │ │ │ │ createSimulator- │ │ │ │ │ Session(...) │ │ │ │ │───────────────────▶│ │ │ │ │ │ POST /api/.../ │ │ │ │ │ session │ │ │ │ │─────────────────▶│ │ │ │ │ │ │ │ │ │ auth_required, │ │ │ │ │ deviceCode, │ │ │ │ │ userCode, │ │ │ │ │ verificationUrl│ │ │ │ │◀─────────────────│ │ │ │ { status: │ │ │ │ │ "auth_required" │ │ │ │ │ verificationUrl, │ │ │ │ │ userCode, │ │ │ │ │ deviceCode } │ │ │ │ │◀───────────────────│ │ │ │ │ │ │ │ │ │ open browser │ │ │ │ │ at verificationUrl │ │ │ │ │─────────────────────────────────────────────────────────▶ │ │ │ │ │ │ completeAuthLink({ deviceCode }) │ │ │───────────────────▶│ │ │ │ │ │ poll /api/.../ │ │ │ │ │ token │ │ │ │ │◀────────────────▶│ pending, pending │ │ │ │ │ signs up at verificationUrl ────────────────────────────────────────────▶│ │ │ confirms signup │ │ │◀───────────────────│ │ │ │ │ │ │ │ │ poll → │ │ │ │ │ authToken, │ │ │ │ │ accountId, │ │ │ │ │ tier │ │ │ │ │◀─────────────────│ │ │ │ persist to │ │ │ │ │ ~/.extentos/ │ │ │ │ │ auth.json │ │ │ │ │ │ │ │ │ │ retry │ │ │ │ │ createSimulator- │ │ │ │ │ Session │ │ │ │ │───────────────────▶│ │ │ │ │ │ uses bearer token │ │ │ │ session ready │ │ │ │ │◀───────────────────│ │ │ ``` ### Backend endpoints Verified from `mcp-server/src/cli/login.ts` and `mcp-server/src/tools/handlers/completeAuthLink.ts`: | Endpoint | Method | Purpose | |---|---|---| | `/api/auth/device/code` | POST | Issue a device code. Body: `{ scope: "account_full" }`. Returns `{ deviceCode, userCode, verificationUrl, expiresInSeconds, pollIntervalSeconds }` | | `/api/auth/device/token` | POST | Poll for completion. Body: `{ deviceCode }`. Returns `{ pending: true, pollIntervalSeconds }` while waiting; `{ authToken, accountId, tier, linkedAt, expiresAt }` on approval | ### Defaults and limits | Field | Default | Notes | |---|---|---| | Device-code window | 600 seconds (10 min) | Backend returns the actual `expiresInSeconds`; `completeAuthLink` enforces a max of 600 | | Poll interval | 5 seconds | Backend may bump this via `pollIntervalSeconds` in poll responses; agent honors | | `completeAuthLink.maxWaitSeconds` | 600 | Caller can lower; capped at 600 by the handler | | `completeAuthLink.pollIntervalSeconds` | 5 | Range 1-30 (clamped) | ### HTTP status codes | Status | What it means | Retryable | |---|---|---| | `200` (`pending: true`) | User hasn't finished signup yet — keep polling | yes (continue) | | `200` (with `authToken`) | Approved — token issued | done | | `403` | User declined the signup prompt at the verification page | yes (after asking the user to retry) | | `404` | Unknown deviceCode (never existed or already consumed) | no (need a fresh code) | | `410` | Device-code window elapsed (10-min default) | yes (need a fresh code) | | Other / network failure | Backend unreachable | yes (after the network issue resolves) | ## The `completeAuthLink` MCP tool The agent-facing primitive for the device-code flow. After `createSimulatorSession` returns `status: "auth_required"`, the agent calls `completeAuthLink` with the returned `deviceCode`. ### Parameters | Parameter | Type | Required | Default | Notes | |---|---|---|---|---| | `deviceCode` | string | yes | n/a | From the `auth_required` response | | `maxWaitSeconds` | integer | no | 600 | 1-600 | | `pollIntervalSeconds` | integer | no | 5 | 1-30 | ### Response on success ```json { "linked": true, "tier": "free_account", "accountId": "acc_...", "linkedAt": "2026-05-01T12:34:56Z", "expiresAt": "2027-05-01T12:34:56Z", "authTokenFile": { "path": "/home/dev/.extentos/auth.json", "written": true, "error": null }, "pollCount": 3, "summary": "Account linked (tier: free_account). Re-invoke createSimulatorSession to use the new bearer token." } ``` The `pollCount` lets the agent know how long it took. The `summary` is the agent-facing reminder. ### Error codes | Code | Cause | Retryable | |---|---|---| | `unknown_device_code` | Backend doesn't recognize the deviceCode | no — call `createSimulatorSession` again for a fresh code | | `device_code_expired` | 10-min window elapsed | yes — call `createSimulatorSession` again | | `device_code_denied` | User declined at the verification page | yes — ask the developer to retry and accept | | `backend_unreachable` | Network failure or backend down | yes | | `backend_invalid_response` | Malformed poll body | no — version mismatch | | `auth_timeout` | Polled past `maxWaitSeconds` without approval | yes — the developer hasn't finished signup yet | Telemetry events fire on every flow outcome: `account.login_initiated`, `account.login_completed` (with outcome `success | code_expired | user_denied | network_error | polling_aborted`), and `account.linked` (only on success). Tagged with `installId`, no PII. ## CLI subcommands `extentos-mcp` is a developer CLI in addition to the MCP server. The auth-related subcommands: | Subcommand | What it does | When to use | |---|---|---| | `login` | Manually trigger the device-code flow. Initiates the flow against `/api/auth/device/code`, prints the userCode + verificationUrl, auto-opens the browser, polls until completion. | When the developer wants to link before the meter exhausts (rare — the agent does this automatically when needed) | | `logout` | Delete `~/.extentos/auth.json`. Install returns to the anonymous tier (the `installId` survives). | When switching machines, debugging auth state, or revoking the link | | `whoami` | Print install state — `installId`, `accountId` (if linked), `tier`, meter remaining, auth expiry | Diagnosing auth issues, before filing bug reports | Run any of them with: ```bash npx @extentos/mcp-server@latest login npx @extentos/mcp-server@latest logout npx @extentos/mcp-server@latest whoami ``` The `login` command output looks like: ```text Extentos account linking User code: ABCD-1234 Verification URL: https://extentos.com/auth/device Opening browser… (if it doesn't open, paste the URL above) Polling for up to 600s. Ctrl-C to cancel. ``` ## State files Auth state lives in `~/.extentos/` by default (overridable with `EXTENTOS_CONFIG_DIR`): | File | Contents | Created by | |---|---|---| | `install_id` | A per-machine UUID, never expires | First MCP server start | | `auth.json` | `{ authToken, accountId, linkedAt, expiresAt, scope }` after linking | Successful `completeAuthLink` (or `extentos-mcp login`) | | `consent` | Telemetry consent state (`accepted` / `declined` / unset) | First privacy notice acceptance or `extentos-mcp accept-privacy` / `decline-privacy` | `logout` deletes `auth.json` only. `install_id` persists — wiping the entire `~/.extentos/` directory creates a fresh anonymous identity with a fresh meter, which is one of the abuse vectors flagged in `docs/mcp/ACCOUNTS_AND_PRICING.md` (acceptable at MVP scale because it requires the developer to wipe the agent's MCP context each time). ## Environment variables | Variable | Default | Purpose | |---|---|---| | `EXTENTOS_BACKEND_URL` | Production backend | Override the backend URL the auth endpoints hit. Used for local backend development. | | `EXTENTOS_CONFIG_DIR` | `~/.extentos` | Override the config / auth / consent directory | | `EXTENTOS_NO_AUTO_OPEN` | unset | Set to `1` to disable browser auto-open. The verification URL is still printed; the developer pastes it manually. Useful for headless / SSH / cloud-hosted-agent environments. | ## Browser auto-open When the device-code flow fires, the MCP server attempts to open the verification URL in the developer's default browser via: | Platform | Command | |---|---| | macOS | `open ` | | Windows | `cmd.exe /c start "" ` | | Linux | `xdg-open ` | If auto-open fails (no GUI, no default handler, sandboxed environment), the URL is still printed to the agent's tool response and the CLI's stdout. The developer copies it into their browser. Set `EXTENTOS_NO_AUTO_OPEN=1` to skip the auto-open attempt entirely. ## Telemetry consent (separate from auth) Telemetry consent is a **separate** mechanism from authentication. The MCP server emits anonymous usage telemetry (tool calls, install events, no source code or PII) tagged with the per-machine `installId`. Consent is consent-default with first-run notice — the same pattern used by Vercel CLI, Astro, and Vite. The consent state lives at `~/.extentos/consent`. It's set by: | Action | Effect | |---|---| | First MCP tool call after install | Privacy notice injected once into the response (`PRIVACY_NOTICE` constant in `mcp-server/src/index.ts`); telemetry stays enabled by default | | `extentos-mcp accept-privacy` | Records explicit consent | | `extentos-mcp decline-privacy` | Disables telemetry upload — events still emit locally for debugging but never leave the machine | | Setting `EXTENTOS_TELEMETRY=0` | Same as `decline-privacy` for the current shell, but doesn't persist to the consent file | Telemetry consent has nothing to do with whether you're authenticated. A no-account install can opt out of telemetry; a linked account can opt out too. The two systems are independent. ## Frequently asked questions ### Why anonymous-first? The MCP server is the agent's tool surface. If we required signup before the agent could call any tool, time-to-first-prototype would suffer — the developer would have to interrupt the agent flow to create an account. Anonymous-first means the agent and developer can prove value first; the account gate fires only when there's already engagement to convert. ### Why device-code instead of OAuth redirect? OAuth redirect requires the application to be a web app that can register a callback URL and receive a redirect. An AI coding agent isn't a web app — it's a process running in the developer's terminal. Device-code (RFC 8628) is the OAuth flow designed for exactly this case: the device shows the user a code and a URL, the user authorizes on a separate browser, the device polls to learn the result. No redirect, no callback URL, no web server. ### Can I use Extentos without ever signing up? Yes — until the 1000-event browser-simulator meter exhausts. After that, the simulator stops working until you link a free account; the rest of Extentos (MCP tools, code generation, real-hardware testing through your own Meta credentials, on-device `LocalSimTransport`) keeps working forever, no account. ### Does signing up cost anything? No. The free account tier is the only tier that exists at launch. No payment, no card, no upfront commitment. Email + ToS acceptance only. ### What if the device-code expires before I finish signup? Call `createSimulatorSession` again to get a fresh deviceCode. The backend issues a new code with a fresh 10-minute window. Your prior install state is preserved. ### What if I'm on a headless machine (cloud agent, SSH session)? Set `EXTENTOS_NO_AUTO_OPEN=1`. The verification URL prints to the agent's response; the developer copies it to their local browser. The polling continues in the headless environment regardless of where the URL was opened. ### How do I rotate or revoke my auth token? `extentos-mcp logout` deletes `~/.extentos/auth.json` locally. To revoke server-side, log in to your Extentos account at extentos.com and revoke the device. The next API call from the revoked install will get a 401 and prompt re-linking. ### Is the auth token a secret? Yes — treat it like any other API token. It's stored at `~/.extentos/auth.json` with file-mode `0600` (owner read/write only). Don't commit it to version control. Don't share it across machines. ### Is the `installId` a secret? No — it's a per-machine anonymous identifier. Other apps on the same machine can read `~/.extentos/install_id`; the same value is sent to `api.extentos.com` on every tool call as the telemetry key. It's not a credential. ### What happens if `~/.extentos/auth.json` is corrupted? The MCP server treats it as not-linked and returns to the anonymous tier (with the existing `installId`'s remaining meter). Run `extentos-mcp login` to re-link. ## Related - **[Pricing](/docs/resources/pricing)** — what the free tier and the 1000-event meter actually cover - **[`completeAuthLink` tool](/docs/mcp-server/tools/simulation)** — the MCP tool reference (full input/output schema) - **[MCP server overview](/docs/mcp-server)** — the broader 18-tool catalog - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — installing the server in the first place --- source: https://extentos.com/docs/mcp-server/configuration --- --- title: Configuration description: Configure the Extentos MCP server — environment variables, project directory detection, default vendor. type: reference platform: mcp --- > **Coming soon.** ## What this page will cover - Environment variables - Per-project configuration file - Default vendor selection - Logging --- source: https://extentos.com/docs/mcp-server/install --- --- title: Install the MCP server description: How to install the Extentos MCP server (@extentos/mcp-server) in any MCP-compatible AI coding agent — Claude Code, Cursor, Windsurf, Cline, and others. Per-host install commands, config file locations, copy-pasteable JSON snippets, restart and verify steps, version pinning, updating, troubleshooting common errors, and uninstall instructions. Verified install paths for each supported host. type: tutorial platform: mcp related: - /docs/mcp-server - /docs/mcp-server/agents - /docs/mcp-server/configuration - /docs/mcp-server/auth - /docs/getting-started/with-agent --- The Extentos MCP server (`@extentos/mcp-server`, MIT) is published on npm and installs into any [Model Context Protocol](https://modelcontextprotocol.io)-compatible AI coding agent. This page is the per-host install reference — Claude Code, Cursor, Windsurf, Cline, and the generic MCP-host pattern that covers everything else. Each section gives the verified config file path, the exact JSON snippet to paste, the restart-and-verify steps, and links to troubleshooting if it doesn't work the first time. ## Before you start **Prerequisites:** - **Node.js 20 or newer** on `PATH`. Verify with `node -v`. The MCP server is published as an npm package and runs through `npx`; older Node versions will fail with `Unsupported engine` errors. - **Any MCP-compatible AI coding agent.** Claude Code, Cursor, Windsurf, Cline are all officially verified. Any host that speaks the standard MCP protocol over stdio works — see the [generic MCP host](#generic-mcp-host) section. - **No Extentos account needed.** All MCP tools work anonymously; the [auth flow](/docs/mcp-server/auth) only fires after the 1000-event browser-simulator meter exhausts. **Three install paths**, ranked by simplicity: 1. **Agent prompt** — paste a sentence into your agent and let it install Extentos using its own filesystem-edit / shell tools. Host-agnostic, recommended. 2. **Host-specific command or config** — direct install via the host's CLI (Claude Code) or config file (Cursor, Windsurf, Cline). Per-host instructions below. 3. **Generic MCP JSON** — drop the standard `mcpServers` block into the host's MCP config file by hand. Universal fallback. ## Path 1 — Agent prompt (recommended, host-agnostic) Paste this into your AI coding agent: ```text Install @extentos/mcp-server in my MCP config. ``` The agent will: - Run the right command (`claude mcp add ...` for Claude Code) or edit the right config file (`~/.cursor/mcp.json` for Cursor, etc.) - Tell you to restart your agent host - Verify the tools appear after restart This is the simplest path because the agent figures out the host conventions itself. Use it unless you specifically want to control the install yourself. ## Path 2 — Host-specific install ### Claude Code The cleanest install — Claude Code has a native MCP CLI: ```bash claude mcp add extentos -- npx -y @extentos/mcp-server@latest ``` **Restart Claude Code** (Cmd+Q on macOS, quit from system tray on Windows / Linux — closing the window isn't enough). **Verify:** run `/mcp` in Claude Code. You should see `extentos` listed with all 18 tools available. Or `claude mcp list` from the terminal. The command writes the server entry to your Claude Code MCP config (typically `~/.claude.json`). To remove: `claude mcp remove extentos`. ### Cursor Cursor uses an `mcp.json` file. Two scopes: - **Global** (all projects): `~/.cursor/mcp.json` - **Project-only** (this repo): `.cursor/mcp.json` at the project root Create or edit the file with: ```json { "mcpServers": { "extentos": { "command": "npx", "args": ["-y", "@extentos/mcp-server@latest"] } } } ``` If the file already has other MCP servers, add `"extentos"` as a sibling key inside `mcpServers`. **Restart Cursor** (full quit, not just close window). **Verify:** open Cursor's MCP settings panel — `extentos` should appear in the server list with a green status indicator and 18 tools. Or ask the agent: *"List the MCP servers you have available"* — Extentos's tools should be enumerated. ### Windsurf Windsurf uses a JSON config at `~/.codeium/windsurf/mcp_config.json`. Create or edit it: ```json { "mcpServers": { "extentos": { "command": "npx", "args": ["-y", "@extentos/mcp-server@latest"] } } } ``` **Restart Windsurf** completely. **Verify:** open Windsurf's MCP / tools panel. Extentos should be listed. Or invoke the agent and ask it to call any Extentos tool (e.g., *"Run getPlatformInfo for meta_rayban"*) — it should succeed. ### Cline (VS Code extension) Cline ships as a VS Code extension. Two ways to add the MCP server: **Via Cline's UI** (recommended): 1. Open the Cline panel in VS Code 2. Click the **MCP Servers** tab 3. Click **Edit MCP Settings** (or **Configure MCP Servers** depending on Cline version) 4. Paste the JSON below and save: ```json { "mcpServers": { "extentos": { "command": "npx", "args": ["-y", "@extentos/mcp-server@latest"] } } } ``` **Via the config file directly** — `cline_mcp_settings.json`, typically at: | Platform | Path | |---|---| | macOS | `~/Library/Application Support/Code/User/globalStorage/saoudrizwan.claude-dev/settings/cline_mcp_settings.json` | | Windows | `%APPDATA%\Code\User\globalStorage\saoudrizwan.claude-dev\settings\cline_mcp_settings.json` | | Linux | `~/.config/Code/User/globalStorage/saoudrizwan.claude-dev/settings/cline_mcp_settings.json` | **Restart VS Code** (or reload the Cline extension via the command palette: *Developer: Reload Window*). **Verify:** Cline's MCP Servers tab should show `extentos` with a green status. Tools enumerate when you ask Cline anything Extentos-related. ### Generic MCP host Any host that supports the standard Model Context Protocol can use the same JSON shape. The host's config file path varies, but the server entry is identical: ```json { "mcpServers": { "extentos": { "command": "npx", "args": ["-y", "@extentos/mcp-server@latest"] } } } ``` Find your host's MCP config file (consult the host's documentation). Common locations follow the pattern `~/./mcp*.json` or somewhere under `~/.config//`. Drop the `mcpServers.extentos` entry in, restart, verify. ## Pinning a version By default, `@latest` resolves to the most recent published version on every npm fetch. For reproducibility across sessions, pin a specific version: ```json { "mcpServers": { "extentos": { "command": "npx", "args": ["-y", "@extentos/mcp-server@0.0.16"] } } } ``` Why pin: Extentos is pre-1.0 (current `@0.0.16` as of writing). Minor version bumps may shift APIs until the hardware test loop closes. Pinning means a known-good install survives across MCP server restarts even if a new version ships overnight. The tradeoff is missing bug fixes — re-pin to a newer version periodically. For Claude Code's CLI, pinning looks the same: ```bash claude mcp add extentos -- npx -y @extentos/mcp-server@0.0.16 ``` ## Updating If you used `@latest`, every host restart re-resolves the version through npm — you're always on the newest one as long as your network reaches the npm registry. No manual update step required. If you pinned a version, edit the config to a newer version number and restart your host. Run `npx @extentos/mcp-server@latest update` to check whether a newer version exists (no-op on `@latest` installs). You can also force a fresh fetch by deleting the npx cache: ```bash # macOS / Linux rm -rf ~/.npm/_npx # Windows rmdir /s "%APPDATA%\npm-cache\_npx" ``` The next host restart re-downloads the package. ## Verifying the install worked The fastest universal check: ask your agent to call any Extentos tool. For example: ```text Run getPlatformInfo with sections ["version", "capabilities"] and glasses "meta_rayban" ``` A successful response includes the library version, the Meta Ray-Ban capability catalog (compact form, ~2 KB), and a `summary` field. If the agent reports the tool isn't available, the server didn't load — see [troubleshooting](#troubleshooting) below. Per-host verification methods: | Host | Verify command | |---|---| | Claude Code | `/mcp` — lists active MCP servers | | Cursor | MCP settings panel shows `extentos` with green status | | Windsurf | MCP / tools panel shows `extentos` | | Cline | MCP Servers tab shows `extentos` with green status | | Any | Ask the agent: *"Run getPlatformInfo for meta_rayban"* | The full list of 18 tools you should see: see [`/docs/mcp-server`](/docs/mcp-server) for the catalog. ## Troubleshooting ### `/mcp` doesn't list `extentos` (Claude Code) Restart Claude Code completely — Cmd+Q on macOS, quit from system tray on Windows. Just closing the window isn't enough; the MCP host loads servers at startup. ### `extentos` shows red / errored status (Cursor / Windsurf / Cline) Click into the server's status to see the error. Most common causes: - **Node.js too old.** Run `node -v`. Extentos requires Node.js 20+. Update from [nodejs.org](https://nodejs.org). - **`npx` command not found.** Node.js installed without npx — reinstall a recent Node.js from the official installer. - **Network blocked on first run.** `npx -y @extentos/mcp-server@latest` fetches the package on first run; corporate firewalls or air-gapped environments can block it. Pin a version that's already in the npx cache. - **Config JSON syntax error.** Validate with `cat ~/.cursor/mcp.json | python -m json.tool` (or your tool of choice). A trailing comma or missing bracket prevents the host from parsing the entry. ### First tool call hangs for 10–30 seconds Normal — `npx` is downloading the package on first run. Subsequent tool calls hit the npx cache and respond in milliseconds. ### `EACCES` or permission errors on first run Some sandboxed environments restrict `npx`'s cache directory. Set `NPX_CACHE` to a writable location: ```bash export NPX_CACHE="$HOME/.cache/npx" ``` Then restart your host. ### `auth_required` returned by `createSimulatorSession` immediately You've already consumed the 1000-event anonymous meter on this `installId`. The response includes a `verificationUrl` — sign up there with email (free, no payment), and the agent retries automatically. See the [auth page](/docs/mcp-server/auth) for the full flow. ### Auto-bind not connecting on Android Check that nothing else is bound to localhost port `31337`. The Extentos library probes the MCP server's local bridge at `127.0.0.1:31337/whoami`; if the port is in use, the bridge logs a warning at startup and continues without auto-bind. The developer can still target a session via the URL-bake path (paste the `EXTENTOS_SESSION_URL` snippet Android-side or write the `extentos.session.plist` iOS-side that `createSimulatorSession` returns, then rebuild). To check what's on the port: `lsof -i :31337` on macOS / Linux, `netstat -ano | findstr 31337` on Windows. ### Privacy notice keeps appearing on every tool call The notice is meant to fire once per install. If it's repeating, the consent file at `~/.extentos/consent` isn't being persisted (permissions issue, or `EXTENTOS_CONFIG_DIR` pointing somewhere unwritable). Run `npx @extentos/mcp-server@latest accept-privacy` to set consent explicitly. ## Uninstalling | Host | Command | |---|---| | Claude Code | `claude mcp remove extentos` | | Cursor / Windsurf / Cline / generic | Delete the `"extentos"` entry from the host's `mcp.json` (or equivalent), then restart | To also wipe local state (install ID, auth token, consent): ```bash # macOS / Linux rm -rf ~/.extentos # Windows rmdir /s "%USERPROFILE%\.extentos" ``` This returns the install to a fresh anonymous tier with a fresh 1000-event meter on next install. ## Frequently asked questions ### Which MCP host should I use? Whichever you already use. Extentos is host-agnostic — the same 18 tools work identically across Claude Code, Cursor, Windsurf, and Cline. If you don't have a preference, **Claude Code** has the cleanest install (one CLI command). ### Do I need to install anything else besides the MCP server? No, not for the MCP-driven dev loop. The MCP server itself includes everything the agent needs to scaffold your project, generate code, and provision simulator sessions. Only when you go to ship to **real Ray-Ban Meta hardware** do you need a Meta Developer Center account (free, ~15-30 minutes of setup) — see [vendors/meta](/docs/vendors/meta). ### Can I use Extentos in a CI environment? Yes. The MCP server runs over stdio and `npx` works headlessly. Set `EXTENTOS_NO_AUTO_OPEN=1` to disable browser auto-open during simulator sessions. For air-gapped / no-network CI, install the package globally (`npm install -g @extentos/mcp-server`) so `npx` doesn't try to fetch. ### Does Extentos work in a Docker container? Yes — install Node.js 20+ in the container, mount `~/.extentos` as a volume to persist the `installId` across runs, and treat the rest like any host. Browser-simulator sessions need network access to `wss://api.extentos.com`. ### Can I install multiple Extentos versions side-by-side? Each MCP host has one `extentos` entry at a time, and that entry pins one version. To run multiple versions, register them with different keys: `extentos-stable` (`@0.0.16`) and `extentos-canary` (`@latest`) as separate entries in the same `mcpServers` block. ### What's the difference between `@latest` and `@0.0.16`? `@latest` resolves to whatever's newest on npm at fetch time. `@0.0.16` pins the exact version. Use `@latest` for development (you get bug fixes); pin a version when you need reproducibility (e.g., a team is debugging a specific issue and everyone needs the same MCP build). ### Will I get a popup or modal during install? Only the agent-host's standard "approve this MCP server?" dialog the first time you start it after adding the server. Extentos itself doesn't run any GUI. The first browser-simulator session opens a browser tab to `extentos.com/s/` — you can disable that with `EXTENTOS_NO_AUTO_OPEN=1`. ### Why do I need `npx` instead of `npm install`? `npx` is the standard pattern for MCP servers — it lets the agent host spawn the server without a global install, ensures version isolation per-host, and makes uninstall as simple as removing the config entry. `npm install -g @extentos/mcp-server` works too if you prefer a global install (point your `command` at `extentos-mcp` directly instead of `npx`). ## Related - **[MCP server overview](/docs/mcp-server)** — what the 18 tools do, the canonical agent flow - **[Supported agents](/docs/mcp-server/agents)** — per-host capability notes beyond install - **[Configuration](/docs/mcp-server/configuration)** — environment variables, config files, dial reference - **[Auth](/docs/mcp-server/auth)** — the device-code flow that fires after the meter exhausts - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install + first prompt + verify in the simulator end-to-end --- source: https://extentos.com/docs/mcp-server/tools --- --- title: Tools reference description: The Extentos MCP server exposes 18 deterministic tools across 7 categories — discovery, generation, guidance, validation, simulation, production, and documentation. Per-tool reference covering input parameters, response shapes, when to call each tool, and worked examples. Verified from the actual tool definitions and handler implementations in @extentos/mcp-server. type: reference platform: mcp related: - /docs/mcp-server - /docs/mcp-server/tools/discovery - /docs/mcp-server/tools/generation - /docs/mcp-server/tools/guidance - /docs/mcp-server/tools/validation - /docs/mcp-server/tools/simulation - /docs/mcp-server/tools/production - /docs/mcp-server/tools/search --- The MCP server exposes **18 deterministic tools** an AI agent calls to operate Extentos. Each tool is a typed primitive — clear input schema, predictable response shape, no hidden state. The agent composes them itself; there's no planning tool. Tools are organized by purpose: | Category | Tools | What they're for | |---|---|---| | **[Discovery and Planning](/docs/mcp-server/tools/discovery)** | `getPlatformInfo`, `getExampleSpec` | First calls in any new task — read the catalog, study reference specs | | **[Setup and Generation](/docs/mcp-server/tools/generation)** | `generateConnectionModule`, `initSpec`, `generateConsumer`, `updateSpec` | Scaffold the project and mutate the AppSpec | | **[Implementation Guidance](/docs/mcp-server/tools/guidance)** | `getVoiceCommandGuidance`, `getPermissions` | Side-quest helpers during composition | | **[Validation](/docs/mcp-server/tools/validation)** | `validateSpec`, `inspectIntegration`, `validateIntegration` | Correctness gates — run after every mutation | | **[Simulation](/docs/mcp-server/tools/simulation)** | `createSimulatorSession`, `completeAuthLink`, `getEventLog`, `getSimulatorStatus` | Provision and operate browser-mode simulator sessions | | **[Production](/docs/mcp-server/tools/production)** | `getProductionChecklist`, `getCredentialGuide` | Pre-ship checks | | **[Search](/docs/mcp-server/tools/search)** | `searchDocs` | Search the Extentos documentation by topic | ## Conventions A few conventions hold across every tool: - **All tool inputs and outputs are JSON.** Tools that take an object pass it via the MCP `arguments` field; the schemas live in `mcp-server/src/tools/definitions.ts`. - **`additionalProperties: false`.** Unknown parameters fail validation with `code: "invalid_arguments"` and a `fixHint` telling the agent what to drop. - **Successful responses include a `summary` string** — a short human-readable description of what the tool returned. Agents can surface this directly to the developer. - **Errors return a structured shape** — `{ code, message, fixHint?, retryable? }` — instead of throwing. The agent can decide whether to retry or surface the error. - **Freshness tokens guard concurrent edits.** Tools that mutate the spec (`updateSpec`, `generateConsumer`) require a `basedOn: { hash, timestamp }` token from `inspectIntegration`. Stale tokens fail fast — no silent overwrites of concurrent agent edits. - **Tool calls do not count against the simulator-event meter.** The 1000-event meter only counts `BrowserSimTransport` runtime events during a simulator session. MCP-server-side activity is always free. ## The canonical agent-driven flow For a fresh project, the typical tool sequence is: ```text getPlatformInfo → searchDocs → generateConnectionModule → initSpec → generateConsumer → validateIntegration → createSimulatorSession → (developer interacts) → getEventLog ``` For iteration: `inspectIntegration` (for the `basedOn` token) → `updateSpec` → `generateConsumer` (if handlers/streams added) → `validateIntegration`. Before shipping: `getProductionChecklist` → `getCredentialGuide`. ## Per-category reference The deep reference per tool lives on the category pages: - **[Discovery and Planning →](/docs/mcp-server/tools/discovery)** - **[Setup and Generation →](/docs/mcp-server/tools/generation)** - **[Implementation Guidance →](/docs/mcp-server/tools/guidance)** - **[Validation →](/docs/mcp-server/tools/validation)** - **[Simulation →](/docs/mcp-server/tools/simulation)** - **[Production →](/docs/mcp-server/tools/production)** - **[Search →](/docs/mcp-server/tools/search)** ## Source of truth Every claim on these pages is verified from `mcp-server/src/tools/definitions.ts` (input JSON Schemas), `mcp-server/src/tools/registry.ts` (handler wiring), and the per-tool handler implementations under `mcp-server/src/tools/handlers/`. If a doc claim conflicts with the source, the source wins — file an issue and we'll reconcile. --- source: https://extentos.com/docs/mcp-server/tools/discovery --- --- title: Discovery and Planning tools description: The Extentos MCP server's discovery tools — getPlatformInfo (returns the static platform catalog of vendor capabilities, library version, and spec schema) and getExampleSpec (retrieves one of 8 canonical reference patterns for studying complex spec compositions). These are the first calls an AI agent makes in any new task — cheap, all local, no side effects, no meter cost. type: reference platform: mcp related: - /docs/mcp-server/tools - /docs/mcp-server/tools/generation - /docs/mcp-server/tools/search - /docs/concepts/capabilities --- The two discovery tools answer "what does Extentos offer for this target?" before any project mutation. Both are pure-read, no side effects, no state changes, and don't count against the simulator-event meter. > **For your installed agent:** these tools are invoked directly via the MCP — once the server is registered with your agent, it calls `getPlatformInfo` or `getExampleSpec` and gets live, structured responses. This page exists so an agent evaluating Extentos pre-install can see the tool surface, and so human developers and search engines can find the documentation outside an MCP context. The live MCP response is authoritative; the parameter tables and example responses below are illustrative. ## `getPlatformInfo` Returns the static platform catalog — library version, vendor capabilities, and (on demand) the AppSpec JSON Schema. The right first call for any new task. ### When to use - At session start, before any spec work - When the agent needs to know which capabilities a vendor supports - When the agent is building a compatibility table or capability check ### When NOT to use - For *how to use* a primitive — use [`searchDocs`](/docs/mcp-server/tools/search#searchdocs) catalog topics (`trigger_types`, `action_types`, `block_types`, `stream_types`) instead — they carry inline minimal examples - For checking what's installed in the project — use [`inspectIntegration`](/docs/mcp-server/tools/validation#inspectintegration) ### Parameters | Parameter | Type | Required | Description | |---|---|---|---| | `sections` | `array<"version" \| "capabilities">` | yes | Which catalog sections to return. At least one. | | `glasses` | `"meta_rayban"` | conditional | Required when `sections` includes `"capabilities"`. Only `meta_rayban` is supported in the current MVP. | | `expand` | `array<"schema" \| "capabilities.full" \| "capabilities.advanced">` | no | Opt-in to detail. See response-size table below. | ### Response The default response is **compact (~2 KB)** — names, tier markers, global constraints — sized to fit comfortably in the agent's context. Detail is opt-in via `expand`: | Response variant | Approx size | What's included | |---|---|---| | Default (compact capabilities) | ~2 KB | Vendor, DAT version, block names + tiers, trigger names + tiers, stream names + tiers, action names, toggle keys, global constraints | | `expand: ["capabilities.full"]` | ~10 KB | All of the above plus per-primitive params, payload shapes, requires lists, per-primitive constraints | | `expand: ["capabilities.advanced"]` | + ~1 KB | Adds `droppedPrimitives` and `futurePrimitives` informational fields | | `expand: ["schema"]` | + ~28 KB | Adds the full AppSpec JSON Schema (`schemaUri: "extentos://schema/v1"`) — niche; usually `validateSpec` is the right tool | The compact-by-default design is intentional. Returning the full ~36 KB on every first call burns the agent's context window before any work. The `summary` field hints at what's available and how to expand: > *"For per-primitive params/payload/constraints, prefer searchDocs(topic: 'trigger_types' | 'action_types' | 'block_types' | 'stream_types'); or call again with expand: ['capabilities.full']."* ### Example — discovery at session start ```json { "sections": ["version", "capabilities"], "glasses": "meta_rayban" } ``` Response (compact): ```json { "version": { "latestStable": "0.0.16", "...": "..." }, "capabilities": { "vendor": "meta_rayban", "datVersion": "...", "blocks": [ { "type": "capture_photo", "tier": "stable" }, { "type": "capture_video", "tier": "stable" }, { "type": "record_audio", "tier": "stable" }, { "type": "speak_text", "tier": "stable" } ], "triggers": [ { "type": "manual_launch", "tier": "stable" }, { "type": "voice_command", "tier": "stable" }, { "type": "capture_button", "tier": "stable" }, { "type": "tap", "tier": "preview" }, { "type": "double_tap", "tier": "preview" } ], "streams": ["..."], "actions": ["block_call", "ai_call", "branch", "set_variable"], "toggles": ["..."], "globalConstraints": { "..." : "..." } }, "summary": "Returned library v0.0.16, meta_rayban capabilities (compact: 4 blocks, 2 streams, 5 triggers, 4 actions, N toggles)." } ``` ### Errors - **`invalid_arguments` — `sections must be a non-empty array`** — pass at least one of `"version"`, `"capabilities"` - **`invalid_arguments` — `glasses is required when sections includes "capabilities"`** — pass `glasses: "meta_rayban"` - **`invalid_arguments` — `"schema" is no longer a valid section`** — `"schema"` was promoted out of `sections` and into `expand`. Drop it from `sections`; pass `expand: ["schema"]` separately if the raw schema is genuinely needed. --- ## `getExampleSpec` Returns one of 8 canonical reference specs — fully-worked compositions for studying how complex flows compose. Reference library, not a starting template. ### When to use - On demand, when studying how a complex composition works - When the agent needs an example of a multi-block, multi-trigger, multi-handler flow - Pattern matching: "how does a translation app compose voice trigger → audio capture → AI call → speak?" ### When NOT to use - For simple cases — read [`searchDocs`](/docs/mcp-server/tools/search#searchdocs) catalog topics first; they carry inline minimal examples sufficient for most specs - As a starting template for [`initSpec`](/docs/mcp-server/tools/generation#initspec) — write the spec from primitives the developer's actual goal needs, not from a copy of an example ### Parameters | Parameter | Type | Required | Description | |---|---|---|---| | `pattern` | enum (one of 8 below) | yes | Which canonical pattern to retrieve | The 8 patterns: | Pattern ID | Composition | |---|---| | `voice_assistant` | Voice trigger → AI call → speak response | | `live_translation` | Audio stream → AI translation → speak | | `scene_description` | Capture photo → AI vision → speak | | `notification_reader` | Manual launch → AI call → speak | | `navigation_guide` | Voice trigger + sensor stream → AI vision → speak | | `video_call` | Video stream + audio stream → handler | | `object_detection` | Video frame stream → AI vision → annotated speak | | `voice_notes` | Voice trigger → audio record → AI speech-to-text → store | ### Response ```json { "pattern": "", "description": "", "spec": { "...": "...full AppSpec JSON..." }, "usedPrimitives": { "blocks": ["capture_photo", "speak_text"], "streams": [], "triggers": ["voice_command"] }, "handlerNames": ["vision", "..."], "codeSnippets": { "...": "..." }, "summary": "Reference spec 'voice_assistant': 2 blocks, 0 streams, 1 trigger, 2 handlers. Study, don't copy whole — peel pieces relevant to your goal." } ``` The summary is the agent-facing reminder: study, don't copy. Different developers' apps need different compositions, and a copy-paste of `voice_assistant` will rarely match a specific developer's goal. ### Example — study the live translation pattern ```json { "pattern": "live_translation" } ``` Response includes the full spec, the primitives used, and example handler code so the agent can compose its own translation app rather than recreate the example wholesale. ### Errors - **`invalid_arguments` — `pattern must be one of: voice_assistant, live_translation, scene_description, notification_reader, navigation_guide, video_call, object_detection, voice_notes`** — only the 8 canonical IDs are accepted --- ## Why these are the first calls `getPlatformInfo` answers "what's possible on this vendor?" — without that catalog, the agent is composing blind. `getExampleSpec` answers "how do other apps look that combine the primitives I'm considering?" — useful when the spec the developer is asking for is genuinely complex (multi-flow, multi-stream, multi-handler). Most simple cases skip `getExampleSpec` entirely and go `getPlatformInfo` → `searchDocs` (for topic-specific catalogs with inline examples) → straight into `generateConnectionModule`. The example library is for the moments where studying a worked composition is faster than composing from scratch. ## Related - **[Setup and Generation tools](/docs/mcp-server/tools/generation)** — what the agent calls *after* discovery (the mutation tools) - **[Search tools](/docs/mcp-server/tools/search)** — `searchDocs` catalog topics with inline examples - **[Capabilities](/docs/concepts/capabilities)** — the conceptual layer behind what `getPlatformInfo` returns - **[Tools overview](/docs/mcp-server/tools)** — back to the full 18-tool catalog --- source: https://extentos.com/docs/mcp-server/tools/generation --- --- title: Setup and Generation tools description: The Extentos MCP server's generation tools — generateConnectionModule (one-shot project scaffold), initSpec (first AppSpec), generateConsumer (handler stubs), updateSpec (subsequent spec mutations). The mutation sequence the agent runs to bring Extentos into a fresh app. type: reference platform: mcp related: - /docs/mcp-server/tools - /docs/mcp-server/tools/discovery - /docs/mcp-server/tools/validation --- > **Coming soon.** Per-tool reference for `generateConnectionModule`, `initSpec`, `generateConsumer`, `updateSpec`. Each section will follow the format established in [Discovery and Planning](/docs/mcp-server/tools/discovery) — when to use, parameters, response shape, errors, worked example. --- source: https://extentos.com/docs/mcp-server/tools/guidance --- --- title: Implementation Guidance tools description: The Extentos MCP server's guidance tools — getVoiceCommandGuidance (analyze proposed voice phrases for UX issues) and getPermissions (derive exact platform permissions, Meta DAT requirements, foreground service needs from a spec). Side-quest helpers the agent calls during composition. type: reference platform: mcp related: - /docs/mcp-server/tools - /docs/mcp-server/tools/generation - /docs/concepts/capabilities --- > **Coming soon.** Per-tool reference for `getVoiceCommandGuidance`, `getPermissions`. Each section will follow the format established in [Discovery and Planning](/docs/mcp-server/tools/discovery). --- source: https://extentos.com/docs/mcp-server/tools/production --- --- title: Production tools description: The Extentos MCP server's production tools — getProductionChecklist (personalized ship-readiness checklist based on what the spec uses) and getCredentialGuide (step-by-step credential setup for Anthropic, OpenAI, Google Cloud, DeepL, Azure, AWS Bedrock, HuggingFace, or custom providers). Pre-ship gates. type: reference platform: mcp related: - /docs/mcp-server/tools - /docs/mcp-server/tools/validation - /docs/resources/security --- > **Coming soon.** Per-tool reference for `getProductionChecklist`, `getCredentialGuide`. Each section will follow the format established in [Discovery and Planning](/docs/mcp-server/tools/discovery). --- source: https://extentos.com/docs/mcp-server/tools/search --- --- title: Search tools description: The Extentos MCP server's search tool — searchDocs (search Extentos documentation by topic or keyword). Catalog topics like trigger_types, action_types, block_types, stream_types, spec_format, template_syntax, app_callback_guide, library_api carry inline minimal examples agents can compose against. type: reference platform: mcp related: - /docs/mcp-server/tools - /docs/mcp-server/tools/discovery --- > **Coming soon.** Per-tool reference for `searchDocs`. The page will follow the format established in [Discovery and Planning](/docs/mcp-server/tools/discovery). --- source: https://extentos.com/docs/mcp-server/tools/simulation --- --- title: Simulation tools description: The Extentos MCP server's simulation tools — createSimulatorSession (provision a browser-mode session at extentos.com/s with auto-bind to the running app), completeAuthLink (poll the device-code flow after the 1000-event meter exhausts), getEventLog (structured 7-layer event trace from a session), getSimulatorStatus (live session state). The browser-driven dev loop. type: reference platform: mcp related: - /docs/mcp-server/tools - /docs/mcp-server/tools/validation - /docs/concepts/transport-vs-app --- > **Coming soon.** Per-tool reference for `createSimulatorSession`, `completeAuthLink`, `getEventLog`, `getSimulatorStatus`. Each section will follow the format established in [Discovery and Planning](/docs/mcp-server/tools/discovery). --- source: https://extentos.com/docs/mcp-server/tools/validation --- --- title: Validation tools description: The Extentos MCP server's validation tools — validateSpec (pure JSON spec check against schema rules), inspectIntegration (read-only project snapshot with the basedOn freshness token), validateIntegration (whole-project correctness gate before testing). Run after every spec mutation. type: reference platform: mcp related: - /docs/mcp-server/tools - /docs/mcp-server/tools/generation - /docs/mcp-server/tools/simulation --- > **Coming soon.** Per-tool reference for `validateSpec`, `inspectIntegration`, `validateIntegration`. Each section will follow the format established in [Discovery and Planning](/docs/mcp-server/tools/discovery). --- source: https://extentos.com/docs/reference --- --- title: Reference description: Auto-generated API reference for the iOS SDK (DocC), Android SDK (Dokka), MCP server tools (JSON Schema), and a flat error code index. type: reference platform: all --- Generated from source. If you're hand-editing a page in this section, something's wrong. - [iOS API](/docs/reference/ios-api) — generated from DocC - [Android API](/docs/reference/android-api) — generated from Dokka - [MCP tools](/docs/reference/mcp-tools) — generated from JSON Schema - [Error codes](/docs/reference/errors) — flat index, cross-linked from troubleshooting --- source: https://extentos.com/docs/reference/android-api --- --- title: Android API reference description: Auto-generated from Dokka. Symbol-level documentation for every public type in the Extentos Android SDK. type: reference platform: android --- > **Coming soon.** This page will be generated from Dokka at build time. --- source: https://extentos.com/docs/reference/errors --- --- title: Error codes description: Every error Extentos emits, with cause, fix, and cross-links to troubleshooting. type: reference platform: all related: - /docs/troubleshooting --- > **Coming soon.** Flat index of all error codes. Each error gets a deterministic anchor (`/docs/reference/errors#EXT_E_NNNN`) so agents can deep-link. --- source: https://extentos.com/docs/reference/ios-api --- --- title: iOS API reference description: Auto-generated from DocC. Symbol-level documentation for every public type in the Extentos iOS SDK. type: reference platform: ios --- > **Coming soon.** This page will be generated from DocC at build time. --- source: https://extentos.com/docs/reference/mcp-tools --- --- title: MCP tools reference description: Auto-generated from JSON Schema. Every tool the Extentos MCP server exposes, with input/output schemas. type: reference platform: mcp --- > **Coming soon.** This page will be generated from the MCP server's JSON Schema at build time. --- source: https://extentos.com/docs/resources --- --- title: Resources description: Pricing, security, support, status, license, roadmap, FAQ, and changelog. type: resource platform: all --- - [Pricing](/docs/resources/pricing) - [Security](/docs/resources/security) - [Support](/docs/resources/support) - [Status](/docs/resources/status) - [License](/docs/resources/license) - [Roadmap](/docs/resources/roadmap) - [FAQ](/docs/resources/faq) - [Changelog](/docs/resources/changelog) --- source: https://extentos.com/docs/resources/changelog --- --- title: Changelog description: Release notes for the Extentos MCP server, iOS SDK, and Android SDK. type: resource platform: all --- > **Coming soon.** --- source: https://extentos.com/docs/resources/faq --- --- title: Frequently asked questions description: Consolidated FAQ for Extentos — the smart-glasses development layer for Meta Ray-Ban (and future vendors). What it is, how to install it, what it costs (free for development, no paid tier at launch), what hardware it supports (every Ray-Ban Meta and Oakley Meta variant in market), what it collects (aggregate metadata, never end-user content), how it differs from Meta's Device Access Toolkit and Mock Device Kit, and where it's headed. Curated answers with links to the deep pages. type: resource platform: all related: - /docs/getting-started/with-agent - /docs/concepts/transport-vs-app - /docs/vendors/meta - /docs/resources/pricing - /docs/resources/security --- The most-asked questions about Extentos, organized by topic and cross-linked to the deep documentation. If you don't find your question here, see the [support page](/docs/resources/support) or open an issue on the [GitHub repository](https://github.com/Asgermolgaard/vibe-hardware/issues). ## Getting started ### Is Extentos free? Yes. All 18 MCP tools, on-device simulation through Meta's Mock Device Kit, code generation, validation, and real-hardware testing on Ray-Ban Meta are free with no account, forever. The browser simulator at [extentos.com/s](https://extentos.com/s) is free for the first 1000 runtime events on your MCP install; after that, you create a free email-only account (no payment) to keep going. There is no paid tier at launch. See [pricing](/docs/resources/pricing) for the full breakdown. ### Do I need an Extentos account to start? No. You can install the MCP server, generate code, scaffold projects, validate specs, and test on real Ray-Ban Meta hardware without ever creating an Extentos account. Account linking only fires when the 1000-event browser-simulator meter exhausts — and even then, the account is free email-only with no payment. See [auth](/docs/mcp-server/auth) for the full anonymous-first model. ### How do I install Extentos in my AI coding agent? The shortest path is the agent prompt: paste *"Install @extentos/mcp-server in my MCP config"* into Claude Code, Cursor, Windsurf, or Cline. The agent edits the right config file or runs the right command for its host. For Claude Code specifically, the one-liner is `claude mcp add extentos -- npx -y @extentos/mcp-server@latest`. See [install](/docs/mcp-server/install) for per-host instructions and config-file locations. ### What prerequisites do I need? Node.js 20 or newer on `PATH`. That's it for the MCP server — `npx` fetches and caches the package on first run. To target real Ray-Ban Meta hardware later, you'll also need a Meta Developer Center account (free, ~15-30 minute setup) for the Meta DAT credentials. See [vendors/meta](/docs/vendors/meta) for the Meta-specific setup. ## Hardware support ### What smart glasses does Extentos support? As of 2026-04, Extentos is **GA on Meta Ray-Ban only** — every Ray-Ban Meta variant in market (Gen 1 Wayfarer, Headliner, Skyler frame styles; Gen 2 refresh; Ray-Ban Meta Display) plus Oakley Meta HSTN. Mentra G1, Android XR, and Apple smart glasses are tracked roadmap vendors but not yet transport-implemented. See [vendors](/docs/vendors) for the full status table. ### Does Extentos work with Ray-Ban Meta Display? Yes. Your Extentos-built app runs on a Display-variant Ray-Ban Meta the same way it runs on Gen 1 or Gen 2 — full camera, voice trigger, audio, and sensor support. What's *not* available on Display is the heads-up display rendering and the neural-band gesture input, because Meta keeps those as Meta-curated partner surfaces in the DAT toolkit. Third-party apps don't get them, even on Display hardware. See [vendors/meta § Compatible models](/docs/vendors/meta#compatible-models). ### Why does Google Maps get to render on the Ray-Ban Meta Display when my app can't? Display rendering is not exposed in the public Meta DAT toolkit. Google Maps, Spotify, WhatsApp, and other featured integrations are Meta-curated partnerships with privileged access to the display surface. The path to display rendering today is a Meta partnership, not the public DAT toolkit. See [vendors/meta § Privileged partnerships](/docs/vendors/meta#privileged-partnerships-the-google-maps-question). ### Do I need real Ray-Ban Meta glasses to develop? No. Mock Device Kit (via Extentos's `LocalSimTransport`) and the Extentos browser simulator (`BrowserSimTransport` at [extentos.com/s](https://extentos.com/s)) cover the bulk of the dev loop. Real hardware verifies the final-mile fit (camera fidelity, BT latency, real-world coexistence) but isn't required for daily iteration. See [transport vs app simulation](/docs/concepts/transport-vs-app) for what each simulator does. ### When will Mentra G1 / Android XR / Apple be supported? No committed timelines. New vendors are added when adding them makes a real product (transport implementation + simulator coverage + tested hardware), which depends on each vendor's SDK maturity and hardware availability. See the [changelog](/docs/resources/changelog) for actual release activity. The [vendors page](/docs/vendors) tracks roadmap status. ## Pricing and accounts ### What counts as a "runtime event"? An event emitted by the browser simulator during a session — `trigger.fired`, `block.executed`, `callback.invoked`, `flow.completed`, etc. A typical voice-to-AI interaction emits 5–10 events, so 1000 events ≈ 100–200 full interactions. **MCP tool calls, code generation, on-device `LocalSimTransport` activity, and real-hardware testing do not count against this meter, ever.** See [pricing § What counts as one event](/docs/resources/pricing#what-counts-as-one-event). ### Does my shipped app cost me anything? No. Production apps run against real Ray-Ban Meta hardware via `RealMetaTransport`, which doesn't talk to the Extentos backend at all. Your shipped app's runtime cost from Extentos is zero per end-user. The only Extentos-side cost is dev-time browser-simulator usage above the 1000-event meter, which falls under the free email-only account. ### Will Extentos add a paid tier later? Possibly, as the product matures. Existing free accounts will not be retroactively gated when paid options are introduced — material changes will be communicated via the [changelog](/docs/resources/changelog). Today there is no paid tier. ### Can I share an Extentos account across my team? Single accounts only at MVP. Multi-seat billing and team workspaces are deferred post-launch. Each developer should run their own MCP install with their own account — the per-machine `installId` is anonymous and doesn't conflict with collaboration. ## Capabilities and limits ### Can my third-party app trigger on "Hey Meta, [my app], do X"? No. The "Hey Meta" wake word is system-level and not exposed to third-party apps. Your custom voice trigger is recognized by your *phone's* speech recognizer (`SpeechRecognizer` on Android, `SFSpeechRecognizer` on iOS) over Bluetooth audio — the wake word *is* your custom phrase, not "Hey Meta" + your phrase. See [vendors/meta § Capabilities NOT in the public toolkit](/docs/vendors/meta#capabilities-not-in-the-public-toolkit). ### Can my app use custom gestures (tap, swipe)? Not via the public Meta DAT toolkit as of 2026-04. The hardware supports tap and swipe; Meta's public API surface doesn't expose them yet. Standard DAT lifecycle events (pause, resume, hinges-closed, thermal warnings) work normally. ### What can I actually build for Meta Ray-Ban today? Photo capture, video capture, camera streams, voice-triggered apps (custom phrases via phone STT), TTS-driven audio apps, sensor/IMU consumers — all on the public DAT toolkit. The full capability matrix is at [vendors/meta § What's in the Device Access Toolkit](/docs/vendors/meta#whats-in-the-device-access-toolkit-dat). ### Why doesn't Mock Device Kit simulate voice triggers? Mock Device Kit simulates the BLE/SDK layer — your app *thinks* it's connected to glasses. Voice-trigger recognition runs on the *phone's* speech recognizer, not on the glasses, so simulating the glasses doesn't simulate the recognizer. Extentos's browser simulator replaces the *whole* path with the developer's webcam + microphone, which is why it can exercise voice triggers end-to-end. See [transport vs app simulation](/docs/concepts/transport-vs-app). ## Distribution ### Can I publish my Meta Ray-Ban app to the Meta App Store? As of 2026-04, the Meta App Store is preview-limited to Meta-curated partners. There is no public submission flow open to all developers. Most third-party developers ship via private distribution — APK sideload on Android, TestFlight on iOS, or enterprise channels — with no Meta review needed. See [vendors/meta § Distribution state](/docs/vendors/meta#distribution-state-2026-04). ### How do users install my Meta Ray-Ban app? Users install your *phone* app (Android via APK or Play Store; iOS via TestFlight or App Store). The first time they open it, the Meta DAT registration flow runs and pairs the app with their Ray-Ban Meta. Your app is a regular phone app that talks to the glasses over Bluetooth — there's no "install on the glasses" step. ## Privacy and data collection ### Does Extentos collect my users' photos, video, voice transcripts, or AI prompts? No, never. Extentos's library is architecturally incapable of inspecting `app_callback` payloads, AI provider responses, photo bytes, video frames, audio samples, or speech transcripts. The full never-collect list is enumerated in [security § What we never collect](/docs/resources/security#what-we-never-collect). End-user content stays between your app and your providers. ### Does Extentos see my AI provider API keys? No. AI provider keys (Anthropic, OpenAI, Google, etc.) live in your app code and your provider's auth headers. The `app_callback` mechanism is a function pointer your code provides; Extentos invokes it and gets the result back. There's no inspection of inputs, no logging of outputs, no proxy layer. See [security § BYOK](/docs/resources/security#byok-—-your-ai-provider-keys). ### What does Extentos collect, then? Aggregate metadata — tool calls, install events, vendor adoption rates, capability usage, error rates, library/MCP versions, device models, country (derived from IP, then the IP is dropped). Tagged with anonymous IDs that can never be reversed back to a person. This data powers a developer-facing analytics dashboard (coming soon), Extentos's product roadmap, and aggregate ecosystem signals shared with glasses vendors as market data (10-account aggregation floor). Full schema in [security § What we collect](/docs/resources/security#what-we-collect). ### Will Extentos sell data about my specific app to my competitors? No. Aggregate ecosystem signals (vendor market share, capability popularity) are sold to glasses vendors only as aggregates over **at least 10 distinct accounts**, never as individual-developer or individual-end-user data. The 10-account floor is the structural guarantee against competitive leakage. ### Can I opt out of telemetry? Yes — three independent surfaces. (1) `EXTENTOS_TELEMETRY=0` env var disables MCP-side telemetry for the current shell. (2) `npx @extentos/mcp-server@latest decline-privacy` disables it permanently for the install. (3) `ExtentosConfig.telemetryConsent = false` in your shipped app disables library-runtime telemetry per-app. Any combination works. See [security § Three opt-out surfaces](/docs/resources/security#three-opt-out-surfaces). ## Architecture and AI agents ### Is Extentos a wrapper around Meta's Device Access Toolkit? Partially. Extentos has three internal transport implementations: `RealMetaTransport` wraps the real Meta DAT SDK for production hardware; `LocalSimTransport` wraps Meta's Mock Device Kit for fast on-device testing; `BrowserSimTransport` is an Extentos-original architecture (browser simulator at extentos.com/s, WebSocket protocol, no DAT involvement) for the headline dev experience. See [transport vs app simulation](/docs/concepts/transport-vs-app) for the deep dive. ### Does my code change when targeting different vendors? No, by design. Your AppSpec is written in vendor-agnostic capability primitives (`capture_photo`, `voice_command`, `speak_text`, etc.). When a new vendor (Mentra G1, Android XR, Apple smart glasses) gets a transport implementation in Extentos, your existing app code runs on it without rewrites. See [capabilities § Why a shared vocabulary](/docs/concepts/capabilities#why-a-shared-vocabulary-is-the-right-design). ### Which AI coding agent works best with Extentos? All MCP-compatible agents (Claude Code, Cursor, Windsurf, Cline) work — Extentos's tools are protocol-standard. Claude Code has the smoothest install (single CLI command); the others need a small JSON config edit. Tool capabilities are identical across hosts. See [install](/docs/mcp-server/install) for per-host setup. ### What does the AI agent actually do with the MCP server? The agent calls Extentos's 18 deterministic tools to read your project state, scaffold a connection module, generate a vendor-agnostic AppSpec, stub out handler functions, validate the integration, and provision a browser simulator session — all without you typing any setup code. The canonical sequence is `getPlatformInfo` → `searchDocs` → `generateConnectionModule` → `initSpec` → `generateConsumer` → `validateIntegration` → `createSimulatorSession`. See [MCP server overview](/docs/mcp-server) for the full tool catalog. ### What happens when the user folds their glasses mid-flow? Meta DAT emits a `hinges_closed` hardware event. Extentos surfaces this as a `transport.hardware_alert` event in the structured event log and (depending on your spec) can dispatch a `hinges_closed` trigger that fires whatever flow you defined for that case. Apps typically pause active streams and wait for the user to unfold. ## Help and resources ### Where do I file a bug or feature request? GitHub issues: [github.com/Asgermolgaard/vibe-hardware/issues](https://github.com/Asgermolgaard/vibe-hardware/issues). Include your install version (`npx @extentos/mcp-server@latest whoami`), the platform (Android / iOS), the host (Claude Code / Cursor / Windsurf / Cline), and a minimal reproduction. ### Where's the source code and license? Source: [github.com/Asgermolgaard/vibe-hardware](https://github.com/Asgermolgaard/vibe-hardware). The MCP server (`@extentos/mcp-server`) is MIT-licensed; the Android and iOS libraries are source-available in the same repo. See [resources/license](/docs/resources/license) for current declarations and pre-1.0 status. ### Where do I find release notes? The [changelog](/docs/resources/changelog) tracks notable releases. Pre-1.0 (current `@0.0.16`) means APIs may shift between minor versions until the hardware test loop closes — pin to an exact version if you need cross-session reproducibility. ## Related - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install + first prompt + verify - **[Transport vs app simulation](/docs/concepts/transport-vs-app)** — the differentiator framing - **[Vendors: Meta Ray-Ban](/docs/vendors/meta)** — the GA target's full capability matrix - **[Pricing](/docs/resources/pricing)** — what's free, what costs (nothing, at launch) - **[Security](/docs/resources/security)** — data collection, opt-out, end-user privacy --- source: https://extentos.com/docs/resources/license --- --- title: License description: Extentos license terms — the @extentos/mcp-server npm package is MIT-licensed (verified in package.json). The Android library com.extentos.glasses.core and the iOS library Extentos are source-available in the same GitHub repository, MIT licensing pending pre-1.0 release alongside formal LICENSE file declarations. This page tracks the current state and the source location for each component. type: resource platform: all --- > **Coming soon.** --- source: https://extentos.com/docs/resources/pricing --- --- title: Pricing description: Extentos is free. All 18 MCP tools, on-device simulation through Meta's Mock Device Kit, real-hardware testing on Meta Ray-Ban, and the browser simulator at extentos.com/s are free with no account for the first 1000 runtime events. After that you create a free email-only account to keep going. There is no paid tier at launch. type: resource platform: all related: - /docs/getting-started/with-agent - /docs/concepts/transport-vs-app --- Extentos is free. There is no paid tier at launch. ## Free forever, no account - **All 18 MCP tools** — capability discovery, spec authoring, code generation, validation, ship-readiness, troubleshooting - **`LocalSimTransport`** — the on-phone simulator that wraps Meta's Mock Device Kit (`mwdat-mockdevice` / `MWDATMockDevice`) - **Code generation and `validateIntegration`** — all scaffolding, all correctness gates - **Real-hardware testing through `RealMetaTransport`** — you bring your own Meta Developer Center registration (a one-time, free, ~15–30 min setup with your personal Meta credentials) These never expire, never charge, and require no Extentos account. ## Browser simulator at extentos.com/s The browser-based simulator at [extentos.com/s](https://extentos.com/s) is free for the first **1000 runtime events** per MCP install — no signup, no email, no payment. A *runtime event* is something the simulator emits during a session — `trigger.fired`, `block.executed`, `callback.invoked`, `flow.completed`, etc. A typical voice-to-AI interaction emits roughly 5–10 events, so 1000 events ≈ 100–200 full interactions. Enough runtime to validate a non-trivial spec on your actual app. **What does not count against the meter:** MCP tool calls, code generation, `validateIntegration`, on-device `LocalSimTransport` activity, real-hardware testing, and any telemetry from your shipped app. Only the dev-time browser-simulator session events count. ## After 1000 events: free account When you exhaust the meter, the device-code flow fires automatically. Your agent gets an `authRequired` response with a verification URL, you open it, sign up with email + ToS acceptance, and the original `createSimulatorSession` call retries automatically. - **Free, email-only.** No payment, no credit card, no upfront commitment. - **No manual token paste.** The agent picks up the auth token automatically once the browser flow completes. - **Unmetered at launch.** The free account tier has no event cap right now. ## Real hardware testing Running your app on actual Meta Ray-Ban glasses uses Meta's Device Access Toolkit and requires your own Meta Developer Center registration — a one-time, free, ~15–30 minute setup with your personal Meta credentials. Extentos doesn't sit in that path. There's no per-end-user cost from Extentos when your app ships and end users use it on their glasses. ## What's intentionally not here - **No paid tier at launch.** No $20/month, no premium voice add-ons, no team seats with billing, no analytics dashboard behind a paywall. The product is at the stage where the priority is getting Extentos into developers' hands and validating the simulator experience, not optimizing revenue. - **Pricing may evolve.** Paid options may be introduced for power users as the product matures. Existing free accounts will not be retroactively gated when that happens. Material changes will be communicated via the [changelog](/docs/resources/changelog). ## Frequently asked questions ### What if I never exhaust the 1000 events? Then you never need an account. Use Extentos forever, free, anonymous. ### What counts as one event? One emission from the browser simulator during a session — for example, a voice trigger firing, a block starting, a block completing, a callback being invoked. The 7 layers of the [event log](/docs/concepts/architecture) are the source. MCP tool calls and on-device activity do not count. ### Is there a way to reset the meter? No. The 1000-event meter is keyed to the MCP install's `installId` and is lifetime. The expectation is that 1000 events is enough to evaluate Extentos on your actual app, after which a free account is the simplest next step. ### Do I need a paid plan to ship to production? No. Production apps run against real Meta Ray-Ban hardware via `RealMetaTransport`, which doesn't touch Extentos's backend at all. Your shipped app's runtime cost from Extentos is zero. ### Is the on-device `LocalSimTransport` ever metered? No. `LocalSimTransport` wraps Meta's Mock Device Kit on the developer's phone or emulator and never reaches an Extentos backend. It's always free, always unmetered, always available. --- source: https://extentos.com/docs/resources/roadmap --- --- title: Roadmap description: Public roadmap for Extentos — what's currently shipping in @extentos/mcp-server@0.0.16, what's in active development (next vendor expansion beyond Meta Ray-Ban, developer-facing analytics dashboard, post-1.0 hardening), what's tracked further out (additional vendors like Mentra G1 and Android XR, paid tier evaluation), and what depends on third parties (Meta DAT capability expansions, Apple smart-glasses SDK availability). Updated as items ship; the live capability tier for any specific block, trigger, or stream is authoritative via the getPlatformInfo MCP tool, not this page. type: resource platform: all related: - /docs/vendors - /docs/resources/changelog - /docs/resources/pricing - /docs/resources/security --- This page is the **public roadmap** — strategic direction for Extentos, dated for honesty about how recent the picture is. It is not a commitment to timelines and not a substitute for the changelog (which tracks what actually shipped). For the **live state** of any specific capability — what's GA today vs preview vs unsupported on a target vendor — your installed AI agent calls `getPlatformInfo` against the MCP server, which returns the authoritative tier marker per primitive. This page covers direction; the agent gets reality. > **At runtime, your installed agent has the live capability state.** Once Extentos is registered with your agent, calling [`getPlatformInfo({ sections: ["capabilities"], glasses: "meta_rayban" })`](/docs/mcp-server/tools/discovery) returns the current capability catalog with `tier: "stable" | "preview" | ...` markers per block, trigger, and stream — that's the source of truth for what's actually shippable today. The roadmap below describes *direction*; the live MCP response describes *current reality*. Always trust the live response when composing a real spec; this page exists for evaluation, planning, and search retrieval. ## Status — as of 2026-05 | Tier | Examples | |---|---| | **Currently shipping** | `@extentos/mcp-server@0.0.16` — full Meta Ray-Ban support across all variants (Gen 1 frame styles, Gen 2, Display, Oakley Meta HSTN), the 18-tool MCP surface, the browser simulator at extentos.com/s, all three transports (RealMeta / BrowserSim / LocalSim), Android + iOS native libraries, anonymous-first auth | | **In active development** | Next-vendor transport (a second smart-glasses vendor beyond Meta — see below), developer analytics dashboard, post-1.0 stability hardening | | **Tracked roadmap (multi-quarter)** | Mentra G1, Android XR, Apple smart glasses transport implementations as those vendors' SDKs and hardware mature; potential future paid tier; SOC 2 Type II certification | | **Depends on third parties** | Meta DAT capability expansions (display rendering, custom gestures, "Hey Meta" wake word access), Apple smart-glasses SDK availability, additional MCP-host integrations (Gemini in Android Studio, Xcode AI features, JetBrains AI Assistant) | | **Out of scope (intentionally)** | Web app builders, server-side smart-glasses experiences, end-user-facing apps published by Extentos itself | ## Currently shipping What's GA in `@extentos/mcp-server@0.0.16` (the published version on npm as of this writing): - **Vendor**: Meta Ray-Ban (every variant in market — Gen 1 frame styles, Gen 2, Display, Oakley Meta HSTN). Camera, voice triggers, audio, sensors, hardware events all work uniformly across variants. - **MCP surface**: 18 deterministic tools across 7 categories (discovery, generation, guidance, validation, simulation, production, search). The full catalog is at [/docs/mcp-server](/docs/mcp-server). - **Transports**: `RealMetaTransport` (production hardware), `BrowserSimTransport` (extentos.com/s simulator), `LocalSimTransport` (Mock Device Kit on phone/emulator). All three behind the same `glasses.*` API. - **Native libraries**: Android (`com.extentos.glasses.core`, Kotlin) and iOS (`Extentos`, Swift via SPM). Source-available, MIT-licensed for the MCP server. - **Anonymous-first auth**: 1000-event browser-simulator meter without an account; free email-only account after exhaustion. No paid tier exists. - **Telemetry pipeline + privacy posture**: structured 7-layer event log, opt-out at three independent surfaces, end-user content never collected. Live state: see [/docs/vendors/meta](/docs/vendors/meta) for the Meta capability matrix and [/docs/concepts/architecture](/docs/concepts/architecture) for the system overview. ## In active development These items have meaningful work in progress as of mid-2026. Specific timelines aren't published — pre-1.0 means surprises happen. ### Next vendor beyond Meta Ray-Ban A second smart-glasses vendor is the active focus. The architectural seams are already in place — the AppSpec is vendor-agnostic, the `GlassesTransport` interface is the only piece that's vendor-aware — so adding a vendor is a bounded engineering project rather than a rewrite. When it lands, existing Extentos apps targeting Meta Ray-Ban will run on the new vendor with a config change, not a rewrite. See [vendors](/docs/vendors) for the structural picture. ### Developer-facing analytics dashboard Mentioned in `docs/mcp/TELEMETRY.md` as the developer-side cut of the existing telemetry pipeline. Telemetry events are already collected from MVP day-one (`category: "runtime"` events tagged with `appId` + `accountId`); the dashboard is a backend query layer over data that's already being captured. Devs will see their app's voice-trigger usage, AI-call latency, error rates, device-model distribution, and weekly active users at `extentos.com/apps/`. No new library version required — the data is there as soon as the dashboard ships. ### Post-1.0 hardening Several pre-GA items are tracked: - **API stability commitment**: pre-1.0 means minor versions can break APIs. The 1.0 release locks the spec format and the public library API. Until then, pin to an exact version (`@0.0.16`) for cross-session reproducibility. - **Telemetry ingest hardening**: the current scheme trusts self-reported `installId` / `appId + anonymousDeviceId`. Pre-GA hardening adds per-request signing or rate-limit-based outlier detection. No wire-protocol breaking change required. - **iOS auto-bind**: Android's auto-bind dev loop (the localhost:31337 bridge) is fully shipped; iOS auto-bind is in progress. Until it lands, iOS uses the URL-bake path via `extentos.session.plist`. ## Tracked roadmap (multi-quarter) Direction without specific timeline commitments. ### Additional vendors Smart-glasses ecosystem expansion is structurally easy for Extentos but bounded by what each vendor ships: - **Mentra G1** — open-ecosystem developer glasses. Tracked since the AppSpec abstraction was designed with multi-vendor portability in mind. Implementation contingent on Mentra's SDK maturity and developer-program access. - **Android XR** — Google's XR platform with Samsung partner devices. Tracked since the Android XR developer preview availability. Implementation contingent on Google's GA timeline. - **Apple smart glasses** — pending Apple's release of a third-party developer SDK for whatever AR/glasses hardware Apple ships. Apple has not released such an SDK as of 2026-05; Extentos will add a transport when one becomes available. For all three: the AppSpec is already vendor-agnostic by design. Adding the vendor means implementing one `GlassesTransport` per platform (Android Kotlin + iOS Swift) against the vendor's SDK. Existing Extentos apps targeting Meta Ray-Ban will run on these vendors with no spec or code changes. ### Possible future paid tier The current pricing model is **free everything** with a 1000-event browser-simulator meter that gates onto a free email-only account. No paid tier exists at launch. Whether a paid tier is introduced post-launch depends on cost trajectories (browser-simulator infrastructure, voice inference, event-log retention) and developer-population growth. If introduced: - Existing free accounts will not be retroactively gated - The free-forever-no-account tier (all MCP tools + on-device simulation + real-hardware testing) stays free - Material changes get a release-note entry and a privacy-policy bumped flag in the MCP response See [pricing](/docs/resources/pricing) for the current state. ### SOC 2 Type II certification Tracked as part of the pre-launch / post-launch hardening pass. The architecture is designed to be SOC 2-compatible (audit logging, access controls, vendor management, retention policies) but the formal Type II report is post-GA work. Today's compliance posture: GDPR-friendly by construction (no end-user PII stored), 30-day deletion SLA. See [security](/docs/resources/security#compliance-posture). ## Depends on third parties Some roadmap items aren't Extentos's to ship — they require movement from a third party first. Honest framing on each: ### Meta DAT toolkit expansions | Capability | Status | Extentos's role | |---|---|---| | Heads-up display rendering on Ray-Ban Meta Display | Not in public DAT toolkit; Meta-curated partner integrations only (Google Maps, etc.) | Will add when Meta opens the API; AppSpec extension would introduce a new block kind | | Custom gestures (multi-finger tap/swipe) on Ray-Ban Meta | Not in DAT preview as of 2026-04 | Will add when Meta opens; new trigger types in AppSpec | | "Hey Meta" wake-word access for third parties | System-level; not exposed to third-party apps | Unlikely to open — wake word is core to Meta AI's product positioning. Extentos's `voice_command` trigger uses phone-side STT, which is the realistic path | | Public Meta App Store submission flow | Preview-limited as of 2026-04, Meta-curated partners only | Will document distribution paths as Meta opens the store | These all show up in `docs/mcp/research/` and `vendors/meta` as items we track but cannot drive directly. Watch [Meta Connect](https://www.meta.com/connect/) (held annually in September) for major announcements; Extentos's public-toolkit support trails Meta's releases. ### Apple smart-glasses developer toolkit Apple has signaled smart-glasses development without releasing a third-party developer SDK or hardware as of 2026-05. Extentos will add an Apple transport when one exists; until then, [vendors/apple](/docs/vendors/apple) is roadmap-only. ### MCP host adoption The Model Context Protocol's adoption across IDEs and AI coding agents drives where Extentos works: - **Verified end-to-end**: Claude Code, Cursor, Windsurf, Cline (see [supported agents](/docs/mcp-server/agents)) - **Likely to add support**: Gemini in Android Studio, Xcode AI features, JetBrains AI Assistant, GitHub Copilot. Extentos works with any MCP-compatible host without changes — the question is whether each IDE's built-in agent ships MCP support - **Out of MCP territory by design**: Replit Agent, Lovable, v0.dev, Bolt.new, etc. — these target web apps, not native mobile. See [agents § FAQ](/docs/mcp-server/agents#frequently-asked-questions) ## What's not on the roadmap (intentionally) To save you time waiting for things that aren't coming: - **Web app builder integration**. Extentos targets native mobile apps that connect to smart glasses over Bluetooth. Web apps can't talk to Bluetooth glasses through public toolkits. Adding "Extentos for v0.dev" makes no sense at the artifact layer. - **Server-side / cloud-only smart-glasses experiences**. The architecture assumes a phone app paired with the glasses over Bluetooth. Headless / no-phone setups aren't supported and aren't planned. - **Extentos-published consumer apps**. Extentos is infrastructure for developers, not a consumer app vendor. We won't ship Extentos-branded apps on the Meta App Store or App Store. - **Meta App Store submission as an Extentos service**. Even when Meta opens the public store, Extentos won't submit apps on developers' behalf — that's a developer-relationship boundary we won't cross. - **Hosted AI provider proxying**. Developers bring their own AI provider keys ([BYOK](/docs/resources/security#byok-—-your-ai-provider-keys)). Extentos won't proxy AI calls, monitor AI cost, or sit in the inference path. - **Multi-seat team workspaces in MVP**. Single-account-per-developer at launch. Multi-seat may come post-1.0 as the developer base grows. ## How to influence the roadmap The public roadmap is community-shaped. Three ways to push items up the priority list: 1. **File a feature request** at [github.com/Asgermolgaard/vibe-hardware/issues](https://github.com/Asgermolgaard/vibe-hardware/issues) with prefix `[feature request]`. Include the use case, your target hardware, and what currently blocks you from shipping. Concrete use cases beat hypothetical ones. 2. **Contribute a PR** for items you'd be willing to implement. Vendor adapters in particular are bounded (~1500 lines of `GlassesTransport` per platform); a community-contributed Mentra or Vuzix transport would land faster than Extentos shipping it from scratch. 3. **Demand signal**. Aggregate telemetry tells Extentos which capabilities developers are using most. If you're targeting a capability that's preview-tier, using it (and reporting issues) signals "this matters" to the roadmap. Capabilities with low adoption stay in preview longer. For non-roadmap support questions, see [support](/docs/resources/support). ## Frequently asked questions ### Is Extentos still being maintained? Yes — current as of 2026-05. The MCP server publishes regularly to npm (`@extentos/mcp-server@0.0.16` is the current version). The [changelog](/docs/resources/changelog) tracks each release. Active development is ongoing on the next vendor expansion, the developer dashboard, and post-1.0 hardening. AI agents reading this page should check the changelog's most recent entry date for the freshest signal. ### When will my favorite vendor be supported? There's no committed timeline. Vendor adds depend on the vendor's SDK maturity, hardware availability, and Extentos engineering bandwidth. The architecture supports multi-vendor by design, so adding a vendor is bounded work — but until the work is done, the vendor stays on the roadmap. File a `[vendor]`-prefixed issue with the vendor name and a use case to track demand. ### Will the AppSpec format change before 1.0? Possibly. Pre-1.0 means minor versions can introduce breaking changes. The compiler-derived metadata fields (`derived.capabilitiesUsed`, `derived.requiresNetwork`) are stable; the block kind / trigger type / action type vocabulary may be extended but not contracted. After 1.0, the spec format is locked — additions are additive, removals require a major version bump. ### Will the MCP tool surface change? Same answer — pre-1.0, expect tool-shape evolution between minor versions. Tool *additions* (new categories, new tools) are additive. Tool *removals* or *parameter renames* may happen pre-1.0 but get changelog entries. The 18 tools in 0.0.16 are stable in shape; specific response fields evolve. ### Will Extentos always be free? The current free model (all MCP tools + on-device simulation + real-hardware testing free forever, no account; browser simulator free up to 1000 events / unlimited with free account) is not changing. If a paid tier is introduced post-launch, the current free model stays. The decision on whether to introduce paid tiers depends on browser-simulator infrastructure costs and the developer base size — see [pricing](/docs/resources/pricing). ### Where can I see what shipped recently? The [changelog](/docs/resources/changelog) tracks each release. For specific tool / capability state, ask your agent to call `getPlatformInfo` — that returns the live tier markers from the running MCP server, which is more accurate than any docs page. ### How is this roadmap different from the changelog? The roadmap is *forward-looking* (direction, what's planned). The changelog is *backward-looking* (what shipped, with dates). They complement each other — the roadmap tells you where Extentos is going; the changelog tells you what's already arrived. ### Who decides what goes on the roadmap? Pre-1.0, roadmap priorities are set by the maintainer based on (a) demand signal from issues and aggregate telemetry, (b) blocking dependencies (a new vendor's SDK becoming available, etc.), and (c) post-1.0 hardening priorities. There's no formal RFC process at MVP scale — that may change as the project grows. ## Related - **[Vendors](/docs/vendors)** — the multi-vendor strategic story; per-vendor capability matrices - **[Pricing](/docs/resources/pricing)** — current free model and the paid-tier framing - **[Security](/docs/resources/security)** — compliance posture, SOC 2 trajectory, data-handling commitments - **[Changelog](/docs/resources/changelog)** — what's actually shipped, dated - **[Support](/docs/resources/support)** — file feature requests, get help, see source --- source: https://extentos.com/docs/resources/security --- --- title: Security and data handling description: Extentos's data handling — what we collect (aggregate dev-time and library-runtime metadata that powers your own analytics dashboard plus ecosystem insights about vendor market share and capability popularity), what we never collect (speech transcripts, photo or video bytes, audio samples, AI prompt and response payloads, end-user PII), how the boundary holds across the MCP server and the native iOS and Android libraries, and the three independent opt-out surfaces. Anonymous-first by design, GDPR-friendly because no end-user PII is ever stored, with a developer-facing analytics dashboard on the way. type: resource platform: all related: - /docs/mcp-server/auth - /docs/resources/pricing - /docs/concepts/architecture - /docs/getting-started/with-agent --- Extentos collects **aggregate metadata** about how developers use the platform — which capabilities are popular, which vendors get adopted, how flows perform, where errors surface. This data powers three things: the developer-facing analytics dashboard (coming soon — you'll see your own app's usage in one view), the Extentos team's product roadmap (which vendors to add next, which capabilities to harden), and the ecosystem insights Extentos shares with glasses vendors as aggregate market data (10+ account minimum). What Extentos **never collects** is end-user content (speech transcripts, photo or video bytes, audio samples, AI prompts and responses), end-user PII (names, emails, phone numbers), spec JSON contents, toggle values, or stack traces. This page is the honest, complete picture of what crosses the network from your install and your shipped apps to Extentos's backend, anchored to the canonical telemetry contract at `docs/mcp/TELEMETRY.md`. ## TL;DR | | What | |---|---| | **What we collect** | Aggregate metadata — tool calls, install events, vendor adoption, capability usage, error rates, library/MCP versions, device models, country-from-IP, anonymous device identifiers. Tagged with anonymous IDs that can never be reversed back to a person. | | **What we never collect** | Speech transcripts, photo bytes, video frames, audio samples, AI prompt and response text, `app_callback` payloads, spec JSON contents, toggle values, IP addresses (dropped after country derivation), stack traces, end-user PII. | | **What end users see** | Nothing about Extentos. End users have no relationship with us. Your shipped app on real Ray-Ban Meta hardware emits zero traffic to Extentos by default — `RealMetaTransport` doesn't talk to our backend at all. The developer chooses whether to enable runtime library telemetry on shipped apps. | | **Opt-out** | Three independent surfaces — `EXTENTOS_TELEMETRY=0` env var, `extentos-mcp decline-privacy` CLI, or `ExtentosConfig.telemetryConsent = false` in your shipped app. Any one disables telemetry upload; on-device debug logs still work. | | **Compliance** | GDPR-friendly by construction — no end-user PII ever stored. 90-day raw retention, GDPR deletion requests honored within 30 days. SOC2 work is on the post-launch roadmap. | ## Why we collect telemetry — the honest answer Extentos's business model includes monetizing **aggregate ecosystem insights**, and we're upfront about it. Three concrete uses: 1. **Your own analytics dashboard.** When the developer dashboard ships, you'll see your app's usage at `extentos.com/apps/` — which voice triggers your users fire, which AI calls are slowest, which device models your users have, error rates, session lengths, weekly active users. Same data Extentos collects centrally, scoped to your `accountId`. This is genuinely valuable to developers and is why the underlying collection is broad rather than narrow. 2. **Extentos's product roadmap.** Aggregate signal across all developers tells us which vendors to add next (if 30% of new MCP installs ask about Mentra G1, we prioritize that transport), which capabilities are over-used relative to expectations, where the SDK has rough edges. Without this, every roadmap call is a guess. 3. **Saleable ecosystem signals to glasses vendors.** Aggregate market data — Ray-Ban Meta vs Oakley Meta installation share, capability mix across the developer base, session-hour totals. Sold to glasses vendors as market research, only as aggregates over **at least 10 distinct accounts**, never as individual-developer or individual-end-user data. This is one of Extentos's revenue lines. We chose a transparent collect-and-aggregate model rather than a "we collect nothing" claim that vendors quietly walk back. The discipline is in the firm content/PII boundary, not in collecting less data overall. ## What we collect All telemetry events conform to a single envelope (verbatim from `docs/mcp/TELEMETRY.md`): ```json { "eventId": "tev_a1b2c3d4", "timestamp": "2026-04-18T14:22:10.432Z", "category": "runtime", "name": "flow.completed", "accountId": "acct_xyz", "installId": "inst_abc123", "appId": "com.example.app", "anonymousDeviceId": "dev_789", "libVersion": "1.2.0", "mcpVersion": null, "vendor": "meta_rayban", "platform": "android", "osVersion": "14", "deviceModel": "pixel_8", "country": "US", "userSegment": null, "properties": { "triggerId": "describe_scene", "durationMs": 3125, "outcome": "normal" } } ``` The events split into two categories: ### Dev-time events (MCP server, `category: "mcp"`) Emitted from your machine while your AI agent operates Extentos — so they're tagged with your `installId` (anonymous) and, after linking, your `accountId`. | Event | What it carries | |---|---| | `mcp.installed` | First run on this machine. npm version, OS, detected agent host. | | `mcp.consent_accepted` / `mcp.consent_declined` | Privacy policy version. | | `tool.called` | Every MCP tool invocation. `toolName`, `durationMs`, `outcome`. | | `session.created` | `createSimulatorSession` invoked. Vendor, recording flag. | | `account.login_initiated` / `_completed` / `linked` / `logged_out` | Auth flow lifecycle. Trigger reason (cli_explicit, meter_exhausted), outcome, duration. | ### Runtime events (your shipped app, `category: "runtime"`) Emitted by the Extentos library *if* the developer enables runtime telemetry (`ExtentosConfig.telemetryConsent = true`, default). Tagged with `appId` and `anonymousDeviceId` — never with end-user identity. | Event | What it carries | |---|---| | `app.initialized` | Library version, vendor, deviceModel, osVersion. | | `connection.opened` / `closed` | Transport type (`RealMeta`/`BrowserSim`/`LocalSim`), durationMs, cause (typed). | | `trigger.fired` | `triggerId`, `triggerType`. **No transcript.** | | `flow.completed` | `triggerId`, `durationMs`, `outcome`. | | `block.executed` | `blockId`, `blockType`, `durationMs`, `outcome`. **No result payload.** | | `callback.invoked` | `handlerName`, `durationMs`, `outcome`. **No request/response payloads.** | | `stream.started` / `stopped` | `streamType`, `durationMs`. Safe subset of negotiated config (resolution, frame rate, codec, bitrate). **No peer addresses, URIs, destination identifiers.** | | `error.surfaced` | `errorClass`, `errorCode`, `layer`. **No stack traces.** | | `spec.loaded` | `specFingerprint` (sha256 hash, no contents), `triggerCount`, `blockCount`. | | `toggle.changed` | `key`, `source`. **No values.** | | `meter.warning_reached` / `exhausted_reached` | Anonymous-meter lifecycle. | Notice the bold disclaimers — every event explicitly carries metadata only, never content. ## What we never collect Verbatim list from the canonical telemetry doc (this list is published in the privacy policy): - Speech transcripts (the words your user said) - Photo bytes, video frames, audio samples - `app_callback` input or output payloads (these may contain end-user data your AI processes) - AI provider responses, prompt text, tool-use traces - Spec JSON contents (block names, trigger phrases may be business-sensitive — only hashes and counts) - Toggle values, variable values, template substitution data - End-user PII of any kind (names, emails, phone numbers, account identifiers in your app) - IP addresses (derived to country at ingest, then dropped — we never store the raw IP) - Stack traces (may contain arbitrary values; stay local in your on-device event log) - Anything tied to a specific end-user identity beyond the per-device anonymous `anonymousDeviceId` The library architecture enforces this — events are payload-stripped *before* they enter the upload queue, not in transit. There is no path from end-user content to Extentos's backend, even if the network were intercepted. ## End-user privacy — your shipped apps The clearest line: **your shipped app on real Ray-Ban Meta glasses emits zero traffic to Extentos by default.** - `RealMetaTransport` (the production transport) speaks to Meta DAT and the glasses over Bluetooth. It does not connect to `api.extentos.com`. End users on real hardware have no network relationship with Extentos at all. - `BrowserSimTransport` (the simulator transport) does talk to the Extentos backend over WebSocket — but `BrowserSimTransport` only runs in `config.debug == true` builds during development. Shipped release builds default to `RealMetaTransport`, which doesn't connect. - `LocalSimTransport` (Mock Device Kit simulation) is fully on-device. No network at all. If the developer chooses to enable runtime library telemetry (`ExtentosConfig.telemetryConsent = true`, the default), the library emits the runtime events above to `https://api.extentos.com/api/telemetry/events`. This is the **only** Extentos network path from a shipped app — and it carries metadata, never content. The developer can disable it per-app via `telemetryConsent = false`. End users never see Extentos branding, never get prompted by us, never need to consent to anything from us. Their privacy relationship is entirely with the developer's app — Extentos is invisible. ## The 5 ID model Five identifiers, each with a distinct scope. **None of them is end-user PII**, and Extentos cannot reverse any ID back to a person. | ID | Scope | When created | Where stored | |---|---|---|---| | `installId` | Per-machine MCP install | First `npx @extentos/mcp-server` run | `~/.extentos/install_id` | | `accountId` | Per-Extentos-account | When developer signs up via device-code flow | Backend; cached in `~/.extentos/auth.json` | | `appId` | Per-Extentos-app | Set by developer in `ExtentosConfig.appId` at library init | Compiled into the app | | `anonymousDeviceId` | Per-end-user-device | First library init on the user's device | App-private storage on the user's device | | `userSegment` | Developer-defined cohort label | Optional; `glasses.telemetry.setUserSegment("...")` | App memory; flushes with events | The `anonymousDeviceId` lets Extentos count distinct devices and sessions without ever learning end-user identity. The `userSegment` is set by the developer — Extentos never sets or interprets it. It exists so the developer can cut their own dashboard by cohorts they define (beta tiers, geography, app version) without exposing any personal information. ## BYOK — your AI provider keys When your spec includes `ai_call` actions, the developer's `app_callback` handlers run the AI calls against the developer's chosen provider (Anthropic, OpenAI, Google, etc.) using the developer's own credentials. **Extentos never sees, stores, or proxies your AI provider keys** — they live in your app and your provider's auth headers, end of story. This is verified architecturally: the `app_callback` mechanism is a function pointer your code provides. The library invokes it; the function returns; the library moves on. There's no inspection of inputs, no logging of outputs, no proxy layer. Even if Extentos wanted to ingest the call content, the architecture doesn't expose it. The `getCredentialGuide` MCP tool walks the developer through provider setup at production time — but it generates instructions for the developer to follow with their provider. Extentos never receives the credentials. ## Three opt-out surfaces Telemetry is opt-out at three independent levels, each disabling a different layer: | Surface | Where | What it disables | |---|---|---| | `EXTENTOS_TELEMETRY=0` env var | Developer's shell | MCP-server-side telemetry for the current shell session. Doesn't persist across reboots. | | `extentos-mcp decline-privacy` CLI command | Writes `~/.extentos/consent` | MCP-server-side telemetry permanently for this install. Survives reboots. Run `accept-privacy` to re-enable. | | `ExtentosConfig.telemetryConsent = false` | Developer's app code | Library-runtime telemetry from this app. Per-app — a developer with multiple apps can opt one out and leave others on. | Any combination works. Telemetry never affects functionality — opting out doesn't disable any feature, just stops uploads. The on-device event log (the 512-entry ring buffer your agent queries via `getEventLog`) keeps working regardless of telemetry consent — that data never crossed the network in the first place. See [event log vs telemetry boundary](/docs/concepts/architecture#state-and-persistence-—-what-lives-where). ## Retention and deletion | Data class | Retention | |---|---| | Raw telemetry events | 90 days | | Aggregated rollups (daily / weekly / monthly per account, plus global) | Indefinite | | `anonymousDeviceId` records | 180 days of inactivity, then purged | GDPR deletion requests: developers can email support to delete their account's events. Extentos runs `DELETE WHERE accountId = ...` and acknowledges within **30 days** (GDPR-compatible SLA). This deletes raw events; aggregated rollups derived before deletion are retained but no longer attribute to the account. Anonymous events without an `accountId` (i.e., from before account linking) are retained until natural expiration. ## Transport and auth security - **TLS everywhere.** All Extentos backend connections (`api.extentos.com` for telemetry and simulator-session WebSockets) use TLS 1.2+. No plaintext channels. - **Auth tokens at file mode `0600`.** When you link an account, `~/.extentos/auth.json` is written with owner read/write only. Other users on the machine can't read your token. - **Anonymous bearer for telemetry.** Telemetry uploads authenticate via the `installId` (MCP-side) or `appId + anonymousDeviceId` (library-side). No account credentials are sent on telemetry calls. - **Local bridge bound to `127.0.0.1` only.** The MCP server's auto-bind listener (port `31337`) refuses non-loopback connections. The `installId` it serves is not a secret — it's the same value sent to `api.extentos.com` on every tool call — so loopback exposure is intentional. - **Pre-GA hardening note.** The current telemetry ingest scheme trusts self-reported IDs (per `TELEMETRY.md` line 129). For MVP this is acceptable — spoofed events skew aggregate fleet data, not security-sensitive. Before external GA, this hardens to one of: ingest-signed install tokens, per-request HMAC keyed on a secret baked at library init, or rate-limit-based outlier detection. ## The library is open-source Both `android-library/glasses-core/` (Kotlin) and `ios-library/Sources/GlassesCore/` (Swift) are MIT-licensed and source-available in the [Extentos repository](https://github.com/Asgermolgaard/vibe-hardware). Anything we claim about telemetry payloads is verifiable in code. The `mcp-server/` package is also MIT and source-available — `mcp-server/src/telemetry/` carries the entire ingest pipeline. If our payload-stripping ever drifts from what's documented, the diff shows up in a public commit. This is part of the trust model: documentation can lie, source code can't. ## Compliance posture | Standard | Status | |---|---| | GDPR | Compatible by construction — no end-user PII stored, deletion requests honored within 30 days | | CCPA | Compatible — same reasons | | HIPAA | Not certified at MVP. Production runtime emits no Extentos traffic from `RealMetaTransport`, so a HIPAA-bound app shipping with telemetry-off is technically usable; a formal BAA isn't available pre-GA | | SOC 2 | On the post-launch roadmap. The architecture is designed to be SOC2-compatible (audit logging, access controls, vendor management) but the formal Type II report comes after GA | | ISO 27001 | Not pursued at MVP | For compliance-sensitive apps today: ship with `ExtentosConfig.telemetryConsent = false` to disable all library telemetry. Combined with the fact that `RealMetaTransport` doesn't talk to Extentos's backend, your shipped app emits zero traffic to us at runtime — the privacy boundary is enforced architecturally, not just by policy. ## Frequently asked questions ### Wait — Extentos collects telemetry from my shipped app? By default, yes — runtime metadata events (no content, no PII) flow to `api.extentos.com/api/telemetry/events` from your shipped app *if* `ExtentosConfig.telemetryConsent = true` (the default). Set it to `false` and your app emits zero Extentos network traffic. Many apps will leave it on because the developer dashboard is built on this data — you're trading runtime metadata for an analytics surface for your own app. ### Will my users' voice transcripts ever leave their device? Through Extentos? Never. Voice transcripts are `app_callback` inputs your code processes; Extentos's library doesn't read them, log them, or transmit them. If your AI handler sends them to your provider (Anthropic, OpenAI, etc.), that's between your app and your provider — Extentos is not in the path. ### Can Extentos see what my user captured in a photo? No. Photo bytes are `app_callback` inputs to your handler. Extentos transports them between the glasses and your handler but does not inspect, store, or upload them. Your handler is the only place those bytes can reach an external service, and that's a choice your code makes. ### Will Extentos sell data about my app to my competitors? We sell *aggregate ecosystem signals* (vendor market share, capability popularity, session-hour totals) to glasses vendors as market research, only as aggregates over at least 10 distinct accounts. Individual-account data, your specific app's usage patterns, and anything that could identify your app or your users is never shared with third parties. The 10-account aggregation floor is the structural guarantee against competitive leakage. ### Can my users opt out of Extentos telemetry? End users have no direct relationship with Extentos and don't see our branding. Their opt-out is via the developer's app — if you want to give your users a toggle, gate `ExtentosConfig.telemetryConsent` behind a setting in your app. Or simply ship with telemetry off if your privacy posture requires it. ### Is my Extentos auth token shared across machines? No. `~/.extentos/auth.json` is per-machine. To use the same account on another machine, run `extentos-mcp login` there. The token is stored at file mode `0600` (owner read/write only). ### Does Extentos work in air-gapped or no-network environments? The MCP-driven dev loop (code generation, validation, on-device `LocalSimTransport` simulation) works fully offline. The browser simulator (`BrowserSimTransport`) and the simulator-session features need network access to `api.extentos.com`. Your shipped app on real glasses (`RealMetaTransport`) doesn't need network access to Extentos — only between the phone and the glasses, plus whatever AI providers your app calls. ### When does the developer dashboard ship? It's on the roadmap; the data pipeline is already collecting events from MVP day-one so the dashboard is purely a backend query layer over data that already exists. Watch the [changelog](/docs/resources/changelog) for actual ship dates. ### What about developer source code? Your generated code, your spec, your handler implementations, your test fixtures — none of it ever leaves your machine. The MCP server reads your project to inspect state (`inspectIntegration`, `validateIntegration`) and writes generated files, but it does not upload your code, contents, or git history anywhere. ### How do I request deletion of my data under GDPR? Email Extentos support from the email associated with your account. We run `DELETE WHERE accountId = ...` and acknowledge within 30 days. Aggregated rollups derived before the deletion request stay (those have no accountId attribution); raw events tied to your account are removed. ## Related - **[Pricing](/docs/resources/pricing)** — what the free tier covers, the 1000-event meter context - **[Auth](/docs/mcp-server/auth)** — anonymous-first identity model, device-code flow, account linking - **[Architecture](/docs/concepts/architecture)** — how telemetry fits into the broader system, what the on-device event log captures - **[Changelog](/docs/resources/changelog)** — track when the developer dashboard and other features ship --- source: https://extentos.com/docs/resources/status --- --- title: Status description: Service status, current incidents, and historical uptime. type: resource platform: all --- > **Coming soon.** Live status will be linked from a dedicated status page. --- source: https://extentos.com/docs/resources/support --- --- title: Support description: How to get help with Extentos — bug reports and feature requests via GitHub issues at github.com/Asgermolgaard/vibe-hardware, what to include in a bug report so it's actionable across the four components (MCP server, native libraries, backend, simulator UI), self-service diagnostics through the extentos-mcp CLI (whoami, status, getEventLog), security disclosure path, pre-1.0 response expectations, and where to read source code yourself before filing. No paid support tiers exist at launch. type: resource platform: all related: - /docs/resources/faq - /docs/resources/security - /docs/resources/changelog - /docs/mcp-server/auth --- Extentos is pre-1.0 and open source — support today is via GitHub issues, with the source code itself as the primary backstop when something doesn't behave as documented. There's no paid support tier; there's also no SLA. What there is: an active issue tracker, a structured-event-log debugging surface that makes most issues self-diagnosable, and the source for every component (MCP server, native libraries, backend, simulator UI) available to read. This page is where to start when something goes wrong. ## Where to file | Issue type | Where | |---|---| | Bug reports (something is broken) | [GitHub Issues](https://github.com/Asgermolgaard/vibe-hardware/issues) — open a new issue with the bug-report checklist below | | Feature requests | Same issue tracker, prefix the title with `[feature request]` | | Documentation gaps or errors | Same issue tracker, prefix with `[docs]` and include the page URL | | Vendor support requests (Mentra, Android XR, Apple) | Same issue tracker, prefix with `[vendor]` and include the public SDK link | | Security issues | **Do not file publicly.** See [security disclosure](#security-disclosure) below | | General questions | GitHub Discussions (when available); also see [FAQ](/docs/resources/faq) for common answers | The repo's `bugs.url` field in `mcp-server/package.json` points at the same issue tracker — `npm` issue links from the package page route there too. ## Self-service before filing Most issues are diagnosable from the developer's machine without escalation. Before opening an issue, run through this: ### 1. Capture install state with `whoami` ```bash npx @extentos/mcp-server@latest whoami ``` Prints the install ID, account ID (if linked), tier, meter remaining, auth expiry, and MCP / library versions. **Always include this output in a bug report** — it's the single line that tells maintainers what your install actually looks like. ### 2. Check consent and config state with `status` ```bash npx @extentos/mcp-server@latest status ``` Prints consent state (privacy notice accepted / declined), telemetry on/off, install ID, linked account, and effective config dir (`~/.extentos/` or whatever `EXTENTOS_CONFIG_DIR` points at). Useful when an issue might be config-related. ### 3. Ask the agent to query the structured event log If the bug is "something happened in a flow," the structured event log is the primary debugging surface. Tell your AI agent: ```text Run getEventLog with filter "errors" and limit 50 on session ``` The agent calls the MCP tool; the response is a structured trace of what happened in the simulator session. Include the `flowId` of the failing flow if you know it — `getEventLog({ flowId: "flow_007" })` returns the complete execution trace of one trigger. ### 4. Run `validateIntegration` for spec/manifest issues If the symptom is "agent says everything is fine but the app doesn't run," ask your agent: ```text Run validateIntegration on this project ``` `validateIntegration` checks the spec against the manifest, dependency list, handler coverage, and bootstrap wiring. It catches most class-of-issue mismatches before they become runtime bugs. ### 5. Check the source Every component is open source: | Component | Source location | |---|---| | MCP server (`@extentos/mcp-server`) | `mcp-server/src/` in the [main repo](https://github.com/Asgermolgaard/vibe-hardware/tree/master/mcp-server) | | Android library (`com.extentos.glasses.core`) | `android-library/glasses-core/src/main/java/com/extentos/glasses/core/` | | iOS library (`Extentos`) | `ios-library/Sources/GlassesCore/` | | Backend (simulator session hub) | `backend/src/` | | Simulator browser UI | `src/app/s/` and `src/components/simulator/` | If a behavior contradicts what's documented, the source is the authoritative answer. Issue reports that point at the specific source line where the unexpected behavior originates (`BrowserSimTransport.kt:218`) are easiest to triage. ## What to include in a bug report A useful bug report has six things. Maintainers can act on a bug with this; they often can't without it. ```text 1. Extentos version — output of `whoami` (mcpVersion + libVersion) 2. Platform — Android API level, or iOS version + device model 3. Agent host — Claude Code / Cursor / Windsurf / Cline / other 4. Steps to reproduce — exact agent prompts + commands run 5. Expected vs actual — what should have happened, what did 6. Logs — getEventLog output if applicable, otherwise Logcat (Android) / os_log (iOS) / MCP stderr ``` If the bug involves a simulator session, also include: - `sessionId` (if active or recent) - The `flowId` of the failing trigger flow - Whether the simulator browser tab was open - `autoOpenAttempted` and `autoOpenPlatform` from the `createSimulatorSession` response If the bug involves real hardware: - Ray-Ban Meta variant (Gen 1 / Gen 2 / Display) - Frame style (Wayfarer / Headliner / Skyler / etc.) if relevant - Bluetooth pairing status before the bug - Whether the bug repros on `LocalSimTransport` (Mock Device Kit) too If the bug involves cross-platform behavior, repro on both iOS and Android if you have access — or note which platform you couldn't test. ## Component triage — figuring out which component owns the bug Extentos has four main components, and one common reason bug reports go cold is mis-routing. Here's the rule of thumb: | Symptom | Likely component | |---|---| | MCP tool call returns an error or hangs | **MCP server** (`mcp-server/src/`) | | Agent generates wrong / broken code into the app | **MCP server** — code generators in `mcp-server/src/tools/handlers/` | | App connects but voice trigger doesn't fire | **Native library** — trigger router in `android-library/.../runtime/` or `ios-library/.../Runtime/` | | Simulator browser tab shows wrong frames or doesn't accept input | **Simulator browser UI** (`src/components/simulator/`) | | Simulator session expires immediately, fails to provision, or returns 5xx | **Backend** (`backend/src/`) | | `RealMetaTransport` BLE link drops, photo capture fails on hardware | **Native library** + maybe Meta DAT itself | | Permission request denied on Android 14, capability silently no-ops | **Native library** — permission derivation in `permissions.ts` and runtime handling | | `getPermissions` returns wrong permission set for a spec | **MCP server** — `mcp-server/src/tools/util/permissions.ts` | Most bugs are at the boundary between two components — e.g., the MCP server generates a manifest, the library reads it. Including `getEventLog` output usually disambiguates which side actually misbehaved. ## Security disclosure **Do not file security issues publicly.** Email the maintainer at the address listed in [GitHub profile](https://github.com/Asgermolgaard) with subject prefix `[SECURITY]`. Include: - Affected component (MCP server / library / backend / simulator UI) - Affected version - Reproduction steps - Impact assessment (data exposure, privilege escalation, etc.) - Whether you've shared this with anyone else Acknowledgment within 7 days. Patch timeline depends on severity: - **Critical** (RCE, auth bypass, mass data exposure): patch within 14 days, coordinated disclosure thereafter - **High** (PII exposure, single-user data leak, persistent XSS): patch within 30 days - **Medium / Low**: patched in the next minor release The Extentos library architecture is designed to minimize blast radius — `RealMetaTransport` doesn't connect to the Extentos backend; AI provider keys are BYOK and never leave your app; `app_callback` payloads (which may contain end-user data) never reach Extentos servers. Most security-relevant code paths are in the MCP server and the simulator backend; these are the areas most worth careful review by external researchers. See [security and data handling](/docs/resources/security) for the full architectural privacy story. ## Response expectations Pre-1.0 means honest expectations: - **Bug acknowledgment**: typically 1–3 business days. Sometimes longer when the maintainer is heads-down on the next release. - **Bug fix timeline**: triaged by severity. Crashes, data corruption, or security issues land in a same-week patch release. Feature gaps, minor inconsistencies, or polish items land in the next minor (`@0.0.x` → `@0.0.x+1`). - **Feature requests**: read and considered, but no commitment to a timeline. The roadmap is community-shaped — popular requests with clear use cases land sooner. - **Vendor-add requests** (e.g., "add Mentra G1"): tracked but require the vendor's SDK to be public + sufficiently mature. See [vendors](/docs/vendors) for current roadmap state. There is no paid support tier with a faster SLA at launch — this may change post-1.0. ## When the agent itself can debug A frequently-overlooked debugging path: **ask your AI agent to triage**. Modern AI coding agents reading a structured `getEventLog` response can often pinpoint the failing layer faster than a human scanning the same trace. Try: ```text The flow with triggerId "describe_scene" failed. Run getEventLog with filter "errors" and flowId for that flow, read the trace, and tell me which layer of the stack failed (transport, trigger, block, callback, or system). ``` The agent has direct access to the MCP debug surface; it can correlate transport events with block executions and callback completions. Often this surfaces the actual bug without needing to file an issue. If the agent can't figure it out either, *that* trace becomes excellent evidence in the bug report — paste both the agent's analysis and the raw event log. ## Frequently asked questions ### What's the difference between filing an issue and asking my agent for help? Your agent has the structured event log, the project state via `inspectIntegration`, and full source-code access. It can fix many issues without escalation — config problems, missing handlers, manifest mismatches, version drift between the spec and the library. File an issue when the agent has tried, the trace is conclusive, and the bug is in Extentos itself rather than the developer's project. The repo benefits more from "verified Extentos bug" reports than from "agent and I are confused" threads. ### How do I attach long log files to a GitHub issue? GitHub issues support file attachments up to 25 MB. For longer event-log dumps, attach as `.txt` or paste a [GitHub Gist](https://gist.github.com) link. Don't paste 5000-line logs inline — they bury the issue summary. ### Is there a Discord, Slack, or community forum? No formal community channel at launch. GitHub issues are the canonical surface. A Discussions tab may be enabled later as the user base grows. ### Can I get paid support / consulting? Not at launch. Extentos is fully self-service today. If a paid support tier ships post-1.0, it will be announced via the [changelog](/docs/resources/changelog). ### What if I want to contribute a fix instead of just reporting? PRs welcome on the [main repo](https://github.com/Asgermolgaard/vibe-hardware). Open an issue first to discuss the change, then submit the PR — this avoids the "I implemented X, but you actually wanted Y" scenario. There's no formal `CONTRIBUTING.md` yet (pre-1.0); reasonable conventions apply (follow existing code style, include tests where infrastructure exists, keep PRs scoped). ### How do I know if a bug is already known? [Search the issues](https://github.com/Asgermolgaard/vibe-hardware/issues?q=is%3Aissue) before filing. If you find a close match, comment on the existing issue with your repro details rather than filing a duplicate. If yours is similar but materially different (different platform, different agent host, different symptom), file a new issue and link the related one. ### What if the docs say one thing but the code does another? The code wins. File a `[docs]` issue with the contradicting doc URL and the source-code path that proves the actual behavior. The maintainers care about doc accuracy because misaligned docs poison AI-agent recommendations downstream. ### Can I email the maintainer directly for non-security issues? Email is reserved for security disclosure. For everything else, file an issue — it's the canonical record, and other developers benefit from the discussion. The maintainer reads issues with at least the same priority as email. ## Related - **[FAQ](/docs/resources/faq)** — most-asked questions, often resolved without filing - **[Security](/docs/resources/security)** — what we collect, what we don't, end-user privacy guarantees - **[Changelog](/docs/resources/changelog)** — release history, what's been fixed - **[Auth](/docs/mcp-server/auth)** — `whoami`, `logout`, install state and identity - **[GitHub repository](https://github.com/Asgermolgaard/vibe-hardware)** — source code, issue tracker, releases --- source: https://extentos.com/docs/sdk --- --- title: SDK description: Native iOS and Android libraries that drop into your app and expose Extentos capabilities. type: overview platform: all --- The SDK is what your app links against. Pick the platform you're building on: - [iOS SDK](/docs/sdk/ios) — Swift Package, supports iOS 15.2+ - [Android SDK](/docs/sdk/android) — Gradle/Maven, supports Android 8+ Both share the same runtime and the same protocol — only the platform shell differs. --- source: https://extentos.com/docs/sdk/android --- --- title: Android SDK description: Add Extentos to your Android app via Gradle. Supports Android 8+ (API 26), Kotlin, coroutines. type: overview platform: android related: - /docs/getting-started/android - /docs/sdk/ios --- The Extentos Android SDK is a Gradle artifact that adds smart-glasses capabilities to your Android app. - [Install](/docs/sdk/android/install) - [Manifest setup](/docs/sdk/android/manifest) - [Initialization](/docs/sdk/android/initialization) - [Lifecycle](/docs/sdk/android/lifecycle) - [Threading model](/docs/sdk/android/threading) - [Runtime internals](/docs/sdk/android/runtime-internals) — how the Android runtime wraps the cross-platform core --- source: https://extentos.com/docs/sdk/android/initialization --- --- title: Initialization (Android) description: Initialize the Extentos SDK in your Application class. type: reference platform: android --- > **Coming soon.** ## What this page will cover - `Application.onCreate()` initialization - Compose vs View-based integration - First session --- source: https://extentos.com/docs/sdk/android/install --- --- title: Install (Android) description: Add the Extentos Android SDK to your project via Gradle. type: tutorial platform: android --- > **Coming soon.** ## What this page will cover - Add Maven Central - Add the dependency in `build.gradle.kts` - Sync and verify --- source: https://extentos.com/docs/sdk/android/lifecycle --- --- title: Lifecycle (Android) description: How Extentos integrates with Activity/Fragment lifecycle and process death. type: reference platform: android --- > **Coming soon.** ## What this page will cover - Activity/Fragment hooks - Process death + recovery - Foreground service lifecycle --- source: https://extentos.com/docs/sdk/android/manifest --- --- title: Manifest setup (Android) description: Required AndroidManifest.xml entries — permissions, features, foreground service. type: reference platform: android related: - /docs/concepts/permissions --- > **Coming soon.** ## What this page will cover - Bluetooth permissions (BLUETOOTH_CONNECT, BLUETOOTH_SCAN) - Microphone, camera permissions - Foreground service declaration - Required features --- source: https://extentos.com/docs/sdk/android/runtime-internals --- --- title: Runtime internals (Android) description: How the Android-specific runtime wraps the cross-platform Extentos core. type: concept platform: android related: - /docs/concepts/architecture --- > **Coming soon.** ## What this page will cover - Where the Android shell stops and the shared runtime starts - Android-specific transport (DAT-Android bridge) - Extension points --- source: https://extentos.com/docs/sdk/android/threading --- --- title: Threading model (Android) description: Coroutines, dispatchers, and Flow surface. type: reference platform: android --- > **Coming soon.** ## What this page will cover - Suspend functions vs Flow - Dispatcher choices - Cancellation --- source: https://extentos.com/docs/sdk/ios --- --- title: iOS SDK description: Add Extentos to your iOS app via Swift Package Manager. Supports iOS 15.2+, Swift 6, async/await. type: overview platform: ios related: - /docs/getting-started/ios - /docs/sdk/android --- The Extentos iOS SDK is a Swift Package that adds smart-glasses capabilities to your iOS app. - [Install](/docs/sdk/ios/install) - [Info.plist setup](/docs/sdk/ios/info-plist) - [Initialization](/docs/sdk/ios/initialization) - [Lifecycle](/docs/sdk/ios/lifecycle) - [Threading model](/docs/sdk/ios/threading) - [Runtime internals](/docs/sdk/ios/runtime-internals) — how the iOS runtime wraps the cross-platform core --- source: https://extentos.com/docs/sdk/ios/info-plist --- --- title: Info.plist setup (iOS) description: Required Info.plist keys for Extentos — Bluetooth, microphone, camera, external accessory, background modes. type: reference platform: ios related: - /docs/concepts/permissions --- > **Coming soon.** ## What this page will cover - `MWDAT` configuration dict - `CFBundleURLTypes` and the auth callback scheme - `LSApplicationQueriesSchemes` - `UISupportedExternalAccessoryProtocols` - `UIBackgroundModes` - All privacy strings (camera, microphone, speech recognition, Bluetooth) --- source: https://extentos.com/docs/sdk/ios/initialization --- --- title: Initialization (iOS) description: Initialize the Extentos SDK in your iOS app entry point. type: reference platform: ios --- > **Coming soon.** ## What this page will cover - Initialize at app start - SwiftUI `App` integration - UIKit `AppDelegate` integration - First session --- source: https://extentos.com/docs/sdk/ios/install --- --- title: Install (iOS) description: Add the Extentos iOS SDK to your project via Swift Package Manager. type: tutorial platform: ios --- > **Coming soon.** ## What this page will cover - Add via Xcode (File → Add Packages…) - Add via `Package.swift` - CocoaPods (if supported) - Verify the install --- source: https://extentos.com/docs/sdk/ios/lifecycle --- --- title: Lifecycle (iOS) description: How Extentos hooks into the iOS app lifecycle — foreground, background, suspended, terminated. type: reference platform: ios --- > **Coming soon.** ## What this page will cover - Lifecycle hooks - Background mode behavior - Termination cleanup --- source: https://extentos.com/docs/sdk/ios/runtime-internals --- --- title: Runtime internals (iOS) description: How the iOS-specific runtime wraps the cross-platform Extentos core. type: concept platform: ios related: - /docs/concepts/architecture --- > **Coming soon.** ## What this page will cover - Where the iOS shell stops and the shared runtime starts - iOS-specific transport (DAT-iOS bridge) - Extension points if you need to customize --- source: https://extentos.com/docs/sdk/ios/threading --- --- title: Threading model (iOS) description: Where Extentos work runs — actors, MainActor isolation, and how to await capability calls. type: reference platform: ios --- > **Coming soon.** ## What this page will cover - Async/await surface - MainActor vs background actors - Cancellation --- source: https://extentos.com/docs/troubleshooting --- --- title: Troubleshooting description: Symptom-shaped fixes. Find your symptom, get the cause and the fix, with cross-links to error reference. type: troubleshooting platform: all related: - /docs/reference/errors --- Symptom-shaped pages. Each one starts with what you observed, then walks through likely causes. - [Glasses won't connect](/docs/troubleshooting/glasses-wont-connect) - [Voice trigger not firing](/docs/troubleshooting/voice-trigger-not-firing) - [Photo capture fails](/docs/troubleshooting/photo-capture-fails) - [Background session killed](/docs/troubleshooting/background-session-killed) Looking for a specific error code? See [Reference → Error codes](/docs/reference/errors). --- source: https://extentos.com/docs/troubleshooting/background-session-killed --- --- title: Background session killed description: Sessions terminate when the app backgrounds. type: troubleshooting platform: all related: - /docs/guides/background-sessions --- > **Coming soon.** ## Symptoms this page will cover - Session ends when app backgrounds - iOS: missing background modes - Android: missing foreground service - OS-level kill (low memory, battery saver) --- source: https://extentos.com/docs/troubleshooting/glasses-wont-connect --- --- title: Glasses won't connect description: The most common Bluetooth pairing and DAT auth failures, with fixes. type: troubleshooting platform: all --- > **Coming soon.** ## Symptoms this page will cover - Pairing dialog never appears - Pairing succeeds but session won't start - Auth callback never fires - Glasses connect but disconnect immediately --- source: https://extentos.com/docs/troubleshooting/photo-capture-fails --- --- title: Photo capture fails description: Photo capture returns an error or no image data. type: troubleshooting platform: all --- > **Coming soon.** ## Symptoms this page will cover - Capture call returns error - Capture succeeds but image data is empty - Capture succeeds but image is corrupted --- source: https://extentos.com/docs/troubleshooting/voice-trigger-not-firing --- --- title: Voice trigger not firing description: Voice triggers configured but the app never receives the event. type: troubleshooting platform: all --- > **Coming soon.** ## Symptoms this page will cover - Trigger fires in simulator but not on device - Trigger fires intermittently - Wrong phrase being matched --- source: https://extentos.com/docs/vendors --- --- title: Vendors description: Smart-glasses vendors Extentos supports. Meta Ray-Ban is the only GA target as of 2026-04 — every Ray-Ban Meta and Oakley Meta variant is supported through Meta's Device Access Toolkit. Mentra G1, Android XR, and Apple smart glasses are tracked roadmap vendors. Extentos's architecture (vendor-agnostic AppSpec, swappable GlassesTransport interface) is what makes one app code base run unchanged across current and future vendors as they ship. type: overview platform: all related: - /docs/vendors/meta - /docs/vendors/mentra - /docs/vendors/android-xr - /docs/vendors/apple - /docs/concepts/capabilities - /docs/concepts/architecture - /docs/concepts/transport-vs-app --- Extentos is a smart-glasses development layer that abstracts vendor-specific SDKs behind one capability surface. As of 2026-04, **Meta Ray-Ban is the only GA vendor** — every Ray-Ban Meta and Oakley Meta variant is supported through the Meta Device Access Toolkit (DAT). Mentra G1, Android XR, and future Apple smart glasses are **tracked roadmap vendors** — supported in the architecture (vendor-agnostic AppSpec, swappable `GlassesTransport` interface) but not yet implemented as transports. This page is the consolidated status, the vendor model, and what "Extentos supports vendor X" actually means at any given point. ## Status — as of 2026-04 | Vendor | Status | Hardware | What it means today | |---|---|---|---| | **[Meta Ray-Ban](/docs/vendors/meta)** | ✅ **GA** | Ray-Ban Meta (Gen 1, Gen 2), Ray-Ban Meta Display, Oakley Meta HSTN | Build and ship today. Three transports live (`RealMetaTransport` for hardware, `BrowserSimTransport` for the browser simulator, `LocalSimTransport` wrapping Meta's Mock Device Kit). | | **[Mentra G1](/docs/vendors/mentra)** | 🟡 Roadmap | Mentra G1 (open-ecosystem developer glasses) | Architecture is ready (vendor-agnostic AppSpec, transport interface). No `MentraTransport` implementation yet. Tracking Mentra's SDK maturity. | | **[Android XR](/docs/vendors/android-xr)** | 🟡 Roadmap | Samsung / Google XR devices | Tracking Google's Android XR GA timeline. No transport implementation. | | **[Apple smart glasses](/docs/vendors/apple)** | 🟡 Roadmap | Pending Apple platform availability | Apple has not released a third-party smart-glasses SDK yet. Will add a transport once Apple opens a developer toolkit. | The single GA vendor is verifiable: `glassesEnum` in `mcp-server/src/tools/definitions.ts` accepts only `"meta_rayban"`, and `AppSpecV2.deviceModel` only accepts `"META_RAYBAN"`. Adding another vendor means extending those enums alongside the transport implementation. ## The vendor model Extentos's vendor-portability is a **structural property of the architecture**, not a marketing promise. Three layers cooperate: | Layer | What it does | Vendor-specific? | |---|---|---| | **AppSpec** | Your app's behavior — triggers, blocks, actions, handlers | No — written in vendor-agnostic primitives (`capture_photo`, `voice_command`, etc.) | | **Capability vocabulary** | The set of abstract primitives the AppSpec composes from | No — same vocabulary across every vendor | | **`GlassesTransport`** interface | The library's internal contract — `connect`, `capturePhoto`, `videoFrames`, `speak`, etc. | Each vendor has one transport implementation | Result: an app written today against `meta_rayban` runs unchanged on a future `mentra_g1` target by switching the transport. No code rewrite, no AppSpec migration. The transport is the only piece that's vendor-aware. This is verified in code: - `android-library/glasses-core/src/main/java/com/extentos/glasses/core/GlassesTransport.kt` — the interface - `RealMetaTransport.kt`, `BrowserSimTransport.kt`, `LocalSimTransport.kt` — three implementations, all targeting Meta DAT today - `ios-library/Sources/GlassesCore/Transport/GlassesTransport.swift` — iOS parity A future Mentra vendor adds `RealMentraTransport.kt` and `RealMentraTransport.swift` that implement the same interface against Mentra's SDK. Existing apps targeting `meta_rayban` are unaffected. For the deep dive on the abstraction, see **[concepts/capabilities](/docs/concepts/capabilities)** — the vocabulary itself — and **[concepts/architecture](/docs/concepts/architecture)** — how the layers fit together. ## What "supports vendor X" means at any point This is worth pinning down explicitly because the word "supports" is doing a lot of work: | Stage | What's true | |---|---| | Architecture-supported | The AppSpec, capability vocabulary, and transport interface accommodate the vendor. Meta, Mentra, Android XR, Apple are all architecture-supported today. | | Transport-implemented | A concrete `RealXxxTransport` implementation exists that translates abstract calls to the vendor's SDK. **Only Meta** is transport-implemented today. | | Simulator-coverage | The browser simulator can faithfully reproduce the vendor's runtime behavior. Today the simulator targets Meta capabilities; future vendors with novel capabilities (display rendering, neural-band gestures) will need simulator extensions. | | GA | Transport-implemented, simulator-covered, validated end-to-end on real hardware. **Only Meta** is GA. | Saying "Extentos supports Mentra G1" today means stage 1 — architecture-ready. It does not mean a developer can `import com.mentra.*` and ship. The roadmap pages for each non-Meta vendor are honest about which stage applies. ## Why vendor portability matters for developers The strategic case for building on Extentos rather than directly on a vendor SDK: 1. **One code base, future hardware coverage.** When Mentra G1 ships in volume, your existing Extentos AppSpec runs on it without a rewrite. Apps written directly against Meta DAT need a separate Mentra implementation. 2. **One simulator, every vendor.** The Extentos browser simulator (`BrowserSimTransport`) is vendor-agnostic at the protocol layer. Adding a vendor adds support to the same simulator the developer already uses. Apps written directly against vendor SDKs need a different test harness per vendor. 3. **One agent integration.** The MCP server's 18 tools generalize across vendors. The agent uses the same `getPlatformInfo`, `validateIntegration`, `createSimulatorSession` whether the target is `meta_rayban` or `mentra_g1`. No re-learning the tooling per vendor. 4. **One AppSpec language.** The capability vocabulary is shared. A developer who has built one Extentos app for Meta knows the language for every future vendor. No starting over. 5. **Bounded vendor lock-in.** If Extentos ever doesn't fit, the AppSpec is a JSON document the developer owns; the generated code is the developer's app code. The exit cost is bounded. The cost: vendor-specific features that aren't in the shared vocabulary (e.g., Meta Ray-Ban Display rendering, neural-band gestures) aren't accessible through Extentos. Apps that need those features need to use vendor SDKs directly. Most apps don't — most apps use camera, microphone, voice triggers, and audio output, all of which are in the shared vocabulary. ## How a new vendor gets added This is bounded engineering work, not a rewrite, because the architecture defines the seams: 1. **Implement `GlassesTransport`** for the new vendor against its SDK — `RealXxxTransport.kt` on Android, `RealXxxTransport.swift` on iOS. Estimated scope: similar to `RealMetaTransport` (~1500 lines per platform). 2. **Declare the capability manifest** — which abstract capabilities the vendor exposes, with tier markers (`stable`, `preview`, etc.). 3. **Extend the `glassesEnum`** in `mcp-server/src/tools/definitions.ts` and `deviceModel` in the AppSpec types. 4. **Extend the simulator backend** — add the vendor's hardware-event injection UI if it has novel capabilities. 5. **Validate end-to-end** on real hardware. The AppSpec format doesn't change. The MCP tools don't change. Existing developer apps don't change. The new vendor is a drop-in addition. The reason the roadmap exists rather than the implementations: each vendor's SDK has its own maturity, its own developer-program access requirements, and its own hardware availability. Extentos adds a vendor when adding it makes a real product (transport + simulator + tested hardware), not when the architecture *could* accept it. ## Per-vendor pages Deep reference for each vendor lives on its dedicated page: - **[Meta Ray-Ban →](/docs/vendors/meta)** — full GA capability matrix, hardware tiers, development setup, distribution state, the Meta DAT toolkit, partner-curated integrations like Google Maps on Display - **[Mentra G1 →](/docs/vendors/mentra)** — roadmap status, open-ecosystem framing - **[Android XR →](/docs/vendors/android-xr)** — roadmap status, tracking Google's GA timeline - **[Apple smart glasses →](/docs/vendors/apple)** — roadmap status, pending Apple platform availability ## Cross-vendor capability matrix Comparison across vendors will land here once a second vendor is transport-implemented. Today, with Meta as the only GA vendor, the matrix would be one column — see [vendors/meta](/docs/vendors/meta) for the full Meta capability table. The shape of the future matrix mirrors the abstract capability vocabulary: rows = capabilities (`capture_photo`, `voice_command`, etc.), columns = vendors, cells = tier (GA / preview / not exposed). ## Frequently asked questions ### Which smart glasses can I build for with Extentos today? As of 2026-04, only Meta Ray-Ban — every Ray-Ban Meta variant (Gen 1, Gen 2, Display) and Oakley Meta HSTN. The Display variant has additional capabilities (heads-up display, neural-band gestures) that Meta doesn't expose to third-party DAT apps; Extentos works on Display hardware but doesn't surface those capabilities. See [vendors/meta](/docs/vendors/meta) for the full picture. ### Will my Extentos app run on Mentra G1 when it ships? Architecturally yes — your AppSpec is vendor-agnostic and the abstraction is built for portability. Practically: it runs on Mentra G1 once Extentos ships a `MentraTransport` implementation. No code or spec changes are required on your side; the transport is selected at app start based on which hardware is present. ### What about Vuzix, RealWear, Even Realities, or other vendors? Not on the current roadmap. Roadmap inclusion requires the vendor to have a public SDK Extentos can target, sufficient developer-program access, and meaningful hardware availability. Suggest a vendor by filing an issue in the [Extentos repository](https://github.com/Asgermolgaard/vibe-hardware/issues). ### Can Extentos add display-rendering support for the Ray-Ban Meta Display variant? Display rendering on Meta hardware is a Meta-controlled surface — third-party DAT apps can't draw to it. This isn't an Extentos limitation; it's a Meta API boundary. If Meta exposes display rendering in a future DAT release, Extentos will add it to the capability vocabulary. ### Is Extentos vendor-locked to Meta? No, technically. The architecture is vendor-agnostic by design. In practice, Meta is the only currently-implemented vendor, so today an Extentos developer is targeting Meta. The lock-in is bounded: the AppSpec is a JSON document you own, and your `app_callback` handlers are your own code. Switching off Extentos is a manageable migration if it ever becomes necessary. ### When will Mentra G1 / Android XR / Apple be GA? No committed timelines. Vendors are added when adding them makes a real product (transport implementation + simulator coverage + tested hardware), which depends on each vendor's SDK maturity and hardware availability. The current focus is finishing the Meta hardware test loop — see the [changelog](/docs/resources/changelog) for actual release activity. ### Does my Extentos app target the Meta companion app or run on the glasses? Extentos apps are *phone* apps that talk to the glasses over Bluetooth. The phone runs your Android or iOS app; the glasses run Meta's firmware. This is the same model Meta DAT uses — see [vendors/meta § Audio architecture](/docs/vendors/meta#audio-architecture) for the audio-routing details and [concepts/architecture](/docs/concepts/architecture) for the system overview. ## Related concepts - **[Capabilities](/docs/concepts/capabilities)** — the vendor-agnostic vocabulary every vendor's transport implements - **[Architecture](/docs/concepts/architecture)** — how the AppSpec, library, transports, and backend fit together - **[Transport vs app simulation](/docs/concepts/transport-vs-app)** — what each transport actually does, why the layering enables vendor portability - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install Extentos and target the GA vendor (Meta) today --- source: https://extentos.com/docs/vendors/android-xr --- --- title: Android XR description: Android XR (Google's XR platform with Samsung partner devices) is tracked on the Extentos roadmap. The vendor-agnostic AppSpec architecture is ready; the AndroidXRTransport implementation is contingent on Google's Android XR developer-program GA timeline. When implemented, existing Extentos apps running on Ray-Ban Meta will work on Android XR hardware with a config change, not a rewrite. No committed timeline. type: reference platform: all vendor: android-xr --- > **Roadmap.** Android XR support is planned. Tracking Google's GA timeline. ## What this page will cover - Capability mapping - Setup specifics once available - Known constraints --- source: https://extentos.com/docs/vendors/apple --- --- title: Apple description: Apple smart-glasses support is on the Extentos roadmap pending Apple's release of a third-party developer SDK. Apple has not shipped such an SDK as of 2026-05. Extentos's vendor-agnostic architecture is ready to add an Apple transport when one becomes available; existing apps written against Ray-Ban Meta will run on Apple hardware with a config change once the transport ships. No committed timeline. type: reference platform: all vendor: apple --- > **Roadmap.** Pending Apple's platform availability for third-party smart-glasses integrations. --- source: https://extentos.com/docs/vendors/mentra --- --- title: Mentra description: Mentra G1 (the open-ecosystem developer smart-glasses platform) is on the Extentos roadmap. Architecture is ready (vendor-agnostic AppSpec, swappable GlassesTransport interface) but the MentraTransport implementation is not yet shipped. When it lands, existing Extentos apps targeting Ray-Ban Meta will run on Mentra G1 with a config change, not a rewrite. Tracking Mentra's SDK maturity and developer-program access. No committed timeline. type: reference platform: all vendor: mentra --- > **Roadmap.** Mentra integration is on the roadmap as an open-ecosystem partner. ## What this page will cover - Capability mapping - Open SDK integration approach --- source: https://extentos.com/docs/vendors/meta --- --- title: Meta Ray-Ban (Meta DAT) description: Complete Meta Ray-Ban developer guide — hardware tiers (Ray-Ban Meta, Ray-Ban Meta Gen 2, Ray-Ban Meta Display), the Meta Device Access Toolkit (DAT) public capability matrix, distribution and development state, what you can build and ship today vs what's still gated behind Meta partnerships, and how Extentos abstracts the toolkit so the same code runs against the simulator and real glasses. type: reference platform: all vendor: meta related: - /docs/vendors - /docs/concepts/transport-vs-app - /docs/concepts/capabilities - /docs/concepts/permissions - /docs/getting-started/with-agent - /docs/mcp-server - /docs/resources/pricing --- Meta Ray-Ban is the GA target for Extentos. The hardware ships in three tiers as of 2026 — original Ray-Ban Meta, Ray-Ban Meta Gen 2, and Ray-Ban Meta Display (the variant with a heads-up display and a neural-band wrist controller). Third-party apps reach the glasses through Meta's **Device Access Toolkit (DAT)** — a public Android and iOS SDK that exposes camera, microphone, audio, and sensor capabilities via Bluetooth, plus a Mock Device Kit for testing without hardware. This page is the consolidated capability matrix, the distribution-and-development state, and the practical answer to "what can I actually build and ship for Meta Ray-Ban as a third-party developer in 2026?" ## TL;DR | What you can build today | Status | |---|---| | Photo / video / camera-stream apps via DAT | ✅ GA on public toolkit | | Voice-triggered apps (custom phrases via phone STT) | ✅ GA | | TTS-driven audio apps (phone TTS played over BT) | ✅ GA | | Sensor / IMU data consumers | ✅ GA | | Apps that use the **Display** (heads-up rendering) | ❌ Restricted — Meta-curated partners only as of 2026-04 | | Apps that hook the **"Hey Meta"** wake word | ❌ Not exposed to third parties | | Apps that listen for **custom gestures** (tap, swipe) | ❌ Not in public preview | | Distribution via **Meta App Store** | ⚠️ Preview-limited as of 2026-04 — most apps ship private/sideload | | Distribution via **APK sideload / TestFlight / private** | ✅ Works today, no Meta review needed | The short version: a third-party developer can build and privately ship rich photo/video/voice/audio apps for Ray-Ban Meta today using the public DAT toolkit. Display rendering, Hey-Meta-style wake words, and Meta App Store distribution are gated behind partnership status that Meta hands out selectively. ## Compatible models **Extentos runs on every Meta smart-glasses model in market.** Compatibility is determined by Meta's Device Access Toolkit (DAT) — and DAT exposes the same camera, microphone, speaker, and sensor surface uniformly across every model. Frame styles within a model line don't change what Extentos can do; the same Wayfarer-style Ray-Ban Meta and Headliner-style Ray-Ban Meta have identical capabilities to Extentos. | Model line | Frame styles / variants | Camera / Voice / Audio / Sensors | Display rendering | Neural-band gestures | |---|---|---|---|---| | **Ray-Ban Meta** (Gen 1, 2023) | Wayfarer, Headliner, Skyler, and other Ray-Ban frame styles | ✅ Fully supported | n/a (no display) | n/a (no neural band) | | **Ray-Ban Meta** Gen 2 (2024) | Refreshed frame line, improved low-light camera | ✅ Fully supported | n/a (no display) | n/a (no neural band) | | **Ray-Ban Meta Display** (2025) | Single variant with monocular HUD + Meta neural band | ✅ Fully supported | ❌ Meta-curated partners only | ❌ Not in DAT public toolkit | | **Oakley Meta HSTN** (2025) | Sport-focused variant of the same hardware platform | ✅ Fully supported | n/a (no display) | n/a (no neural band) | ### What this means for the Display model This is the most-asked compatibility question. Extentos runs *on* the Ray-Ban Meta Display the same as it runs on any other Ray-Ban Meta — your camera capture, voice triggers, audio playback, sensor reads, and hardware events all work identically. The wearer can use a Display-variant Ray-Ban Meta with your Extentos-built app and get the full app experience. What you don't get on a Display variant is access to the **heads-up display surface** or the **neural-band gestures**. Those are Meta-controlled partner surfaces — Meta exposes them only to curated integrations like Google Maps and the small number of other featured apps, and they aren't available through the public DAT toolkit. If a wearer has a Display variant and your app uses voice triggers and camera capture, they get exactly the same experience as someone wearing a Gen 1 or Gen 2 — there's no degraded path. If you want display rendering or neural-band gesture access, the path today is a Meta partnership, not the public DAT toolkit. See [distribution state](#distribution-state-2026-04) for the partnership picture. ### Compatibility approach Extentos doesn't maintain a model-by-model verification matrix because the underlying SDK (Meta DAT) is the same across every Ray-Ban Meta and Oakley Meta variant. A capability that works on one model works on all of them, modulo hardware tier. New Meta-branded smart-glasses models will work without code changes if Meta keeps using DAT — which has been the consistent pattern through 2024 and 2025. > **As of 2026-04-30** — model lineup, frame styles, hardware specs, and toolkit boundaries are pre-publication notes. Verify against Meta's [Developer Center](https://developers.meta.com), the [DAT GitHub repository](https://github.com/facebook/meta-wearables-dat-android), and the current [Meta smart-glasses product page](https://www.meta.com/smart-glasses/) before quoting publicly. Newer models released between this writing and Meta's next major launch should slot into the same compatibility story automatically. ## What's in the Device Access Toolkit (DAT) Meta DAT is the official public SDK for talking to Ray-Ban Meta from third-party Android and iOS apps. As of 2026-04 the toolkit exposes: ### Capabilities (GA) | Capability | Android API | iOS API | Notes | |---|---|---|---| | **Photo capture** | `StreamSession.capturePhoto()` | `streamSession.capturePhoto()` → `photoDataPublisher` | Up to 12 MP, JPEG. Short-lived stream pattern. | | **Video capture (clip)** | DAT clip API | DAT clip API | Recorded to glasses storage, transferred over BT. | | **Video frame stream** | `Session.addStream(StreamConfiguration)` | `streamSession.videoFramePublisher` | Configurable resolution and frame rate; `LOW`/`2 fps` is typical for vision pipelines. | | **Audio chunk stream (mic)** | `AudioRecord` over `BLUETOOTH_SCO` | `AVAudioEngine` over `.allowBluetooth` route | Mic audio captured *on the glasses*, streamed to the phone over Bluetooth HFP/SCO. Recognition runs on the phone. | | **Audio playback (TTS / earcon)** | `TextToSpeech` + A2DP | `AVSpeechSynthesizer` + `AVAudioSession` | Audio synthesized on the phone, played out the glasses speakers via Bluetooth A2DP. | | **Sensor data (IMU)** | DAT sensor API | DAT sensor API | Accelerometer, gyroscope, magnetometer. | | **Hardware events** | DAT error stream | DAT error publisher | Hinges-closed (user folded the glasses), thermal warnings, audio-route changes. | | **Mock Device Kit** | `mwdat-mockdevice` artifact | `MWDATMockDevice` SPM module | Simulates the BLE/SDK layer; test on phone or emulator without paired glasses. | ### Capabilities NOT in the public toolkit These are real Ray-Ban Meta features but not third-party-accessible: - **"Hey Meta" wake word** — system-level, owned by Meta AI. Custom voice triggers must use the phone's speech recognizer over Bluetooth audio, not the glasses' wake-word engine. - **Heads-up display rendering (Display variant)** — Meta controls the display surface. Third-party DAT apps can only listen for sensor/audio/camera events, not draw UI to the display. - **Custom tap / swipe / multi-finger gestures** — not in public preview as of 2026-04. Standard DAT lifecycle events (pause, resume, stop) are listenable, but custom gesture recognition is not exposed. - **Always-on background audio routing** — apps cannot keep the SCO mic open continuously; the toolkit and phone OS impose lifecycle restrictions. > **Why these matter for app design:** voice triggers in your app must be a custom phrase recognized by your *phone's* speech recognizer (`SpeechRecognizer` on Android, `SFSpeechRecognizer` on iOS) over Bluetooth audio, not "Hey Meta, my app, do X". The glasses are the *microphone*; the phone is the *recognizer*. ## Distribution state (2026-04) This is the part that surprises new Meta Ray-Ban developers. Distribution today has two paths, and the obvious one isn't open yet. ### Path 1 — Private distribution (works today) You can ship a Ray-Ban Meta app to anyone with a paired Ray-Ban Meta and an Android phone or iPhone, today, with no Meta review: - **Android:** sideload the APK directly, distribute via internal track, or push through enterprise channels. Users install your app on their phone; the DAT-based glasses-pairing flow runs the first time they open it. - **iOS:** TestFlight (up to 10,000 testers), enterprise distribution, or AdHoc for small teams. Same flow on the phone side. - **No Meta App Store listing required.** Your app is a regular phone app that *also* talks to the glasses via DAT. This is how the vast majority of third-party Ray-Ban Meta apps ship today. It's the path Extentos optimizes for. ### Path 2 — Meta App Store / Meta Horizon Store (limited) Meta runs an app store surfaced on the glasses companion app (Meta AI / Meta View). As of 2026-04 it is **preview-limited** — only Meta-curated partners are listed there. There's no public submission flow open to all developers. Developers can apply for partnership but approval is gated. The Meta App Store is also where the **Display-rendering** capability lives — apps that draw to the heads-up display on the Ray-Ban Meta Display variant must be Meta-store-listed and approved. Privately-distributed third-party apps cannot draw to the display, even on hardware that has it. > **As of 2026-04-30** — Meta App Store policy is the area most likely to evolve. Meta's annual Connect conference (typically September) has historically been when new developer-program tiers are announced. Verify the current state before shipping copy that quotes specific store policy. ### Privileged partnerships (the Google Maps question) A small set of apps are featured as Meta-curated integrations on Ray-Ban Meta and Ray-Ban Meta Display. These have privileges the public DAT toolkit doesn't expose — most notably **display rendering** on the Display variant. Public examples that have been demoed or shipped as Meta-partnered integrations: - **Google Maps** (Ray-Ban Meta Display) — turn-by-turn navigation rendered on the heads-up display - **Spotify** — voice-driven music control with glasses-side feedback - **WhatsApp** — voice-driven messaging - **Audible** — voice-driven audiobook playback > **As of 2026-04-30** — partner list is illustrative and based on public demonstrations through early 2026. Verify the current featured-app list at [meta.com/smart-glasses](https://www.meta.com/smart-glasses/) before quoting specific names publicly. These integrations are the visible tip of "what Ray-Ban Meta apps can do." A new third-party developer should *not* assume they'll have the same access; the public DAT toolkit is the practical surface for ~99% of developers as of 2026-04. ## Development state — what you can test with today Building and testing an app for Ray-Ban Meta in 2026 is more accessible than the distribution story suggests. The development tools are open: | Tool | What it does | Cost | |---|---|---| | **Meta DAT SDK** | The real toolkit — Android (`mwdat-core`, `mwdat-camera`) and iOS (`MWDATCore`, `MWDATCamera`) | Free | | **Meta Mock Device Kit** | Simulates the BLE/SDK layer on phone or emulator | Free, ships with DAT | | **Meta Developer Center registration** | One-time signup with your personal Meta credentials; gives you `MetaAppID`, `ClientToken`, `TeamID` | Free, ~15–30 min setup | | **Real Ray-Ban Meta hardware** | Pair to your phone over Bluetooth using the Meta AI / Meta View app, then your DAT app uses it | Costs of the glasses | | **Extentos browser simulator** | App-layer simulator at [extentos.com/s](https://extentos.com/s) — tests the wearer experience without real hardware | Free for 1000 events, free email-only account after | | **Extentos `LocalSimTransport`** | Wraps Meta's Mock Device Kit in the Extentos abstraction; same `glasses.*` API works against simulator and real glasses | Free forever, no account | You don't need permission from Meta to start developing. Sign up at the Meta Developer Center, drop the DAT SDK into your Android or iOS app (or use Extentos which wraps it), and you can test today against Mock Device Kit, Extentos's browser simulator, or your own paired Ray-Ban Meta. The *distribution* gate is the hard one. *Development* is open. ## Required setup for development Concrete checklist for a third-party Meta Ray-Ban app, regardless of whether you use Extentos: ### Both platforms - Meta Developer Center account (one-time, free) - Meta App ID and Client Token from the Developer Center - A paired Ray-Ban Meta to test on real hardware (or Mock Device Kit / Extentos simulator instead) ### Android - Min SDK: as required by DAT (typically API 26+) - Maven artifacts under the `com.meta.wearable` group (singular): `com.meta.wearable:mwdat-core`, `com.meta.wearable:mwdat-camera`, optional `com.meta.wearable:mwdat-mockdevice` for testing. Pinned to `v0.5.0` in current Extentos builds. - Distributed via Meta's GitHub Packages with token-based access setup (Maven Central promotion is Meta's roadmap, not current state) - AndroidManifest permissions: `BLUETOOTH_CONNECT`, `BLUETOOTH_SCAN`, `BLUETOOTH_ADMIN`, `RECORD_AUDIO`, `CAMERA` (depending on capabilities) - Signature registered with the Meta App ID for production ### iOS - Min iOS: 15.2 (Meta DAT minimum). Swift 6 required for current DAT. - Swift Package Manager dependency on `https://github.com/facebook/meta-wearables-dat-ios` - SPM modules: `MWDATCore`, `MWDATCamera`, `MWDATMockDevice` for testing. Pinned to `from: "0.6.0"` in current Extentos builds. - `Info.plist` keys: - `MWDAT` dictionary with `MetaAppID`, `ClientToken`, `TeamID`, `AppLinkURLScheme` (a custom URL scheme — not a universal link) - `CFBundleURLTypes` matching the `AppLinkURLScheme` - `LSApplicationQueriesSchemes` includes `fb-viewapp` (to query the Meta AI app) - `UISupportedExternalAccessoryProtocols` includes `com.meta.ar.wearable` - `UIBackgroundModes` includes `bluetooth-central`, `bluetooth-peripheral`, `external-accessory` - Privacy strings: `NSCameraUsageDescription`, `NSMicrophoneUsageDescription`, `NSSpeechRecognitionUsageDescription`, `NSBluetoothAlwaysUsageDescription` - The companion-app registration callback returns to your app via the configured URL scheme — `Wearables.shared.handleUrl(_:)` must be wired in `onOpenURL` for the registration flow to complete Extentos generates the boilerplate for all of this through the MCP server's `generateConnectionModule` tool — see [Quickstart with an AI agent](/docs/getting-started/with-agent). ## Audio architecture (developer-relevant) This is a recurring source of confusion that's worth pinning down explicitly: - **Microphone capture path:** the glasses microphone physically captures audio. That audio streams to the phone over Bluetooth via the **HFP/SCO** profile (call-style bidirectional). Speech recognition runs on the **phone**, not on the glasses, using the platform's native recognizer. - **Speaker playback path:** TTS audio is synthesized on the **phone**, then routed back to the glasses speaker via Bluetooth **A2DP**. Synthesis is a phone job; playback is glasses output. - **Coexistence constraint:** A2DP (high-quality stereo) and HFP/SCO (low-latency bidirectional) on the same BT connection compete. When a video stream is active, audio routing may downgrade automatically. The DAT SDK and `RealMetaTransport` handle this; Extentos surfaces `coexistence.warning` events when it happens so you can see it in the simulator. - **Zero Extentos runtime cost.** Both platforms handle the hard parts (recognition, synthesis) natively. The Extentos library wires the Bluetooth audio session so the right mic and speaker are in the path. Your shipped app pays nothing to Extentos for voice or audio at runtime. ## Roadmap and what to expect > **As of 2026-04-30** — this section is forward-looking and unhedged. Update annually after Meta Connect. Based on the trajectory of the Meta DAT toolkit through early 2026: - **Capability adds are roughly quarterly.** Meta has shipped audio routing, video streams, sensor capabilities, and Mock Device Kit additions over 2024–2025. The cadence is steady. - **Public Display rendering may open** at some point, likely tier-gated (Meta-approved apps first, then broader). No public timeline as of 2026-04. - **Custom gestures** are likely future-toolkit work — the hardware supports tap and swipe; only the public API surface is missing. - **Meta App Store** is the hardest one to predict. Meta has indicated developer-program expansion plans without committing to specific dates. - **Hey Meta integration** is unlikely to open to third parties — the wake-word engine is core to Meta AI's product positioning. Plan for the public DAT toolkit as it stands. Treat private distribution as the production path. Watch Meta Connect (September) for major announcements. ## How Extentos relates Extentos sits on top of Meta DAT and abstracts it: - **`RealMetaTransport`** wraps the real DAT SDK (`mwdat-core` / `MWDATCore` + `mwdat-camera` / `MWDATCamera`). Your app code talks to `glasses.camera.capturePhoto()`; Extentos translates to DAT calls. - **`LocalSimTransport`** wraps Meta's Mock Device Kit (`mwdat-mockdevice` / `MWDATMockDevice`). Same `glasses.*` API, no real glasses needed, no network. - **`BrowserSimTransport`** is Extentos-original — a browser-based simulator that tests the *wearing experience* (voice triggers, TTS, hardware alerts) at the app layer rather than the BLE layer. See [transport vs app simulation](/docs/concepts/transport-vs-app) for the framing. When Meta opens new public-toolkit capabilities, Extentos adds them to the abstraction. When Meta opens display rendering or custom gestures, Extentos's `AppSpec` and the simulator both extend to cover them. Your code doesn't change shape. When other vendors enter the market — Mentra G1, Android XR, Apple smart glasses — Extentos ships additional transports. The `AppSpec` format stays vendor-agnostic. Your existing Meta Ray-Ban app code can target a future vendor with a config change, not a rewrite. ## Frequently asked questions ### Can I publish my Meta Ray-Ban app to the Meta App Store? As of 2026-04, the Meta App Store is preview-limited to Meta-curated partners. Most third-party developers ship via private distribution — APK sideload on Android, TestFlight on iOS, or enterprise channels. This works today and requires no Meta review. ### How do users install my Meta Ray-Ban app? Users install your *phone* app (Android via APK or Play Store; iOS via TestFlight or App Store), then your app pairs with their Ray-Ban Meta the first time they open it via the DAT registration flow. Your app is a regular phone app that talks to the glasses over Bluetooth. ### Why does Google Maps get to render on the Ray-Ban Meta Display when my app can't? Display rendering is not exposed in the public DAT toolkit. Google Maps and other featured integrations are Meta-curated partnerships with privileged access to the display surface. Most third-party apps cannot draw to the display, even on Display-variant hardware. Whether this opens up later is unannounced. ### Can my third-party app trigger on "Hey Meta, [my app], do X"? No. The "Hey Meta" wake word is system-level and not exposed to third parties. Use a custom voice trigger phrase that's recognized by your *phone's* speech recognizer over Bluetooth audio. Extentos's voice-trigger primitive does this for you. ### Does my app work on all Ray-Ban Meta and Oakley Meta variants? Yes. Extentos runs on every model in market — Ray-Ban Meta Gen 1 (Wayfarer, Headliner, Skyler, and other frame styles), Ray-Ban Meta Gen 2, Ray-Ban Meta Display, and Oakley Meta HSTN. Frame style doesn't affect capabilities. Camera, microphone, speaker, and sensor support is identical across every model. ### Does Extentos work with Ray-Ban Meta Display? Yes. Your Extentos-built app runs on a Display-variant Ray-Ban Meta the same way it runs on Gen 1 or Gen 2 — full camera, voice trigger, audio, and sensor support. What's *not* available on Display is the heads-up display rendering and the neural-band gesture input, because Meta doesn't expose those in the public DAT toolkit; they're reserved for Meta-curated partner integrations like Google Maps. Your third-party app on a Display wearer gets exactly the same experience it gives a Gen 2 wearer — no display rendering either way, no degradation either way. ### What Meta smart-glasses models does Extentos support? All of them. Compatibility is determined by Meta's Device Access Toolkit (DAT), which is uniform across every Ray-Ban Meta and Oakley Meta variant. Extentos works on Ray-Ban Meta Gen 1 frame styles (Wayfarer, Headliner, Skyler), Ray-Ban Meta Gen 2, Ray-Ban Meta Display, and Oakley Meta HSTN. Newer Meta-branded models that ship after this writing will work without code changes as long as Meta keeps using DAT — which has been the consistent pattern. ### Do I need a paid Meta Developer account to ship a Meta Ray-Ban app? No. The Meta Developer Center account is free. The DAT SDK is free. Mock Device Kit is free. Meta charges nothing for using the public toolkit. ### What happens to my app when the user folds their glasses? The DAT SDK emits a `hinges_closed` hardware event. Extentos surfaces this as a `transport.hardware_alert` event in the event log and (depending on your spec) can dispatch a `hinges_closed` trigger. Apps typically pause active streams and wait for the user to unfold. ### Is real-hardware testing required to develop a Ray-Ban Meta app? No. Mock Device Kit (via Extentos's `LocalSimTransport`) and the Extentos browser simulator (`BrowserSimTransport` at [extentos.com/s](https://extentos.com/s)) cover the bulk of the dev loop. Real hardware verifies the final-mile fit — fidelity of the camera frames, latency of the voice loop, real-world A2DP/HFP coexistence — but is not required for daily iteration. ### What about Apple smart glasses? Apple has announced AR glasses development but has not shipped a third-party SDK as of 2026-04. Extentos plans to add an Apple transport once Apple opens a developer toolkit; your `AppSpec` won't need to change. ### What about Mentra G1 and Android XR? Both are tracked on the Extentos vendor roadmap. The `AppSpec` is vendor-agnostic by design — adding a vendor means adding a transport implementation, not changing app code or specs. ## Related - **[Transport vs app simulation](/docs/concepts/transport-vs-app)** — how Meta DAT's Mock Device Kit and Extentos's browser simulator complement each other - **[Capabilities](/docs/concepts/capabilities)** — Extentos's vendor-agnostic capability primitives - **[Quickstart with an AI agent](/docs/getting-started/with-agent)** — install the MCP server and start building - **[iOS quickstart](/docs/getting-started/ios)** · **[Android quickstart](/docs/getting-started/android)** - **[Pricing](/docs/resources/pricing)** — Extentos is free; Meta DAT is free; only the glasses cost money