Architecture

Extentos is a five-component system: an AI agent (Claude Code, Cursor, Windsurf, Cline) drives the MCP server (@extentos/mcp-server, a deterministic tool surface for discovery + scaffolding + simulation), the MCP server scaffolds Extentos into the developer's app, the developer (or their agent) writes handler classes that subscribe to capability primitives on the Extentos native library (Android com.extentos:glasses on Maven Central or the iOS Swift package from github.com/extentos/swift-glasses — products GlassesCore, GlassesUI, GlassesDebug, GlassesLifecycle, GlassesTesting), the library exposes one stable glasses.* API over four swappable transport implementations (the vendorless system-audio baseline, real Meta DAT, the browser simulator at extentos.com/s, or Extentos's own local in-memory simulator), and the Extentos backend brokers simulator sessions over WebSocket. The same handler code runs identically against all four transports — simulator and production — because the library translates capability calls into transport-specific operations underneath. This page is the system diagram, the data flow, and the design rationale.

The three-layer model

Extentos models app development in three conceptual layers, top-down:

This model is intentional: one capability vocabulary drives everything below it. There's no "browser version" and "production version" of your handler — there's one class, and the library dispatches each capability call against whichever transport is active. Edit your handler in Kotlin/Swift, rebuild + reinstall, and the same code runs in all four runtimes; the simulator picks up the new binary automatically thanks to auto-bind.

Component diagram

              ┌─────────────────────────────────────┐
              │  AI agent                           │
              │  (Claude Code, Cursor, Windsurf,    │
              │   Cline, or any MCP-compatible host)│
              └────────────────┬────────────────────┘
                               │ MCP stdio
                               ▼
         ┌──────────────────────────────────────────┐
         │  @extentos/mcp-server (npm)              │
         │                                          │
         │   Deterministic tools:                   │
         │   • discovery: getPlatformInfo, get-     │
         │     CapabilityGuide, getCodeExample,     │
         │     searchDocs                           │
         │   • scaffold: generateConnectionModule   │
         │   • validation: inspectIntegration,      │
         │     validateIntegration                  │
         │   • simulation: createSimulatorSession,  │
         │     getSimulatorStatus, getEventLog,     │
         │     completeAuthLink                     │
         │   • production: getProductionChecklist,  │
         │     getCredentialGuide,                  │
         │     getVoiceCommandGuidance,             │
         │     getPermissions                       │
         └─────┬─────────────────────┬──────────────┘
               │                     │
               │ writes              │ HTTPS
               │ code                │
               ▼                     ▼
     ┌─────────────────────┐  ┌──────────────────────────┐
     │  Dev's app code     │  │  Extentos backend        │
     │  (Android / iOS)    │  │  api.extentos.com        │
     │                     │  │                          │
     │  - Handler classes  │  │  - Session store         │
     │  - Connection page  │  │  - WebSocket hub         │
     │  - Bootstrap (gen)  │  │  - Device-code auth      │
     │  - Extentos library │  │  - Event log retention   │
     └──────────┬──────────┘  └────────┬─────────────────┘
                │                      │
                │ links library        │ WSS sessions
                │ (4 transports)       │
                ▼                      ▼
     ┌──────────────────────┐  ┌────────────────────────┐
     │  GlassesTransport    │  │  Browser surrogate     │
     │  (internal interface)│  │  extentos.com/s/{id}   │
     │  ┌────────────────┐  │  │                        │
     │  │ RealMeta       │──┼──┼──► (none)              │
     │  │ wraps DAT SDK  │  │  │                        │
     │  ├────────────────┤  │  │  - real webcam frames  │
     │  │ BrowserSim     │──┼──┼──► WSS to backend ─────┼─┐
     │  │ WebSocket      │  │  │  - real mic audio      │ │
     │  ├────────────────┤  │  │  - hardware injection  │ │
     │  │ LocalSim       │──┼──► (none, no network)     │ │
     │  │ in-memory sim  │  │  │  - replay              │ │
     │  └────────────────┘  │  └────────────────────────┘ │
     └──────────┬───────────┘                             │
                │                                         │
                │ Bluetooth (BLE / HFP / A2DP)            │
                ▼                                         │
     ┌──────────────────────┐                             │
     │  Real Ray-Ban Meta   │                             │
     │  (production)        │                             │
     └──────────────────────┘                             │
                                                          │
              ┌───────────────────────────────────────────┘
              │
              ▼
      same Extentos backend ─► getEventLog reads from here

The agent talks to the MCP server. The MCP server scaffolds Extentos into the developer's app. The developer (or their agent) writes handler classes against the library. The library routes the same glasses.* API through one of four transports based on config. The browser simulator and the library both connect to the backend over WebSocket for the same sessionId, putting them in the same room.

What runs where

A typical dev loop, end to end

This is the data flow agents and developers care about — what happens when the agent says "add a voice-driven photo capture to my app":

1. Developer:   "Add a wake phrase 'describe this' that takes a photo and reads what it sees."
2. Agent:       Calls getPlatformInfo({ glasses: "meta_rayban" })
                → MCP returns capability catalog (audio, camera, ...)
3. Agent:       Calls getCodeExample({ pattern: "photo_describe_voice" })
                → MCP returns the full Kotlin + Swift composition
4. Agent:       Calls getCapabilityGuide for any primitives it needs to look up
                (capture_photo, transcription_incremental, speak)
5. Agent:       Calls generateConnectionModule({ platform: "android", ... })
                → MCP writes ExtentosBootstrap.kt, manifest, Gradle wiring
6. Agent:       Writes CoachHandler.kt (or PhotoDescribeHandler.kt) — subscribes
                to glasses.audio.transcriptions(), matches "describe this",
                calls glasses.camera.capturePhoto(), forwards to vision LLM,
                speaks the response via glasses.audio.speak(). Updates
                extentos.manifest.json's `capabilities` array.
7. Agent:       Calls validateIntegration()
                → MCP checks dependency declared, bootstrap calls
                  ExtentosGlasses.create(), permissions cover capabilities
                → Returns ✓ all good
8. Agent:       Calls createSimulatorSession()
                → MCP HTTPs the Extentos backend, gets sessionId
                → Backend opens WSS endpoint
                → MCP auto-opens browser to extentos.com/s/{sessionId}
9. Library:     On app start, reads BuildConfig.EXTENTOS_SESSION_URL (Android) or
                extentos.session.plist (iOS), or auto-binds via MCP probe;
                selects BrowserSimTransport, opens WSS to api.extentos.com/
                ws/{sessionId} with role: "app". CoachHandler subscribes to
                glasses.audio.transcriptions().
10. Browser:    Opens extentos.com/s/{sessionId}, joins same session as role:
                "browser". Backend now has both halves in the same room.
11. Developer:  Speaks "describe this" into laptop mic
                Browser → transcript "describe this" relayed over WSS
                Library emits Final transcript on the transcriptions Flow
                CoachHandler matches the string, calls glasses.camera
                  .capturePhoto() and glasses.audio.speak("Let me see…")
                BrowserSimTransport sends capture_photo, gets back photo bytes
                Handler forwards the photo to the customer's vision LLM
                Handler calls glasses.audio.speak(answer) — TTS plays in browser
12. Agent:      Calls getEventLog({ filter: "voice" }) or filter:"errors" if
                something didn't work as expected
                → MCP queries backend, returns the structured trace
                → Agent confirms the flow worked end to end

Every capability call in step 11 emits structured events — audio.transcriptions_subscribed, audio.record_discrete_started, camera.capture_photo_started, camera.capture_photo_completed, speak.started, speak.completed, transport.frame.relayed — into the library's ring buffer, forwarded to the backend over the WebSocket session, queryable by the agent via getEventLog. The same flow on real glasses (step 11 with RealMetaTransport) produces the same event shapes — only the transport changes.

The four transports

The library exposes one internal interface (GlassesTransport) and ships four implementations. Selection happens once at ExtentosGlasses.create() time based on config; the developer's code never branches on which transport is active.

• SystemAudioTransport — the vendorless baseline. Microphone and speaker over the phone's own Bluetooth audio routing, with no vendor SDK in the path; serves the complete agent runtime on any hands-free audio route (smart glasses, ordinary earbuds, or the phone itself). No camera, no display — those are what a vendor transport adds.
• RealMetaTransport — wraps the real Meta DAT SDK (mwdat-core + mwdat-camera + mwdat-display on Android, MWDATCore + MWDATCamera on iOS). Production path. BLE link to real Ray-Ban Meta hardware.
• BrowserSimTransport — Extentos-original. WebSocket to api.extentos.com, browser surrogate at extentos.com/s drives real webcam, microphone, and TTS playback. The headline dev-loop simulator.
• LocalSimTransport — Extentos's own in-memory deterministic transport (no DAT SDK, no network). On-device simulation for unit tests, the fast inner loop, and CI.

The same glasses.camera.capturePhoto(), glasses.audio.recordDiscrete(), glasses.audio.speak("hello") calls work identically against all four. See transport vs app simulation for the deep dive on why this layering matters and what each transport simulates.

The capability vocabulary — the universal contract

Every Extentos app composes handler code against the same capability primitives — see concepts/capabilities for the full vocabulary. The contract:

Vendor-agnostic by design — the API doesn't reference Meta DAT, BLE, or any specific hardware. Your handler says for await t in glasses.audio.transcriptions() { … }. The Android XR and Brilliant Labs transports proved it: a second vendor with a completely different app model runs the same handler, and only the transport translation changes.

extentos.manifest.json's top-level capabilities array (the list of SDK features your handler uses) drives getPermissions and getProductionChecklist. Each declared capability has a known set of platform-permission requirements; the toolchain writes the right Android manifest entries and iOS Info.plist keys for you.

State and persistence — what lives where

Extentos is intentionally lean about what persists:

• The customer's handler code lives in the developer's repo. Source-controlled, durable, the source of truth for behavior.
• extentos.manifest.json lives in the developer's repo. Records library version, declared capabilities, permissions, and per-platform build metadata.
• The on-device event log is a 512-entry ring buffer in memory. ~256 KB. FIFO eviction. Does not persist across app restarts — it's debugging data, not telemetry.
• The backend session log is forwarded copies of the on-device events, stored per sessionId for 48 hours after each event is recorded. After that it's gone. This is what getEventLog queries.
• ~/.extentos/install.json holds a per-machine install ID (inst_…). Persists forever. Identifies the install when the browser-simulator gate triggers the device-code flow (account-required for simulator minting, the generateConnectionModule scaffold step, and the account-scoped project tools).
• ~/.extentos/auth.json is the auth token after a free-account device-code flow. Persists forever until extentos-mcp logout.
• No end-user data is persisted by Extentos in production. RealMetaTransport doesn't touch the backend. Your shipped app's runtime emits zero traffic to Extentos servers.

This matters for the privacy and compliance story: the only thing Extentos's backend ever sees is dev-time simulator session activity. End-user runtime activity on real glasses never leaves the developer's app.

The auto-bind dev loop

When the developer's app is built with config.debug = true and the Extentos library is linked, the library opens a persistent WebSocket to the backend's /ws/pending endpoint and probes the host machine's local bridge at 127.0.0.1:31337/whoami (or 10.0.2.2:31337 from the Android emulator, or localhost on iOS Simulator). The MCP server, when running, listens on the same port.

Result: every time the agent calls createSimulatorSession, the backend pushes a session_attached message over the pending socket, and the running app instantly switches to the new session — no rebuild, no URL paste, no developer typing. The library logs which transport it picked and why; the simulator UI opens automatically in the browser.

If the bridge can't be reached (cellular phone on a different network, headless CI, cloud-hosted agent, port 31337 already in use), the developer uses the URL-bake path: paste the BuildConfig.EXTENTOS_SESSION_URL snippet (Android) or write the extentos.session.plist payload (iOS) that createSimulatorSession returns, then rebuild the app once. The happy-path experience (auto-bind on the same machine) is "agent asks for a session, app and browser are both already connected"; the URL-bake path adds one rebuild step but works on any topology.

Why this architecture

A few design choices that recur across the components:

• One handler, many runtimes. The same handler code runs against the browser simulator, the on-device local simulator, and real Ray-Ban Meta hardware. No "dev version" vs "prod version" — there's one class, and the library dispatches each capability call against whichever transport is active.
• Same glasses.* API in simulation and production. Developer code is portable across SystemAudio, RealMeta, BrowserSim, and LocalSim. Failures in simulation are real failures (validateIntegration errors, handler errors, hardware-ready gating) — not transport artifacts. Because the simulator runs the same library code as production, it's designed to behave identically; the final-mile fidelity (camera quality, BT latency, real-world A2DP/HFP coexistence) is being validated on real hardware now — treat a clean simulator run as strong evidence, not a hardware guarantee.
• Vendor-agnostic SDK. Your handler code doesn't reference Meta or DAT. Each vendor gets its own transport implementation inside Extentos, and existing apps reach it with a config change rather than a rewrite — as Android XR did in 2026-07.
• Agent-native. The MCP server's deterministic tools (the full catalog) are designed for an AI agent to compose. There's no planning tool — agents are better planners than regex; the tools are deterministic primitives the agent calls in sequence (generateConnectionModule → write handler classes → validateIntegration → createSimulatorSession).
• Honest simulator. BrowserSimTransport doesn't mock Bluetooth — it uses a different transport (WebSocket) through the same interface. Hardware-ready gating, permission denials, coexistence warnings, and session-expired states all surface as real events. The simulator is not a fake; it's a different but truthful runtime.
• Production runtime cost is zero. RealMetaTransport doesn't talk to Extentos's backend. Voice and audio use the platform-native STT and TTS over Bluetooth. Your shipped app pays Extentos nothing per end-user. The cost base scales with developer population, not user population.

Related concepts

• Transport vs app simulation — the deep dive on what each transport simulates and why both layers exist
• Capabilities — the vendor-agnostic capability primitives your handler composes against
• Vendors: Meta Ray-Ban — the GA target, what the Meta DAT toolkit exposes, and the distribution state
• MCP server — the deterministic tool surface and the agent-driven flow
• Quickstart with an AI agent — install the MCP server and walk through a real dev loop

Related