--- title: Protocol overview description: The Extentos simulator wire protocol — WebSocket roles, hardware-ready gating, the inject envelope, event log, and voice proxy. The SDK speaks it for you. type: concept platform: all related: - /docs/concepts/transport-vs-app - /docs/concepts/sessions - /docs/concepts/architecture --- You almost never need this page. The protocol here is the **WebSocket wire format between the SDK's `BrowserSimTransport` and the Extentos backend** during simulator mode — the SDK speaks it for you. It matters only when you're debugging a simulator session (via [`getEventLog`](/docs/mcp-server/tools/simulation)) or reasoning about *why* the simulator behaves exactly like hardware. On real glasses (`RealMetaTransport`) and the in-process mock (`LocalSimTransport`) no backend protocol is involved at all. For the bigger picture of why the simulator runs the same app code as production, read [transport vs app simulation](/docs/concepts/transport-vs-app). ## The backend hub The backend is a custom Node server (`backend/server.ts`) that wraps Next.js and a `ws.WebSocketServer`. Every simulator session is a room on that hub, addressed by `sessionId`. Three kinds of client connect to a room: | Role | Who it is | Singleton? | |---|---|---| | `app` | The consumer's running app (the SDK's `BrowserSimTransport`) | One per session | | `browser` | The hardware surrogate — the simulator tab at `extentos.com/s/` that renders the wearer view, captures webcam/mic, and exposes the click-to-fire panels | One per session | | `observer` | A read-only spectator (a dashboard, a second tab) | Up to 8 | The hub enforces the `app` and `browser` singletons — a second live client of either role is rejected with a role conflict — and caps observers at 8. ## Hardware-ready gating Capability traffic doesn't flow until the surrogate is ready. The hub flips `hardware_ready` true once both the app and browser roles are attached, and the SDK gates capability calls on that signal client-side — `capturePhoto()` before the surrogate is up returns a clean "not ready" locally instead of putting a frame on the wire. This mirrors a real device that isn't connected yet. ## The inject envelope The simulator's click-to-fire panels (and the agent-driven [`injectTranscript`](/docs/mcp-server/tools/simulation) / `injectAssistantUtterance` / `injectInput` / `injectHardwareButton` tools) deliver synthetic input wrapped in an **inject envelope**. The hub unwraps it and forwards it **browser → app only**, so an injected transcript reaches the app exactly as a real STT result would and handler dispatch is identical to hardware. Injected input never travels app → browser. ## Camera stream state The surrogate mirrors the glasses' capture-button model: the first shared-stream use (a frame-grab photo, a video recording, or `videoFrames`) arms the camera stream, and the capture button — the sim page's Hardware-buttons panel or `injectHardwareButton` — drives it exactly like the right temple on real glasses: **tap = pause/resume, hold = stop**. The SDK core owns this state machine and announces transitions as `camera_stream_opened` / `camera_stream_paused` / `camera_stream_resumed` / `camera_stream_closed` events (camera chip); the sim page's capture LED tracks them (lit = streaming, dark = paused or closed, mirroring the privacy LED on the physical glasses). Every camera consumer follows the same state: `videoFrames` delivery halts while paused (the app's picture freezes) and dies on close, and an in-flight `captureVideo` recording captures no footage while paused (the clip splices across the gap; a close ends it with the footage so far). While paused, stream-needing captures fail with `CaptureError.StreamPaused` and a `capture_denied` event (errors chip) records why — the same shared gate that fires on hardware. One surfaced substrate delta: on real glasses a hold stops the whole device session (the connection drops and the SDK auto-recovers over a few seconds); the sim closes the stream without the connection blip. After a stop, the next camera use re-arms the stream on both substrates. ## The event log Everything in a session lands in a structured event log. The SDK keeps a 512-entry ring buffer on-device and mirrors it to the backend during simulator mode, where it's queryable per `sessionId` with [`getEventLog`](/docs/mcp-server/tools/simulation) for 48 hours after each event is recorded. Each event carries exactly one **chip** — there are seven: `errors` · `voice` · `camera` · `display` · `ai` · `lifecycle` · `custom` `errors` is special: any event at severity ≥ warn lands there regardless of modality, so a failed photo capture lands in `errors` (not `camera` — chips are mutually exclusive); read both chips to see the request-in-`camera` plus failure-in-`errors` flow. `voice` covers mic/STT, TTS, voice triggers, and Phase-4 assistant lifecycle; `ai` is reserved for customer-side BYOK calls wrapped in `glasses.observability.aiCall(...)`. Filter with `getEventLog({ sessionId, filter: "voice" })` and union chips client-side when you need more than one. ## The voice proxy Simulator STT and TTS run through the backend at `/api/voice/{stt, tts}` so the browser surrogate doesn't need its own model keys — the dev STT/TTS path is one of the deliberately-contained [substrate deltas](/docs/concepts/transport-vs-app) between simulator and hardware. App-level behavior is identical; only the provider differs. ## Versioning The protocol is internal and versioned with the backend; the SDK and hub negotiate a compatible version on connect. There's no third-party-implementable spec today — if you're building an Extentos app you consume capability primitives, never this wire format. A public protocol spec would only ship if a third party needed to implement an alternate hub. ## Related - [Transport vs app simulation](/docs/concepts/transport-vs-app) — what the simulator swaps and what it keeps identical - [Sessions](/docs/concepts/sessions) — roles, lifetime, and the account gate on session minting - [getEventLog](/docs/mcp-server/tools/simulation) — reading the event log while debugging