Protocol overview

You almost never need this page. The protocol here is the WebSocket wire format between the SDK's BrowserSimTransport and the Extentos backend during simulator mode — the SDK speaks it for you. It matters only when you're debugging a simulator session (via getEventLog) or reasoning about why the simulator behaves exactly like hardware. On real glasses (RealMetaTransport) and the in-process mock (LocalSimTransport) no backend protocol is involved at all.

For the bigger picture of why the simulator runs the same app code as production, read transport vs app simulation.

The backend hub

The backend is a custom Node server (backend/server.ts) that wraps Next.js and a ws.WebSocketServer. Every simulator session is a room on that hub, addressed by sessionId. Three kinds of client connect to a room:

The hub enforces the app and browser singletons — a second live client of either role is rejected with a role conflict — and caps observers at 8.

Hardware-ready gating

Capability traffic doesn't flow until the surrogate is ready. The hub flips hardware_ready true once both the app and browser roles are attached, and the SDK gates capability calls on that signal client-side — capturePhoto() before the surrogate is up returns a clean "not ready" locally instead of putting a frame on the wire. This mirrors a real device that isn't connected yet.

The inject envelope

The simulator's click-to-fire panels (and the agent-driven injectTranscript / injectAssistantUtterance / injectInput / injectHardwareButton tools) deliver synthetic input wrapped in an inject envelope. The hub unwraps it and forwards it browser → app only, so an injected transcript reaches the app exactly as a real STT result would and handler dispatch is identical to hardware. Injected input never travels app → browser.

Camera stream state

The surrogate mirrors the glasses' capture-button model: the first shared-stream use (a frame-grab photo, a video recording, or videoFrames) arms the camera stream, and the capture button — the sim page's Hardware-buttons panel or injectHardwareButton — drives it exactly like the right temple on real glasses: tap = pause/resume, hold = stop. The SDK core owns this state machine and announces transitions as camera_stream_opened / camera_stream_paused / camera_stream_resumed / camera_stream_closed events (camera chip); the sim page's capture LED tracks them (lit = streaming, dark = paused or closed, mirroring the privacy LED on the physical glasses). Every camera consumer follows the same state: videoFrames delivery halts while paused (the app's picture freezes) and dies on close, and an in-flight captureVideo recording captures no footage while paused (the clip splices across the gap; a close ends it with the footage so far). While paused, stream-needing captures fail with CaptureError.StreamPaused and a capture_denied event (errors chip) records why — the same shared gate that fires on hardware. One surfaced substrate delta: on real glasses a hold stops the whole device session (the connection drops and the SDK auto-recovers over a few seconds); the sim closes the stream without the connection blip. After a stop, the next camera use re-arms the stream on both substrates.

The event log

Everything in a session lands in a structured event log. The SDK keeps a 512-entry ring buffer on-device and mirrors it to the backend during simulator mode, where it's queryable per sessionId with getEventLog for 48 hours after each event is recorded.

Each event carries exactly one chip — there are seven:

errors · voice · camera · display · ai · lifecycle · custom

errors is special: any event at severity ≥ warn lands there regardless of modality, so a failed photo capture lands in errors (not camera — chips are mutually exclusive); read both chips to see the request-in-camera plus failure-in-errors flow. voice covers mic/STT, TTS, voice triggers, and Phase-4 assistant lifecycle; ai is reserved for customer-side BYOK calls wrapped in glasses.observability.aiCall(...). Filter with getEventLog({ sessionId, filter: "voice" }) and union chips client-side when you need more than one.

The voice proxy

Simulator STT and TTS run through the backend at /api/voice/{stt, tts} so the browser surrogate doesn't need its own model keys — the dev STT/TTS path is one of the deliberately-contained substrate deltas between simulator and hardware. App-level behavior is identical; only the provider differs.

Versioning

The protocol is internal and versioned with the backend; the SDK and hub negotiate a compatible version on connect. There's no third-party-implementable spec today — if you're building an Extentos app you consume capability primitives, never this wire format. A public protocol spec would only ship if a third party needed to implement an alternate hub.

Related

• Transport vs app simulation — what the simulator swaps and what it keeps identical
• Sessions — roles, lifetime, and the account gate on session minting
• getEventLog — reading the event log while debugging

Related