Capabilities

A capability is a vendor-agnostic SDK primitive your handler subscribes to — audio.transcriptions(), camera.capturePhoto(), audio.speak(), the assistant, display, hardware events — and the same handler code runs across every supported vendor because the transport, not your code, adapts.

A capability in Extentos is a vendor-agnostic SDK primitive your handler code subscribes to — audio.transcriptions() for continuous STT, audio.recordDiscrete() for a silence-VAD-bounded audio clip (raw audio — transcribe it via audio.transcriptions() or the assistant), audio.speak() for TTS, camera.capturePhoto() for a still, camera.videoFrames() for a frame stream, plus hardware-event flows for thermal / hinges / call-state / lifecycle. The capability vocabulary is the contract between your handler code (which calls capability primitives) and the underlying transport (which translates those calls into platform-specific operations — Meta DAT on Meta smart glasses, Google's Jetpack Projected on Android XR). This page is the technology behind that contract: the full vocabulary, how per-vendor manifests narrow it, how platform permissions derive automatically, and why a shared vocabulary plus a standard transport interface is the design that makes the same code run across every supported smart-glasses vendor.

The capability layer

Three coordinated layers turn an abstract capability into running code on real hardware:

Your handler is written in layer 1 — pure capability calls, no vendor names. The MCP server's validateIntegration tool checks your extentos.manifest.json's declared capability list against layer 2 for the target vendor — flagging anything the vendor doesn't expose. At runtime, the library's selected transport (layer 3) translates your calls into actual platform operations. Same code, different transport — that's how a handler written for Meta Ray-Ban can later target a vendor with a different SDK shape without rewrites.

At runtime, your installed agent has these live. Once Extentos's MCP server is registered with your agent, the agent calls getPlatformInfo for the capability catalog scoped to the current vendor, getCapabilityGuide(feature) for per-feature call shapes, and getCodeExample(pattern) for canonical compositions in Kotlin + Swift. The static tables on this page are the human-readable reference for pre-install evaluation, SEO, and out-of-context lookup; the live MCP response is authoritative when composing real handler code.

Audio primitives

The audio surface is the most-used part of the SDK. The audio primitives:

The canonical voice-Q&A pattern composes these: register a wake phrase with glasses.voice.onPhrase(phrase, label, stops) { ... } (sugar over transcriptions() that also surfaces the phrase on the connection page and the simulator's click-to-fire panel, with automatic handler cancellation when a stops phrase fires), then call speak() to acknowledge, recordDiscrete() to capture the user's question, and speak() again for the answer. Customers needing regex / stateful matching skip onPhrase and subscribe to transcriptions() directly, optionally calling glasses.voice.registerHint(...) to keep the UI affordance visible. getCodeExample(pattern: "voice_qa_assistant") returns the full ~100-line composition in Kotlin and Swift. For new voice-AI apps, prefer the assistant runtime over hand-composing this loop — onPhrase becomes the assistant's wake trigger.

Camera primitives

Photo URI helpers (Photos.loadBase64(uri), Photos.loadBytes(uri), Photos.loadBitmap(uri), Photos.mediaTypeFromUri(uri) on Android; Photo.loadImage() on iOS) bridge the data-URI / file-URI scheme variance across transports — write your handler against the helpers and the same code runs on BrowserSim, LocalSim, and real Meta Ray-Ban transports.

Three videoFrames guarantees worth knowing for streaming pipelines:

• Timestamps are strictly increasing per stream (timestampMs on Android, presentationTimeUs on iOS). The hardware's presentation clock can rebase during internal session recovery; the SDK normalizes so your pipeline can safely assume monotonic order. Don't treat frame timestamps as wall-clock.
• Requested resolution is a ceiling, not a guarantee. On Meta hardware the tiers are LOW 360×640 / MEDIUM 504×896 / HIGH 720×1280, but Bluetooth bandwidth drives an automatic quality ladder (resolution steps down first, then frame rate, never below 15 fps) plus adaptive per-frame compression — effective quality can change mid-stream with no error. The stream is also armed once per app session and kept warm, so the first camera use fixes the quality tier. Read frame.width / frame.height for the effective resolution.
• Starting the camera never exits Active. Arming the capture session is internal; the connection state stays Active, with its camera status refining through starting → ready (broken means a capture genuinely failed and a reconnect is needed). Treat only Disconnected — or a demotion out of Active — as a lost connection; never tear down app resources on !Active.

Speech output

glasses.audio.speak() routes through the phone's native TTS engine (TextToSpeech on Android, AVSpeechSynthesizer on iOS) and plays audio through the glasses speaker over Bluetooth HFP/SCO — the hands-free voice profile, not A2DP music streaming (a concurrent video stream can force an audio-route change; see audio streaming). The phone is the synthesizer; the glasses are the speaker. This is intentional — TTS quality is bound by the phone engine, which matters when comparing against premium voice providers like ElevenLabs (for which you'd fetch audio yourself from the customer's handler and play through the phone speaker until direct-audio-bytes routing to the glasses lands as a future SDK feature).

cancelSpeak() interrupts the active utterance immediately for the barge-in flow. See getCodeExample(pattern: "barge_in_speak") for the canonical TaskGroup / structured-concurrency pattern that cancels speak the moment a Final transcript arrives.

The assistant runtime — voice AI

For voice-driven AI, the canonical surface is the Phase-4 assistant runtime: glasses.assistant.start(provider) { tool(name, description) { … } }. The model owns wake detection, turn-taking, and intent; you write tool bodies that act on app state. It runs through Extentos's managed AI gateway by default — no API key in your app — and you wake it from any trigger (canonically glasses.voice.onPhrase("hey …") { session?.wake() }). It has shipped in com.extentos:glasses since 1.4.0 and in the iOS SDK's GlassesCore product from github.com/extentos/swift-glasses. Full detail: the assistant runtime. The lower-level audio primitives above remain for fine-grained, non-assistant control.

Display

On Ray-Ban Display, glasses.display.show { … } renders a small UI tree (text, buttons, images, full-surface video) the wearer sees, and glasses.display.isAvailable tells you whether the connected device has a display so you can branch. It never throws on non-display glasses — show() simply no-ops. Display support is in beta: the full surface ships in both SDKs and is fully tested in the simulator on both platforms; on-glasses rendering on Meta Ray-Ban Display hardware is in its verification phase. Full detail: the display capability.

Hardware-event flows (mostly planned)

Hardware events describe the world changing: temperature, hinge state, audio routing, call state, app lifecycle, notifications, location. This is a forward-compatible vocabulary — the event types are defined, but most are not yet delivered to handler code on shipping hardware. Today the only world-state your handler actually observes is connection state, via glasses.connection.state (a Disconnected value carries ThermalCritical or HingesClosed as its cause). The browser simulator can inject the rest so you can write handlers ahead of the runtime wiring, but they don't reach a consumer API yet.

What glasses.runtime.events (Flow<RuntimeEvent> / AsyncStream<RuntimeEvent>) emits today is the SDK's own runtime stream — toggle changes, assistant lifecycle, and logs — not the hardware events in the table below.

Write forward-compatible handlers, but gate on what's live. Only connection.state surfaces world-state today; the simulator injects the rest for testing. See the Hardware events guide for exactly what's subscribable now versus planned.

searchDocs(topic: 'connection_state_model') and searchDocs(topic: 'event_log_schema') cover the event types and the diagnostic surface in full.

Toggles — runtime gates the user controls

Eight runtime toggles gate capabilities at runtime; the user owns them via the connection page UI; your handler reads them via glasses.toggles.state. They're session-scoped — held in memory and re-seeded to defaults on each ExtentosGlasses.create(), not persisted across app restarts.

searchDocs(topic: 'toggles') covers each toggle's enforcement status and gotchas in depth.

Permission derivation

Each capability declares its platform-permission requirements once, centrally. The MCP server's getPermissions(capabilities, platform) returns the exact set:

The "BT permissions" column above means the library-declared Bluetooth pair, which is merged into your manifest automatically — it is never something you add. A voice-only app needs no Bluetooth Info.plist key and no MWDAT keys on iOS: it reaches the mic and speaker through the OS's own audio routing, not through the vendor SDK. speak needs nothing of your own on either platform — TTS runs through the platform engine. The authoritative per-capability derivation is permissions, or getPermissions for your exact list.

Update extentos.manifest.json's capabilities array when you add a primitive to your handler — the next getPermissions call surfaces the new keys; validateIntegration confirms they land in the manifest and Info.plist. A developer who never had to think about iOS speech-recognition entitlements still ends up with a correct NSSpeechRecognitionUsageDescription because the capability said "transcription_incremental" and the toolchain knew what that meant on iOS.

The declared footprint — one source, three surfaces

Your capabilities array is the single source of truth for what your app uses. generateConnectionModule writes it once (into the manifest and the compiled ExtentosConfig.usedCapabilities) from a single input, so the three surfaces that need it can never disagree:

• OS permissions — derived as above (getPermissions).
• The connection page — one tile per declared capability, lit when the connected glasses provide it and dimmed when they don't (e.g. Display on a non-display Ray-Ban). Declare nothing and the page shows no capabilities section — it never guesses a default set ("declare nothing, show nothing").
• Your dashboard — the project Overview shows your declared capabilities as "what the app uses," with a per-capability tested marker once it's been exercised in the simulator.

The footprint is declared, not detected. The SDK doesn't scan your code to infer it — static analysis can't reliably tell which capabilities a build actually uses (it would silently miss the camera and ship a runtime crash), which is why every platform's permission model is declare-then-use, not derive. Keep the list current by re-running generateConnectionModule with the capability when you add it.

When a capability is absent

One build ships to every device, and the hardware a user has connected changes minute to minute — glasses in a case, ordinary earbuds instead, nothing at all. (Those are two different states, and capabilities distinguishes them — see below.) Absent capabilities are therefore a normal production state, not an error path, and the SDK treats them that way.

Three rules:

1. Capability calls are always callable and always safe. An absent capability never throws. Discrete calls (camera.capturePhoto()) return a typed error naming the cause; stream calls complete rather than hanging; display calls are a silent no-op.
2. Your control flow is the same in development and production. Only the signal changes — loud in dev (event-log warning, simulator banner), quiet in prod (a counter on your dashboard).
3. Ask before you call. glasses.capabilities is the flat, per-connection truth: camera, microphone, speaker, display.

if (glasses.capabilities.camera) {
    val photo = glasses.camera.capturePhoto()
}

That guard is the whole pattern. Branch on the capability, never on a model name — a new device or a new vendor then needs no code change from you.

What this looks like with no glasses connected

A camera app running on a phone whose glasses aren't there keeps working — but "no glasses" is two different states, and they report differently:

capabilities reports what the resolved transport's vendor provides, not what is powered on right now. A Bluetooth bond survives the glasses being switched off, so the vendor arm still claims and capabilities.camera stays true with them in the case. That's why it is not a presence check: for "can I capture right now," pair it with connection.state.

val canCapture = glasses.capabilities.camera &&
    glasses.connection.state.value is GlassesState.Active

You lose exactly the capabilities the hardware was providing, and nothing else. See choose your path for how this shapes an integration.

Validation and capability negotiation

Three MCP tools work together to keep a handler aligned with what the target vendor can actually do:

• getPlatformInfo({ glasses: "<vendor>" }) — returns the vendor's capability manifest. Which audio / camera / hardware-event primitives it supports. Which it doesn't. Which are GA versus preview.
• validateIntegration() — checks the project against the vendor's capability list for the configured target. Flags capabilities the manifest declares that the vendor doesn't expose, missing permissions, dependency drift. Returns structured errors the agent can act on.
• getProductionChecklist() — late-stage gate. Verifies permissions are wired, credentials are set, foreground-service hints are present for continuous-capture flows, edge cases are handled. Run before shipping.

In the typical agent flow, getPlatformInfo is the first call (discovery), validateIntegration runs after every structural change (correctness gate), and getProductionChecklist runs once the developer is preparing to ship. The capability layer is what lets these tools be deterministic — they answer yes/no against the manifest rather than guessing.

The transport contract

Each vendor provides a GlassesTransport implementation — the code that translates abstract capability calls into platform-specific API calls. The interface is identical across vendors:

GlassesTransport
├─ connect(deviceId)
├─ capturePhoto(config)
├─ captureVideo(config)
├─ recordAudio(config)            ─► returns AudioRecording (transcript + bytes)
├─ videoFrames(config)            ─► continuous stream
├─ audioChunks(config)            ─► continuous stream
├─ transcriptions(config)         ─► continuous stream (Partial + Final)
├─ speak(text, config)
├─ cancelSpeak()
├─ earcon(sound, volume)
└─ events                         ─► transport state, hardware alerts, errors

A vendor that supports a capability implements the corresponding method against its SDK. A vendor that doesn't support a capability either fails fast at validateIntegration (preferred — caught before runtime) or surfaces a typed Result error at runtime (fallback for capabilities that depend on runtime state, like permissions).

This is the engineering boundary that makes "add a vendor = implement the interface" a clean, bounded task — not a sprawling rewrite. For the deep dive on how transports work and what each implementation does, see transport vs app simulation.

Why a shared vocabulary is the right design

Five reasons the capability layer is shaped this way:

1. Vendor portability is structural, not negotiated. Because handler code is written against the capability primitives instead of vendor-specific calls, an app targeting Meta Ray-Ban today can target a future vendor by switching the transport — no code rewrite. The portability is a property of the architecture, not something an individual developer has to engineer per project.
2. Validation is deterministic. The capability vocabulary is finite and the per-vendor manifest is a known set. validateIntegration answers "does this app run on this vendor?" with a yes/no plus structured errors. That determinism is what lets an AI agent confidently mutate the integration — every change has a clear validation outcome.
3. Permissions derive automatically. Each capability declares its platform-permission requirements once, centrally. Add transcription_incremental to your handler and extentos.manifest.json's capabilities array; the iOS Info.plist and the Android manifest get the right keys without the developer learning what NSSpeechRecognitionUsageDescription is.
4. Simulators are honest. The browser simulator and the on-device local simulator both implement the same capability vocabulary the production transports do. There's no "simulator-only" or "production-only" capability — anything you can run in simulation runs in production, and vice versa.
5. New vendor onboarding is bounded work. Adding a vendor is "implement the GlassesTransport interface against the new SDK and declare the capability manifest." No SDK shape changes, no migrations for existing developer handlers, no new MCP tools. Android XR is the worked example: a genuinely different app model (a projected phone activity, not a Bluetooth session) reached the same capability surface without moving it. Brilliant Labs then exercised the other half — degradation — with the first device that has a display and no speaker at all (Brilliant Frame), plus a round panel and hardware that cannot traverse focus. None of those needed a new concept; they are absent capabilities and panel geometry, handled by the rules already here.

Targeting multiple vendors

The capability layer is what makes this technically possible. The strategic story — what supported and roadmap vendors are, when each ships, how to think about portability when planning your app — lives on /docs/vendors as the section landing page, with the Extentos build guide at /docs/vendors/meta and neutral platform pages for Android XR, Brilliant Labs and Apple in the ecosystem landscape.

A future page will cover the runtime semantics of multi-vendor apps — graceful degradation when a target vendor doesn't expose a capability the handler uses, fallbacks, validation policies for "this app must run on at least N of these vendors." That's deferred until a second vendor is shipping, when the rules will be concrete enough to commit to. For now: target one vendor at a time, let validateIntegration confirm fit, and rely on the capability layer to keep your handler code portable when the time comes.

Frequently asked questions

Can I add a new capability that isn't in the vocabulary?

Not directly — the capability vocabulary is a coordinated contract across the SDK, the validator, both simulators, and every vendor's transport. Extending it is an Extentos library change. If a capability you need doesn't exist, the path is to file an issue describing the use case; it's added when there's a cross-vendor primitive worth standardizing.

For app-specific behavior that doesn't need a new SDK primitive — custom AI processing, business logic, network calls — that's just code in your handler class. The handler is your code; you can do anything in it. See searchDocs(topic: 'custom_handlers') for the canonical handler shape and searchDocs(topic: 'custom_extensions') for the framing of how to compose around missing primitives (e.g., custom on-device STT against audio.audioChunks()).

How does the manifest know what permissions to derive on iOS vs Android?

The capability vocabulary has a per-platform permission map baked in. transcription_incremental declares NSSpeechRecognitionUsageDescription on iOS and RECORD_AUDIO on Android, plus the Bluetooth keys both platforms need. The MCP server's getPermissions tool returns the current set given the capability list; generateConnectionModule writes them into the manifest and Info.plist on initial scaffold.

Are streams metered the same as one-shot calls?

The library emits stream.started and stream.stopped events into the structured event log regardless of one-shot vs continuous shape. Browser-simulator session minting requires a free email-only account (Google or email + password — see pricing); once linked, sessions and the events they emit are unlimited. The discovery, validation, guidance, and search MCP tools work anonymously; besides simulator minting, only the generateConnectionModule scaffold step and the account-scoped project tools need the linked account.

How is glasses.audio.transcriptions() different from "Hey Meta"?

"Hey Meta" is Meta's system-level wake word — third-party apps can't hook it. glasses.audio.transcriptions() is the continuous-transcript primitive: the glasses microphone captures audio, streams it to the phone via Bluetooth HFP/SCO, on-device recognition on the phone (Vosk on real hardware; the platform recognizer on the dev path) emits Partial + Final transcripts, and your handler matches strings against them to detect wake phrases. No "Hey Meta" prefix; the wake word is whatever string your handler matches against. See vendors/meta for the full audio-architecture story and searchDocs(topic: 'voice_ux_guide') for phrase-design rules.

Does the capability layer add runtime overhead?

Negligible. The library is a thin translation between abstract calls and the vendor's SDK. There's no extra serialization, no extra IPC, no proxy layer. Capability indirection is compile-time (the transport dispatch is a single method dispatch); runtime is direct SDK calls.

Related concepts

• Architecture — how the capability layer fits into the broader system (MCP, library, backend, transports)
• Transport vs app simulation — the deep dive on how each transport implements the capability vocabulary
• Vendors — the strategic multi-vendor story; per-vendor capability manifests
• Vendors: Meta Ray-Ban — the GA target's full capability manifest
• Quickstart with an AI agent — install the MCP server and see capabilities in action

Related