--- title: Discovery and SDK reference tools description: The Extentos MCP server's discovery + reference tools — getPlatformInfo (the static catalog of vendor capabilities), getCapabilityGuide (per-feature Kotlin + Swift call shape with gotchas), and getCodeExample (full canonical compositions for the common voice-glasses patterns). These are the first calls an AI agent makes in any new task — cheap, all local, no side effects, free and anonymous. type: reference platform: mcp related: - /docs/mcp-server/tools - /docs/mcp-server/tools/generation - /docs/mcp-server/tools/search - /docs/concepts/capabilities --- These discovery + reference tools answer "what does Extentos offer for this target?" and "how do I actually call it?" before any project mutation — plus `getMigrationGuide` for apps already built on raw Meta DAT. All are pure-read, no side effects, no state changes — free and anonymous. > **For your installed agent:** these tools are invoked directly via the MCP — once the server is registered with your agent, it calls `getPlatformInfo` / `getCapabilityGuide` / `getCodeExample` and gets live, structured responses. This page exists so an agent evaluating Extentos pre-install can see the tool surface, and so human developers and search engines can find the documentation outside an MCP context. The live MCP response is authoritative; the parameter tables and example responses below are illustrative. ## `getPlatformInfo` Returns the static platform catalog — library version + the list of SDK capabilities the glasses expose (audio.transcriptions, audio.recordDiscrete, audio.speak, camera.capturePhoto, camera.videoFrames, audio.audioChunks, toggles, connection.state, …). The right first call for any new task. ### When to use - At session start, before scaffolding or writing handler code - When the agent needs to know which capabilities a vendor supports - When the agent is building a capability declaration for the manifest ### When NOT to use - For *how to call* a primitive — use `getCapabilityGuide(feature)` for the per-feature minimal usage + gotchas - For end-to-end compositional examples — use `getCodeExample(pattern)` - For project-installed state — use `inspectIntegration` ### Parameters | Parameter | Type | Required | Description | |---|---|---|---| | `sections` | `array<"version" \| "capabilities">` | yes | Which catalog sections to return. At least one. | | `glasses` | `"meta" \| "meta_rayban" \| "android_xr" \| "brilliant"` | no | Optional — defaults to `meta` (canonical token; `meta_rayban` is a legacy alias for the same vendor). `android_xr` is experimental/preview — it returns the Android XR capability catalog for discovery, while scaffolding and simulator sessions remain Meta-only. | | `expand` | `array<"schema" \| "capabilities.full" \| "capabilities.advanced">` | no | Opt-in to detail. See response-size table below. | ### Response The default response is **compact (~2 KB)** — names, categories, tier markers — sized to fit comfortably in the agent's context. Detail is opt-in via `expand`: | Response variant | Approx size | What's included | |---|---|---| | Default (compact capabilities) | ~2 KB | Vendor, DAT version, feature names + categories + tiers, toggle keys, global constraints | | `expand: ["capabilities.full"]` | ~10 KB | All of the above plus per-feature params, payload shapes, requires lists, per-feature constraints | | `expand: ["capabilities.advanced"]` | + ~1 KB | Adds `droppedPrimitives` and `futurePrimitives` informational fields | The compact-by-default design is intentional. Returning the full ~10 KB on every first call burns the agent's context window before any work. The `summary` field hints at what's available and how to expand. ### Example — discovery at session start ```json { "sections": ["version", "capabilities"], "glasses": "meta_rayban" } ``` Response (compact): ```json { "version": { "latestStable": "2.1.2", "...": "..." }, "capabilities": { "vendor": "meta_rayban", "datVersion": { "android": "0.8.0", "ios": "0.8.0" }, "features": [ { "name": "capture_photo", "category": "media", "tier": "dat_native" }, { "name": "capture_video", "category": "media", "tier": "phone_side_workaround" }, { "name": "video_frames", "category": "stream", "tier": "dat_native" }, { "name": "record_audio", "category": "media", "tier": "phone_side_workaround" }, { "name": "audio_chunks", "category": "stream", "tier": "dat_native" }, { "name": "transcription_incremental", "category": "stream", "tier": "phone_side_workaround" }, { "name": "speak", "category": "voice", "tier": "sdk_method" }, "...etc" ], "toggles": [{ "key": "listening_mode", "gates": ["transcription_incremental", "..."], "description": "..." }, { "key": "camera_streaming_enabled", "gates": ["..."], "description": "..." }], "globalConstraints": ["...", "..."] }, "summary": "Returned library + DAT versions and the meta_rayban capability catalog (compact: N audio/camera primitives, 8 toggles)." } ``` ### Errors - **`invalid_arguments` — `sections must be a non-empty array`** — pass at least one of `"version"`, `"capabilities"` --- ## `getCapabilityGuide` Per-feature SDK usage guide — minimal Kotlin + Swift snippet, config args, gotchas, and which `getCodeExample` patterns exercise the feature. Pairs with `getPlatformInfo` (which lists feature names + categories) by adding the actual idiom for using each. ### When to use - When you know which feature you need (from `getPlatformInfo`) and want the canonical call shape - When you're hitting a confusing failure — the gotchas list typically covers it - When mapping out the manifest's `capabilities` array — confirm each entry's call shape before declaring ### When NOT to use - For a complete compositional pattern — use `getCodeExample` - For capability discovery — use `getPlatformInfo` ### Parameters | Parameter | Type | Required | Description | |---|---|---|---| | `feature` | enum | yes | Feature name from `getPlatformInfo.features[].name`. Plus `connection_state` and `toggles` for the cross-cutting SDK surfaces. | Accepted feature names span the capability primitives (`capture_photo`, `capture_video`, `record_audio`, `transcription_incremental`, `speak`, `video_frames`, `audio_chunks`, `voice_command`, `earcon`), the glasses-display surface (`display`), the cross-cutting SDK surfaces (`connection_state`, `toggles`), the **Phase 4 assistant runtime** (`assistant_runtime`, `assistant_start`, `assistant_tool`, `assistant_provider_openai`, `assistant_vision`, `assistant_session_runtime`), and the deprecated Phase 3 conversation runtime (`conversation_runtime`, `conversation_on_wake`, `conversation_listen`, `conversation_speak`, `conversation_ai_complete`). The live input-schema enum is derived from the guide catalog, so the schema is the authoritative list. ### Response ```json { "feature": "", "summary": "", "kotlin": "", "swift": "", "gotchas": ["", "", "..."], "relatedExamples": ["voice_qa_assistant", "barge_in_speak", "..."] } ``` --- ## `getCodeExample` Full canonical SDK compositions in Kotlin + Swift for the common voice-glasses patterns. Each pattern is a complete handler implementation the agent can peel from when writing real customer code. ### When to use - When you're about to write handler code and want the canonical shape - When a customer asks for a feature that maps cleanly to one of the patterns - When you need a paste-ready BYOK client (currently: Anthropic via `byok_anthropic`) — the LLM-client wrapper that the voice-Q&A and vision patterns reference as `AnthropicClient` ### When NOT to use - For capability discovery — use `getPlatformInfo` - For per-feature minimal usage — use `getCapabilityGuide` ### Parameters | Parameter | Type | Required | Description | |---|---|---|---| | `pattern` | enum (see the pattern table below — the live schema enum is derived from the served catalog) | yes | Which compositional pattern to fetch | The patterns: | Pattern ID | Composition | |---|---| | `assistant_agent_loop` | **Phase 4 canonical — start here for new voice-assistant work.** `glasses.assistant.start(provider) { tool(name, description) { body } }` plus wake wiring; the model owns wake/turn-taking/intent/confirmation, the customer writes tool bodies against app state. | | `agent_driven_e2e_full_loop` | The agent-side E2E test workflow for Phase 4 assistants — explicit two-step wake, multi-tool sweep, four-channel verification (event log + adb logcat + screencap + library state). Drives the REAL OpenAI Realtime provider as well as Mock. Companion to `assistant_agent_loop`. | | `display_browse_detail` | The glasses-display two-view navigation pattern (Ray-Ban Display): browse cards ⇄ full-screen detail, Neural-Band slide/pinch/mid-pinch and assistant tools driving one app-side state machine, with `show(onBack = ...)` back routing and the agent verification recipe. | | `display_media_gallery` | The glasses-display media-gallery pattern — paged media cards (image + caption) on the Ray-Ban Display with Neural-Band navigation (slide to page, pinch to select), one app-side paging state machine, and the agent verification recipe. | | `video_frames_ml` | Continuous `videoFrames()` → an on-device / remote ML model, with the throttling + backpressure gotchas (drop-frames-when-busy, don't queue, decode off the main loop) that keep an ML pipeline from falling behind the frame stream. | | `voice_qa_assistant` | Wake transcript match → speak prompt → recordDiscrete (silence-VAD) → LLM call → speak answer. The canonical multi-turn voice-Q&A flow. | | `barge_in_speak` | Speak with cancellation — TaskGroup-style race between `speak()` completion and a fresh Final transcript that interrupts. | | `photo_describe_voice` | Wake transcript match → speak prompt → capturePhoto → vision LLM → speak description. | | `live_transcription_ui` | Subscribe to `transcriptions()`, stream partials/finals into a Compose / SwiftUI live caption UI. | | `voice_notes` | Wake transcript match → recordDiscrete with longer max-duration → persist transcript + audio bytes to local storage. | | `connection_page_setup` | Minimum-viable scaffold — instantiate `ExtentosGlasses`, mount `ExtentosConnectionPage`, log connection state. | | `byok_anthropic` | Paste-ready Anthropic Claude API client (text + Vision, OkHttp / URLSession). Returns a sealed `LlmResult` with `Ok` / `KeyMissing` / `KeyRejected` / `Connectivity` / `Transient` / `ClientBug` / `Empty` variants so the handler can give a distinct user-facing message per failure mode. Referenced by `voice_qa_assistant` and `photo_describe_voice` as `AnthropicClient`. | | `agent_test_loop` | LEGACY (pre-Phase-4) three-surface verification recipe — for Phase 4 assistants use `agent_driven_e2e_full_loop` instead. | | `conversation_agent_loop` | LEGACY Phase 3 conversation runtime (`glasses.conversation.onWake { listen() / speak() / ai.complete() }`) — removed on Android in 1.4.0. Historical migration reference only; new code uses `assistant_agent_loop`. | ### Response ```json { "pattern": "", "title": "", "description": "<2-3 sentence framing>", "code": { "kotlin": "", "swift": "" }, "explanation": "", "gotchas": ["", "", "..."], "relatedFeatures": ["transcription_incremental", "record_audio", "speak", "..."], "requiredDependencies": { "android": [{ "configuration": "implementation", "notation": "", "appliedIn": "app/build.gradle.kts" }], "iosSwiftPackages": [{ "url": "", "from": "", "products": [""] }] }, "requiredPlugins": { "android": [{ "id": "", "version": "", "appliedIn": "root build.gradle.kts", "applyFalse": true }] } } ``` `requiredDependencies` and `requiredPlugins` are optional fields populated when a pattern needs Gradle / SPM deps beyond the SDK + stdlib (currently only `byok_anthropic`, which needs OkHttp + kotlinx-serialization-json on Android plus the Kotlin serialization plugin in BOTH the root and app `build.gradle.kts`). Agents iterating multiple patterns merge the union of all `requiredDependencies` / `requiredPlugins` into the host app's build files. ### Example — peel the voice-Q&A composition ```json { "pattern": "voice_qa_assistant" } ``` The response returns a full Kotlin `CoachHandler` class (and matching Swift) that registers a wake phrase via `glasses.voice.onPhrase` (sugar over `glasses.audio.transcriptions()`), calls `glasses.audio.speak("What would you like to know?")`, then `glasses.audio.recordDiscrete(...)` with silence-VAD, calls an LLM, and speaks the answer. ### Errors - **`invalid_arguments`** — `pattern` must be one of the ids in the pattern table above. The live input-schema enum is derived from the served catalog, so the schema (not this page) is the authoritative list. --- ## `getMigrationGuide` The entry point for a developer who **already built their app against raw Meta DAT** and is adopting the Extentos SDK. Because Extentos's production transport calls the same DAT underneath, migration is a call-site swap, not a rewrite. Returns a map **keyed by your existing DAT symbol** (`Wearables.createSession`, `photoDataPublisher`, `addStream`, `Display`, raw `CaptureError`, …) → the Extentos primitive that replaces it, each naming the `feature` to drill into with `getCapabilityGuide`, plus an ordered cutover plan. ### When to use - When the project already imports `com.meta.wearable.dat.*` (Android) / `MWDATCore` / `MWDATCamera` (iOS) and you're moving it onto Extentos ### When NOT to use - For a greenfield project with no existing DAT code — start at `getPlatformInfo` → `generateConnectionModule` ### Parameters | Parameter | Type | Required | Description | |---|---|---|---| | `from` | `"meta_dat"` | no | Which SDK you're migrating from. Defaults to `meta_dat` — the only mapped source today; the enum is forward-compatible for other wrappers as they land. | ### Response Returns `approach` (`"full_cutover"`), a `coexistenceNote` (side-by-side operation is **not** supported today — the Meta `DeviceSession` is single-owner), an ordered `plan`, and `mappings[]` (each `{ area, datSymbols, extentosCall, extentosFeature, whatChanges }`). The conceptual companion is [Migrating from raw Meta DAT](/docs/guides/migrating-from-meta-dat); see also [Extentos vs Meta DAT](/docs/concepts/vs-meta-dat). --- ## Why these are the first calls `getPlatformInfo` answers "what's possible on this vendor?" — without that catalog, the agent is composing blind. `getCapabilityGuide` answers "how do I call this specific feature?" — the agent calls it once per primitive it plans to use. `getCodeExample` answers "what does a complete real flow look like end-to-end?" — useful for the catalogued shapes; for ad-hoc business logic, the agent composes from primitives directly. A typical first-time flow: ```text 1. getPlatformInfo({ sections: ["version", "capabilities"], glasses: "meta_rayban" }) 2. getCodeExample({ pattern: }) 3. getCapabilityGuide({ feature: }) 4. generateConnectionModule(...) 5. 6. validateIntegration() ``` ## Related - **[Setup and Generation tools](/docs/mcp-server/tools/generation)** — what the agent calls *after* discovery - **[Search tools](/docs/mcp-server/tools/search)** — `searchDocs` for conceptual topics - **[Capabilities](/docs/concepts/capabilities)** — the conceptual layer behind what `getPlatformInfo` returns - **[Tools overview](/docs/mcp-server/tools)** — back to the full tool catalog