The display capability

Display support is in beta. The full display surface — the builder DSL, select/back input, per-device gating — ships in both SDKs and is fully tested in the simulator on both platforms. On-glasses rendering on Meta Ray-Ban Display hardware is in its verification phase; APIs are stable, with refinements expected as hardware verification completes.

The display capability (glasses.display.*) renders UI on a glasses screen — today, the Ray-Ban Display. You compose a node tree with a small builder DSL (Kotlin or Swift) and hand it to glasses.display.show { ... }; the connected device renders it. On real hardware the tree is delivered over Meta DAT's mwdat-display path; in the browser simulator the same tree renders in the browser tab. There is no WebView, no canvas, no local-bitmap drawing — you declare a tree of containers and leaves, and the platform renders it.

Ships in both SDKs; fully sim-testable on both. glasses.display.* ships in the Kotlin and Swift SDKs, and the DSL, input routing, and event log are exercised end-to-end in the simulator on both platforms. On-glasses rendering via Meta DAT (mwdat-display on Android, MWDATDisplay on iOS, DAT 0.8.0) ships on BOTH platforms. Neither has been confirmed against real Meta Ray-Ban Display hardware yet — the tree, the translation and the input routing are exercised end to end in the simulator and in unit tests on both, and the on-glasses render is the step still outstanding. The local-media roots (image(photo) / video(clip), prepareVideo, forgetHostedVideo / forgetHostedImage) ship on both platforms. See the same tree in Swift for the Swift shapes.

Per-device, never-throw

Display is a capability of the specific device model, not the vendor. Within Meta Ray-Ban, the Ray-Ban Display has a screen and the Ray-Ban Meta / Oakley models do not. So you branch on a runtime signal, never on a model name:

val isAvailable: Boolean = glasses.display.isAvailable   // true only on a display-capable device

if (glasses.display.isAvailable) {
    glasses.display.show { /* ... rich display UX ... */ }
} else {
    glasses.audio.speak("Here's what I found …")          // graceful voice fallback
}

The model is never-throw: show() on a display-less device is a safe silent no-op — degradation never crashes and never diverges between the simulator and hardware. The gate is live: in the simulator, switching the device model pushes the new profile to the running app with no reconnect, so the same session can see isAvailable flip from true to false mid-run. Good UX guards rich display features with isAvailable and falls back to speech.

The builder DSL

show(onBack) { ... } runs against a DisplayRootScope. You nest containers and place leaves; show {} replaces the entire display each call (there are no incremental updates — own your view state app-side and re-render whole views on every transition). show() and clear() are suspend.

Containers:

• column(...) { ... } — a vertical stack.
• row(...) { ... } — a horizontal stack.

Both take gap, padding, mainAlign, crossAlign, background, and an optional onClick that makes the whole container tappable. A clickable container needs an explicit id for a stable, agent-targetable handle — otherwise it falls back to a positional id (box-0, box-1, …) that shifts when you reorder the tree.

Leaves:

• text(text, style, color, align)
• image(url, size, cornerRadius, align) — a remote image fetched by URL.
• button(text, style, icon, align, id, onClick) — pass an explicit id.
• icon(name, style, align) — a vendor-specific icon by name.

Root-only, full-surface (these render as the entire display and must be the sole top-level node — nesting them inside a column / row is a compile error, and the builder scope inside a container simply doesn't expose them):

• video(url) — a remote MP4 by URL.
• video(clip) — a locally-recorded VideoClip (the SDK hosts it; see below).
• image(photo) — a locally-captured Photo rendered full-surface (the SDK hosts it).

glasses.display.show(onBack = { scope.launch { glasses.display.clear() } }) {
    column(gap = 8, padding = 12) {
        text("My notes", style = TextStyle.HEADING)
        column(
            background = Background.CARD,
            onClick = { scope.launch { openNote(note) } },
            id = "note-${note.id}",          // stable identity-based id
        ) {
            text(note.title, style = TextStyle.BODY)
            text(note.preview, style = TextStyle.CAPTION, color = TextColor.SECONDARY)
        }
    }
}

Enum vocabulary

The same tree in Swift

The Swift SDK builds the identical node tree — the tree vocabulary, wire JSON, and root-normalization rules are core-owned, so a tree composed in Swift renders exactly like its Kotlin twin. The builder differs in a few idioms:

• show(onBack:content:) and clear() are async — await glasses.display.show { ... }.
• Containers: one flexBox(direction:) method (.column / .row) instead of the column / row sugar — same gap, padding (an EdgeInsets), mainAlign, crossAlign, background, and optional onClick.
• Leaves: text(_:style:color:align:), image(url:size:cornerRadius:align:), button(_:style:icon:align:onClick:), icon(_:style:align:) — and video(url:) on the root scope.
• Ids auto-assign. There is no explicit id: parameter — tappable nodes get stable-within-a-show ids automatically each show.

await glasses.display.show(onBack: { Task { await glasses.display.clear() } }) { root in
    root.flexBox(direction: .column, gap: 8,
                 padding: EdgeInsets(top: 12, right: 12, bottom: 12, left: 12)) { col in
        col.text("My notes", style: .heading)
        col.button("Open latest") { Task { await openLatest() } }
    }
}

Android-only: the column / row container sugar (iOS uses flexBox directly). Everything else here works the same on both platforms, including the local-media roots image(photo) / video(clip), prepareVideo, and forgetHostedVideo / forgetHostedImage.

Additive-light design rule

The Ray-Ban Display is a waveguide that adds light to the world — it can only brighten, never occlude. Dark pixels render transparent. So design bright-on-dark: bright text and strokes read well; dark fills disappear into the scene. The same applies double to video: bright-on-dark footage reads, dark scenes mostly vanish. The simulator composites the display additively (a screen blend) so previews match the optics.

One consequence to know: a dead or unfetchable media URL emits no light, so the panel shows nothing even while the tree-level checks look green — the one failure additive optics can't surface visually. See Errors and observability.

Content that does not fit

Panels differ by more than size. Meta Ray-Ban Display is 600x600 and scrolls; Android XR display glasses are 450x394; Brilliant Halo is a 256px circle whose usable area is the inscribed square, and neither Brilliant device can scroll at all — Halo has one button and Frame has one tap gesture. So there is no row count that is right everywhere, and an app that picks one is wrong on every panel it was not tuned on.

You do not pick one. Every container takes an overflow policy, and the default answers it per device:

On a panel that scrolls, the SDK hands the whole tree over and the device's own view host handles travel. On a panel that cannot, the SDK splits the tree into pages that fit and appends a page button to each one — so page travel rides selection, the one action every display device can make. Your onClick map never sees it, and the wearer can see that there is more.

Pages break at the root container's direct children and no deeper, so a card never loses its title to a page break. A child too tall for any page gets a page to itself rather than being skipped. The last page wraps to the first, because on a one-button device there may be no way back.

The same split runs in the simulator and on glass, from the same panel table, so a page break you see in the browser is the page break the wearer gets. Switch the simulator's device to Brilliant Halo to see your own layout page.

If you want more than the default gives you, branch on the connected device (glasses.device.type) and render a different tree — the default path is meant to be good, not to be the only option.

Full-surface video and photos, with hosting

video(...) is the one true-motion channel third-party content has — declarative tree updates are link-bound (a few Hz), but a root video plays on the glasses themselves. Playback auto-starts once the tree is sent and stops on the next show() / clear().

The glasses can only fetch an http(s) URL — a local data: / file: clip is rejected on hardware. So:

• video(url) / image(url) — for media already hosted at an http(s) URL.
• video(clip) / image(photo) — for a clip or photo you just captured. The SDK uploads it to a copy on Extentos's media host at show() time and renders the hosted URL; you never hand-roll hosting. A photo hosts in well under a second; a multi-MB clip takes a few seconds (you'll see a brief "Preparing…" only for video).

// Show a just-captured photo full-surface — the SDK hosts the local capture.
glasses.display.show(onBack = { scope.launch { showList() } }) {
    image(photo)
}

The hosted copy intentionally outlives the on-screen display, so re-showing the same clip/photo reuses it instead of re-uploading. There is no time-based expiry — the copy lives at an unguessable public URL on Extentos's media host until your app removes it. That makes releasing it your responsibility, not a background job's:

• forgetHostedVideo(clip) / forgetHostedImage(photo) — tear down the hosted copy. Idempotent and safe (a no-op if never hosted, already forgotten, or no display-video delivery is wired); does not touch the local file.
• prepareVideo(clip): Boolean — speculatively pre-upload a clip (e.g. when you render a list of recordings) so a later show { video(clip) } plays instantly. Best-effort, idempotent, uploads at most once. There's no prepareImage sibling — photos host fast enough that there's no latency to move off the critical path.

Build the Photo / VideoClip the same way for show and for forget, so both resolve to the same hosted copy. Use platform-hosted test clips for quick verification: https://extentos.com/sample-videos/mountain-bike.mp4 and https://extentos.com/sample-videos/run-in-city.mp4.

Input model

The display has two input rails, both driven the same way in the simulator (injectInput) as on hardware:

• Selection — a Neural Band index-pinch on hardware (injectInput(action: "select", targetId) in the sim) fires the node's registered onClick, routed by node id. Give every clickable node a stable id.
• Back — the back gesture (Neural Band mid-pinch on hardware; injectInput(action: "back") in the sim) fires the current show's onBack. Back is view-contextual, so register it per show: a detail view's onBack typically re-renders the browse view; a root view's onBack clears the display; a show with no onBack ignores the gesture. clear() drops the current handlers so a stale onBack can't fire after a wipe.

onClick / onBack are plain () -> Unit — launch a coroutine inside them for the suspend show() / clear() calls. The slide (focus highlight) moves device/sim-side; your app doesn't see focus changes, but the focus target is observable in the event log. Keep an explicit Back button alongside onBack for discoverability. The back-gesture wiring is sim-verified; the real-hardware mid-pinch lands with the DAT display validation pass.

Agent-driven verification reads the live tree with getDisplayState and drives input with injectInput — see the MCP tools reference.

Permissions

The display capability contributes zero extra Android permissions and no iOS plist keys — rendering is outbound over the existing DAT connection (the standard Bluetooth set every glasses app already declares). If your app already connects to the glasses, you can render to the display — on real hardware the scaffold also flips the DAM_ENABLED manifest flag for you when Display is in your declared capability footprint (the Extentos SDK ships it off by default).

Errors and observability

There is no DisplayError return type. show() never throws; display delivery problems surface as runtime log events on glasses.runtime.events — display.video_delivery_failed, display.image_delivery_failed, display.video_error — and on the errors chip in the simulator event log (they are warn severity). A dead media URL additionally raises a visible "video failed to load" placeholder in the simulator viewport and a display_error entry on the errors chip — when a display flow looks green but shows nothing, check that chip first. App code can react to display.video_error (e.g. fall back to a text view). See the error reference.

Related

• Render on the display — the task guide: capture → store → browse, plus the assistant-driven variant
• Capabilities — the full vendor-agnostic SDK vocabulary and the per-device capability model
• The assistant runtime — assistant tools can drive glasses.display.* by voice
• Error reference — the no-DisplayError model and the runtime log events
• Vendors: Meta Ray-Ban — the device family and the Ray-Ban Display