Render on the display

This guide builds a small media gallery on the Ray-Ban Display with no LLM at all: capture a photo, save it to your own store, render a browsable list of cards, and show one full-surface when the user pinches it. Then it points at the assistant-driven variant, where the same display DSL is driven by voice.

Read the display capability for the full DSL, the additive-light rule, and the input model. This page is the task walkthrough.

Android-first; iOS port pending; real-hardware rendering being validated. The display surface is shipped and sim-verified on Android — the iOS DisplayClient doesn't exist yet, and real-hardware rendering plus the mid-pinch back gesture are being validated against DAT now. On a display-less device every call is a safe no-op.

Gate on isAvailable

Display is per-device: the Ray-Ban Display has a screen, the Ray-Ban Meta doesn't. Branch on the runtime signal, never on a model name, and fall back to voice when there's no screen:

if (!glasses.display.isAvailable) {
    glasses.audio.speak("Saved — you've got ${'$'}{store.all().size} photos.")
    return
}
glasses.display.show { /* ... */ }

show() on a display-less device is a silent no-op, so the guard is about UX, not safety — give the user a voice path. The gate is live in the simulator: switching the device model flips isAvailable mid-session with no reconnect.

Capture → store → browse (no LLM)

A self-contained gallery driven by direct display calls plus a button or voice trigger. show {} replaces the entire surface each call, so own your nav state app-side (list vs detail) and re-render whole views on every transition. Each show registers its own onBack because back is view-contextual.

import com.extentos.glasses.core.Background
import com.extentos.glasses.core.ExtentosGlasses
import com.extentos.glasses.core.TextStyle
import com.extentos.glasses.core.valueOrNull
import kotlinx.coroutines.*

// 'store' is YOUR persisted gallery (Room / DataStore / files). SavedPhoto.toPhoto()
// rebuilds a Photo the SAME way for show AND forget, so both hit the same hosted copy.
class GalleryDisplay(
    private val glasses: ExtentosGlasses,
    private val store: PhotoStore,
    private val scope: CoroutineScope = CoroutineScope(SupervisorJob() + Dispatchers.Main),
) {
    // Wire this to a button onClick or a voice trigger.
    fun capture() = scope.launch {
        val photo = glasses.camera.capturePhoto().valueOrNull() ?: return@launch
        store.add(photo)
        showList()
    }

    // Root view: one card per saved photo.
    fun showList() = scope.launch {
        if (!glasses.display.isAvailable) return@launch
        val items = store.all().take(5)
        glasses.display.show(onBack = { scope.launch { glasses.display.clear() } }) {
            column(gap = 8, padding = 12) {
                text("Photos (${'$'}{store.all().size})", style = TextStyle.HEADING)
                for (p in items) {
                    column(
                        onClick = { showDetail(p) },
                        id = "photo-${'$'}{p.id}",        // stable id — gestures/agents target it
                        background = Background.CARD,
                    ) {
                        text(p.label, style = TextStyle.BODY)
                    }
                }
            }
        }
    }

    // Detail view: the photo full-surface; back returns to the list.
    private fun showDetail(p: SavedPhoto) = scope.launch {
        glasses.display.show(onBack = { showList() }) {
            image(p.toPhoto())   // auto-hosts the local capture, then renders the hosted URL
        }
    }

    // On delete, release the hosted display copy (built the SAME way as the show).
    fun delete(p: SavedPhoto) = scope.launch {
        store.remove(p)
        glasses.display.forgetHostedImage(p.toPhoto())
        showList()
    }
}

What's happening:

• Cards are clickable column containers with a stable id ("photo-" + p.id). Without an explicit id, a clickable container falls back to a positional box-0 / box-1 that shifts on reorder — coupling input routing to render order.
• image(photo) is root-only and full-surface. The glasses fetch only http(s) URLs, so the SDK hosts your local capture at show() time and renders the hosted URL — you never hand-roll hosting. (A photo hosts in well under a second; the video equivalent video(clip) takes a few seconds.)
• onBack is registered per show: the detail view's onBack re-renders the list; the root's clears. The back gesture is the Neural Band mid-pinch on hardware. onClick / onBack are plain () -> Unit, but show() / clear() are suspend — launch a coroutine inside the handler.
• forgetHostedImage tears down the hosted copy when the user deletes the photo. Rebuild the Photo the same way for show and forget so both resolve to the same copy. It's idempotent and doesn't touch the local file.

Add a Back button to the detail view for discoverability alongside the gesture:

button("Back", style = ButtonStyle.OUTLINE, id = "back") { showList() }

Test it in the simulator

A fresh sim session defaults to rayban_meta (no display) — set the device first, or every show is a silent no-op:

await setSimDevice({ sessionId, device: "rayban_display" });

// Read EXACTLY what's on the glasses — structured tree, no pixels, no browser needed:
const d1 = await getDisplayState({ sessionId });
//   d1.tree = the rendered DisplayNode tree; d1.interactiveIds = the pinchable card ids

// Pinch a card the way the Neural Band would (needs the browser tab — ensureSimulatorBrowser):
await injectInput({ sessionId, action: "select", targetId: d1.interactiveIds[0] });
const d2 = await getDisplayState({ sessionId });   // the detail view

// Back via the gesture — mid-pinch on hardware, injected back in the sim:
await injectInput({ sessionId, action: "back" });  // onBack fires, re-renders the list

getDisplayState reads the live hub snapshot and works headless; injectInput needs the browser tab attached (it owns the rendered tree + focus). The durable trace is getEventLog(filter: "display") — display_show, display_select, display_navigate, display_back — where event order proves causality. Test both branches: rayban_display renders; setSimDevice({ device: "rayban_meta" }) flips isAvailable to false and your voice fallback runs. See the MCP tools reference.

Dead URLs show nothing. Additive light can't render a failed media fetch — the panel stays dark while the tree checks look green. The simulator shows a "video failed to load" placeholder and a display_error on the errors chip; on hardware a display.video_error runtime event fires so you can fall back to a text view. Check the errors chip first when a display flow looks green but shows nothing. (Errors — there is no DisplayError return type.)

The assistant-driven variant

The same display DSL can be driven by voice instead of buttons: an assistant tool calls glasses.display.show(...), and "show my notes" / "open the groceries note" / "go back" drive the same app-side state machine the Neural Band does. One state machine, two input rails, no conflict — gate the tool on isAvailable and return the reason in ToolResult.Err so the model explains it ("These glasses have no display — want me to read it aloud?").

tool(
    "show_notes",
    "Show the browsable list of the user's notes on the glasses display. " +
        "Also call when the user asks to go back to the list from an open note.",
) {
    if (!glasses.display.isAvailable) {
        return@tool ToolResult.Err("These glasses have no display. Offer to read the notes aloud instead.")
    }
    showBrowse()
    ToolResult.Ok("Notes are on the display — slide between them, pinch one to open it.")
}

For the full two-view browse ⇄ detail pattern with both rails wired, see the assistant runtime and getCodeExample(pattern: "display_browse_detail") via the MCP server.

Related

• The display capability — the full DSL, additive-light rule, hosting, and input model
• The assistant runtime — drive the display by voice from assistant tools
• Capabilities — the per-device capability model and the full SDK vocabulary
• Error reference — the no-DisplayError model and the display runtime events
• Vendors: Meta Ray-Ban — the device family and the Ray-Ban Display

Related