Capture a photo

The camera is the strongest public capability on Meta Ray-Ban, and a one-shot still is the simplest way to use it. From any handler you call glasses.camera.capturePhoto(), get back an ExtentosResult<Photo, CaptureError>, and either pattern-match the result or take the common .valueOrNull() shortcut. This page covers the call shape, the typed failure modes, and the canonical "photo → vision LLM → spoken answer" flow.

Kotlin leads the snippets and is the source of truth. iOS has the same capturePhoto surface (case .success(let photo)) with fewer bundled helpers (only photo.loadImage()) — see the iOS notes inline.

The minimal call

import com.extentos.glasses.core.PhotoConfig
import com.extentos.glasses.core.PhotoFormat
import com.extentos.glasses.core.Photos
import com.extentos.glasses.core.Resolution
import com.extentos.glasses.core.valueOrNull

suspend fun takePhoto(glasses: ExtentosGlasses) {
    val photo = glasses.camera.capturePhoto(
        PhotoConfig(resolution = Resolution.MEDIUM, format = PhotoFormat.JPEG),
    ).valueOrNull() ?: return  // null = the Err branch — see "Handling failures" below

    val uri = photo.uri ?: return          // Photo.uri is String? (nullable)
    val bitmap = Photos.loadBitmap(uri)    // android.graphics.Bitmap? — for Compose
}

PhotoConfig takes typed enums, not strings: Resolution is LOW | MEDIUM | HIGH, PhotoFormat is JPEG | HEIC. Both live in the flat com.extentos.glasses.core package.

By default a photo grabs a frame off the shared camera stream — the reliable path: a frame-grab can't trip Meta's frame-stall watchdog, and it behaves the same whether or not a recording or live-view is already running. It comes back at stream resolution (resolution sets that — 720×1280 / ~0.9 MP at HIGH, 504×896 at MEDIUM), which is ample for a vision model. For a separate dedicated still, set PhotoConfig(dedicatedCapture = true): instead of a frame off the stream, it fires DAT's discrete capturePhoto, whose resolution is set by DAT independently of the stream (typically higher, but fixed — DAT's capture takes no resolution argument), traded against reliability (it can intermittently stall while a stream is live). Reach for it on a deliberate, human-facing photo; leave it off for assistant/vision and rapid captures. It's per-call.

Photo.uri is transport-shaped and nullable

Photo.uri is a String?. Two things to know:

• It is typed String? — a blank frame on capturePhoto is surfaced as CaptureError.PlatformError("no_image_uri") rather than an Ok with a null uri, but null-check before using it anyway (a Photo you construct from other sources can carry null).
• Its scheme varies by transport — data: in the browser simulator, file: on real glasses (RealMetaTransport) and LocalSim. Don't try to AsyncImage(uri) or URL(uri) it directly. Use the Photos helpers, which bridge both schemes uniformly:

All except mediaTypeFromUri (synchronous, pure CPU) are suspend and run their decode / stream-copy on Dispatchers.IO. They return null/false on an unreadable URI, unrecognized scheme, decode failure, or OOM — they never throw. When you save a captured photo, use Photos.copyToFile(uri, dst) rather than hand-rolling loadBytes + writeBytes; it handles data: / file:// / absolute paths, creates parent dirs, and overwrites the destination.

iOS: the bundled helper is photo.loadImage() (returns a UIImage/NSImage from either scheme). There is no Photos.copyToFile / mediaTypeFromUri equivalent yet — you encode the image and infer the media type yourself.

Handling failures

capturePhoto() returns ExtentosResult<Photo, CaptureError>. .valueOrNull() collapses the failure to null, which is fine for fire-and-forget, but to give the user a meaningful message — or to react to a closed toggle — pattern-match with when. Don't wrap the call in runCatching: it only catches thrown exceptions and would mask the typed error.

import com.extentos.glasses.core.CaptureError
import com.extentos.glasses.core.ExtentosResult

when (val r = glasses.camera.capturePhoto()) {
    is ExtentosResult.Ok -> use(r.value)
    is ExtentosResult.Err -> when (val e = r.error) {
        is CaptureError.NotConnected       -> promptToConnectGlasses()
        // Declared for future transports; the current capture path does not emit
        // these three. Handle them so the `when` stays exhaustive, but don't rely
        // on them firing — a denied CAMERA grant arrives as PlatformError.
        is CaptureError.PermissionDenied   -> requestCameraPermission()
        is CaptureError.ThermalCritical    -> glasses.audio.speak("The glasses are too hot — let them cool.")
        is CaptureError.HingesClosed       -> glasses.audio.speak("Open the glasses to take a photo.")
        is CaptureError.CoexistenceBlocked -> glasses.audio.speak("Camera's busy with a video stream — try again.")
        is CaptureError.StreamPaused       -> glasses.audio.speak("The camera is paused — tap the right temple to resume.")
        is CaptureError.DisabledByUser     -> glasses.audio.speak("Camera is switched off (${e.toggle}).")
        is CaptureError.PlatformError      -> log(e.code, e.message)
    }
}

CaptureError.DisabledByUser carries the gating toggle key as a payload — capture is gated by camera_streaming_enabled (default true) and privacy_mode (default false); when either closes you get this variant with the toggle name. The full variant list and payload shapes are in the error reference. CaptureError.CoexistenceBlocked fires when an outgoing video stream holds the camera — capture and outgoing_video_stream are mutually exclusive (camera exclusivity).

CaptureError.StreamPaused is different from a toggle: the wearer paused the camera with the glasses' capture button (a single temple tap) — a deliberate hardware privacy gesture. Meta's DAT gives your app no way to resume the stream programmatically (a single tap pauses and resumes; tap-and-hold stops), so the honest, platform-respecting response is to ask the wearer to tap the temple again, then retry. Don't fight it with a teardown-and-reopen — that drops the whole connection. The wearer taps to resume and the next capturePhoto() succeeds.

You can exercise this whole path in the simulator, before touching hardware: the sim page's Hardware-buttons panel has the same capture button (tap = pause/resume, press-and-hold = stop — the capture LED on the glasses view goes dark while paused or stopped), and an agent drives it headlessly with the injectHardwareButton MCP tool. It's the same shared gate, so the error your handler sees is byte-for-byte what real glasses produce. After a hold-to-stop, the next capture re-arms the stream — on real glasses that stop also drops the connection for a few seconds while the SDK auto-recovers; the sim skips that blip.

Trigger it with a voice phrase

capturePhoto() is just a suspend call — fire it from anywhere: a button, the assistant via a tool, or a voice trigger. The voice-trigger shape:

glasses.voice.onPhrase(phrase = "take a photo", label = "Take a photo") {
    val photo = glasses.camera.capturePhoto().valueOrNull() ?: run {
        glasses.audio.speak("I couldn't take a photo.")
        return@onPhrase
    }
    val uri = photo.uri ?: return@onPhrase
    val dst = File(context.filesDir, "photo-${System.currentTimeMillis()}.jpg")
    Photos.copyToFile(uri, dst)
    glasses.audio.speak("Saved.")
}

Photo + vision LLM

The canonical "AI describes what you see" flow is: wake → capture → vision model → speak. You bring the LLM client; the SDK gives you the photo and the URI helpers to feed it.

glasses.voice.onPhrase(phrase = "describe what you see", label = "Describe scene") {
    val photo = glasses.camera.capturePhoto(
        PhotoConfig(resolution = Resolution.MEDIUM, format = PhotoFormat.JPEG),
    ).valueOrNull() ?: run {
        glasses.audio.speak("I couldn't take a photo.")
        return@onPhrase
    }
    val uri = photo.uri ?: return@onPhrase
    val base64 = Photos.loadBase64(uri) ?: return@onPhrase            // bytes for the request body
    val mediaType = Photos.mediaTypeFromUri(uri) ?: "image/jpeg"      // the media_type field

    // `vision` is YOUR client. capturePhoto gives you the inputs; the model call is yours.
    val description = vision.describe(base64, mediaType, "Describe this scene briefly.")
    glasses.audio.speak(description)
}

Two honest ways to make that vision.describe(...) call:

• Bring-your-own-key in handler code. Write a small client (OkHttp + your provider's /v1/messages-style endpoint). getCodeExample(pattern: "byok_anthropic") is a paste-ready Anthropic Claude Vision wrapper with retry + typed failure variants; getCodeExample(pattern: "photo_describe_voice") is the full composition.
• The assistant's vision tool. If you're already on the Phase-4 assistant runtime, don't hand-roll a vision call — capture the photo inside a tool body and hand the URI to session.includeImage(uri, prompt). The model sees the image, speaks about it in its configured voice, and remembers it for follow-ups, all on the managed AI gateway (no key in your app).

A vision round-trip is 2–8 s p95 and the user is staring at the glasses waiting — play glasses.audio.earcon(EarconSound.START) at capture time so they know it worked, and note that speak() routes over Bluetooth HFP (see audio streaming).

Verify it in the simulator

capturePhoto() emits a capture_photo_returned event when it resolves. Pull it with getEventLog(filter: "all") (the *_returned row classifies under the custom chip) — on success the row carries duration_ms, mechanism, width, height, format, and has_uri. No row means the call is still in flight or never started. mechanism is frame_grab or dedicated — which path your PhotoConfig requested. This is especially useful in the simulator: the sim returns the same image regardless of dedicatedCapture (it has no live-stream vs dedicated-capture distinction to fake), so mechanism is how you confirm which path your code would take on real hardware, where the resolution and reliability actually differ.

The camera-stream lifecycle is visible too: the first frame-grab photo (or video/frames use) arms the shared stream — camera_stream_opened lands under the camera chip and the sim page's capture LED lights. A capture-button tap emits camera_stream_paused / camera_stream_resumed, and a capture attempted while paused logs capture_denied under the errors chip with the same "tap the right temple" message your handler receives. The browser simulator drives the photo from your real webcam, so the whole flow is exercisable before you touch hardware — the simulator is designed to behave like the glasses and that fidelity is under active validation, so confirm capture timing on real glasses too.

Related

• Capture video — bounded clips and the continuous frame stream.
• Audio streaming — speak(), transcripts, and the HFP routing the snippets above rely on.
• The managed AI gateway — where assistant vision calls are metered.
• Error reference — every CaptureError variant and payload.
• Photo capture fails — when capturePhoto returns an error you didn't expect.
• getCapabilityGuide(feature: "capture_photo") — the minimal call shape + the full gotcha list.

Related