iOS API reference

The iOS SDK installs from github.com/extentos/swift-glasses (install). This page is a hand-maintained reference of the surface you touch first — the entry point, the configuration struct, and the GlassesUI types; a generated symbol-level DocC reference will replace it as the package line grows. For per-primitive call shapes (camera, audio, voice, …), your agent's getCapabilityGuide(feature) serves Swift snippets generated from source.

Entry point

import GlassesCore

// usedCapabilities picks the transport on iOS — declare your real footprint.
let glasses: any ExtentosGlasses = Extentos.create(config: ExtentosConfig(
    usedCapabilities: [.microphone, .speaker]   // + .camera / .display if used
))
// Camera/display apps forward the Meta auth callback from their App scene.
// A voice app has no DAT pairing, so it needs neither of these lines:
.onOpenURL { url in Task { _ = await glasses.handleUrl(url) } }

ExtentosGlasses exposes the typed sub-clients (connection, camera, audio, runtime, toggles, voice, telemetry, observability, assistant, display), plus capabilities: DeviceCapabilitySet, usedCapabilities: [DeclaredCapability], and a root connect() convenience — what the generated bootstrap calls, equivalent to connection.connect(). See initialization.

Return shapes vary by kind of call: lifecycle and capture ops return an ExtentosResult to pattern-match (threading); continuous primitives are AsyncStream / AsyncThrowingStream; display show / clear return Void; and assistant methods throw.

camera.videoFrames(config:) is an AsyncThrowingStream, and the two errors it can throw both arrive at the start of iteration:

do {
    for try await frame in glasses.camera.videoFrames() { render(frame) }
} catch let e as CameraUnavailable {
    log("no camera on this transport: \(e.message)")
} catch is CameraStreamPaused {
    showHint("Tap the right temple of your glasses to resume the camera")
}

To branch before starting a stream, read glasses.capabilities.camera. Note the Android counterpart adds one cause iOS cannot have — a build missing com.extentos:glasses-meta — because on iOS every vendor transport ships inside GlassesCore, so there is no vendor dependency to omit.

camera.streamState() is an AsyncStream<CameraStreamState> reporting what the camera stream is doing — .idle, .arming, .streaming, .paused, .stopping. It yields the current phase immediately on iteration, then only on real transitions. Pair it with activeStreamInfo(), which reports the config a running stream locked at.

This is a different axis from the connection state's camera: CameraStatus, which is health (will a capture work, and should the UI prompt the wearer). Both matter at once: a latched capture failure while a stream is running is a real state, which is why they are not one enum.

Follow streamState() — not the connection state — for anything that forwards frames onward, such as a live relay, a recorder, or a frame pipeline. A stream can stop delivering while the connection stays perfectly healthy: the wearer folds the glasses, taps the temple, or another app takes the device session. .paused is the case with no other signal at any layer, because frames simply stop.

for await phase in glasses.camera.streamState() {
    switch phase {
    case .streaming: relay.resume()
    case .paused:    relay.hold("folded, or another app took the camera")
    case .idle:      relay.stop()
    default:         break
    }
}

ExtentosConfig

All fields have defaults — ExtentosConfig() is a valid development configuration.

Display

glasses.display renders declarative trees on display-capable glasses (Ray-Ban Display). display.isAvailable reports whether the connected device has a display — a display call on a no-display device is a silent no-op, never a crash. Display ships in both SDKs and is fully simulator-testable; on-glasses rendering via Meta DAT ships on both platforms, not yet confirmed on Display hardware.

if glasses.display.isAvailable {
    await glasses.display.show(onBack: { /* back gesture for this show */ }) { scope in
        scope.flexBox(direction: .column, gap: 8) { col in
            col.text("Hello from the app")
            col.button("Done") { /* fires on a real tap or a sim-injected select */ }
        }
    }
}
await glasses.display.clear()

Each show replaces the whole display (latest wins); tappable nodes get auto-assigned ids, stable within a show. The node vocabulary is text, image(url:) (the glasses fetch http(s) themselves), button, icon, and flexBox(direction:) containers; the root scope adds video(url:) — a hosted video as the sole, full-surface node. The local-media roots (image(photo:), video(clip:)) and the hosted-media helpers (prepareVideo, forgetHostedVideo, forgetHostedImage) behave exactly as on Android — the SDK hosts the local capture at show() time and renders the hosted URL, because the glasses fetch http(s) only.

Named sounds

glasses.audio also pushes raw PCM to the glasses speaker alongside speak and earcon:

See bring your own realtime model.

VideoConfig and Photo.loadImage

captureVideo(config:) takes a VideoConfig with five fields: resolution, maxDurationSeconds (nil = no time cap — the capture ends on cancellation), format, includeAudio (default true), and frameRate (default 24 — video and live view share one warm camera stream, so it takes effect only when the recording is the session's first camera use).

Photo.loadImage() is the one iOS photo helper: async, decodes file:// and data: URIs uniformly across transports, and returns a PlatformImage? (UIImage on iOS) — nil on unrecognized scheme, missing file, or decode failure.

Assistant

glasses.assistant is the voice-AI runtime, routed through the Extentos managed gateway (no API key in code). Two forms: the sugar start(provider:_:) (builds the config inline, returns a started session) and the raw createSession(config:) + session.start(). Sessions are singleton-active per instance; assistant methods throw (AssistantError).

let session = try await glasses.assistant.start(provider: .managed()) {
    $0.instructions = "You are a helpful assistant on smart glasses."
    $0.tool("take_picture", description: "Take a photo with the glasses camera.") {
        if case .success = await glasses.camera.capturePhoto() { return .ok("photo saved") }
        return .err("camera failed")
    }
}

An AssistantSession has a wake/sleep cycle inside it: start() lands it Dormant, wake() opens the realtime connection, sleep() closes it but keeps history/tools ready, stop() tears it down for good. In-session ops: say(_:), greet(_:), includeImage(uri:prompt:), setVoice(_:), setModel(_:), updateInstructions(_:), cancelSpeak() (barge-in), and the history ops (conversationHistory(limit:), clearHistory(), appendHistory(_:), replaceHistory(_:)). session.state observes the 8-state lifecycle: idle, dormant, activating, active, reconnecting, sleeping, stopping, stopped.

Voice and audio helpers

• glasses.voice.onPhrase(phrase:label:stops:handler:) — the library matches the phrase (case-insensitive substring on final transcripts) and runs your handler. While it runs, hearing any stops phrase cancels the handler's Task — plain structured concurrency, so your defer cleanup runs. Returns a VoiceRegistration whose cancel() removes the hint.
• glasses.voice.registerHint(phrase:label:stops:) — announce the phrase to the simulator + connection-page UI while doing your own matching.
• glasses.audio.cancelSpeak() — barge-in: stops in-flight speak TTS immediately; idempotent.
• glasses.audio.startPushToTalk() — returns a PushToTalkSession; await session.stopAndFlush() ends the capture and returns the concatenated final transcripts.

GlassesUI

The escape hatch — rememberExtentosState(_:) and its ObservableObject form ExtentosStateModel — is gated behind an SPI import: @_spi(ExtentosEscapeHatch) import GlassesUI. The opt-in is deliberate: once you bypass ExtentosConnectionPage, you're responsible for rendering any surfaces the library adds later.

import GlassesUI

ExtentosConnectionPage(glasses: glasses)

// Restyled + trimmed:
ExtentosTheme(appearance: myAppearance) {
    ExtentosConnectionPage(
        glasses: glasses,
        config: ConnectionPageConfig(sections: SectionVisibility(voiceCommands: false))
    )
}

Local tier

Separate SPM products — see on-device models. Linking GlassesLocal raises the package's deployment floor to iOS 17.

// GlassesLocal
public enum ExtentosLocalTier {
    public static func register()                       // BEFORE Extentos.create(...)
    public static func models() -> [LocalModelInfo]
    public static func autoChoice() -> AutoChoice
    public static func deviceFit(for dashboardId: String) -> DeviceFit
    public static func download(
        modelId: String,
        onProgress: @escaping @Sendable (Double) -> Void = { _ in }
    ) async throws
    @discardableResult
    public static func delete(modelId: String) throws -> Bool

    public struct LocalModelInfo {
        public let id: String
        public let displayName: String
        public let requiredMb: Int
        public let isInstalled: Bool
        public let autoEligible: Bool
    }
    public struct AutoChoice {
        public let modelId: String?
        public let isLocal: Bool
        public let cloudReason: CloudFallbackReason?
        public let downloadTarget: String?
    }
    public struct DeviceFit {
        public let modelId: String
        public let requiredMb: Int
        public let availableMb: Int
        public var fits: Bool
    }
    public enum LocalTierError: Error { case unknownModel(String) }
}

// GlassesLocalVoice
public enum ExtentosLocalVoice {
    public static func register()
    public static var modelDirectory: URL? { get }
}

public enum KokoroVoiceModel {
    public static var totalBytes: Int { get }
    public static func isInstalled() -> Bool
    public static func delete() throws
    public static func download(
        progress: @escaping @Sendable (Double) -> Void
    ) async throws
}

Two shape differences from Android: no Context parameter on any call, and download's progress is a single Double fraction rather than Android's (bytesDownloaded, totalBytes) pair.

register() must run before Extentos.create(...). Because Swift runs stored-property initializers ahead of init(), declare the handle without an initializer and assign it inside init() after the register() calls — see on-device models.

Differences from Android

Shipping both platforms? These are the ones that bite. Names and shapes first:

Swift has, Android doesn't: progressiveResponse, premiumVoice, interceptors, externalTaskGroup on ExtentosConfig; a root glasses.connect(); Extentos.default(); Extentos.requestSpeechRecognitionAuthorization(). (KokoroVoiceModel — isInstalled / download / delete for the on-device voice — exists on both platforms; Swift takes no Context.)

Android has, Swift doesn't: VoiceScope / onPhrase(firesWhen:) — so an iOS wake phrase can re-fire mid-conversation and you gate it in your handler; camera.stopVideo(); camera.preferredStreamConfig; display.prepareVideo / forgetHostedVideo / forgetHostedImage; the video(clip:) and image(photo:) display overloads; applicationContext / activityProvider; and transportChosen / selectionSource / device / connectionConfig on the root handle.

Neither platform has an assistant-runtime language setting — see what language does the assistant speak.

Related

• Initialization — entry points, sub-clients, auth handoff, speech authorization
• Threading — the concurrency contract
• Info.plist setup and lifecycle
• Android API — the same surface in Kotlin