iOS API reference

The iOS SDK installs from github.com/extentos/swift-glasses (install). This page is a hand-maintained reference of the surface you touch first — the entry point, the configuration struct, and the GlassesUI types; a generated symbol-level DocC reference will replace it as the package line grows. For per-primitive call shapes (camera, audio, voice, …), your agent's getCapabilityGuide(feature) serves Swift snippets generated from source.

Entry point

import GlassesCore

// usedCapabilities picks the transport on iOS — declare your real footprint.
let glasses: any ExtentosGlasses = Extentos.create(config: ExtentosConfig(
    usedCapabilities: [.microphone, .speaker]   // + .camera / .display if used
))
// Camera/display apps forward the Meta auth callback from their App scene.
// A voice app has no DAT pairing, so it needs neither of these lines:
.onOpenURL { url in Task { _ = await glasses.handleUrl(url) } }

ExtentosGlasses exposes the typed sub-clients (connection, camera, audio, runtime, toggles, voice, telemetry, observability, assistant, display), plus capabilities: DeviceCapabilitySet, usedCapabilities: [DeclaredCapability], and a root connect() convenience — what the generated bootstrap calls, equivalent to connection.connect(). See initialization.

Return shapes vary by kind of call: lifecycle and capture ops return an ExtentosResult to pattern-match (threading); continuous primitives are AsyncStream / AsyncThrowingStream; display show / clear return Void; assistant methods throw; and registerSound / soundNames are synchronous.

camera.videoFrames(config:) is an AsyncThrowingStream, and the two errors it can throw both arrive at the start of iteration:

do {
    for try await frame in glasses.camera.videoFrames() { render(frame) }
} catch let e as CameraUnavailable {
    log("no camera on this transport: \(e.message)")
} catch is CameraStreamPaused {
    showHint("Tap the right temple of your glasses to resume the camera")
}

To branch before starting a stream, read glasses.capabilities.camera. Note the Android counterpart adds one cause iOS cannot have — a build missing com.extentos:glasses-meta — because on iOS every vendor transport ships inside GlassesCore, so there is no vendor dependency to omit.

ExtentosConfig

All fields have defaults — ExtentosConfig() is a valid development configuration.

Display

glasses.display renders declarative trees on display-capable glasses (Ray-Ban Display). display.isAvailable reports whether the connected device has a display — a display call on a no-display device is a silent no-op, never a crash. Display ships in both SDKs and is fully simulator-testable; on-glasses rendering via Meta DAT is live on Android, with iOS on-glasses delivery in progress.

if glasses.display.isAvailable {
    await glasses.display.show(onBack: { /* back gesture for this show */ }) { scope in
        scope.flexBox(direction: .column, gap: 8) { col in
            col.text("Hello from the app")
            col.button("Done") { /* fires on a real tap or a sim-injected select */ }
        }
    }
}
await glasses.display.clear()

Each show replaces the whole display (latest wins); tappable nodes get auto-assigned ids, stable within a show. The node vocabulary is text, image(url:) (the glasses fetch http(s) themselves), button, icon, and flexBox(direction:) containers; the root scope adds video(url:) — a hosted video as the sole, full-surface node. Android-only today: the local-media roots (image(photo:), video(clip:)) and the hosted-video helpers (prepareVideo, forgetHostedVideo).

Named sounds

glasses.audio plays named sounds through the glasses speaker alongside speak and earcon:

Dashboard-uploaded sounds register automatically at assistant start; code registrations win on name collisions.

VideoConfig and Photo.loadImage

captureVideo(config:) takes a VideoConfig with five fields: resolution, maxDurationSeconds (nil = no time cap — the capture ends on cancellation), format, includeAudio (default true), and frameRate (default 24 — video and live view share one warm camera stream, so it takes effect only when the recording is the session's first camera use).

Photo.loadImage() is the one iOS photo helper: async, decodes file:// and data: URIs uniformly across transports, and returns a PlatformImage? (UIImage on iOS) — nil on unrecognized scheme, missing file, or decode failure.

Assistant

glasses.assistant is the voice-AI runtime, routed through the Extentos managed gateway (no API key in code). Two forms: the sugar start(provider:_:) (builds the config inline, returns a started session) and the raw createSession(config:) + session.start(). Sessions are singleton-active per instance; assistant methods throw (AssistantError).

let session = try await glasses.assistant.start(provider: .managed()) {
    $0.instructions = "You are a helpful assistant on smart glasses."
    $0.tool("take_picture", description: "Take a photo with the glasses camera.") {
        if case .success = await glasses.camera.capturePhoto() { return .ok("photo saved") }
        return .err("camera failed")
    }
}

An AssistantSession has a wake/sleep cycle inside it: start() lands it Dormant, wake() opens the realtime connection, sleep() closes it but keeps history/tools ready, stop() tears it down for good. In-session ops: say(_:), greet(_:), includeImage(uri:prompt:), setVoice(_:), setModel(_:), updateInstructions(_:), cancelSpeak() (barge-in), and the history ops (conversationHistory(limit:), clearHistory(), appendHistory(_:), replaceHistory(_:)). session.state observes the 8-state lifecycle: idle, dormant, activating, active, reconnecting, sleeping, stopping, stopped.

Voice and audio helpers

• glasses.voice.onPhrase(phrase:label:stops:handler:) — the library matches the phrase (case-insensitive substring on final transcripts) and runs your handler. While it runs, hearing any stops phrase cancels the handler's Task — plain structured concurrency, so your defer cleanup runs. Returns a VoiceRegistration whose cancel() removes the hint.
• glasses.voice.registerHint(phrase:label:stops:) — announce the phrase to the simulator + connection-page UI while doing your own matching.
• glasses.audio.cancelSpeak() — barge-in: stops in-flight speak TTS immediately; idempotent.
• glasses.audio.startPushToTalk() — returns a PushToTalkSession; await session.stopAndFlush() ends the capture and returns the concatenated final transcripts.

GlassesUI

The escape hatch — rememberExtentosState(_:) and its ObservableObject form ExtentosStateModel — is gated behind an SPI import: @_spi(ExtentosEscapeHatch) import GlassesUI. The opt-in is deliberate: once you bypass ExtentosConnectionPage, you're responsible for rendering any surfaces the library adds later.

import GlassesUI

ExtentosConnectionPage(glasses: glasses)

// Restyled + trimmed:
ExtentosTheme(appearance: myAppearance) {
    ExtentosConnectionPage(
        glasses: glasses,
        config: ConnectionPageConfig(sections: SectionVisibility(voiceCommands: false))
    )
}

Local tier

Separate SPM products — see on-device models. Linking GlassesLocal raises the package's deployment floor to iOS 17.

// GlassesLocal
public enum ExtentosLocalTier {
    public static func register()                       // BEFORE Extentos.create(...)
    public static func models() -> [LocalModelInfo]
    public static func autoChoice() -> AutoChoice
    public static func deviceFit(for dashboardId: String) -> DeviceFit
    public static func download(
        modelId: String,
        onProgress: @escaping @Sendable (Double) -> Void = { _ in }
    ) async throws
    @discardableResult
    public static func delete(modelId: String) throws -> Bool

    public struct LocalModelInfo {
        public let id: String
        public let displayName: String
        public let requiredMb: Int
        public let isInstalled: Bool
        public let autoEligible: Bool
    }
    public struct AutoChoice {
        public let modelId: String?
        public let isLocal: Bool
        public let cloudReason: String?
        public let downloadTarget: String?
    }
    public struct DeviceFit {
        public let modelId: String
        public let requiredMb: Int
        public let availableMb: Int
        public var fits: Bool
    }
    public enum LocalTierError: Error { case unknownModel(String) }
}

// GlassesLocalVoice
public enum ExtentosLocalVoice {
    public static func register()
    public static var modelDirectory: URL? { get }
}

public enum KokoroVoiceModel {
    public static var totalBytes: Int { get }
    public static func isInstalled() -> Bool
    public static func delete() throws
    public static func download(
        progress: @escaping @Sendable (Double) -> Void
    ) async throws
}

Two shape differences from Android: no Context parameter on any call, and download's progress is a single Double fraction rather than Android's (bytesDownloaded, totalBytes) pair.

register() must run before Extentos.create(...). Because Swift runs stored-property initializers ahead of init(), declare the handle without an initializer and assign it inside init() after the register() calls — see on-device models.

Differences from Android

Shipping both platforms? These are the ones that bite. Names and shapes first:

Swift has, Android doesn't: progressiveResponse, premiumVoice, interceptors, externalTaskGroup on ExtentosConfig; a root glasses.connect(); Extentos.default(); Extentos.requestSpeechRecognitionAuthorization(). (KokoroVoiceModel — isInstalled / download / delete for the on-device voice — exists on both platforms; Swift takes no Context.)

Android has, Swift doesn't: VoiceScope / onPhrase(firesWhen:) — so an iOS wake phrase can re-fire mid-conversation and you gate it in your handler; camera.stopVideo(); camera.preferredStreamConfig; display.prepareVideo / forgetHostedVideo / forgetHostedImage; the video(clip:) and image(photo:) display overloads; applicationContext / activityProvider; and transportChosen / selectionSource / device / connectionConfig on the root handle.

Neither platform has an assistant-runtime language setting — see what language does the assistant speak.

Related pages

• Initialization — entry points, sub-clients, auth handoff, speech authorization
• Threading — the concurrency contract
• Info.plist setup and lifecycle
• Android API — the same surface in Kotlin

Related