Audio streaming

The audio sub-client is symmetric. In: glasses.audio.audioChunks (raw PCM), glasses.audio.transcriptions() (continuous speech-to-text), and glasses.audio.recordDiscrete(...) (a silence-bounded clip). Out: glasses.audio.speak(text) (text-to-speech) with cancelSpeak(), and glasses.audio.sendAudio(...) (raw PCM) with stopAudio() — the pair you want when your own realtime model produces the audio. This page covers each and the one Bluetooth-audio gotcha that bites everyone.

Kotlin leads the snippets and is the source of truth. iOS ships every surface on this page (for await chunk in ..., case .success).

Speak (text-to-speech)

glasses.audio.speak("Got it.")  // suspends until the TTS engine finishes by default

speak() synthesizes with the phone's platform TTS engine (Android TextToSpeech / iOS AVSpeechSynthesizer) and routes the audio to the glasses speaker. It returns ExtentosResult<Unit, AudioError> and blocks the calling coroutine until the engine finishes by default — to listen and speak at the same time, launch it in its own scope.

A better voice, with no assistant involved

The platform engine is fine for short confirmations and robotic on longer text. If you've added the on-device voice module, name a voice in SpeakConfig and the same call speaks in a natural one:

glasses.audio.speak(articleSummary, SpeakConfig(voice = "kokoro-emma"))

_ = await glasses.audio.speak(articleSummary, config: SpeakConfig(voice: "kokoro-emma"))

No assistant session is required, and the text can come from anywhere — a cloud model's reply, your own content, a notification. It works on every transport including the vendorless audio baseline, so a voice-only app with no glasses paired gets the same voice through the phone or earbuds. Setup and the eleven speakers: on-device models.

Routing is per call and never errors: an unknown voice id, or a model that hasn't finished downloading, serves the platform voice for that utterance instead. cancelSpeak() interrupts a local synthesis exactly as it does platform TTS. rate and pitch apply to the platform engine only — the local voice speaks at its model's own prosody.

To interrupt mid-utterance (barge-in), call cancelSpeak() from another coroutine:

val job = launch { glasses.audio.speak(longText) }
// ...on detected user speech:
glasses.audio.cancelSpeak()  // fire-and-forget; the speaker falls silent within ~tens of ms
job.cancel()

The full race-and-cancel barge-in composition is getCodeExample(pattern: "barge_in_speak").

speak() and earcon() are never gated by a toggle — audio output always plays. No toggle, not even privacy_mode, silences output (the toggles gate audio/camera input + STT). A DisabledByUser result is therefore impossible for speak/earcon; build your own mute switch if you need one.

Speak quality is bound by the phone's TTS engine, not the glasses. For a premium voice you have three options: name an on-device voice in SpeakConfig (above), use the Phase-4 assistant runtime (cloud TTS over the managed gateway), or synthesize wherever you like and push the bytes yourself with sendAudio.

A2DP vs HFP coexistence

speak() routes audio over the Bluetooth HFP profile (voice-band — the link negotiates wideband speech, mSBC at 16 kHz mono), the same profile the glasses use as a microphone — not A2DP. The consequence: while speak() is active, high-quality A2DP music playback drops to voice-band HFP mono for the duration of the speech. The same is true for transcriptions(), recordDiscrete(), audioChunks, and captureVideo(VideoConfig(includeAudio = true)) — any time the glasses act as a mic, you are on HFP and A2DP is suspended.

This is a hardware property of Meta DAT, not an Extentos choice. Surface a "listening…" / "speaking…" indicator so the user understands why their music quality dipped, and avoid overlapping a music-playback feature with continuous capture. Coexistence is a real-hardware behavior — the browser simulator has no A2DP music path, so verify this on actual glasses rather than expecting the sim to reproduce it.

Continuous transcription

transcriptions() is a Flow<Transcript> of partial + final speech-to-text — the primitive behind live captions and string-matched wake phrases:

import com.extentos.glasses.core.Transcript
import com.extentos.glasses.core.TranscriptionConfig

glasses.audio.transcriptions(TranscriptionConfig(language = "en-US")).collect { t ->
    // Transcript is a sealed class: Partial | Final. Partial fires 5–20x/sec.
    if (t !is Transcript.Final) return@collect
    handleFinalText(t.text)
}

Three things that trip people up:

• Always pattern-match is Transcript.Final before reading t.text. Partial fires 5–20 times a second and partial text can match a wake phrase mid-utterance, causing double-fires.
• The library does not normalize. t.text is exactly what the recognizer returned, in the casing the user spoke. Match against t.text.lowercase() (and strip punctuation/apostrophes) yourself.
• Subscribe once. Each subscription is a separate recognizer instance — subscribing per-loop or per-trigger means duplicate transcripts and battery drain.

Transcription is gated by listening_mode (default treated as on; "off" is the user's hard kill-switch), audio_capture_enabled, and privacy_mode. (transcription_enabled is declared in the catalog but not consumed by the library yet — don't chase it.) If transcripts stop arriving, check those first — see Wake phrase not matching. For matching a wake phrase you usually want voice triggers (glasses.voice.onPhrase) rather than a raw collector; drop to transcriptions() directly only for live-caption feeds or custom (regex / stateful) matching. The live-captions-into-Compose shape is getCodeExample(pattern: "live_transcription_ui").

Language, and where it is and isn't honored

TranscriptionConfig(language = …) is the only language control in the SDK, and it does not take effect on every transport.

It is honored everywhere on iOS, and only on the simulator transports on Android.

iOS runs Apple's SFSpeechRecognizer and builds it from Locale(identifier: language), so every locale Apple supports works on every transport. Android runs a Vosk recognizer on real glasses and on the audio baseline — pinned to an en-US acoustic model that never reads the field — and the platform SpeechRecognizer (which does read it) only on the simulator transports.

Raw microphone chunks

When you need raw PCM — your own on-device STT model, audio analysis, custom keyword spotting — subscribe to audioChunks:

import com.extentos.glasses.core.AudioChunkConfig

glasses.audio.audioChunks(
    AudioChunkConfig(
        chunkMillis = 20,    // 20ms is the standard opus/CELT frame — don't change unless your model needs it
        sampleRate = 16000,  // REQUESTED rate; see the caveat below
    ),
).collect { chunk ->
    // feed chunk.data (ByteArray) to your STT / VAD / classifier. Also chunk.sampleRate, chunk.timestampMs.
}

Use transcriptions() instead if you just want speech-to-text — the SDK's recognizer is cheaper and pre-routed (on Android hardware, on-device Vosk; on iOS hardware, Apple's SFSpeechRecognizer; the platform recognizer on the dev/sim path). audioChunks is for cases that genuinely need PCM.

sampleRate is a request, not a guarantee. Meta DAT delivers mic audio over the Bluetooth HFP link, which negotiates wideband speech (mSBC, 16 kHz mono) — on real hardware the effective rate is the HFP rate, not necessarily your config value. For any rate-sensitive DSP (FFT bins, VAD windows), read the per-chunk chunk.sampleRate rather than assuming the config value. The stream buffers with drop-oldest overflow — consumers that can't keep up lose frames rather than back-pressuring the mic.

audioChunks is gated by audio_capture_enabled and privacy_mode (but not listening_mode — it's a raw-audio path, not STT).

It is safe to subscribe during a live assistant session

audioChunks does not take the microphone. The SDK holds a single tap on the input node and fans the buffers out to every consumer, because the platform allows only one tap per input — the assistant's own capture, transcriptions(), recordDiscrete() and captureVideo(includeAudio = true) are all already sharing it.

So an app can meter or analyse the wearer's own voice while glasses.assistant is mid-conversation, which is what a live input-level visualiser needs:

glasses.audio.audioChunks(AudioChunkConfig(chunkMillis = 40))
    .collect { chunk -> meter.value = rms(chunk.data) }   // runs happily alongside the assistant

This is worth stating because the alternative a developer reaches for — standing up their own AVAudioEngine / AudioRecord tap — fails on exactly the one-tap rule the shared input exists to solve, and the failure looks like the assistant going deaf rather than like a tap conflict.

For the other direction — how loudly the assistant itself is currently speaking — see assistant audio levels.

Bring your own realtime model

If a realtime model already runs your conversation — Gemini Live, an OpenAI Realtime session on your own key, something self-hosted — it hands you finished audio, not text. speak(text) is the wrong shape for that: it would throw away audio the model already generated so the SDK could re-synthesize the words. sendAudio is the outbound half of audioChunks, so the loop closes with the SDK owning only the device I/O at either end:

import com.extentos.glasses.core.OutgoingAudioFidelity
import kotlinx.coroutines.launch

// Ask what the speaker path can carry, so your model generates a format
// the link will not just resample away.
val rate = if (glasses.audio.outputFidelity == OutgoingAudioFidelity.HiFi) 24000 else 8000

// Mic in -> your model.
launch {
    glasses.audio.audioChunks().collect { chunk ->
        myRealtimeModel.sendUserAudio(chunk.data, chunk.sampleRate)
    }
}

// Your model -> the glasses speaker. The mic stays open the whole time,
// so the wearer can talk over the reply.
launch {
    myRealtimeModel.audioOut.collect { pcm ->
        if (myRealtimeModel.wearerInterrupted()) {
            glasses.audio.stopAudio()  // barge-in: drop what is still queued
        } else {
            glasses.audio.sendAudio(pcm16 = pcm, sampleRate = rate)
        }
    }
}

stopAudio() matters more than it looks. A realtime model emits faster than realtime, so by the time the wearer starts talking, seconds of reply can already be buffered ahead of the speaker. Without it, playback continues until that backlog drains and the interruption feels ignored. It is the same drain the managed assistant uses for its own barge-in. Note that cancelSpeak() does not cancel sendAudio audio — the two output paths are independent.

You do not have to mute the mic while playing. Capture runs through the platform's voice-communication unit (Voice-Processing I/O on iOS, VOICE_COMMUNICATION on Android), which subtracts the device's own output from the input. That is what makes an always-open mic workable: the echo disappears, a real interruption still arrives.

Format contract. pcm16 is signed 16-bit little-endian mono. An empty chunk is a no-op rather than an error, and a dangling odd byte is dropped — the same convention the named-sound registry uses. The one rejection is a non-positive sampleRate, which returns AudioError.PlatformError(code = "invalid_sample_rate"). Those rules live in the shared Rust core, so both platforms behave identically.

outputFidelity reports HiFi on real Meta glasses (the HFP link negotiates wideband mSBC) and in the browser simulator (WebAudio, full band). Read it rather than assuming — it is a per-transport value, and it exists so your model can be asked for the right output format up front.

Playback is not gated by audio_capture_enabled or privacy_mode; those govern capture. speak() behaves the same way.

If you would rather not own the conversation at all, use the assistant runtime instead — it brings the model, turn taking, barge-in and error handling with it. The two tiers are deliberate: either Extentos runs the loop, or you do. There is nothing in between to half-configure.

Bounded capture (record a turn)

For a one-shot "ask a question" capture, recordDiscrete(...) records until the user goes silent and returns the clip (rawAudioUri); on real hardware the transcript field is empty today — transcription there is your app's job (the browser sim populates it). It is covered with the voice flows: see getCapabilityGuide(feature: "record_audio") for the silence-VAD knobs and getCodeExample(pattern: "voice_notes") for a wake → record → persist composition.

Verify it in the simulator

The browser simulator drives transcription from your real laptop microphone and plays speak() output through your speakers, so you can rehearse the whole audio loop before hardware. recordDiscrete emits a record_audio_returned event (getEventLog(filter: "voice")). The audio chips in the event log are voice (transcripts, recordings, speech) — note there is no audio filter. The simulator is designed to behave like the glasses, but the audio substrate differs (24 kHz PCM in sim vs the glasses' wideband 16 kHz HFP) and that fidelity is under active validation — confirm audio behavior on real glasses.

Related

• Voice triggers — glasses.voice.onPhrase, the structured wrapper over transcriptions().
• Capture a photo — the vision-LLM flow that pairs capture with speak().
• Capabilities — the full audio capability catalog.
• Error reference — every AudioError variant.
• getCapabilityGuide(feature: "speak") / getCapabilityGuide(feature: "transcription_incremental") / getCapabilityGuide(feature: "audio_chunks") — call shapes + full gotchas.