Audio streaming

The audio sub-client has three streaming surfaces and one output surface. In: glasses.audio.audioChunks (raw PCM), glasses.audio.transcriptions() (continuous speech-to-text), and glasses.audio.recordDiscrete(...) (a silence-bounded clip). Out: glasses.audio.speak(text) (text-to-speech), with cancelSpeak() for barge-in. This page covers each and the one Bluetooth-audio gotcha that bites everyone.

Kotlin leads the snippets and is the source of truth. iOS ships every surface on this page (for await chunk in ..., case .success) — and additionally exposes playSound / registerSound / soundNames on its audio client.

Speak (text-to-speech)

glasses.audio.speak("Got it.")  // suspends until the TTS engine finishes by default

speak() synthesizes with the phone's platform TTS engine (Android TextToSpeech / iOS AVSpeechSynthesizer) and routes the audio to the glasses speaker. It returns ExtentosResult<Unit, AudioError> and blocks the calling coroutine until the engine finishes by default — to listen and speak at the same time, launch it in its own scope.

A better voice, with no assistant involved

The platform engine is fine for short confirmations and robotic on longer text. If you've added the on-device voice module, name a voice in SpeakConfig and the same call speaks in a natural one:

glasses.audio.speak(articleSummary, SpeakConfig(voice = "kokoro-emma"))

_ = await glasses.audio.speak(articleSummary, config: SpeakConfig(voice: "kokoro-emma"))

No assistant session is required, and the text can come from anywhere — a cloud model's reply, your own content, a notification. It works on every transport including the vendorless audio baseline, so a voice-only app with no glasses paired gets the same voice through the phone or earbuds. Setup and the eleven speakers: on-device models.

Routing is per call and never errors: an unknown voice id, or a model that hasn't finished downloading, serves the platform voice for that utterance instead. cancelSpeak() interrupts a local synthesis exactly as it does platform TTS. rate and pitch apply to the platform engine only — the local voice speaks at its model's own prosody.

To interrupt mid-utterance (barge-in), call cancelSpeak() from another coroutine:

val job = launch { glasses.audio.speak(longText) }
// ...on detected user speech:
glasses.audio.cancelSpeak()  // fire-and-forget; the speaker falls silent within ~tens of ms
job.cancel()

The full race-and-cancel barge-in composition is getCodeExample(pattern: "barge_in_speak").

speak() and earcon() are never gated by a toggle — audio output always plays. No toggle, not even privacy_mode, silences output (the toggles gate audio/camera input + STT). A DisabledByUser result is therefore impossible for speak/earcon; build your own mute switch if you need one.

Speak quality is bound by the phone's TTS engine, not the glasses. For a premium voice, either use the Phase-4 assistant runtime (cloud TTS over the managed gateway) or wait for the outgoing_audio_stream playback primitive (future).

A2DP vs HFP coexistence

speak() routes audio over the Bluetooth HFP profile (voice-band — the link negotiates wideband speech, mSBC at 16 kHz mono), the same profile the glasses use as a microphone — not A2DP. The consequence: while speak() is active, high-quality A2DP music playback drops to voice-band HFP mono for the duration of the speech. The same is true for transcriptions(), recordDiscrete(), audioChunks, and captureVideo(VideoConfig(includeAudio = true)) — any time the glasses act as a mic, you are on HFP and A2DP is suspended.

This is a hardware property of Meta DAT, not an Extentos choice. Surface a "listening…" / "speaking…" indicator so the user understands why their music quality dipped, and avoid overlapping a music-playback feature with continuous capture. Coexistence is a real-hardware behavior — the browser simulator has no A2DP music path, so verify this on actual glasses rather than expecting the sim to reproduce it.

Continuous transcription

transcriptions() is a Flow<Transcript> of partial + final speech-to-text — the primitive behind live captions and string-matched wake phrases:

import com.extentos.glasses.core.Transcript
import com.extentos.glasses.core.TranscriptionConfig

glasses.audio.transcriptions(TranscriptionConfig(language = "en-US")).collect { t ->
    // Transcript is a sealed class: Partial | Final. Partial fires 5–20x/sec.
    if (t !is Transcript.Final) return@collect
    handleFinalText(t.text)
}

Three things that trip people up:

• Always pattern-match is Transcript.Final before reading t.text. Partial fires 5–20 times a second and partial text can match a wake phrase mid-utterance, causing double-fires.
• The library does not normalize. t.text is exactly what the recognizer returned, in the casing the user spoke. Match against t.text.lowercase() (and strip punctuation/apostrophes) yourself.
• Subscribe once. Each subscription is a separate recognizer instance — subscribing per-loop or per-trigger means duplicate transcripts and battery drain.

Transcription is gated by listening_mode (default treated as on; "off" is the user's hard kill-switch), audio_capture_enabled, and privacy_mode. (transcription_enabled is declared in the catalog but not consumed by the library yet — don't chase it.) If transcripts stop arriving, check those first — see Wake phrase not matching. For matching a wake phrase you usually want voice triggers (glasses.voice.onPhrase) rather than a raw collector; drop to transcriptions() directly only for live-caption feeds or custom (regex / stateful) matching. The live-captions-into-Compose shape is getCodeExample(pattern: "live_transcription_ui").

Language, and where it is and isn't honored

TranscriptionConfig(language = …) is the only language control in the SDK, and it does not take effect on every transport.

It is honored everywhere on iOS, and only on the simulator transports on Android.

iOS runs Apple's SFSpeechRecognizer and builds it from Locale(identifier: language), so every locale Apple supports works on every transport. Android runs a bundled Vosk recognizer on real glasses and on the audio baseline — pinned to an en-US acoustic model that never reads the field — and the platform SpeechRecognizer (which does read it) only on the simulator transports.

Raw microphone chunks

When you need raw PCM — your own on-device STT model, audio analysis, custom keyword spotting — subscribe to audioChunks:

import com.extentos.glasses.core.AudioChunkConfig

glasses.audio.audioChunks(
    AudioChunkConfig(
        chunkMillis = 20,    // 20ms is the standard opus/CELT frame — don't change unless your model needs it
        sampleRate = 16000,  // REQUESTED rate; see the caveat below
    ),
).collect { chunk ->
    // feed chunk.data (ByteArray) to your STT / VAD / classifier. Also chunk.sampleRate, chunk.timestampMs.
}

Use transcriptions() instead if you just want speech-to-text — the SDK's recognizer is cheaper and pre-routed (on Android hardware, on-device Vosk; on iOS hardware, Apple's SFSpeechRecognizer; the platform recognizer on the dev/sim path). audioChunks is for cases that genuinely need PCM.

sampleRate is a request, not a guarantee. Meta DAT delivers mic audio over the Bluetooth HFP link, which negotiates wideband speech (mSBC, 16 kHz mono) — on real hardware the effective rate is the HFP rate, not necessarily your config value. For any rate-sensitive DSP (FFT bins, VAD windows), read the per-chunk chunk.sampleRate rather than assuming the config value. The stream buffers with drop-oldest overflow — consumers that can't keep up lose frames rather than back-pressuring the mic.

audioChunks is gated by audio_capture_enabled and privacy_mode (but not listening_mode — it's a raw-audio path, not STT).

Bounded capture (record a turn)

For a one-shot "ask a question" capture, recordDiscrete(...) records until the user goes silent and returns the clip (rawAudioUri); on real hardware the transcript field is empty today — transcription there is your app's job (the browser sim populates it). It is covered with the voice flows: see getCapabilityGuide(feature: "record_audio") for the silence-VAD knobs and getCodeExample(pattern: "voice_notes") for a wake → record → persist composition.

Verify it in the simulator

The browser simulator drives transcription from your real laptop microphone and plays speak() output through your speakers, so you can rehearse the whole audio loop before hardware. recordDiscrete emits a record_audio_returned event (getEventLog(filter: "voice")). The audio chips in the event log are voice (transcripts, recordings, speech) — note there is no audio filter. The simulator is designed to behave like the glasses, but the audio substrate differs (24 kHz PCM in sim vs the glasses' wideband 16 kHz HFP) and that fidelity is under active validation — confirm audio behavior on real glasses.

Related

• Voice triggers — glasses.voice.onPhrase, the structured wrapper over transcriptions().
• Capture a photo — the vision-LLM flow that pairs capture with speak().
• Capabilities — the full audio capability catalog.
• Error reference — every AudioError variant.
• getCapabilityGuide(feature: "speak") / getCapabilityGuide(feature: "transcription_incremental") / getCapabilityGuide(feature: "audio_chunks") — call shapes + full gotchas.

Related