> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.empiriolabs.ai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.empiriolabs.ai/_mcp/server.

# Realtime Voice API

> Full-duplex speech-to-speech over WebSocket. Stream microphone audio to the model and receive its spoken reply on the same connection, interrupt it mid-response, and let server-side voice activity detection manage turns.

Realtime models hold a two-way voice conversation over a single WebSocket connection. You stream microphone audio to the model and receive its spoken reply on the same connection. Either side can speak at any time, so the model can be interrupted mid-response.

Text-to-speech sessions use the same endpoint the other way round: you stream text in, for example a language model's reply while it is being written, and receive speech while the text is still arriving. See [Text to speech](#text-to-speech).

> **Note**
>
> Realtime models use this WebSocket endpoint instead of HTTP. `stream: true` on the other endpoints streams output only and cannot carry microphone audio to the model.

## Endpoint

```text
wss://api.empiriolabs.ai/v1/realtime?model=<model>
```

Authenticate with your API key in the `Authorization` header of the WebSocket handshake:

```text
Authorization: Bearer $EMPIRIOLABS_API_KEY
```

> **Warning**
>
> Never put your API key in the query string or in browser code. To connect from a web page, use a client secret.

## Connect from a browser

A browser cannot set headers on a WebSocket connection, so a web page connects with a short-lived client secret instead of your API key.

On your server, create a client secret with [`POST /v1/realtime/client_secrets`](/api-reference/api-reference/audio/create-realtime-client-secret):

```bash
curl https://api.empiriolabs.ai/v1/realtime/client_secrets \
  -H "Authorization: Bearer $EMPIRIOLABS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-8-omni-flash-realtime", "expires_after": {"seconds": 600}}'
```

```json
{
  "object": "realtime.client_secret",
  "value": "ek_...",
  "expires_at": 1791414431,
  "model": "qwen3-8-omni-flash-realtime",
  "url": "wss://api.empiriolabs.ai/v1/realtime?model=qwen3-8-omni-flash-realtime"
}
```

Send `value` and `url` to the page. The page opens the socket with two subprotocols, then uses the events described below:

```js
const ws = new WebSocket(secret.url, ["empirio.realtime.v1", `empirio-token.${secret.value}`]);
```

* A client secret works only for the model it was created for.
* `expires_after.seconds` sets how long the secret can open new sessions: 10 to 7200 seconds, 600 by default. A session that is already open continues after the secret expires.
* Sessions bill to your account like any other API usage and appear in usage under the API key that created the secret. Deactivating or deleting that key stops its client secrets from opening new sessions.
* Creating a client secret counts as one request toward your account rate limit.
* If a session cannot start, for example because the secret has expired or the account has no remaining balance, the socket sends an `error` event with a `code` and `message`, then closes with code `1008`, or `1013` when you can retry shortly.
* The page sends session settings, such as instructions and voice, in `session.update`, so the page can read them. To keep instructions private, connect from your server and relay audio to the page instead.

## Models

Speech-to-speech:

| Model                                                                            | Conversation languages                                                       | Best for                                                                                  |
| -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| [`gemini-3-8-live`](/models/gemini-3-8-live)                                     | Multilingual; the model replies in the language you speak                    | Low-latency conversations with live video input, 30 voices and tool calling               |
| [`gemini-3-8-live-extended-thinking`](/models/gemini-3-8-live-extended-thinking) | Multilingual; the model replies in the language you speak                    | Conversations that need deeper reasoning, set with `reasoning_effort`                     |
| [`stepaudio-3-realtime`](/models/stepaudio-3-realtime)                           | Chinese, English                                                             | Latest StepAudio generation, with adaptive reasoning during the conversation              |
| [`stepaudio-2-5-realtime`](/models/stepaudio-2-5-realtime)                       | Chinese, English                                                             | Paralinguistic understanding of tone, pacing and hesitation                               |
| [`qwen3-8-omni-flash-realtime`](/models/qwen3-8-omni-flash-realtime)             | Multilingual, 36 languages and dialects for speech output                    | Conversations with live video input, 56 voices and tool calling                           |
| [`qwen-audio-3-1-realtime-plus`](/models/qwen-audio-3-1-realtime-plus)           | Multilingual                                                                 | Conversations that need web search, with 27 voices                                        |
| [`qwen3-5-omni-flash-realtime`](/models/qwen3-5-omni-flash-realtime)             | Multilingual, including Spanish, French, German, Japanese, Korean and Arabic | Conversations that switch between languages, with text and audio tokens priced separately |
| [`qwen3-5-omni-plus-realtime`](/models/qwen3-5-omni-plus-realtime)               | Multilingual, including Spanish, French, German, Japanese, Korean and Arabic | The larger Omni tier, at a higher rate than Flash                                         |

Speech translation:

| Model                                                                                  | Conversation languages                                                                         | Best for                                                                                             |
| -------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| [`qwen3-8-livetranslate-flash-realtime`](/models/qwen3-8-livetranslate-flash-realtime) | Multilingual, including Chinese, English, Spanish, French, German, Japanese, Korean and Arabic | Simultaneous spoken translation. Set the target language with `session.update` before you send audio |

Live transcription:

| Model                                                              | Purpose                                                                                                       |
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------- |
| [`stepaudio-2-5-asr-stream`](/models/stepaudio-2-5-asr-stream)     | Live transcription. Transcript events arrive while the speaker is still talking                               |
| [`deepgram-nova-3-stream`](/models/deepgram-nova-3-stream)         | Live transcription with interim results, billed per minute of audio sent                                      |
| [`openai-whisper-1`](/models/openai-whisper-1)                     | Live transcription with the same model used for file transcription, billed per minute in both cases           |
| [`qwen-audio-3-1-asr-stream`](/models/qwen-audio-3-1-asr-stream)   | Live captions. Partial words arrive while the speaker is talking, and the settled sentence at each pause      |
| [`qwen-audio-3-1-asr-message`](/models/qwen-audio-3-1-asr-message) | Voice messages and voice input. Each utterance arrives whole once the speaker pauses, with no partial results |

Text to speech:

| Model                                              | Languages                                                                               | Best for                                                                                                                  |
| -------------------------------------------------- | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| [`qwen-audio-3-1-tts`](/models/qwen-audio-3-1-tts) | English with 15 voices; Chinese, English and other languages with 4 multilingual voices | Speaking text while it is still being written, with natural-language delivery direction and inline emotion and sound tags |

Each transcription model accepts audio differently, so check this before you build a client. `stepaudio-2-5-asr-stream` follows the event protocol below: base64 audio inside `input_audio_buffer.append`. `deepgram-nova-3-stream` takes **raw binary frames** of 16 kHz mono 16-bit PCM, sent directly over the socket without a JSON wrapper, and returns Deepgram `Results` events carrying `is_final`. Add `&language=` to the connection URL to set the language. `openai-whisper-1` follows the event protocol but requires **24 kHz** audio and rejects 16 kHz. `qwen-audio-3-1-asr-stream` and `qwen-audio-3-1-asr-message` follow the event protocol with 16 kHz audio: transcripts arrive as `conversation.item.input_audio_transcription.delta` while the speaker talks (stream only) and `conversation.item.input_audio_transcription.completed` when each sentence settles. Do not send a commit: voice activity detection determines when each phrase ends.

## Your first session

```python
import asyncio, json, os, websockets

URL = "wss://api.empiriolabs.ai/v1/realtime?model=stepaudio-3-realtime"

async def main():
    async with websockets.connect(
        URL,
        additional_headers=[("Authorization", f"Bearer {os.environ['EMPIRIOLABS_API_KEY']}")],
        max_size=None,
    ) as ws:
        print(json.loads(await ws.recv())["type"])          # session.created

        await ws.send(json.dumps({
            "type": "session.update",
            "session": {"voice": "lively-girl",
                        "instructions": "You are a warm, concise support agent."},
        }))

        await ws.send(json.dumps({
            "type": "conversation.item.create",
            "item": {"type": "message", "role": "user",
                     "content": [{"type": "input_text", "text": "What can you help me with?"}]},
        }))
        await ws.send(json.dumps({"type": "response.create"}))

        async for raw in ws:
            event = json.loads(raw)
            if event["type"] == "response.audio.delta":
                ...  # base64 pcm16 chunk, append to your playback buffer
            elif event["type"] == "response.done":
                break

asyncio.run(main())
```

## Text to speech

Connect with `?model=qwen-audio-3-1-tts`, send text with `input_text_buffer.append` in pieces of any size, and read audio from `response.audio.delta`. Speech starts while the text is still arriving.

```python
import asyncio, base64, json, os, websockets

URL = "wss://api.empiriolabs.ai/v1/realtime?model=qwen-audio-3-1-tts"

async def main():
    async with websockets.connect(
        URL,
        additional_headers=[("Authorization", f"Bearer {os.environ['EMPIRIOLABS_API_KEY']}")],
        max_size=None,
    ) as ws:
        print(json.loads(await ws.recv())["type"])          # session.created

        await ws.send(json.dumps({
            "type": "session.update",
            "session": {"voice": "Ava_v3.1", "instruction": "Warm, like a late-night radio host"},
        }))

        async def speak():
            for piece in ["Good evening, ", "and welcome to the late edition. ", "Here is tonight's news."]:
                await ws.send(json.dumps({"type": "input_text_buffer.append", "text": piece}))
            await ws.send(json.dumps({"type": "session.finish"}))

        sender = asyncio.create_task(speak())
        audio = bytearray()
        async for raw in ws:
            event = json.loads(raw)
            if event["type"] == "response.audio.delta":
                audio.extend(base64.b64decode(event["delta"]))   # 16-bit PCM, 24,000 Hz, mono
            elif event["type"] == "session.finished":
                break
        await sender

asyncio.run(main())
```

**Sending text.** `input_text_buffer.append` adds text. `input_text_buffer.commit` speaks everything sent so far, and `session.finish` speaks the rest and ends the session with `session.finished`. Inline tags such as `[excited]` or `[whispers]` work as they do on [`POST /v1/audio/speech`](/models/qwen-audio-3-1-tts); SSML does not.

**Session settings.** `session.update` accepts `voice`, `instruction`, `response_format` (`pcm`, the default, `mp3` or `opus`), `sample_rate` (24,000 Hz by default), `speed`, `pitch`, `volume`, `seed`, `language_hints`, `pronunciation`, `replace` and `enable_markdown_filter`, with the same values as the HTTP endpoint. A change applies from the next segment.

**Segments.** Text is spoken in segments that end at sentence boundaries. Each segment starts with `response.created` and ends with `response.audio.done` and a `response.done` event carrying its usage. A segment also ends after 8 seconds without new text, and the delivery tag in effect carries into the next segment.

## Audio format

Audio is exchanged in both directions as base64-encoded, 16-bit signed little-endian PCM. Send microphone audio with `input_audio_buffer.append` and read model audio from `response.audio.delta`.

The format identifier for the same bytes differs by model: StepAudio models use `pcm16`, the Qwen3.5 Omni models use `pcm`, and `qwen3-8-omni-flash-realtime` and `qwen-audio-3-1-realtime-plus` report `pcm16` for input and `pcm24` for output. The Gemini 3.8 Live models accept `pcm16` (16 kHz) or `pcm24` (24 kHz) input, set with `input_audio_format`, and return `pcm24`. The `session.created` event reports the session's `input_audio_format` and `output_audio_format`; read the defaults from it rather than hard-coding them.

## Video frames

`gemini-3-8-live`, `gemini-3-8-live-extended-thinking` and `qwen3-8-omni-flash-realtime` accept video frames alongside audio. Send frames as base64 JPEG or PNG with `input_image_buffer.append` while you append audio, and ask about them in the same turn:

```json
{"type": "input_image_buffer.append", "image": "<base64 frame>"}
```

Frames use their own buffer, not a conversation item. On `qwen3-8-omni-flash-realtime` the model retains up to 50 turns and 240 seconds of video history, and up to 100 turns and 600 seconds of audio. The other realtime models on this page accept audio only.

## Voices

Set the voice with `session.update` before the model produces any audio; it cannot be changed after the model has spoken. Each model accepts its own list of voices. An unsupported value is rejected when the model first responds, not when you set it, so verify the first response after changing voices.

StepAudio models (`stepaudio-3-realtime`, `stepaudio-2-5-realtime`):

| Voice ID                | Voice                  |
| ----------------------- | ---------------------- |
| `lively-girl`           | Bright, lively female  |
| `livelybreezy-female`   | Light, breezy female   |
| `elegantgentle-female`  | Elegant, gentle female |
| `soft-spoken-gentleman` | Calm, gentle male      |
| `magnetic-voiced-male`  | Deep, magnetic male    |
| `vibrant-youth`         | Youthful, energetic    |
| `zixinnansheng`         | Confident male         |

Qwen models (`qwen3-5-omni-flash-realtime`, `qwen3-5-omni-plus-realtime`), which default to `Tina`:

| Voice ID |
| -------- |
| `Tina`   |
| `Serena` |
| `Ethan`  |
| `Dylan`  |
| `Sunny`  |
| `Peter`  |
| `Kiki`   |
| `Eric`   |

LiveTranslate (`qwen3-8-livetranslate-flash-realtime`) uses `Tina` by default:

| Voice ID |
| -------- |
| `Tina`   |
| `Serena` |
| `Ethan`  |
| `Cindy`  |

`qwen3-8-omni-flash-realtime` has its own set of 56 voices and defaults to `Tina`. The set differs from the Qwen3.5 Omni list: `Ethan` is not available, and `Cici`, `Zane` and `longanlingxin` are:

| Voice ID                                                                                                                    |
| --------------------------------------------------------------------------------------------------------------------------- |
| `Tina`, `Serena`, `Maia`, `Mia`, `Cindy`, `Cici`, `Qiao`, `Momo`, `Angel`, `Mione`, `Katerina`, `Jennifer`, `Liora Mira`    |
| `Raymond`, `Zane`, `Ryan`, `Evan`, `Wil`, `Aiden`, `Gold`, `Joyner`, `Theo Calm`, `Li Cassian`                              |
| `Sunny`, `Dylan`, `Eric`, `Peter`, `Marcus`, `Li`, `Rocky`, `Kiki`, `Joseph Chen`, `longanlingxin`                          |
| `Sohee`, `Ono Anna`, `Rizky`, `Hana`, `Arda`, `Roya`, `Alek`, `Jakub`, `Eliška`, `Griet`                                    |
| `Sonrisa`, `Bodega`, `Andre`, `Radio Gol`, `Dolce`, `Lenn`, `Marina`, `Siiri`, `Ingrid`, `Sigga`, `Bea`, `Chloe`, `Emilien` |

`qwen-audio-3-1-realtime-plus` has a separate set and defaults to `longanqian_v3.1`:

| Voice ID                                                                                                           |
| ------------------------------------------------------------------------------------------------------------------ |
| `longanqian_v3.1`, `longanhuan_v3.1`, `longanlingxin_v3.1`, `longanfengyue_v3.1`, `xunanchuan_v3.1`                |
| `beth_v3.1`, `betty_v3.1`, `cally_v3.1`, `daniel`, `echo`, `hannah`, `sherry`, `loongmary`, `loongjohn`            |
| `longanqian`, `longanlingxin`, `longanlufeng`, `longanlingxi`, `longanxiaoxin`, `longanfengyue`, `longanyuanfei`   |
| `longanhuan_v3.6`, `longjielidou_v3.6`, `longpaopao_v3.6`, `longhuohuo_v3.6`, `longchuanshu_v3.6`, `loongeva_v3.6` |

Both models report `Chelsie` as the session voice on connect but do not accept it. Set a voice from the lists above, or leave it unset to use the model's default.

The Gemini 3.8 Live models (`gemini-3-8-live`, `gemini-3-8-live-extended-thinking`) share 30 voices and default to `Puck`. Any other value is refused as soon as you set it:

| Voice ID                                                                                                                       |
| ------------------------------------------------------------------------------------------------------------------------------ |
| `Zephyr`, `Puck`, `Charon`, `Kore`, `Fenrir`, `Leda`, `Orus`, `Aoede`, `Callirrhoe`, `Autonoe`                                 |
| `Enceladus`, `Iapetus`, `Umbriel`, `Algieba`, `Despina`, `Erinome`, `Algenib`, `Rasalgethi`, `Laomedeia`, `Achernar`           |
| `Alnilam`, `Schedar`, `Gacrux`, `Pulcherrima`, `Achird`, `Zubenelgenubi`, `Vindemiatrix`, `Sadachbia`, `Sadaltager`, `Sulafat` |

**Session settings on Gemini 3.8 Live.** The voice, instructions, turn detection, tools, temperature and `reasoning_effort` are fixed once the conversation starts. Send them in a `session.update` before the first audio, video frame or message.

## Reasoning

`gemini-3-8-live-extended-thinking` reasons in the background while it talks. Set how much with `reasoning_effort` before the conversation starts: `low`, `medium` (the default) or `high`.

```json
{"type": "session.update", "session": {"reasoning_effort": "high"}}
```

The model often acknowledges a question first and answers in a second response. Each response arrives with its own `response.done` and is billed on its own.

## Function calling

`gemini-3-8-live`, `gemini-3-8-live-extended-thinking`, `qwen3-8-omni-flash-realtime` and `qwen-audio-3-1-realtime-plus` can call functions you declare. Declare them in `session.update` with JSON Schema parameters:

```json
{"type": "session.update", "session": {"tools": [{"type": "function", "name": "get_weather",
  "description": "Get the current weather for a city.",
  "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}]}}
```

When the model calls a function you receive `response.function_call_arguments.done` with its `call_id`, `name` and JSON `arguments`. Run the function, return the result as a `function_call_output` item, then send `response.create`:

```json
{"type": "conversation.item.create", "item": {"type": "function_call_output", "call_id": "<call_id>",
  "output": "{\"forecast\": \"light rain\", \"temperature_c\": 12}"}}
```

On the Gemini 3.8 Live models, `tool_choice` accepts `auto` or `none`.

## Web search

`qwen-audio-3-1-realtime-plus` can search the web during a conversation. Web search is off by default; enable it with `session.update`:

```json
{"type": "session.update", "session": {"enable_search": true}}
```

Web search and function calling cannot be enabled in the same session. Search results are added to the conversation and count as input tokens, so a turn that searches reports higher `input_tokens`. The other models on this page do not support web search.

## Events

The protocol follows the widely used realtime event format, so clients built for that format work here.

| Direction | Event                                                   | Meaning                                                                       |
| --------- | ------------------------------------------------------- | ----------------------------------------------------------------------------- |
| Server    | `session.created`                                       | Connection is ready; carries the session defaults                             |
| Client    | `session.update`                                        | Set voice, instructions, modalities, turn detection                           |
| Client    | `input_audio_buffer.append`                             | Append a base64 pcm16 chunk of microphone audio                               |
| Client    | `input_image_buffer.append`                             | Append a base64 JPEG or PNG video frame (models that accept video)            |
| Client    | `conversation.item.create`                              | Add a text turn, or return a `function_call_output`                           |
| Client    | `response.create`                                       | Request a response from the model                                             |
| Server    | `response.audio.delta`                                  | A base64 pcm16 chunk of the reply                                             |
| Server    | `response.audio_transcript.delta`                       | Text transcript of what the model is saying (conversation models)             |
| Server    | `response.audio_transcript.text`                        | Translated text on LiveTranslate (this model also emits `response.text.text`) |
| Server    | `conversation.item.input_audio_transcription.completed` | Transcript of what the caller said                                            |
| Server    | `response.function_call_arguments.done`                 | The model called a function; carries the `call_id`, `name` and `arguments`    |
| Server    | `response.done`                                         | The turn is complete; includes that turn's token usage                        |
| Server    | `error`                                                 | A request was rejected; the session remains open                              |
| Client    | `input_text_buffer.append`                              | Add text to speak (text-to-speech sessions)                                   |
| Client    | `input_text_buffer.commit`                              | Speak everything sent so far (text-to-speech sessions)                        |
| Client    | `session.finish`                                        | Speak the rest, then end the session (text-to-speech sessions)                |
| Server    | `input_text_buffer.committed`                           | The committed text has been spoken                                            |
| Server    | `session.finished`                                      | The session has ended after `session.finish`                                  |

## Turn taking

**Turn detection is off by default on the conversation models.** `session.created` reports `turn_detection: null` on the StepAudio and Qwen Omni realtime models, and a session that only appends audio is transcribed but never answered. Choose one of the two modes below. LiveTranslate starts with turn detection enabled, so do not also send `input_audio_buffer.commit` and `response.create`. The Gemini 3.8 Live models also start with turn detection enabled; set `turn_detection` to null before the conversation starts to manage turns yourself.

```json
{"type": "session.update", "session": {"turn_detection": {"type": "server_vad"}}}
```

With `server_vad` enabled, keep streaming audio; the model detects when the speaker has finished and responds automatically. Detection requires a short period of trailing silence, so keep sending audio briefly after speech ends. Speaking while the model is responding interrupts it.

To manage turns manually, leave turn detection off and send `input_audio_buffer.commit` followed by `response.create` when the turn ends.

## Billing

Speech-to-speech models bill each completed turn separately, based on the token usage reported in that turn's `response.done`, at the model's published input and output rates. The Qwen and Gemini models, including LiveTranslate, price text and audio tokens separately, so a spoken turn and a typed turn of the same length cost different amounts. On the Gemini 3.8 Live models, video frames bill as input tokens, reasoning bills as output tokens, and a reply you interrupt is billed for the part the model generated.

Most transcription models bill by the duration of audio sent, at each model's published rate: `stepaudio-2-5-asr-stream` per hour, `deepgram-nova-3-stream` per minute. `qwen-audio-3-1-asr-stream` and `qwen-audio-3-1-asr-message` bill by tokens instead: each completed sentence is billed on its own from the usage the model reports, at the published input and output rates. An open connection is not billed while no audio is sent.

`qwen-audio-3-1-tts` bills each segment from the usage in its `response.done`: input tokens for the text and output tokens for the audio, at the model's published rates.

There is no charge for keeping a session open; you pay only for completed turns and the audio you send. Current rates are on each model page and on the [pricing page](https://empiriolabs.ai/pricing).

## Limits

* Conversation languages differ by model; see the table above.
* A closed socket ends the session. Reconnecting starts a new one with no memory of the previous conversation, so keep your own transcript if you need continuity.
* A close with code `1012` or `1001` means you can reconnect right away. From a browser, reconnect with the same client secret while it is unexpired, or create a new one.
* One connection carries one conversation. Open a socket per concurrent caller.