Realtime Voice API

Realtime models hold a two-way voice conversation over a single WebSocket connection. You stream microphone audio to the model and receive its spoken reply on the same connection. Either side can speak at any time, so the model can be interrupted mid-response.

Endpoint

wss://api.empiriolabs.ai/v1/realtime?model=<model>

Authenticate with your API key in the Authorization header of the WebSocket handshake:

Authorization: Bearer $EMPIRIOLABS_API_KEY

Connect from a browser

A browser cannot set headers on a WebSocket connection, so a web page connects with a short-lived client secret instead of your API key.

On your server, create a client secret with POST /v1/realtime/client_secrets:

curl https://api.empiriolabs.ai/v1/realtime/client_secrets \
  -H "Authorization: Bearer $EMPIRIOLABS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-8-omni-flash-realtime", "expires_after": {"seconds": 600}}'

{
  "object": "realtime.client_secret",
  "value": "ek_...",
  "expires_at": 1791414431,
  "model": "qwen3-8-omni-flash-realtime",
  "url": "wss://api.empiriolabs.ai/v1/realtime?model=qwen3-8-omni-flash-realtime"
}

Send value and url to the page. The page opens the socket with two subprotocols, then uses the events described below:

const ws = new WebSocket(secret.url, ["empirio.realtime.v1", `empirio-token.${secret.value}`]);

• A client secret works only for the model it was created for.
• expires_after.seconds sets how long the secret can open new sessions: 10 to 7200 seconds, 600 by default. A session that is already open continues after the secret expires.
• Sessions bill to your account like any other API usage and appear in usage under the API key that created the secret. Deactivating or deleting that key stops its client secrets from opening new sessions.
• Creating a client secret counts as one request toward your account rate limit.
• If a session cannot start, for example because the secret has expired or the account has no remaining balance, the socket sends an error event with a code and message, then closes with code 1008, or 1013 when you can retry shortly.
• The page sends session settings, such as instructions and voice, in session.update, so the page can read them. To keep instructions private, connect from your server and relay audio to the page instead.

Models

Speech-to-speech:

Speech translation:

Live transcription:

Each transcription model accepts audio differently, so check this before you build a client. stepaudio-2-5-asr-stream follows the event protocol below: base64 audio inside input_audio_buffer.append. deepgram-nova-3-stream takes raw binary frames of 16 kHz mono 16-bit PCM, sent directly over the socket without a JSON wrapper, and returns Deepgram Results events carrying is_final. Add &language= to the connection URL to set the language. openai-whisper-1 follows the event protocol but requires 24 kHz audio and rejects 16 kHz. qwen-audio-3-1-asr-stream and qwen-audio-3-1-asr-message follow the event protocol with 16 kHz audio: transcripts arrive as conversation.item.input_audio_transcription.delta while the speaker talks (stream only) and conversation.item.input_audio_transcription.completed when each sentence settles. Do not send a commit: voice activity detection determines when each phrase ends.

Your first session

import asyncio, json, os, websockets
URL = "wss://api.empiriolabs.ai/v1/realtime?model=stepaudio-3-realtime"
async def main():
    async with websockets.connect(
        URL,
        additional_headers=[("Authorization", f"Bearer {os.environ['EMPIRIOLABS_API_KEY']}")],
        max_size=None,
    ) as ws:
        print(json.loads(await ws.recv())["type"])          # session.created
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {"voice": "lively-girl",
                        "instructions": "You are a warm, concise support agent."},
        }))
        await ws.send(json.dumps({
            "type": "conversation.item.create",
            "item": {"type": "message", "role": "user",
                     "content": [{"type": "input_text", "text": "What can you help me with?"}]},
        }))
        await ws.send(json.dumps({"type": "response.create"}))
        async for raw in ws:
            event = json.loads(raw)
            if event["type"] == "response.audio.delta":
                ...  # base64 pcm16 chunk, append to your playback buffer
            elif event["type"] == "response.done":
                break
asyncio.run(main())

Audio format

Audio is exchanged in both directions as base64-encoded, 16-bit signed little-endian PCM. Send microphone audio with input_audio_buffer.append and read model audio from response.audio.delta.

The format identifier for the same bytes differs by model: StepAudio models use pcm16, the Qwen3.5 Omni models use pcm, and qwen3-8-omni-flash-realtime and qwen-audio-3-1-realtime-plus report pcm16 for input and pcm24 for output. The Gemini 3.8 Live models accept pcm16 (16 kHz) or pcm24 (24 kHz) input, set with input_audio_format, and return pcm24. The session.created event reports the session’s input_audio_format and output_audio_format; read the defaults from it rather than hard-coding them.

Video frames

gemini-3-8-live, gemini-3-8-live-extended-thinking and qwen3-8-omni-flash-realtime accept video frames alongside audio. Send frames as base64 JPEG or PNG with input_image_buffer.append while you append audio, and ask about them in the same turn:

{"type": "input_image_buffer.append", "image": "<base64 frame>"}

Frames use their own buffer, not a conversation item. On qwen3-8-omni-flash-realtime the model retains up to 50 turns and 240 seconds of video history, and up to 100 turns and 600 seconds of audio. The other realtime models on this page accept audio only.

Voices

Set the voice with session.update before the model produces any audio; it cannot be changed after the model has spoken. Each model accepts its own list of voices. An unsupported value is rejected when the model first responds, not when you set it, so verify the first response after changing voices.

StepAudio models (stepaudio-3-realtime, stepaudio-2-5-realtime):

Qwen models (qwen3-5-omni-flash-realtime, qwen3-5-omni-plus-realtime), which default to Tina:

LiveTranslate (qwen3-8-livetranslate-flash-realtime) uses Tina by default:

qwen3-8-omni-flash-realtime has its own set of 56 voices and defaults to Tina. The set differs from the Qwen3.5 Omni list: Ethan is not available, and Cici, Zane and longanlingxin are:

qwen-audio-3-1-realtime-plus has a separate set and defaults to longanqian_v3.1:

Both models report Chelsie as the session voice on connect but do not accept it. Set a voice from the lists above, or leave it unset to use the model’s default.

The Gemini 3.8 Live models (gemini-3-8-live, gemini-3-8-live-extended-thinking) share 30 voices and default to Puck. Any other value is refused as soon as you set it:

Session settings on Gemini 3.8 Live. The voice, instructions, turn detection, tools, temperature and reasoning_effort are fixed once the conversation starts. Send them in a session.update before the first audio, video frame or message.

Reasoning

gemini-3-8-live-extended-thinking reasons in the background while it talks. Set how much with reasoning_effort before the conversation starts: low, medium (the default) or high.

{"type": "session.update", "session": {"reasoning_effort": "high"}}

The model often acknowledges a question first and answers in a second response. Each response arrives with its own response.done and is billed on its own.

Function calling

gemini-3-8-live, gemini-3-8-live-extended-thinking, qwen3-8-omni-flash-realtime and qwen-audio-3-1-realtime-plus can call functions you declare. Declare them in session.update with JSON Schema parameters:

{"type": "session.update", "session": {"tools": [{"type": "function", "name": "get_weather",
  "description": "Get the current weather for a city.",
  "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}]}}

When the model calls a function you receive response.function_call_arguments.done with its call_id, name and JSON arguments. Run the function, return the result as a function_call_output item, then send response.create:

{"type": "conversation.item.create", "item": {"type": "function_call_output", "call_id": "<call_id>",
  "output": "{\"forecast\": \"light rain\", \"temperature_c\": 12}"}}

On the Gemini 3.8 Live models, tool_choice accepts auto or none.

Web search

qwen-audio-3-1-realtime-plus can search the web during a conversation. Web search is off by default; enable it with session.update:

{"type": "session.update", "session": {"enable_search": true}}

Web search and function calling cannot be enabled in the same session. Search results are added to the conversation and count as input tokens, so a turn that searches reports higher input_tokens. The other models on this page do not support web search.

Events

The protocol follows the widely used realtime event format, so clients built for that format work here.

Turn taking

Turn detection is off by default on the conversation models. session.created reports turn_detection: null on the StepAudio and Qwen Omni realtime models, and a session that only appends audio is transcribed but never answered. Choose one of the two modes below. LiveTranslate starts with turn detection enabled, so do not also send input_audio_buffer.commit and response.create. The Gemini 3.8 Live models also start with turn detection enabled; set turn_detection to null before the conversation starts to manage turns yourself.

{"type": "session.update", "session": {"turn_detection": {"type": "server_vad"}}}

With server_vad enabled, keep streaming audio; the model detects when the speaker has finished and responds automatically. Detection requires a short period of trailing silence, so keep sending audio briefly after speech ends. Speaking while the model is responding interrupts it.

To manage turns manually, leave turn detection off and send input_audio_buffer.commit followed by response.create when the turn ends.

Billing

Speech-to-speech models bill each completed turn separately, based on the token usage reported in that turn’s response.done, at the model’s published input and output rates. The Qwen and Gemini models, including LiveTranslate, price text and audio tokens separately, so a spoken turn and a typed turn of the same length cost different amounts. On the Gemini 3.8 Live models, video frames bill as input tokens, reasoning bills as output tokens, and a reply you interrupt is billed for the part the model generated.

Most transcription models bill by the duration of audio sent, at each model’s published rate: stepaudio-2-5-asr-stream per hour, deepgram-nova-3-stream per minute. qwen-audio-3-1-asr-stream and qwen-audio-3-1-asr-message bill by tokens instead: each completed sentence is billed on its own from the usage the model reports, at the published input and output rates. An open connection is not billed while no audio is sent.

There is no charge for keeping a session open; you pay only for completed turns and the audio you send. Current rates are on each model page and on the pricing page.

Limits

• Conversation languages differ by model; see the table above.
• A closed socket ends the session. Reconnecting starts a new one with no memory of the previous conversation, so keep your own transcript if you need continuity.
• A close with code 1012 or 1001 means you can reconnect right away. From a browser, reconnect with the same client secret while it is unexpired, or create a new one.
• One connection carries one conversation. Open a socket per concurrent caller.