Realtime Voice API | EmpirioLabs AI Docs

Realtime models hold a two-way voice conversation over a single WebSocket. You stream audio up, the model streams audio back, and either side can talk at any moment, so the model can be interrupted mid-sentence the way a person can.

This is the only EmpirioLabs model surface that is not HTTP. Streaming with stream: true on the regular endpoints sends data one way, from us to you, so it can deliver generated audio but has no channel to carry your microphone audio up. Full duplex needs a socket.

Endpoint

wss://api.empiriolabs.ai/v1/realtime?model=<model>

Authenticate with the same API key you use everywhere else, as an Authorization header on the handshake:

Authorization: Bearer $EMPIRIOLABS_API_KEY

Browsers cannot set headers on a WebSocket, so a browser cannot open this connection directly. Connect from your server and relay audio to the browser over your own socket. Do not put your API key in the query string or ship it to a browser.

Models

Model	Best for
`stepaudio-3-realtime`	Latest generation, adaptive reasoning during the conversation
`stepaudio-2-5-realtime`	Paralinguistic understanding of tone, pacing and hesitation

Conversation is available in Chinese and English.

Your first session

import asyncio, json, os, websockets
URL = "wss://api.empiriolabs.ai/v1/realtime?model=stepaudio-3-realtime"
async def main():
    async with websockets.connect(
        URL,
        additional_headers=[("Authorization", f"Bearer {os.environ['EMPIRIOLABS_API_KEY']}")],
        max_size=None,
    ) as ws:
        print(json.loads(await ws.recv())["type"])          # session.created
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {"voice": "lively-girl",
                        "instructions": "You are a warm, concise support agent."},
        }))
        await ws.send(json.dumps({
            "type": "conversation.item.create",
            "item": {"type": "message", "role": "user",
                     "content": [{"type": "input_text", "text": "What can you help me with?"}]},
        }))
        await ws.send(json.dumps({"type": "response.create"}))
        async for raw in ws:
            event = json.loads(raw)
            if event["type"] == "response.audio.delta":
                ...  # base64 pcm16 chunk, append to your playback buffer
            elif event["type"] == "response.done":
                break
asyncio.run(main())

Audio format

Audio travels in both directions as base64-encoded pcm16: 16-bit signed little-endian PCM. Send microphone audio with input_audio_buffer.append and read model audio from response.audio.delta.

Voices

Set the voice with session.update before the model produces any audio. It cannot be changed once the model has spoken, and any value outside this list is rejected.

Voice ID	Voice
`lively-girl`	Bright, lively female
`livelybreezy-female`	Light, breezy female
`elegantgentle-female`	Elegant, gentle female
`soft-spoken-gentleman`	Calm, gentle male
`magnetic-voiced-male`	Deep, magnetic male
`vibrant-youth`	Youthful, energetic
`zixinnansheng`	Confident male

Events

The protocol follows the widely used realtime event shape, so a client written against that convention works here.

Direction	Event	Meaning
Server	`session.created`	Connection is ready; carries the session defaults
Client	`session.update`	Set voice, instructions, modalities, turn detection
Client	`input_audio_buffer.append`	Append a base64 pcm16 chunk of microphone audio
Client	`conversation.item.create`	Add a text or audio turn
Client	`response.create`	Ask the model to reply
Server	`response.audio.delta`	A base64 pcm16 chunk of the reply
Server	`response.audio_transcript.delta`	Text transcript of what the model is saying
Server	`response.done`	The turn finished; carries that turn’s token usage
Server	`error`	Something was rejected; the session stays open

Turn taking

Server-side voice activity detection decides when you have stopped speaking and when the model should reply. Speaking while the model is talking interrupts it, which is what makes the conversation feel natural rather than walkie-talkie.

Billing

Each completed turn is billed on its own, from the token usage reported in that turn’s response.done, at the model’s published input and output rates. Holding a session open costs nothing by itself; you pay for the turns you generate. Live rates are on each model page and on pricing.

Limits

Conversation is Chinese and English only.
A closed socket ends the session. Reconnecting starts a new one with no memory of the previous conversation, so keep your own transcript if you need continuity.
One connection carries one conversation. Open a socket per concurrent caller.