Bring Your Own LLM (BYOL)

Overview

BYOL lets you replace the default platform LLM with any inference engine you control — an on-premise model, a fine-tuned model, a Google ADK agent, a LangGraph workflow, or anything else that can speak the Vaani WebSocket protocol. When BYOL is enabled for an agent, every response turn is routed to your WebSocket server instead of the built-in provider (OpenAI, Google, Groq, etc.). The rest of the pipeline — STT, TTS, telephony, transcripts, Langfuse tracing — stays exactly the same.

How to enable BYOL

Open your agent in the dashboard.
Go to Brain → Reasoning Language Model (LLM).
Switch to the Bring your Own LLM (BYOL) tab.
Paste your WebSocket URL (e.g. wss://your-server.example.com/chat/stream).
(Optional) Enter an Auth Token in the Auth Token field. When set, Vaani sends Authorization: Bearer <token> as an HTTP header during the WebSocket upgrade handshake. The token is stored Fernet-encrypted at rest and only decrypted at call time — the UI always displays a masked value (****<last4>). Leave blank for unauthenticated connections.
Click Test Connection to verify reachability, then Save URL.
Choose a Fallback LLM — either No fallback or Use platform LLM.

The URL is stored under agent_config.persona.senses_capabilities.brain.llm.extra_params.llm_websocket_url and takes effect immediately on the next call.

WebSocket protocol

Your server must implement the following JSON message exchange over a persistent WebSocket connection. Vaani opens one connection per call (identified by session_id / room name) and sends one request per agent turn.

Connection handshake

Immediately after the WebSocket is accepted, your server must send two JSON frames in order:

{ "interaction_type": "config",    "content": "Server ready" }
{ "interaction_type": "greeting",  "content": "Hello" }

These frames are consumed by the Vaani agent and discarded — they are only used to confirm the connection is live. The content strings may be anything.

Agent → Your server (request)

For every agent turn (after the user finishes speaking) Vaani sends a JSON frame over the WebSocket. Here is a real example captured from a live call:

{
  "interaction_type": "response_required",
  "response_id": 2,
  "call_id": "outbound-1782291018-65b6af82",
  "transcript": [
    {
      "role": "system",
      "content": "You are speaking with {{customer_name}}, who has a {{account_type}} account valid until {{subscription_end}}.\n\n[... full agent system prompt ...]"
    },
    { "role": "assistant", "content": "I am at your service, how can I help?" },
    { "role": "user", "content": "What is this call regarding?" },
    { "role": "user", "content": "What is this call regarding?" }
  ],
  "req_body": {
    "agent_id": "535d0c34-8086-419e-b39d-ee549fc28e93",
    "medium": "telephony",
    "contact_number": "+917893209830",
    "name": "John",
    "voice": "",
    "dnd_check_skipped": true,
    "voice_gender": "female",
    "primary_language": "hi",
    "secondary_language": "en",
    "welcome_interruptible": true,
    "bg_noise_enabled": false,
    "bg_noise_volume": 60,
    "voice_speed": 1.0,
    "x_agent_id": "my-custom-agent-id",
    "modify_agent": {
      "persona": {
        "metadata": {
          "customer_name": "John Doe",
          "account_type": "Premium",
          "subscription_end": "Dec 31, 2026"
        },
        "identity": {
          "system_prompt": "You are speaking with {{customer_name}}, who has a {{account_type}} account valid until {{subscription_end}}."
        }
      },
      "training": null,
      "experience": null,
      "analysis": null
    }
  }
}

Field	Type	Description
`interaction_type`	`"response_required"`	Always this value for a normal turn
`response_id`	integer	Monotonically increasing; echoed back in every response chunk
`call_id`	string	The room/call identifier for this session
`transcript`	array	Full conversation history for this call, including the system prompt as the first `"system"` message
`req_body`	object or `null`	Original trigger-call request body (all top-level fields including `modify_agent`); `null` if the call was not triggered via the external API
`req_body.x_agent_id`	string or `null`	Value of the `X-Agent-Id` header passed to `/trigger-call/`; `null` or absent if the header was not sent

The system prompt built from your agent configuration is always the first entry with "role": "system". Your server must apply it as the LLM’s instruction/system message for each turn. If you create an ADK LlmAgent or a LangChain chain, pass this text as the agent instruction or system message so the configured persona is honoured.

req_body.modify_agent.persona.metadata contains the per-call template variables (e.g. customer_name, account_type) passed by the caller. If your system prompt uses {{ variable }} placeholders, populate your LLM session state from this object so the placeholders resolve correctly.

Your server → Agent (streaming response)

Stream back one or more chunks, each as a JSON frame:

{
  "response_type": "response",
  "response_id": 1,
  "content": "Yes, we carry ",
  "content_complete": false
}

Send a final frame with "content_complete": true and optionally empty content to signal end of turn:

{
  "response_type": "response",
  "response_id": 1,
  "content": "beautiful Pelikan M800 models.",
  "content_complete": true
}

Field	Type	Description
`response_type`	`"response"`	Always this value
`response_id`	integer	Must match the `response_id` from the request
`content`	string	Text chunk to speak; may be empty on the final frame
`content_complete`	boolean	`true` on the last frame of a turn

You may optionally add "end_call": true on the final frame to signal that the agent should hang up the call after speaking the response.

Keep-alive (ping/pong)

If your server sends a keep-alive frame Vaani will echo it back immediately:

{ "response_type": "ping_pong" }

WebSocket connection authentication

Vaani supports an optional Bearer token for authenticating the WebSocket connection to your BYOL server. When configured, the token is sent as the standard Authorization HTTP header during the WebSocket upgrade handshake:

Authorization: Bearer <your-token>

Your server can validate this header before accepting the connection (e.g. using FastAPI’s websocket dependency or any standard WebSocket middleware).

How it works

Step	What happens
Configure	Enter the token in the Auth Token field on the BYOL settings tab and click Save URL.
Storage	The token is encrypted using Fernet symmetric encryption (key derived from `SECRET_KEY`). The UI always shows only a masked value (`****<last4 chars>`).
At call time	The agent decrypts the token in memory and attaches it to every new WebSocket connection as `Authorization: Bearer <token>`.
No token	If the field is left blank, the WebSocket connection is made without an `Authorization` header.

X-Agent-Id header forwarding

If the X-Agent-Id HTTP header is present on the /trigger-call/ request, Vaani forwards its value as an X-Agent-Id header on the WebSocket upgrade handshake:

X-Agent-Id: <your-agent-id>

This lets your BYOL server identify which logical agent is initiating the connection without having to parse the request body. The value is also available inside every turn payload as req_body.x_agent_id.

When it is sent	The `X-Agent-Id` header is only present on the WebSocket connection when the caller included `X-Agent-Id` in the original `/trigger-call/` request.

Example server-side usage (FastAPI)

from fastapi import WebSocket, WebSocketDisconnect, HTTPException, status

EXPECTED_TOKEN = "my-secret-token"

@app.websocket("/chat/stream/{session_id}")
async def websocket_endpoint(websocket: WebSocket, session_id: str):
    auth = websocket.headers.get("authorization", "")
    if not auth.startswith("Bearer ") or auth[7:] != EXPECTED_TOKEN:
        await websocket.close(code=status.WS_1008_POLICY_VIOLATION)
        return

    agent_id = websocket.headers.get("x-agent-id")  # present only when sent by caller
    await websocket.accept()
    # use agent_id to route to the correct agent / workflow
    # ... handle the session

Fallback behaviour

You can configure what happens when your server is unreachable or returns an error:

Setting	Behaviour
No fallback	The turn fails silently; the agent waits for the next user utterance
Use platform LLM	The agent falls back to the primary provider configured in the From Providers tab; if that also fails, it tries the fallback provider

The fallback mode is stored as extra_params.fallback in the agent config ("none" or "platform").

Session management

Each call opens a new connection at <ws_url>/<call_id> (the call_id / room name is appended automatically). Your server should use the session_id or the path component to isolate per-call state (e.g. conversation memory, tool state). When the call ends Vaani closes the WebSocket cleanly. You can also expose a DELETE /session/{session_id} endpoint on your server so Vaani can explicitly clean up state (see the example ADK server).

Reference implementation

The vaani-adk-byol-example repository contains a complete FastAPI server (adk_server.py) that:

Implements the full WebSocket protocol above
Runs a Google ADK agent (LlmAgent) with the system prompt injected per session
Exposes REST (/chat), SSE (/chat/sse), and WebSocket (/chat/stream/{session_id}) endpoints
Can be deployed locally and exposed with ngrok in under 5 minutes.

# Install
cd vaani-adk-byol-example
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
cp example.env .env   # add your GOOGLE_API_KEY

# Run
uvicorn adk_server:app --host 0.0.0.0 --port 8090 --reload

# Expose publicly
ngrok http 8090
# → copy wss://xxxx.ngrok-free.app and paste into the BYOL tab

Getting Started

Guides

Overview

How to enable BYOL

WebSocket protocol

Connection handshake

Agent → Your server (request)

Your server → Agent (streaming response)

Keep-alive (ping/pong)

WebSocket connection authentication

How it works

X-Agent-Id header forwarding

Example server-side usage (FastAPI)

Fallback behaviour

Session management

Reference implementation

​Overview

​How to enable BYOL

​WebSocket protocol

​Connection handshake

​Agent → Your server (request)

​Your server → Agent (streaming response)

​Keep-alive (ping/pong)

​WebSocket connection authentication

​How it works

​X-Agent-Id header forwarding

​Example server-side usage (FastAPI)

​Fallback behaviour

​Session management

​Reference implementation

Overview

How to enable BYOL

WebSocket protocol

Connection handshake

Agent → Your server (request)

Your server → Agent (streaming response)

Keep-alive (ping/pong)

WebSocket connection authentication

How it works

X-Agent-Id header forwarding

Example server-side usage (FastAPI)

Fallback behaviour

Session management

Reference implementation