> ## Documentation Index > Fetch the complete documentation index at: https://docs.vaanivoice.ai/llms.txt > Use this file to discover all available pages before exploring further. # Bring Your Own LLM (BYOL) > Connect your own LLM to a Vaani agent via a streaming WebSocket server ## Overview BYOL lets you replace the default platform LLM with **any inference engine you control** — an on-premise model, a fine-tuned model, a Google ADK agent, a LangGraph workflow, or anything else that can speak the Vaani WebSocket protocol. When BYOL is enabled for an agent, every response turn is routed to your WebSocket server instead of the built-in provider (OpenAI, Google, Groq, etc.). The rest of the pipeline — STT, TTS, telephony, transcripts, Langfuse tracing — stays exactly the same. *** ## How to enable BYOL 1. Open your agent in the dashboard. 2. Go to **Brain → Reasoning Language Model (LLM)**. 3. Switch to the **Bring your Own LLM (BYOL)** tab. 4. Paste your WebSocket URL (e.g. `wss://your-server.example.com/chat/stream`). 5. *(Optional)* Enter an **Auth Token** in the Auth Token field. When set, Vaani sends `Authorization: Bearer ` as an HTTP header during the WebSocket upgrade handshake. The token is stored **Fernet-encrypted** at rest and only decrypted at call time — the UI always displays a masked value (`****`). Leave blank for unauthenticated connections. 6. Click **Test Connection** to verify reachability, then **Save URL**. 7. Choose a **Fallback LLM** — either *No fallback* or *Use platform LLM*. The URL is stored under `agent_config.persona.senses_capabilities.brain.llm.extra_params.llm_websocket_url` and takes effect immediately on the next call. *** ## WebSocket protocol Your server must implement the following JSON message exchange over a persistent WebSocket connection. Vaani opens **one connection per call** (identified by `session_id` / room name) and sends one request per agent turn. ### Connection handshake Immediately after the WebSocket is accepted, your server **must** send two JSON frames in order: ```json theme={null} { "interaction_type": "config", "content": "Server ready" } { "interaction_type": "greeting", "content": "Hello" } ``` These frames are consumed by the Vaani agent and discarded — they are only used to confirm the connection is live. The content strings may be anything. *** ### Agent → Your server (request) For every agent turn (after the user finishes speaking) Vaani sends a JSON frame over the WebSocket. Here is a real example captured from a live call: ```json theme={null} { "interaction_type": "response_required", "response_id": 2, "call_id": "outbound-1782291018-65b6af82", "transcript": [ { "role": "system", "content": "You are speaking with {{customer_name}}, who has a {{account_type}} account valid until {{subscription_end}}.\n\n[... full agent system prompt ...]" }, { "role": "assistant", "content": "I am at your service, how can I help?" }, { "role": "user", "content": "What is this call regarding?" }, { "role": "user", "content": "What is this call regarding?" } ], "req_body": { "agent_id": "535d0c34-8086-419e-b39d-ee549fc28e93", "medium": "telephony", "contact_number": "+917893209830", "name": "John", "voice": "", "dnd_check_skipped": true, "voice_gender": "female", "primary_language": "hi", "secondary_language": "en", "welcome_interruptible": true, "bg_noise_enabled": false, "bg_noise_volume": 60, "voice_speed": 1.0, "x_agent_id": "my-custom-agent-id", "modify_agent": { "persona": { "metadata": { "customer_name": "John Doe", "account_type": "Premium", "subscription_end": "Dec 31, 2026" }, "identity": { "system_prompt": "You are speaking with {{customer_name}}, who has a {{account_type}} account valid until {{subscription_end}}." } }, "training": null, "experience": null, "analysis": null } } } ``` | Field | Type | Description | | --------------------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | `interaction_type` | `"response_required"` | Always this value for a normal turn | | `response_id` | integer | Monotonically increasing; echoed back in every response chunk | | `call_id` | string | The room/call identifier for this session | | `transcript` | array | Full conversation history for this call, including the system prompt as the first `"system"` message | | `req_body` | object or `null` | Original trigger-call request body (all top-level fields including `modify_agent`); `null` if the call was not triggered via the external API | | `req_body.x_agent_id` | string or `null` | Value of the `X-Agent-Id` header passed to `/trigger-call/`; `null` or absent if the header was not sent | The **system prompt** built from your agent configuration is always the first entry with `"role": "system"`. Your server must apply it as the LLM's instruction/system message for each turn. If you create an ADK `LlmAgent` or a LangChain chain, pass this text as the agent instruction or system message so the configured persona is honoured. `req_body.modify_agent.persona.metadata` contains the per-call template variables (e.g. `customer_name`, `account_type`) passed by the caller. If your system prompt uses `{{ variable }}` placeholders, populate your LLM session state from this object so the placeholders resolve correctly. *** ### Your server → Agent (streaming response) Stream back one or more chunks, each as a JSON frame: ```json theme={null} { "response_type": "response", "response_id": 1, "content": "Yes, we carry ", "content_complete": false } ``` Send a final frame with `"content_complete": true` and optionally empty `content` to signal end of turn: ```json theme={null} { "response_type": "response", "response_id": 1, "content": "beautiful Pelikan M800 models.", "content_complete": true } ``` | Field | Type | Description | | ------------------ | ------------ | ---------------------------------------------------- | | `response_type` | `"response"` | Always this value | | `response_id` | integer | Must match the `response_id` from the request | | `content` | string | Text chunk to speak; may be empty on the final frame | | `content_complete` | boolean | `true` on the last frame of a turn | You may optionally add `"end_call": true` on the final frame to signal that the agent should hang up the call after speaking the response. *** ### Keep-alive (ping/pong) If your server sends a keep-alive frame Vaani will echo it back immediately: ```json theme={null} { "response_type": "ping_pong" } ``` *** ## WebSocket connection authentication Vaani supports an optional **Bearer token** for authenticating the WebSocket connection to your BYOL server. When configured, the token is sent as the standard `Authorization` HTTP header during the WebSocket upgrade handshake: ``` Authorization: Bearer ``` Your server can validate this header before accepting the connection (e.g. using FastAPI's `websocket` dependency or any standard WebSocket middleware). ### How it works | Step | What happens | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Configure** | Enter the token in the **Auth Token** field on the BYOL settings tab and click **Save URL**. | | **Storage** | The token is encrypted using [Fernet](https://cryptography.io/en/latest/fernet/) symmetric encryption (key derived from `SECRET_KEY`). The UI always shows only a masked value (`****`). | | **At call time** | The agent decrypts the token in memory and attaches it to every new WebSocket connection as `Authorization: Bearer `. | | **No token** | If the field is left blank, the WebSocket connection is made without an `Authorization` header. | *** ## X-Agent-Id header forwarding If the `X-Agent-Id` HTTP header is present on the `/trigger-call/` request, Vaani forwards its value as an `X-Agent-Id` header on the WebSocket upgrade handshake: ``` X-Agent-Id: ``` This lets your BYOL server identify which logical agent is initiating the connection without having to parse the request body. The value is also available inside every turn payload as `req_body.x_agent_id`. | When it is sent | The `X-Agent-Id` header is **only** present on the WebSocket connection when the caller included `X-Agent-Id` in the original `/trigger-call/` request. | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Example server-side usage (FastAPI) ```python theme={null} from fastapi import WebSocket, WebSocketDisconnect, HTTPException, status EXPECTED_TOKEN = "my-secret-token" @app.websocket("/chat/stream/{session_id}") async def websocket_endpoint(websocket: WebSocket, session_id: str): auth = websocket.headers.get("authorization", "") if not auth.startswith("Bearer ") or auth[7:] != EXPECTED_TOKEN: await websocket.close(code=status.WS_1008_POLICY_VIOLATION) return agent_id = websocket.headers.get("x-agent-id") # present only when sent by caller await websocket.accept() # use agent_id to route to the correct agent / workflow # ... handle the session ``` *** ## Fallback behaviour You can configure what happens when your server is unreachable or returns an error: | Setting | Behaviour | | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | **No fallback** | The turn fails silently; the agent waits for the next user utterance | | **Use platform LLM** | The agent falls back to the primary provider configured in the *From Providers* tab; if that also fails, it tries the fallback provider | The fallback mode is stored as `extra_params.fallback` in the agent config (`"none"` or `"platform"`). *** ## Session management Each call opens a **new** connection at `/` (the call\_id / room name is appended automatically). Your server should use the `session_id` or the path component to isolate per-call state (e.g. conversation memory, tool state). When the call ends Vaani closes the WebSocket cleanly. You can also expose a `DELETE /session/{session_id}` endpoint on your server so Vaani can explicitly clean up state (see the example ADK server). *** ## Reference implementation The [vaani-adk-byol-example](https://github.com/mohammad-vaaniresearch/vaani-byol-example) repository contains a complete FastAPI server (`adk_server.py`) that: * Implements the full WebSocket protocol above * Runs a Google ADK agent (`LlmAgent`) with the system prompt injected per session * Exposes REST (`/chat`), SSE (`/chat/sse`), and WebSocket (`/chat/stream/{session_id}`) endpoints * Can be deployed locally and exposed with **ngrok** in under 5 minutes. ```bash theme={null} # Install cd vaani-adk-byol-example python -m venv venv && source venv/bin/activate pip install -r requirements.txt cp example.env .env # add your GOOGLE_API_KEY # Run uvicorn adk_server:app --host 0.0.0.0 --port 8090 --reload # Expose publicly ngrok http 8090 # → copy wss://xxxx.ngrok-free.app and paste into the BYOL tab ```