About This Book
Who this book is for
This book is for AI engineers, applied scientists, voice and telephony developers, and senior software engineers who are building production voice agents, live translation, and streaming transcription on Python services. If you are moving from browser demos to deployed backend systems, or integrating the OpenAI Realtime API into existing telephony or web stacks, you will find the architecture, code patterns, and operational guidance you need. It is particularly useful for teams that must own server-side logic for security, tool access, and session management rather than relying only on client-side JavaScript.
What you will learn and build
You will learn how to build server-side Python voice agents using the GA Realtime API and its three production model IDs: gpt-realtime-2 for reasoning, gpt-realtime-whisper for streaming speech-to-text, and gpt-realtime-translate for live multilingual conversations. Through progressive chapters, you will construct a reusable audio I/O layer, implement WebSocket and SIP transports, manage session lifecycle and history, handle interruptions and function calling, and add observability and cost controls. You will also learn to evaluate reasoning effort levels, manage audio token budgets, and implement reconnection strategies for long-running sessions. By the end, you will have a complete multilingual voice support agent that handles web and phone inbound traffic with translation, tool use, and production hardening.
How to use this book (recommended reading path)
Read Part I to establish foundations: Realtime API orientation, Python setup, and session architecture. Part II guides you through your first WebSocket agent, turn detection, and tool integration. Part III covers specialized models for transcription and translation, plus telephony attachment with SIP. Part IV brings everything together with browser WebRTC boundaries, observability, deployment, scaling, and a full end-to-end case study. Each chapter includes runnable Python modules and VS Code debugging configurations you can use directly. If you are migrating from a preview-era app, jump to Chapter 12 for the beta-to-GA migration path.
High-level prerequisites
You should be comfortable with Python, asynchronous programming with asyncio, and basic audio concepts such as sampling rate and PCM16. Familiarity with WebSockets, REST APIs, and cloud deployment practices will help. Experience with the OpenAI Python SDK is useful but not mandatory; we cover the Realtime-specific surface from the ground up. No prior experience with voice or telephony stacks is required; we build those layers step by step in Python.
