Phone Agent from Zero

Treat the phone agent as a business process

A phone agent is not a chatbot with a phone number attached. Calls are recorded, transcribed, billed, transferred, interrupted, and often tied to appointments, support, sales, screening, or account changes. The first decision is not the model. It is what the agent is allowed to do, what it must refuse, when it must transfer, and how it obtains recording or data-processing consent.

Start with bounded jobs

Good first workflows include appointment confirmation, inbound triage, FAQ, simple lead qualification, follow-up calls, and internal helpdesk routing. Avoid fully automated medical diagnosis, financial advice, complex complaints, conversations with minors, or emotionally charged disputes. High-risk calls should start as human-assist, not full replacement.

Write the call flow before the prompt

A production call flow has an opening, identity disclosure, consent line, goal confirmation, data collection, tool use, recovery prompts, handoff, and close. Prompt wording matters, but the call flow decides what happens when the user is silent, refuses information, asks for a human, changes their mind, or hits a backend error.

Build the first production path

The fastest route is a managed platform such as Retell AI or Vapi: buy or connect a number, configure model, voice, transcription, tools, and webhooks, then call it from real phones. The controlled route is LiveKit or Pipecat with providers such as Deepgram, Cartesia, ElevenLabs, or OpenAI, but then you own media transport, workers, retries, and observability.

Handle number, consent, and recording early

Recording, outbound calling, automated disclosure, and marketing rules vary by region. Even in an MVP, identify the agent, explain recording or transcription where needed, and provide an exit path. Store consent state with call logs, script version, and launch date.

Keep tool calls read-only at first

Start with lookup tools: order status, appointment availability, FAQ retrieval, ticket creation drafts. Writes such as rescheduling, cancellation, SMS sending, CRM updates, or payment changes need confirmation and failure handling. If the user interrupts, the agent can stop speaking, but it should not silently cancel an irreversible action.

Use 30 real calls as the gate

Before launch, run at least 30 calls across quiet rooms, speakerphones, car audio, accents, long pauses, interruptions, refusal, tool timeouts, handoff, and hangups. Review transcript, recording, event timeline, cost, and outcome for each. The embedded Tech With Tim Vapi tutorial is a useful build reference; your launch standard should be based on these failure calls.

Define success before the model improvises

For each call type, write the success condition in plain language. An appointment bot succeeds when it confirms the right person, date, time, timezone, and callback path. A lead screener succeeds when it collects the required fields and marks uncertainty rather than inventing missing data. A support triage bot succeeds when it routes correctly, not when it keeps the user talking longest. These definitions become your evaluation rubric.

Choose tools by stage

Retell AI is the quickest production-oriented managed path when call analytics, simulations, and phone operations matter. Vapi is strong for builders who want provider choice, frontend integration, and tool wiring. Deepgram is worth evaluating when speech recognition and voice-agent API cost are central. OpenAI Realtime is compelling for native speech-to-speech tests. LiveKit and Pipecat are for teams ready to own realtime media.

Price the whole call

Retell currently lists AI voice agents at $0.07-$0.31 per minute. Deepgram lists Voice Agent API from $0.075 per minute and Nova or Flux STT from $0.0077 per minute. Those are components, not your full bill. Add telephony, LLM, TTS, silence, retries, call storage, QA, and human handoff.

Use managed first, migrate only with evidence

Most teams should prove the call flow on Retell or Vapi before self-hosting. Move to LiveKit or Pipecat when cost, privacy, region, vendor control, or custom media routing becomes the main constraint. Preserve prompt versions, tool schemas, call samples, failure labels, and consent records so migration is possible.

Plan the operator console

Even a simple phone agent needs an operator view: live status, recent calls, transcript, recording link, extracted fields, tool-call errors, handoff reason, and estimated cost. Without that view, the team learns about failures from angry users. Start with a manual dashboard if needed; the important part is that support and engineering can inspect the same call without digging through provider logs.

Launch softly and monitor failure

The first production week is not about maximizing automation. Watch task completion, handoff rate, repeat-request rate, missed interruption, average and P95 cost, tool failures, complaints, and hangup points. Sample failed calls daily and turn them into regression tests.

Keep a human escape hatch

Every phone agent needs a clear sentence like “I can transfer you to a person.” If live transfer is not available, create a ticket, schedule a callback, or send a confirmation. Automation without an escape path turns system failure into user frustration.

Pre-launch checklist

Before expanding traffic, verify the opening disclosure, recording consent, max call duration, retry limit, timeout phrase, transfer behavior, blocked topics, and webhook failure response. Check that every write action has confirmation, every call has a trace ID, and every failed call can be replayed. Then watch the first live cohort closely; a phone agent teaches you more in its first 100 real calls than in a week of synthetic demos.

Keep the first launch narrow. One phone number, one call reason, one transfer queue, and one daily review ritual will produce cleaner learning than a broad rollout with five half-tested intents. Scale after the failure categories stabilize.