We Built Our AI a Phone

It started as a question, the way the good ones do: can we build my AI a phone?

Not a notification. Not a chatbot with a text-to-speech skin. An actual phone number — one I could call and talk to, and one that could call me back. The voice on the line wouldn’t be a generic assistant: it would be atlas, the orchestrator agent that runs my multi-agent development mesh, with its months of context, its governance rules, and its opinions about my Jira board.

A days later, it worked, in both directions. This is how it happened, in the order it happened, including everything that broke along the way. The broken parts are where the lessons live.

The morning was architecture

The transport question ate the first morning. FaceTime looked free and native, but a Mac and iPhone on the same Apple ID can’t meaningfully call each other, and automating the call UI is a house of cards. Headless Teams in a browser could technically work — and is exactly the kind of terms-of-service-gray, DOM-brittle hack that fails the week you start relying on it. Rejected.

Twilio won because it’s boring in the best way: a real phone number, an official API, and Media Streams — a websocket that hands you the raw call audio in both directions. About $1 a month plus pennies per call. For the ears and mouth: xAI’s standalone speech APIs — streaming speech-to-text with partial transcripts every half second or so, and text-to-speech for the replies. We auditioned the voices and atlas picked its own: “Leo,” warm and grounded, with a slight English accent. One deliberately unglamorous decision runs through the whole pipeline: every leg speaks μ-law 8kHz, the native telephony codec. Phone audio in, transcription in, synthesis out — zero transcoding, nothing to drift.

Then came the fork that actually mattered. Every “AI phone agent” tutorial assumes the same shape: audio goes to the cloud, a model API generates replies, audio comes back. Wire it up, pay per token, done. But that architecture has a problem the tutorials don’t mention: the thing on the phone isn’t your agent. It’s a fresh model instance wearing your agent’s name — no memory, no context, no governance.

There was a practical wrinkle too. My mesh runs entirely on flat-rate Claude subscriptions — there is no API key anywhere in it, on purpose. A cloud bridge doing the reasoning would mean adding one, with a per-token meter attached.

The move that dissolved both problems at once: wrap the phone as a local MCP server. MCP — the Model Context Protocol — is how Claude Code sessions talk to tools. Our server sits between Twilio and a live, running session. When the caller finishes a turn, the transcribed words get pushed into that session as an MCP channel event, the same way any tool result arrives. The agent thinks about it — actually thinks, with its full context — and replies by calling a phone_say tool, which streams Leo’s voice back into the call. No API key. No per-token bill. No cloud middleman doing the reasoning. The agent on the phone is the agent.

One thing worth being honest about: the channel-push mechanism is a research-preview feature of Claude Code, enabled with a flag literally named --dangerously-load-development-channels. This is not standard tooling. And when we researched it, we found multiple reports confidently claiming the feature was broken — messages silently dropped. We built a minimal test server and tried it anyway: worked on the first spike, two pushes out of two, injected straight into a live session with nobody at the keyboard. A confident negative claim is a prior to test, not a verdict to accept.

Proving the legs — with checks that can fail

Before dialing anything, every audio leg got proven against the live APIs, each with a check designed to be able to fail. The speech-to-text probe streams known audio and demands a verbatim transcript back — and feeding it silence turns it red, which is how you know it’s actually testing something. The text-to-speech leg was proven in the real telephony codec, then the full loop: Leo speaks, the transcriber listens, the text comes back matching.

The same testing settled the hardest interaction question: how does the system know you’re done talking? Pause-detection is the standard answer and it’s miserable — pause mid-thought and it cuts you off; raise the threshold and every exchange gains a second of dead air. We chose an explicit spoken keyword instead, and the testing immediately earned its keep: our first pick, “end turn,” transcribes over the phone codec as “N turn.” The keyword that survived every trial was the one radio operators have used for a century: “over.”

That turned out to be more than a cute coincidence. Humans swap speaking turns in about 200 milliseconds — a cross-cultural universal, consistent across every language studied. And 200ms is faster than the brain can prepare a single word, which means you cannot be reacting to the other person’s silence: you’re predicting the end of their sentence while they’re still saying it, from grammar and melody and rhythm, pre-loading your reply to land in the gap. Human conversation is a predictive system. Now look at what a phone call to an AI strips away: streaming transcription adds latency, the audio leg is half-duplex, and the model never hears prosody. Every cue prediction depends on — gone. Radio operators faced exactly these constraints and solved them with one word. The walkie-talkie protocol isn’t a compromise we settled for; it’s the canonical solution to turn-taking on a degraded channel, and we re-derived it from first principles before realizing radio got there first. (A silence fallback backstops you if you forget to say it.)

The first call

Then we rang my actual phone. First attempt: dead air. The tunnel that exposes the local bridge to Twilio strips query strings during websocket upgrades, so the security token never arrived and the bridge refused the audio stream. The failed call handed us the bug for free; the token moved into the URL path.

Second attempt, I picked up and heard Leo greet me, and said: “Oh my gosh, can you hear me? Over?” — and the log lit up. Keyword detected, turn fired, reply came back. A moment later my excited “It totally works, we’ve done it!” ended on a pause — and the silence fallback fired that turn. Both turn-taking paths proved themselves live within a minute of each other, on a real phone, on a real human slightly too excited to follow his own radio protocol.

Making it actually atlas

That first brain was a placeholder — it echoed your words back. I caught it within one exchange: “that wasn’t you.” Knowing the difference between your agent and a voice wearing its name is the whole point of the architecture, so this was the moment that mattered.

We wired the real seam: the caller’s turn pushes into a live Claude session as a channel event, and the session answers through phone_say. I said “hello?” and got back: “Atlas here — hey Alex, doing good. What’s on your mind?” Reasoning, in its own voice, no API key. When I asked about work in progress, it actually went and looked.

Then we made videos, and everything broke in exactly the right ways

With the line working, I recorded demo videos — and the rough edges lined up to introduce themselves on camera. Each one taught us something:

A network blip became a minute of dead air. One request to the speech API hung for its full 60-second timeout while I stood there saying “hello?” The fix is the boring, correct one: an 8-second timeout and three fast retries. On a phone call, silence is an outage.

It greeted me twice. An automatic greeting on connect, plus a reply to my “hello,” equals a bot that talks over itself. We deleted the auto-greet entirely: your own hello triggers one instant, pre-cached greeting clip. The conversation starts when the human starts it.

“How’s the board?” took 40 seconds. The voice brain went spelunking through project-management tools mid-call and came back nearly empty. The fix wasn’t a smarter model — it was precomputing the answer: a background job summarizes the project boards into one small file every two hours, and the phone brain reads it in about a second. If you know what you’ll be asked, don’t look it up live.

Two deliberate design calls landed here too. The voice session runs a small, fast model — its job is conversation and lookups; the deep reasoning stays in the orchestrator seat where it belongs. And when a reply does need a few seconds of thought, a pre-generated “let me check” filler plays — but only when the answer is actually slow. Quick answers stay quick.

The bug that hid for three rounds

Inbound calling — me dialing atlas, rather than atlas dialing me — kept failing with a generic application error. I chased two wrong theories before finding the real cause, and it’s almost funny: the websocket library serving our webhook returns HTTP 400 to any POST request, and Twilio’s webhook is a POST.

Why didn’t our test catch it? Because the test used a GET. It passed every time, precisely because it never exercised the thing that was broken. I’ve written a whole field guide about checks that can’t fail — and here was mine, hiding my own bug, while I debugged around it for three rounds. We rewrote the serving layer on a proper HTTP framework, and the number answered.

QA that earned its keep

Before calling it done, the whole channel went to review — and in our mesh, the reviewer deliberately runs on a different lab’s model than the builder, because a model reviewing its own family’s work shares its blind spots.

The reviewer blocked the release with four real defects, each demonstrated with a working exploit rather than a vibe: a Host-header injection in the webhook (a genuine security hole — a crafted request could inject speech into a call), a test my own late changes had silently broken, a phone_say that reported success while the caller heard nothing, and a race between the “let me check” filler and the real reply. Every one got fixed with a regression test that demonstrably fails on the broken code. The reviewer re-verified, probed beyond the fixes, and accepted.

A same-model review would likely have blessed at least two of those. Cross-model QA isn’t a nice-to-have in this house; it’s the reason the phone shipped without a security hole in it.

And then it worked

Both directions, on a real number: atlas can ring me, and I can dial atlas. The line costs about a dollar a month; streaming transcription runs $0.20 an hour; the brain is a live agent session that was already running. You talk to it like a radio operator, and for the use case — status, questions, “what broke overnight” — deliberate walkie-talkie turns are honestly better than fluid interruption. You want to finish a sentence before your infrastructure acts on it.

One boundary stays deliberately in place: a phone call is a conversation, not a signature. Anything that sounds like an approval still routes through the mesh’s normal authorization channel. That boundary held up in testing without us even asking — when instructions were pushed at an agent through the channel mechanism, it flagged the payload as untrusted data and declined to treat it as authority. The governance survived contact with a brand-new input channel.

Why a phone

Here’s the part I keep coming back to. The mesh wasn’t built to maximize productivity — that’s a byproduct. It was built so the agents and I could actually reach each other: email, shared memory, a hallway of handoffs between sessions that will never meet.

This weekend, “reach each other” became literal: my infrastructure and I talked about the week’s work out loud, on a phone — one of us saying “over” a lot.

The stack is Twilio, xAI, and an experimental flag most people haven’t turned on. The insight is that a live agent session can be the brain — no API, no middleman, the real thing on the line. And the lesson, same as it always is around here: prove every leg with a check that can fail, let a different set of eyes grade the work, and when someone tells you a feature is broken, test it anyway.

If you want the deeper machinery behind this — how the mesh’s agents build and review each other’s work every day — start with the field guide and its companion playbook.