An AI agent can decide that a customer needs a human without ending the call. The difficult part is preserving the interaction around that decision: the customer’s media session, the context the human needs, the transcript that explains what happened, and the recording artifacts created along the way.
The architecture usually follows one of two patterns. In a room-based warm handoff, the human joins the existing call before the AI leaves. In a media-leg transfer, the original connection is replaced or redirected into a new call. Both can be valid. The right choice depends on whether the system’s priority is preserving the live session or crossing a telephony boundary cleanly.
Start with the handoff decision
An AI agent should escalate when the customer explicitly asks for a person, the agent repeatedly fails to resolve the request, or the interaction reaches a sensitive billing or account-security issue. These are examples documented in Twilio’s escalation guidance; they are useful operating triggers rather than a universal policy.
Once escalation is approved, the system needs to answer a more specific question: does the customer remain in the current session while a human is added, or does the application create a replacement call?
That distinction affects the customer’s experience immediately. It also determines whether the application must reconcile one room with a changing participant set or join two separate call records after the transfer.
Pattern one: keep the customer in the room
A room-based warm handoff keeps the customer connected while the human agent is introduced. The human joins the existing room, receives the AI’s summary and relevant state, and the AI participant can leave after the human is ready.
In a documented conference pattern, a warm transfer adds a new participant to the conference and then removes the participant who initiated the transfer. Twilio describes this behavior in its Voice Conference documentation. A room-based video or audio system can use the same general shape: preserve the room, add the human, and retire the AI participant only after the handoff succeeds.
This pattern has a direct customer benefit. The customer does not have to accept a new call or explain the problem again merely because responsibility moved from software to a person. The human can enter while the AI remains available to summarize the interaction or answer a clarifying question.
It also creates an application responsibility. The human interface needs the handoff context before, or at the moment, the human joins. The room alone does not tell the human why the escalation happened, what the customer already tried, or which answer the AI gave incorrectly.
What the human should receive
The handoff payload should be structured rather than treated as a block of unprocessed transcript text. Useful fields include the interaction identifier, the customer identity, the reason for escalation, the current topic, unresolved questions, important variables, and a concise summary of what has already happened.
Microsoft’s Copilot Studio handoff documentation describes a product-specific set of context elements that can be passed to a live-agent system, including the conversation, relevant variables, the last topic, conversation ID, agent message, language, and user-defined topic variables. Twilio documents a different product-specific payload that can carry custom attributes, a conversation ID, and a profile ID.
The important design lesson is to define the contract around the receiving human-support system’s needs instead of copying field names from another product. Version that payload and make the handoff event queryable after the call.
Pattern two: replace the media leg
A media-leg transfer moves the customer from the original connection into a new call. The higher-level interaction may remain the same in the application, but the media path has a transfer boundary.
For SIP, Twilio’s <Refer> documentation says that the instruction initiates a SIP Refer and replaces the Twilio leg with the new call. In its SIP transfer documentation, Twilio also describes the new call as a child call associated with the original call. That child-call relationship gives the application identifiers to reconcile, but it does not merge the recordings, transcripts, or application state automatically.
A media-leg transfer can be the cleanest option when the human agent sits behind a SIP or contact-centre boundary that cannot join the original room directly. It can also fit an existing telephony operation that already knows how to route calls, apply queue logic, and record each call leg.
The cost is continuity work. The customer may experience a hold, a new connection, or a transition between call legs. The application must carry the interaction context across the boundary and associate the original call with the replacement call. If that correlation is missing, the human may receive a new call with no reliable explanation of what happened before it arrived.
Choose the room-based pattern when continuity is the priority
The recommendation is to use a room-based warm handoff when the customer’s current session can remain active and the human can join the same media environment. It keeps the customer connected, allows the AI to leave only after the human is present, and gives the application one session in which to associate participants and events.
This recommendation has a clear limit: it only works when the human-support endpoint can participate in the room or can be connected to it through a compatible gateway. A room-based design also does not guarantee a single recording file or a complete human-facing transcript. Those outcomes still depend on the recording and transcript pipeline.
Use a media-leg transfer when the human must be reached through SIP, PSTN, or a separate contact-centre system that cannot join the existing room. The transfer boundary is acceptable if the application deliberately carries the context and correlates the resulting call records.
That choice has its own limit: replacing the media leg can interrupt the customer’s path, and the application cannot assume that identifiers, recordings, or transcript state will carry across automatically.
Keep media continuity and context continuity separate
A customer can remain in the same room while the human receives poor context. A transfer can create a new call while the application preserves a complete interaction record. Media continuity and context continuity are related, but they are separate design problems.
For an AI agent to human handoff, create a stable interaction record before escalation begins. Attach the room ID or call-leg ID, participant identities, timestamps, the escalation event, and the handoff reason to that record. Store the AI summary and the relevant transcript reference there rather than relying on the receiving system to reconstruct the interaction from media events.
LiveKit’s handoff documentation makes the same problem visible inside an agent workflow: a new agent or task starts with fresh conversation history unless the previous context is explicitly passed, copied, truncated, or summarized. Its chat-context documentation also describes session history as available to the application. A human-support interface still needs a deliberate mechanism to display or attach that history.
The handoff event should therefore be a first-class record. It should say when escalation was requested, when the human was invited, when the human accepted, when the AI left, and whether the customer had to reconnect. Those events make the interaction auditable and give operations a way to measure failures without guessing from a transcript.
Design the transcript as a correlated record
A transcript is useful only when the human can tell who said what and where the escalation occurred. In a room-based system, preserve participant identity and track identity with each transcript segment. LiveKit’s transcription documentation describes messages that carry the transcribed track ID and the identity of the participant associated with the track.
When a SIP transfer creates a child call, the application must perform the equivalent correlation across call legs. Keep the original interaction ID, the original call identifier, the replacement call identifier, participant or endpoint identity, and timestamps together. The exact identifiers differ by platform, but the correlation requirement remains part of the application contract.
Do not send the human an unbounded transcript by default. Give the operator a short summary, the customer’s stated goal, unresolved questions, the reason for escalation, and a way to open the underlying transcript when detail is needed. This reduces the time spent searching while retaining the source record for review.
Decide what the recording should become
Recording continuity requires its own design. A room-based recording can include participants who join later when the recording rules cover all tracks. Twilio’s recording rules documentation states that later participants can also be recorded when recording is enabled for all tracks.
That does not mean the system will produce one uninterrupted file. Twilio also documents that a new recording identifier is created each time track recording starts. A room can therefore produce multiple recording artifacts if recording stops and restarts.
A telephony conference introduces another boundary. Twilio’s Conferences resource documentation distinguishes conference recordings from recordings associated with individual call legs. A pipeline that uses both must query and reconcile both resource types.
Choose the output deliberately:
- One continuous recording: keep the interaction in a recording model that can capture the relevant participants and produce the required final artifact.
- Linked recordings: retain separate room, conference, or call-leg files and connect them through the interaction ID and handoff event.
- One post-processed artifact: preserve the original files, then assemble a review copy after the session while keeping the source boundaries and timestamps.
The correct choice depends on how the recording will be reviewed, searched, redacted, exported, or retained. Decide before implementation whether “the recording” means a file, a set of linked artifacts, or a derived review output. The choice between composite and per-track recording affects that decision, especially when speaker identity and later processing matter.
Measure the whole handoff
Call-connected status is too narrow to show whether the handoff worked. Measure the time from escalation decision to human acceptance, the time the customer spends waiting, reconnection failures, whether the human received the intended context, and whether the customer had to repeat the request.
For a room-based handoff, also record whether the human joined the intended room, whether the AI left at the correct point, and whether the customer’s media continued through the transition. For a media-leg transfer, record the parent and child call identifiers, the transfer outcome, and whether the transcript and recording references were attached to the same interaction.
These measurements should be generated by the handoff state machine rather than inferred later from provider dashboards. A successful media connection is only one event in the sequence. The operational outcome is whether the customer reached the right human with enough context to continue the conversation.
Bottom Line
Use a room-based warm handoff when preserving the customer’s current session is the primary requirement and the human can join that room. Use a media-leg transfer when the human endpoint requires SIP or a separate contact-centre boundary, and make the context, identifiers, transcript, and recording relationship explicit across the transfer.
In both designs, treat the handoff as an application workflow composed of several coordinated parts. The media path, context payload, transcript correlation, recording output, and operational events must be designed together.
What’s Next
The handoff architecture changes again when the customer begins on a phone line rather than in a browser. The companion post, Voice Agents on a Phone Line vs in the Browser, examines how SIP and PSTN affect the audio path and the architecture around a voice agent.
FAQs
What is an AI agent to human handoff?
An AI agent to human handoff moves an active customer interaction from an automated agent to a human operator while carrying the context needed to continue the conversation. The handoff can keep the customer in the same room or create a replacement call, but the application must explicitly transfer the summary, identifiers, and relevant interaction state.
Should the human join the same room as the AI agent?
Yes, when the human endpoint can join the existing room and customer continuity is the priority. A room-based warm handoff lets the human enter before the AI leaves, but it still requires a context payload and a recording plan.
Does a warm handoff create one continuous recording?
No. A room-based recording can include participants who join later when the recording rules cover all tracks, but stopping and restarting recording can create new recording identifiers. The application should choose between one recording, linked recordings, or a post-processed artifact.
What happens to the transcript during a SIP transfer?
A SIP transfer can create a new child call and replace the original media leg. The application must correlate the original and replacement call identifiers and attach both transcript segments to the same interaction record; the transfer mechanism does not automatically merge transcript state.
What context should an AI agent send to a human?
Send the interaction identifier, customer identity, escalation reason, current topic, unresolved request, concise summary, relevant variables, and a reference to the transcript. Product documentation from Twilio and Microsoft shows different implementations, so define a versioned payload that matches the receiving human-support system.
How should teams test an AI agent to human handoff?
Test whether the customer remains connected or reconnects as designed, whether the human receives the intended context, whether transcript segments retain speaker identity, and whether recording artifacts can be found through the interaction ID. Also measure wait time, reconnection failures, and repeat-explanation rate in the target architecture rather than assuming a successful call connection proves the handoff worked.