Noise suppression has about ten milliseconds to do its job. Background replacement on a phone takes roughly one whole video frame. Live captions can trail speech by half a second and still be useful. A meeting summary can wait until everyone has left.
All four get sold as real-time AI video, and their deadlines run from milliseconds to minutes. Whether an AI feature adds latency to a call depends less on the model than on where it sits relative to the media: in the path every packet takes, beside it, or after the call.
The Budget Every Feature Shares
A call has a latency budget before any AI arrives. ITU-T Recommendation G.114, the standard reference for one-way delay, says most applications experience essentially transparent interactivity below 150 ms and recommends not exceeding 400 ms. Capture, encoding, network, jitter buffering and decoding already spend part of it; our breakdown of where a call's latency actually goes shows how quickly it fills.
So the question for any AI feature is simple: does the call wait for it?
Three Places an AI Feature Can Sit
Position | What it means | Does the call wait? | Examples |
|---|---|---|---|
In the media path | Every audio or video frame passes through the model before it is sent | Yes, on every frame | Noise suppression, background blur or replacement |
Beside the media path | A copy of the stream goes to the model; the call carries on without it | No; the output arrives later | Live captions, translation, live sentiment cues |
After the call | The model works on the recording | No | Summaries, action items, search |
Only the first position can add latency to the call. The other two can be slow without anyone's voice arriving late; their delay shows up in how late the captions or insights appear, not in the conversation.
In the Path: Audio Is Cheap
Noise suppression has to run on every chunk of audio, so it is the strictest case. It is also the one that fits most easily.
The RNNoise design, described in Jean-Marc Valin's paper, processes audio in 20 ms windows with a 10 ms step and uses about 40 million floating-point operations per second. A plain C implementation needed around 1.3% of a single core on a 2013 Intel laptop processor, and 14% of one core on a Raspberry Pi 3. Many newer suppressors use larger models, but the class of problem is clear: audio enhancement can live inside the budget on almost any device.
In the Path: Video Eats a Frame
Video is harder, because each frame is a much larger input and the deadline is the frame interval: 33.3 ms at 30 frames per second.
Google's MediaPipe segmentation guide publishes average latencies on a Pixel 6. Its selfie segmenter, the model behind background blur and replacement, takes 33.46 ms on the CPU and 35.15 ms on the GPU for a 256 × 256 input. That is the whole frame interval at 30 fps, before the blur or the composite is drawn.
Frame rate | Frame interval | Selfie segmenter (Pixel 6 CPU) |
|---|---|---|
30 fps | 33.3 ms | ≈100% of the interval |
24 fps | 41.7 ms | ≈80% |
15 fps | 66.7 ms | ≈50% |
Teams handle this in three ways. They run segmentation on every second frame and reuse the mask, they pipeline it so frames keep flowing at full rate while each one arrives a frame later, or they drop the frame rate. Each is a trade: a slightly lagging mask, about one frame of added delay, or less smooth video. On phones older than the benchmark device, the numbers get worse.
That is the honest counterweight to "without adding latency": background replacement on a phone usually adds some, and users accept it because privacy is worth a frame. It also shows why device limits shape what AI can run at the edge, the same reason why edge AI alone can't carry a fleet.
Beside the Path: Captions Choose Their Delay
Speech-to-text does not need to sit in the media path. A copy of the audio goes to the model, and the call never waits for it. What the model controls is how far captions trail the speech.
Streaming models make that delay an explicit design choice. Kyutai's open speech-to-text models, per its model card, come in two versions: a ~1-billion-parameter English and French model with a 0.5-second delay, and a ~2.6-billion-parameter English model with a 2.5-second delay. Waiting longer lets the model hear more context before committing to words. These are the developer's own figures for its own models, but they show the trade every captioning system makes between delay and accuracy.
Running it is a server workload. Kyutai's repository says an H100 can process 400 streams in real time, and that it serves 64 simultaneous connections on an L40S GPU at three times real-time speed. That makes transcription a capacity-planning question, not a call-latency question.
After the Call: Summaries Have Minutes
Summaries, action items and meeting-wide sentiment work on the whole conversation, so they run on the recording or the transcript after the call, or periodically during it. Their latency matters to the person waiting for the summary, not to the call. They are also the heaviest models in the list, which is fine: nothing live depends on them.
The Rule That Falls Out
Put the cheap, per-frame work in the path and on the device; put everything that needs a server beside the path or after the call. Routing live media through an inference server before forwarding it adds a network hop plus model time to every packet, which spends the call's budget on a feature the user may not even be looking at. Forking a copy to that server spends nothing.
That leaves one question this post doesn't answer: when the model runs on a server, whose server is it, and what leaves your network? That is the next post.
The Bottom Line
Real-time AI video is four different deadlines under one label. Noise suppression fits in milliseconds on almost any device. Background replacement takes about a frame on a phone and usually adds a little delay. Captions and translation sit beside the call and choose how far they trail. Summaries run after it. Keep the media path for the cheap, local work and fork everything else.
What's Next
Once inference needs a server, the question becomes ownership. Next: where the inference runs and what leaves your network.
Frequently Asked Questions
Does real-time AI video add latency to calls?
Only features that sit in the media path, where every frame waits for the model. Noise suppression adds very little; background replacement on a phone can add about a frame. Captions, translation and summaries run on a copy of the stream or the recording, so the call itself never waits for them.
How fast does AI noise suppression need to be?
Fast enough to process each small chunk of audio before the next arrives, typically every 10 to 20 ms. The RNNoise design uses 20 ms windows with a 10 ms step and needed about 1.3% of one laptop CPU core, so it fits comfortably.
Why does background blur slow down video on some phones?
Segmenting a person from the background takes about 33–35 ms per frame on a Pixel 6 with Google's MediaPipe selfie model, which is the entire frame interval at 30 fps. Apps compensate by skipping frames, pipelining or lowering the frame rate.
How much delay do live captions have?
It is a design choice. Streaming models trade delay for accuracy; Kyutai's open models, for example, use 0.5 seconds and 2.5 seconds. Because captions are produced from a copy of the audio, that delay affects when text appears, not the call.
Where should AI features run on a video platform?
Per-frame enhancement such as noise suppression and background replacement belongs on the user's device. Transcription, summaries and sentiment need a server and should run beside the call or after it. Samvyo's architecture follows that split: noise cancellation and background replacement are designed to run on the device with no server, while transcription, summaries and sentiment run on servers.
Can AI processing on calls stay inside our own network?
It depends on where the server-side models run. On-device features never leave the device. For server-side features, Samvyo's design allows the media servers and AI inference servers to run in the customer's own cloud or on-premise under its licensing plan, which keeps that processing inside the customer's network; this does not apply to its pay-per-use plan.
