Turning on live captions can send every word of a board meeting to another company's servers. So can meeting summaries, sentiment cues and translation. The video platform may run entirely on your infrastructure, and the AI on top of it may not.
On-prem AI video means keeping that inference inside your own network. Whether you need it, and what it costs, comes down to one question for each feature: where does the model run? Our post on which AI features can run without adding latency sorted features by deadline. This one sorts them by what leaves the building.
Three Places Inference Can Run
On the device | On servers you control | Third-party AI API | |
|---|---|---|---|
What leaves your network | Nothing | Nothing, if the media servers are yours too | Audio, video or text sent to the API |
Who processes the data | The user's device | You | The provider, under its terms |
Fits | Per-frame work: noise suppression, background replacement | Transcription, translation, summaries, sentiment | The same server-side features |
Capacity | Limited by the weakest device | GPUs you buy and run | The provider's problem |
Model quality and updates | Small models | Open or licensed models you update | Usually the newest models, updated for you |
On-device features are private by construction: the audio or frame is processed where it was captured. They are also limited to small models, so the heavier features land on a server. Which server is the decision that matters.
What a Third-Party API Actually Keeps
Sending audio to an AI API is not the same as giving it away. The terms are detailed, and they are getting better. Take OpenAI's API data policy as it read on 14 October 2026:
- API data is not used to train its models unless you opt in.
- Abuse-monitoring logs are kept for up to 30 days by default on most endpoints, including the Realtime API; the audio transcription endpoint keeps none.
- Zero data retention is available for eligible endpoints, including transcription and Realtime, but requires approval.
- Data storage can be kept in several regions, including India, but regional processing is listed only for the US, the EU and the UAE.
That last line is the kind of detail a security review catches. A company in India can keep stored data in India and still have its meeting audio processed abroad. Other providers' terms differ, so read the endpoint-level detail for the provider you use, not the headline policy.
Even with the best terms, the data still leaves your network and the provider becomes a processor of it. For many organisations that is acceptable with the right agreement. For others, the rules about where your media legally has to live make it a non-starter.
Shrink What Leaves Before You Decide Where
The choice isn't all or nothing. Most AI features need far less than the full media:
What the feature needs | Examples | What could stay inside |
|---|---|---|
Video frames | Background replacement, gesture or object detection | Run on the device |
Audio only | Transcription, translation, voice sentiment | Video never leaves |
Transcript only | Summaries, action items, search | Audio never leaves |
Metadata only | Talk-time, participation, call quality | Content never leaves |
A design that runs transcription in-house and sends only text to an external summariser exposes far less than one that streams the full call to an API. Removing names or account numbers from the transcript before it leaves narrows it further. Decide the minimum each feature needs, then decide where that minimum is processed.
Running Inference Yourself: What It Costs
Keeping server-side AI in-house turns a per-minute API bill into a capacity plan. Transcription is a GPU workload. Kyutai's open speech-to-text project, in its repository, reports serving 64 simultaneous connections on one L40S GPU at three times real-time speed, and says an H100 can process 400 streams in real time. These are the developer's own figures for its own models.
On those figures, 200 concurrent captioned streams need four L40S GPUs, or one H100 at the higher figure, before redundancy. Peaks matter here as much as for the media servers: if the all-hands is captioned, the GPUs have to cover it.
The other costs are less visible. Someone has to choose, update and evaluate the models. Quality may trail the newest commercial APIs. Summaries and sentiment need language models, which bring their own hardware and tuning. In exchange, nothing leaves, which for some deployments is the only acceptable answer: a deployment with zero internet cannot call an external API at all.
The Condition That Makes "Never Leaves" True
"Video never leaves your network" is a strong claim, and it is only true when every piece that touches the media is inside:
- The media servers. If calls route through a vendor's cloud, the media has already left, whatever the AI does.
- The AI inference servers. Every server-side model, including transcription, summaries and sentiment.
- Recording and transcript storage. Where the output lands matters as much as where it was produced.
- Client-side features. On-device processing keeps them local by design.
If any one of these is a third-party service, the claim no longer holds. When a vendor says media never leaves, ask which deployment and which plan that applies to. It is rarely all of them.
The counterweight belongs here too. For most organisations, a third-party API with zero data retention, regional processing and a processing agreement is a reasonable choice, and often the better model. On-prem AI video is for the cases where the data itself must not leave: regulated industries, government, sovereignty requirements and air-gapped sites.
The Bottom Line
On-prem AI video is a question of where inference runs, feature by feature. On-device work never leaves. Server-side work stays inside only if both the media servers and the inference servers are yours. Third-party APIs send data out under terms that are better than they used to be but still terms. Shrink what each feature needs, then put it where your rules allow.
What's Next
Where the AI runs is one part of a broader deployment choice. Next: on-prem, managed cloud or SaaS, and what each does to your data story.
Frequently Asked Questions
What is on-prem AI video?
Running the AI features of a video platform, such as transcription, summaries and sentiment, on servers inside your own network instead of sending media to an external AI service. It keeps meeting content inside, at the cost of running the GPUs and models yourself.
Does using an AI API for transcription send my meeting audio outside?
Yes. The audio goes to the provider for processing. Terms vary by provider and endpoint. OpenAI, for example, does not train on API data by default; its Realtime API keeps abuse-monitoring logs for up to 30 days unless you are approved for zero data retention, and regional processing is listed only for the US, EU and UAE.
Which AI features can run without any server?
Per-frame features like noise suppression and background replacement run on the user's device, so nothing leaves it. Transcription, summaries and sentiment need server-side models. Samvyo's architecture follows this split: noise cancellation and background replacement are designed to run on the device, while transcription, summaries and sentiment run on servers.
Can a video platform guarantee that video never leaves our network for AI?
Only if the media servers, the AI inference servers and storage all run in your environment. With Samvyo, that claim applies only on its licensing plan, where the customer runs both the media servers and the AI inference servers in its own cloud or on-premise. It does not apply to pay-per-use customers.
How many GPUs does in-house transcription need?
It depends on the model and concurrency. One open-source project reports 64 simultaneous streams on an L40S GPU and 400 real-time streams on an H100. Size for your peak, including large captioned meetings, and add redundancy.
Is a third-party AI API with zero data retention enough?
For many organisations, yes, combined with regional processing and a data processing agreement. Where rules require that media never leaves your control, or the deployment is air-gapped, inference has to run on infrastructure you operate.
