In this post I describe how I create a on-premise solution to handle with videos that needs scenes segmented, faces identified, subtitles generated, thumbnails picked, metadata pulled into the catalog. This post is about how we stopped trying to win that math with people and started winning it with our own AI.
TL;DR: How SOFA Digital built MIDAS, evolving from cloud APIs to on-premise infrastructure with our own computer vision models (cast recognition, nudity detection) and VLMs like BLIP-2, plus four lessons that probably apply to your stack too.
Stack: Computer Vision, FaceNet, YOLO, CLIP, BLIP-2, Python, FFmpeg, on-premise GPU workstations
Level: Advanced
Reading time: ~6 min
Media is a brutal workload, and cloud AI APIs solve the technical side of it. They introduce two new problems though: cost that scales linearly with volume, and data that leaves your infrastructure to be processed somewhere else. For an aggregator that ingests thousands of hours of licensed content from third parties, neither problem is a detail.
This whole thing started in my own study hours, more curiosity than strategy. The POC worked better than I expected, which says less about my code than about how generous open models have become. The content team adopted it before engineering had finished arguing about whether it was production-ready. That was the signal it was going to be real.
Start in the cloud (and do not feel bad about it)
The first version of MIDAS lived entirely on managed services: Azure for some pieces, AWS for others, vendor APIs glued together with Python. That was the right call, even with the per-video cost being uncomfortably high from day one. You validate a pipeline against real content first, then you optimize the bill. Reversing the order is how teams build elegant infrastructure for problems they have not proved are worth solving.
Cloud APIs also taught us where the real differentiation was. Transcoding and OCR were commodities. Scene segmentation tuned for our catalog and face recognition against our specific cast database were not. The harder we leaned into our actual content (turkish telenovelas, latin american movies, regional formats), the wider the gap got between “good enough” generic models and “actually useful for our business.”
Insight one: you do not leave the cloud because of the price, you leave because of the slope
If your cloud AI bill is flat month over month, on-prem rarely pays off. If your bill is growing because usage is growing and every new feature multiplies API calls per content unit, look at the slope, not the current number. We did not migrate because the bill was too high. We migrated because in twelve months it would be punishing, and the math for hardware suddenly worked. Cloud bills, like compound interest, are friendlier to whoever is collecting them.
The other half of the trigger was strategic. Licensors care where their content gets processed. Some explicitly. Some implicitly, in the clauses that surface at contract renewal. Owning the inference path gave us a story that “we use AWS for vision” never could.
Building the vision stack: insight two
The on-prem move unlocked the second phase of the project: building our own models for the work that mattered most. Cast recognition against our catalog. Nudity detection pipelines for content moderation. Scene-level understanding using VLMs like BLIP-2 to extract captions, context, and visual semantics that older CNN-only stacks could not reach.
We started with open architectures (FaceNet, YOLO, CLIP, BLIP-2) and fine-tuned them against our own data. The data was the hard part. Curating clean training sets for our actors and presenters, annotating scenes against our editorial taxonomy, building reliable benchmarks for sensitive-content detection (yes, somebody has to watch every frame that needs the right label): that work is unglamorous, and it is exactly where the value compounds.
This is also where my PhD work in computer vision started paying back into the day job in a direct way. Reading papers slowly, understanding what a published benchmark actually tells you, picking a backbone for the right reasons instead of the trendy ones: turns out research habits are a transferable skill, and nobody warned me about that during qualifying exams.
The model is commodity. The annotated dataset is the moat. Everyone has access to the same architectures. Almost nobody has access to your catalog, cleanly labeled, with edge cases documented. Spend more time on annotation than on model selection. You will outperform teams chasing the latest paper while their training data is still a mess.
The name MIDAS turned out to fit better than I planned: in the original myth, everything King Midas touched turned to gold; in our pipeline, every minute of unprocessed footage the system touches turns into structured metadata, recognized faces, scene boundaries, and searchable transcripts. The metaphor is on the nose, and I am okay with that.
Insight three: pick the cheapest model that clears your quality bar
VLMs like BLIP-2 are a good example of the heuristic we ended up trusting. For high-volume, domain-narrow work (tagging scenes, classifying frames, generating captions against a fixed vocabulary), a fine-tuned open model running on your hardware outperforms a frontier API for the work, at a fraction of the cost per inference. Save the frontier-API budget for the long-tail of low-volume, high-reasoning tasks where it actually earns its price. Paying per token to a model that knows everything about the internet but nothing about your catalog is a strange place to put money.
The next bet on this front is a small language model trained on SOFA’s catalog, running locally next to the vision stack. We have not built it yet. The reasoning is the same as everywhere else in the pipeline: a 7B model trained on your data should beat a frontier model on the open internet for the high-volume, domain-specific language work (synopses, recommendations, taxonomy classification), and the cost curve makes the trade-off comfortable. That one is on the roadmap, not in production. Yet.
Insight four: the metric you do not want to look at is the one that decides everything
People love benchmarks. The model that wins the leaderboard, the latest jump on HumanEval, the screenshot of a clever output. None of that runs your business. Two boring numbers run it: throughput (how much work per hour) and cost per inference (how much each output costs you). Track both from day one. Make decisions against both. Ignore the rest until those two are healthy.
The architecture that survives is the one that lets you swap models without rewriting the business logic, because today’s best model will not be the best model in six months (and probably not the best model next Tuesday, the way things are moving). Layer it: ingestion normalizes raw content, inference runs the models, enrichment turns outputs into business entities, orchestration coordinates everything. Each layer is replaceable. The only thing that should not move easily is your training data.
Next steps
- Audit your AI bill against your trajectory: Compare your last twelve months against the next twelve and decide whether the slope still makes sense.
- Spend a week on your annotation pipeline: Almost nobody does this seriously, and it is where the moat lives.
- Pick the boring metric: Track throughput and cost per inference from day one. Everything else is a vanity number.
- Build for swap, not for the model: A pipeline that lets you replace models without rewriting business logic is worth more than a clever integration with whatever is on top of the leaderboard.