You have a landscape video and you need it vertical for Reels. You center crop it, look at the result, and it is fine. Then the person takes two steps to the left and spends the next fifty seconds talking to the edge of the screen. Fine, you will just follow the speaker automatically. That was the exact moment you signed, without reading it, a contract for an estimation problem over a noisy signal. This post is about what is inside that contract.
TL;DR: The signal processing and computer vision fundamentals behind automatic vertical reframing: scene segmentation, landmark geometry, variance as a speech proxy, band limited energy gating, IIR filtering and hysteresis.
Stack: Python, NumPy, OpenCV, MediaPipe Face Mesh, PySceneDetect, FFmpeg
Level: Advanced
Reading time: ~9 min
The real problem is not finding faces. Finding faces has been solved for a decade. The problem is that reframing means deciding, frame by frame, where to point a virtual camera, and every input to that decision lies to you in a different way: the detector disappears for half a second, mouths move with nobody speaking, and the audio confirms that somebody is talking without saying which of the four people in the shot it is. Worse, the decision is temporal. A choice that is correct on an isolated frame can be wrong as a sequence, because the viewer does not perceive position, they perceive motion. That is how you get 100 percent of frames right and still ship video that makes people queasy.
I learned that the expensive way. My crop kept sweeping across the frame and back, and I spent a good while convinced MediaPipe had broken. It had not. When I stopped guessing and printed the candidate scores, the four faces in that shot were separated by 0.0002. The detector was flawless. I was the one who was wrong, and specifically my decision rule was.
Segment on the edit, not on the clock
Before you track anything: where did the camera cut? A cut is a discontinuity, and every temporal filter in the pipeline has to be reset there. Scene detection reduces each frame to a compact descriptor, usually an HSV histogram, measures the distance between consecutive frames, and flags the peaks.
The naive version compares that distance to a fixed threshold. It breaks on the first real scene you feed it. A dark dialogue scene has tiny frame to frame distances, so genuine cuts slip through. A handheld shot in daylight blows past the threshold on its own, with no cut at all. A fixed threshold only works on the video you tuned it against, which is another way of saying it does not work.
Adaptive detection changes the question:
score = distance[t] / (rolling_average(distance, window) + epsilon)
is_cut = score > adaptive_threshold
You stop asking “is this change big” and start asking “is this change big for this part of this video”. Same trick as anomaly detection, and it comes back three sections from now, with audio.
A face becomes a coordinate system
A modern face mesh does dense landmark regression: it maps an image region to a fixed set of points, 468 of them for MediaPipe, in a canonical topology. Index 13 is the inner upper lip in every frame of every video, forever. That fixed indexing is what makes the rest possible, because it turns “a face” into something addressable.
Two properties matter more than the model architecture. Coordinates come back normalized from 0 to 1 against the frame, which decouples every downstream calculation from resolution. And the model returns no identity: two faces in shot become two landmark sets in an arbitrary order that changes between frames.
So it is a detector, not a tracker. Confusing the two is the most common bug in this kind of pipeline, and I say that with the authority of someone who shipped exactly that bug.
Speech is a mouth changing, not a mouth open
You can estimate “this person is talking” from geometry alone. The temptation is to measure how open the mouth is and threshold it. That approach fails on anyone whose mouth rests slightly open, and fails again on anyone who talks through their teeth.
So you measure the aperture over a short window and take its dispersion:
aperture = [abs(lm[13][1] - lm[14][1]) / face_height(lm) for lm in window]
speech_proxy = np.std(aperture)
Two things do the heavy lifting in those two lines. Dividing by face height makes the measure scale invariant, so the person glued to the lens and the person in the background compete on equal terms. Without it you are not measuring speech, you are measuring distance to the camera. And the standard deviation, instead of the mean, is what encodes “changing” rather than “open”.
Now the honest part: this is a weak, purely visual correlate. Chewing scores well. Laughing scores very well. Yawning is practically a keynote.
Audio answers when, video answers who
A single mixed audio track cannot tell you which person is speaking. The standard reaction is to reach for diarization, which costs you WhisperX, Pyannote, a HuggingFace token and a few gigabytes of model. It works well, to be fair. But look at what you are buying: the “who”. And you already have the “who”, from the lips. What is missing is the “when”, and “when” is cheap.
Band limit to the voice range and take short time energy:
# after band-passing 200 to 3500 Hz and downmixing to mono
rms = np.sqrt(np.mean(chunk ** 2))
RMS is the standard way to summarize the amplitude of an oscillating signal, because the waveform swings symmetrically around zero and a plain average collapses to nothing. Square it so everything is positive, average for the power, take the root to get back to the original units.
Then comes the decision that separates what works on your test clip from what works in production: never compare that number to an absolute threshold. A podcast mastered to broadcast levels and a phone in someone’s pocket have nothing in common in absolute terms. Normalize against a high percentile of the file itself:
gate = np.clip(rms / np.percentile(rms, 95), 0.0, 1.0)
Now the number means “loud relative to how loud this file gets”. The 95th percentile and not the max, because the max is one clipped transient away from destroying your entire scale.
The fusion itself is a multiplication. You multiply the visual proxy by the gate, and mouth movement with no voice behind it collapses toward zero, while the choice between faces stays entirely with vision. Two sensors, two different questions, one product. That is sensor fusion at the score level, and it costs you a band-pass and one array operation.
Smoothing does not fix errors, it makes them elegant
Raw per frame position jitters. The standard fix is an exponential moving average, and it is worth recognizing what it actually is: a first order IIR low pass filter.
smooth = alpha * target + (1 - alpha) * smooth
One line, one state variable, no buffer. The useful intuition is the time constant: the filter covers about 63 percent of a step change after 1/alpha frames. At alpha 0.12 and 24 fps, that is eight frames to react and close to a second to settle.
Here is the catch. Low alpha is beautiful against jitter and brutal against legitimate change, because the filter cannot tell noise you want gone from real motion you want followed. Every wrong jump in the input becomes a slow, deliberate, extremely visible camera move. If your tracker is wrong for four frames, a heavy filter turns that into a second of drift that reads like a directorial decision. It is worse than the original twitch.
Argmax is a terrible tracker
Picking the best candidate every frame produces an estimator with no memory. Output stability is exactly the gap between the top two scores, nothing more. When that gap approaches the noise floor, the selection oscillates at whatever rate you sample.
My four faces at 0.0002 apart were not an edge case. They were the predicted behavior of an argmax over a near tie. I was the one expecting something else.
The fix is to stop treating each frame as an independent decision and charge a toll for changing your mind:
score = quality(candidate) - penalty * distance(candidate, current_target)
A candidate near the one you already follow pays almost nothing, so tracking a subject who walks keeps working. A candidate across the frame has to be clearly better to win. That asymmetry between staying and switching is hysteresis, a word James Alfred Ewing coined in the 1880s from the Greek for “lagging behind”, describing magnetic material whose state depends on its history and not only on the field applied right now. Your crop should lag behind in the same sense.
Tune the penalty by measuring, and expect a knee rather than a ramp. In my sweep the largest jump inside a shot fell from 0.629 to 0.184 as the penalty went from 0 to 0.30, then stayed flat all the way to 2.0. Past the knee you are only buying stubbornness.
Finding the picture inside the frame
A letterboxed source hides a quiet geometry bug. A 2.4:1 film inside a 16:9 container has black bars that, as far as your crop math is concerned, are part of the frame. Result: the vertical window spends close to a quarter of its pixels rendering black, and nobody complains because it looks intentional.
Detecting it is easy, scan for rows uniformly below a black threshold. But two decisions separate what works from what destroys footage in silence.
First, sample at several points in the video and take the union, never the intersection. A sample landing on a fade to black reports a tiny box, and cropping to it eats real picture. Union cannot fail in that direction.
Second, reject thin trims. Vignetting and dark corners make the detector shave a few pixels off video with no bars at all. Require any bar to be a meaningful fraction of the dimension before believing it. A real letterbox on a 2.4:1 film sits at 12 percent of the height. A vignette sits under 2. The gap between them is wide enough to sleep on.
Let the encoder move the window
The last piece is architectural. You have a trajectory, one crop position per frame. The obvious implementation is a Python loop that decodes, crops and re-encodes frame by frame. It is slow, and not because of Python’s speed: it is because you are hauling every frame into your process and back out, all for a parameter change.
The alternative is to keep the pixels inside the encoder and send only the parameters. FFmpeg’s sendcmd filter reads a timestamped command file and mutates another filter’s arguments as the timeline plays:
0.000 crop x 148, crop y 44;
0.042 crop x 149, crop y 44;
Your code describes the path, the encoder walks it, and the whole thing becomes one pass. The principle is worth stealing outside video too: when the heavy machinery already has a scheduler, hand it a declarative description of what you want over time instead of driving it by hand from outside.
What you learned
You now have the vocabulary for every decision in a vertical cropper: adaptive thresholds instead of fixed ones, dense landmarks as a coordinate system, dispersion instead of magnitude, relative instead of absolute normalization, multiplication as fusion of two sensors answering different questions, an IIR filter with a time constant you can defend, hysteresis so the estimator stops changing its mind, and a declarative handoff to the encoder at the end.
Underneath all of it, reframing is estimating a trajectory over a noisy signal. None of this is new. Most of it predates the neural network in the middle of the stack by decades, and it is still the part that decides whether the video is watchable.
Next steps
- Identity tracking: match detections across frames by bounding box overlap, so per person history stops being indexed by list position. It is the root fix for the problem hysteresis only papers over.
- Profile detection: landmark asymmetry tells you when a face is turned away, which helps when the largest face in the shot is the back of someone’s head.
- Reset on discontinuity: a temporal filter crossing a cut is averaging two unrelated signals. Segment first, filter second.
- Measure motion, not position: score by accumulated displacement and direction reversals. Per frame accuracy will happily hide a crop that never sits still.
Questions or feedback? Find me on LinkedIn or GitHub.