How We Made AI Clipping Actually Work on Twitch VODs

Twitch support was the most requested feature we've ever shipped, and by far the hardest. On paper it sounds simple: we already find clips in YouTube videos, and a Twitch VOD is just another video. In practice, almost nothing from our YouTube pipeline survived contact with Twitch. This post is the story of what broke, and the machine we built so an 8-hour stream comes out the other end as a handful of accurate, ready-to-post clips.

Why Twitch Is a Different Beast Than YouTube

When you submit a YouTube video, we can hand the AI a link and it watches the video directly. No download, no upload, no storage. YouTube also gives us captions, with timing, for most videos.

Twitch gives us neither. There's no way to point an AI model at a Twitch VOD, so every second the AI watches is a second we have to fetch, compress, and upload ourselves. And there are no captions at all, so if we want to know what was said (we very much do, more on that later), we have to transcribe it ourselves.

Now add the shape of the content. A typical YouTube submission is a 1 to 2 hour podcast. A typical Twitch submission is a 4 to 8 hour stream. Anything mildly expensive on YouTube gets brutally expensive on Twitch unless you're careful about what you actually touch.

Downloading Only the Minutes We Need

The first rule of the Twitch pipeline: never download the whole VOD. If the best moment of an 8-hour stream happens in hour 6, we should pay for a few minutes of footage, not 8 hours.

Twitch stores a VOD as thousands of small video pieces, so in theory you can grab just the pieces you want. In practice there's a trap: the very first piece of the video contains the instructions a video player needs to decode all the others. The standard downloading tools skip straight to the part you asked for, leaving those instructions behind, and you get a file that downloads fine, looks fine, and plays exactly zero frames.

So we wrote our own fetcher. It figures out exactly which pieces overlap the window we want, grabs them in parallel, retries the stragglers, and always stitches that critical first piece back on. The payoff: cost scales with the size of the clip, not the length of the stream. Hour 7 costs the same as minute 2.

Then we double-check everything, because Twitch downloads fail in creative ways. A “successful” file can be an empty shell, or have a silent hole in the middle. Our favorite: right after a stream ends, Twitch sometimes only serves the tiny preview thumbnails, so a naive downloader proudly delivers a slideshow of JPEGs instead of a video. Every file has to prove it actually plays before the pipeline trusts it.

Writing the Transcript Ourselves

With no captions to lean on, we build our own transcript. Twitch exposes an audio-only version of every VOD, a tiny fraction of the video's size, so that's all we pull for this step.

We split the audio into 20-minute pieces and transcribe them all in parallel with a speech model that returns the timing of every single word. That word-level timing turns out to be the most valuable data in the whole pipeline, and the next section is why.

This step bends instead of breaking. If one 20-minute piece fails to transcribe, we keep going with a gap rather than killing the job. A stream with 20 corrupted minutes still gets clipped from the other 7 hours.

The AI Finds the Moment, the Transcript Finds the Time

Here's the uncomfortable truth about using AI on long videos: models are great at spotting the moment and unreliable at telling you when it happened. Ask one for a timestamp and it can be off by seconds, sometimes minutes. When a clip is 45 seconds long, seconds are everything.

Our fix: the AI is never allowed to be the source of truth for time. When it finds a highlight, it has to give us the moment as a full word-for-word quote of what was said. We then match the start and end of that quote against our transcript, forgiving filler words and small differences, and read the exact start and end times off the words themselves. The AI's timestamp is only a hint about where to look, never the answer.

This one idea is what makes Twitch clips land on the actual moment. In testing, it caught the model confidently placing a highlight 40 seconds away from where it really happened. And when a quote can't be matched (it happens), we fall back to the AI's timestamp instead of failing, and we record which method we used for every clip.

Assume Everything Fails, Resume Anywhere

A long VOD job talks to Twitch, our download workers, a transcription service, an AI model, and storage, hundreds of times. Something will hiccup. So the whole pipeline runs on one principle: track progress in small units, and never redo finished work.

Every VOD is analyzed in chapter-sized segments, roughly 25 to 40 minutes each, and each segment keeps its own status: waiting, working, done, or permanently failed. If a job dies at segment 9 of 14, retrying picks up at segment 9. The transcript, the chapters, and the 8 finished segments are all reused. Segments also move independently, so segment 1 can be downloading its clips while segment 4 is still being analyzed.

The last piece is honest failure. A subscriber-only VOD tells you it's subscriber-only. A stream that ended 20 minutes ago, which Twitch hasn't finished processing, says “not ready yet, try again shortly” instead of a generic error. Getting those messages right ended a whole class of pain: people resubmitting jobs that could never work, and people giving up on jobs that would have worked an hour later.

The Bottom Line

Making AI clipping work on Twitch wasn't one hard problem. It was five medium ones stacked in a trench coat: no direct AI access to the video, no captions, downloads that fail silently, AI timestamps you can't trust, and jobs so long that something always breaks. The answer to each was the same instinct: touch as little data as possible, verify everything, and let every step fail without taking the job down with it. If you stream on Twitch, all of this machinery sits behind one paste of a VOD link.