Transcoding pipeline - 480 tasks, one file

A pipeline that turns one source file into a set of renditions, on machines that can be taken away at any moment, without ever producing a file that is quietly wrong.

Functional

• Accept a job naming a source and a target ladder, and return an id immediately.
• Split the source into independently decodable segments and encode them in parallel.
• Produce every rung of the ladder, in the codecs required, and stitch each one back into a single continuous rendition.
• Retry a failed segment without redoing the job, and stop retrying when the failure is clearly not transient.
• Report per-job progress that is honest - derived from completed tasks, not estimated from elapsed time.
• Verify output quality before publishing: a VMAF score per rendition and a scan for black or frozen frames.
• Account for what each job cost, in core-minutes, per rung.

Non-functional

• Throughput: 480 million tasks a day, 5,555 a second average and three times that at peak.
• A job of 8 minutes completes in under 10 minutes at p50 and under 45 at p99. The gap between those two is almost entirely the slowest task in each job.
• No task is executed twice in a way that can corrupt output. At-least-once delivery plus idempotent, content-addressed writes, never exactly-once as a promise.
• Correctness over throughput at every choice: a wrong rendition that is published is far worse than a slow one that is not.
• The fleet runs at 70 percent or more on preemptible capacity, and losing a machine mid-task is a routine event with no operator involvement.

Questions to ask before designing this

• What is the segment length, and who decided it? It sets the retry cost, the row count, the parallelism ceiling and the number of joins that can click. It is the single most consequential number here and it is usually inherited from a tutorial.
• Is the ladder fixed or per-title? Per-title encoding saves 20 to 30 percent of compute and adds an analysis pass that delays every job start.
• Are GPUs allowed? They are 20 times faster and worse at the same bitrate, so the answer differs per rung, and using them for archival masters is a decision you cannot undo without re-encoding the catalogue.
• What is the acceptable cost of a failed job - a retry, or a phone call? That decides whether the poison queue has an automatic consumer or a human one.
• How much does a job know about its deadline? Interactive uploads and catalogue backfills in one queue with no priority means the backfill wins by being first, which is exactly backwards.
• Is audio segmented too? Segmenting audio on video keyframe boundaries is how you get an audible click at every join, and it is found late.