How a video editor actually works

September 23, 2026 · 5 min read · by Nobody

Every video editor, from a phone app to a Hollywood suite, does the same four jobs: it keeps a timeline of what should appear when, it renders the picture at any moment, it encodes those pictures into a compact stream, and it packages everything into a file. Once you see those four jobs, every editor starts to look familiar.

A video is a stack of pictures

A video is a sequence of still images (frames) shown fast enough that your brain sees motion.

Those pictures are enormous. One 1080p frame is 1920 × 1080 pixels, and each pixel needs 3 bytes for red, green and blue:

• One frame: 1920 × 1080 × 3 = 6.2 MB
• One second at 30 fps: 187 MB
• One minute: about 11 GB

YouTube recommends uploading 1080p at 30 fps with a bitrate of about 8 Mbps (YouTube Help), which is 1 MB per second. So a typical encoded video is roughly 180 times smaller than the raw pictures.

That gap is why the rest of this post exists.

Job 1: the timeline

The timeline is the editor's model of your video. It's a set of tracks stacked on top of each other, and each track holds clips placed at a start time, with an in point and an out point into their source.

Nothing in the timeline is pixels yet. It's a description: "from 6.0s to 10.0s, show the title on top". Editing (trimming, moving, reordering) only changes this description, which is why it feels instant.

Job 2: rendering a frame

To show a frame, the editor asks one question for a moment in time: what's under the playhead? Then it builds the picture from the bottom track up.

flowchart TD
  T["Time t"] --> F["Find clips under the playhead"]
  F --> D["Decode each source frame"]
  D --> E["Apply effects<br/>(crop, color, motion)"]
  E --> C["Composite bottom to top"]
  C --> P["One finished frame"]

Preview plays this loop as fast as it can, often at lower quality so it keeps up. Export runs the same loop for every single frame at full quality, which is why exporting takes real time: a one-minute video at 30 fps is 1,800 frames to build.

Job 3: encoding (the compression trick)

Encoding turns finished frames into a compact stream. The key idea is that neighboring frames are almost identical, so most frames only store what changed.

• I frames (keyframes) are complete pictures. They're large, but anything can start from them.
• P frames store only the differences from an earlier frame.
• B frames borrow from frames before and after them, so they're the smallest.

This is also why jumping around in a video isn't instant. To show a frame in the middle of a group, the player has to decode from the previous I frame. Encoders place keyframes every couple of seconds as a balance between file size and how quickly you can seek.

Common video codecs are H.264 (plays almost everywhere), VP9 and AV1 (smaller files, newer). Audio has its own codecs, like AAC and Opus.

Job 4: the container

A codec compresses; a container packages. An .mp4 file isn't "H.264"; it's a box that holds an H.264 video stream, an AAC audio stream, and an index that says which bytes belong to which moment.

One detail matters for the web: an MP4's index (the moov box) can sit at the start or the end of the file. When it's at the start ("fast start"), a browser can begin playing before the whole file has downloaded.

Where it's all going

Classic editors decode footage: every frame starts as pixels from a camera. A different kind of editor is growing fast, where frames are computed from a description instead: shapes, text and motion rules. There's no footage to decode, the picture is sharp at any size, and the same frame always comes out identical.

That's the kind of editor I'm building: Vectorcut, which turns a script into a 2D animated video. It launches soon. Next in this series: how to render MP4 video entirely inside a web browser.

Nobody