Stop Piping Video Through Canvas: WebCodecs Has Real Codec Control
For years, the only way to encode video in the browser was to draw frames onto a canvas and let MediaRecorder do the rest. You picked a MIME type, and the browser picked the codec. You wanted VP9 at a specific bitrate, or to drop the first frame, or to transcode on the fly. None of that was possible without a WASM build of ffmpeg, which is a couple of megabytes to download and a per-frame tax on top of that.
WebCodecs changes that. It is a small set of browser-native interfaces that hand you hardware-accelerated, per-frame access to video and audio codecs. You feed it raw frames, it hands you compressed chunks, and you decide the codec, the bitrate, and the keyframe cadence. There is no canvas round trip and no WASM to ship, because the GPU and the browser's native codec stack do the work.
The four objects that matter
Everything in WebCodecs comes down to four types. A VideoFrame is a single frame of raw pixels, backed by GPU memory, with a timestamp, a duration, and a pixel format. An EncodedVideoChunk is the compressed version of that same frame, backed by regular memory. A single chunk typically holds 10 to 100 times less data than its raw frame, and that ratio is the whole reason the API exists. The VideoEncoder turns frames into chunks; the VideoDecoder does the reverse. Audio has the same shape, with AudioData and AudioEncoder standing in for the video types. A typical AudioData block holds 1024 samples, and you read the raw floats out with copyTo(). There is no direct bridge into the Web Audio API, so you copy the samples into an AudioBuffer yourself if you want to play them back.
The encoder and decoder are asynchronous. Each one keeps an internal processing queue, and the methods you call are just messages you append to the back of that queue. configure(), encode(), and decode() all queue work and return a promise. flush() queues a barrier and resolves once everything ahead of it has finished. reset() and close() are the only synchronous calls, and they purge the queue. The practical upshot is that you are managing a pipeline, not calling a function and reading the result.
A minimal encode loop
const encoder = new VideoEncoder({
output: (chunk, meta) => {
chunks.push(chunk);
// mux into MP4 or WebM here
},
error: (e) => console.error(e),
});
encoder.configure({
codec: 'vp09.00.10.08',
width: 1280,
height: 720,
bitrate: 2_500_000,
});
// for each frame:
const frame = new VideoFrame(source, {
timestamp: now * 1_000_000,
duration: 16_667,
});
encoder.encode(frame, { keyFrame: frameIndex === 0 });
frame.close();
Two details trip people up. First, the timestamp is in microseconds, not milliseconds, so multiply by a million. Second, you must call frame.close() to release the GPU memory. Frames are not reclaimed the way you expect, and holding onto them leaks. The keyFrame hint asks the encoder to emit a key frame, which is how you get a chunk whose type is "key" instead of "delta".
Where it actually shines
The obvious win is browser-based editing. Tools like Clipchamp and a few other web editors use it to transcode, trim, and export without shipping a native binary. A "key" chunk is a self-contained frame you can seek to; a "delta" chunk only makes sense relative to the last key frame. That is exactly the model a timeline editor needs, and it is something MediaRecorder never exposed.
The less obvious win is live streaming and conferencing. Because you own the encoder, you can adapt the bitrate to the network, drop frames under load, or encode at a resolution the source camera does not natively produce. WebRTC still owns the transport, but the codec decisions are yours now.
The gotchas
Codec support is the big one. H.264 is the most widely supported and is what most MP4 files use. VP9 compresses better and lives in WebM, but older Safari does not decode it. AV1 is the newest and the best per bitrate, but encoder support is still patchy on Apple hardware. Check VideoEncoder.isConfigSupported() before you commit to a config, and fall back rather than assume.
Then there is the queue. If you encode faster than the hardware can keep up, the queue grows, your frames pile up behind each other, and your real-time pipeline quietly becomes a buffer. Watch encoder.encodeQueueSize and apply backpressure instead of firing and forgetting.
When to reach for it
If you need to pick a codec, control the bitrate, emit key frames on a schedule, or transcode per frame, WebCodecs is the right tool and there is no good alternative. If you just need to record a tab or a camera into an MP4, MediaRecorder is simpler and good enough. WebCodecs is a low floor rather than a finished product: it gives you the codec primitives and leaves the muxer, the seek index, and the streaming logic to you. That is more work, but it is the work that used to require a native build.
Comments