We wrote a video codec to beat H.264

Every Livsho show is a video, and every viewer pays for it twice: in mobile data to download it, and in battery to decode it. Today those videos are H.264 or HEVC, whichever the viewer's phone handles best. Both are excellent, and neither was designed for what a Livsho show actually looks like. So we started writing our own.

Meet fdv, short for Fast Decode Video. It's a video codec written from scratch in C11: one header, zero dependencies, no borrowed code, and it's open source at github.com/yhbsh/fdv. It isn't Livsho's codec yet. This post is about why we want one, how far fdv has come, and how we plan to tune it for Livsho shows specifically.

What a Livsho show looks like to a codec

Watch any show on Livsho and you'll see the same shape. A seller sits in front of a wall, a shelf or a desk that never moves. One product at a time is held up to the camera: a trading card, a figure, a game. The seller's hand turns it, tilts it, points at the corner that matters. Every few minutes the product changes and the auction starts again.

To a codec that's three very different jobs:

• The room, which is most of the frame and never changes. It should cost almost nothing: big blocks, copied from the previous frame.
• The product, which is small but is the reason anyone is watching. Card text, holo foil, the paint on a figure. That's where the bits should go.
• The hand, which moves all the time. A codec shouldn't redraw it, it should describe the move.

General codecs handle all three, but they're tuned for everything at once: films, sports, video calls. A codec tuned for this one shape can spend almost nothing on the room and put the savings into the product. For a buyer deciding whether a card is near-mint, that difference is the whole point.

Why it matters to people watching

• Sharper products at the same data. If the room is nearly free, the same bitrate buys more detail where the buyer is looking.
• Shows that hold up on weak networks. Fewer bits for the same picture means fewer stalls on a crowded 4G cell.
• Less battery per hour. People watch shows for an hour or more. A decoder that does less work per frame is battery the viewer keeps.
• Better clips and video invoices. Our clip pipeline decodes every recording. Faster decoding makes clips faster.

fdv today, by the numbers

• 8,482 lines of C in a single header (fdv.h), with no dependencies beyond libm and pthreads.
• At live-streaming settings (720p, 30 fps, 1 Mbps, 2-second GOP), fdv beats x264's low-latency preset on all 23 test scenes, by 7.38 dB of PSNR on average.
• It decodes a 720p frame in 0.83 ms. libavcodec's H.264 decoder took 1.20 ms in our measurement.
• A repeated frame has its own fast path and encodes in 0.27 ms. That matters a lot for a room that never moves.

The one rule: decode wins

A show is encoded once, on the seller's phone, and decoded by every viewer. So every time a design question came up, the answer that made decoding cheaper won, even when it made encoding harder. In practice that means table-driven, branch-light entropy decoding, small integer transforms that fit SIMD registers, and frames that split into bands which decode in parallel on separate cores.

Under the hood

Blocks that split where it pays

Key frames are cut into 64×64 coding tree units, and each one is split recursively (64 → 32 → 16 → 8 → 4) only where splitting saves more bits than it costs. That's exactly the behaviour a Livsho show wants: one big block for a patch of wall, a cascade of 4×4 blocks around the edge of a card. 4×4 blocks pick from 9 directional prediction modes, and larger ones use DC, vertical, horizontal or plane. Modes are ranked by SATD, a cheap Hadamard-transformed error, before anything is actually coded.

Motion, for the hand

Inter frames use 16×16 macroblocks that can split into 8×8, quarter-pixel motion with a 6-tap luma filter, a diamond search seeded from neighbouring vectors, and two reference frames. Unchanged blocks are SKIPped, and a SKIP names which of up to 3 neighbouring motion vectors it copies (HEVC calls this merge), which cut bitrate by 6.8% on its own. On our divergent scene, with opposed motion inside single blocks (think fingers), fdv needs 17% fewer bits than x264.

Two entropy coders, and the smaller one wins

Every frame's symbols are split into separate streams (16 on inter frames, 10 on key frames), and each stream is coded twice:

• A static rANS coder, interleaved two ways, that lands within 0.6% of the theoretical entropy limit.
• An adaptive range coder primed from what has already been decoded, for short streams where a frequency table would cost more than it saves.

Whichever is smaller gets written. The adaptive coder is slower to decode, so it gets a hard budget of 65,536 serial symbols per frame, spent on the shortest streams first. Decode time stays bounded no matter what the seller points the camera at.

If two kinds of symbol need different sentences to describe, they want different models.

A lesson we measured four separate times

The rest of the pipeline

• Integer 4×4 and 8×8 transforms chosen per 8×8 region, with a dead-zone quantizer and a lightweight rate-distortion-optimised quantization pass.
• A deblocking filter on the 8×8 grid, which saved 2.2% of bitrate and made decoding faster (0.83 → 0.78 ms per frame).
• Rate control that holds every frame to its own bit budget and re-encodes a frame that comes out more than 1.5× over, up to 3 times. A 1,000 kbps target on real camera footage landed at 1,006 kbps.
• Hand-written NEON SIMD for the hot loops: inverse transforms, dequantisation, SAD, chroma motion compensation, and the deblocking filter.
• Frames split into horizontal bands that encode and decode on separate cores. Real 720p camera footage encodes in 45.8 ms per frame on one thread and 7.8 ms on 8 bands, which is comfortably live.

The scoreboard

At live-streaming settings, the ones a Livsho show actually uses, fdv is ahead on every scene we have. The harder fight is x264's slower "medium" preset, which has time to think. Here's how many more or fewer bits fdv needs than x264 medium for the same quality, scene by scene:

fdv wins 9 of 23 outright, some by a lot (stress −88%, chroma −32%, plaza −28%, grain −20%). On average it still needs 9% more, down from a 64% median gap when this work started. Notice where it does well: grain, colour, layered and busy scenes. That's what a show under studio lights looks like.

Test clips are descriptions, not downloads

fdv ships its own renderer: a small scene language that draws 23 deterministic clips, so every test produces identical frames on every machine. Each scene stresses one thing: grain, rotation, hard cuts, a flight over ray-marched mountains. The next scene we'll add is the one that matters most: a desk, a shelf, a card held up to the camera, and a hand that won't keep still.

Bugs worth framing

• The grain cliff. Film grain at QP 32 needs 22.2 Mbps, at QP 34 1.80 Mbps, and at QP 36 just 114 kbps. A 1 Mbps target doesn't exist between those steps. Giving the operating point a fractional part, carried on λ, took the encoder from 522 to 922 kbps against that target.
• False sharing. Threads publishing their progress on the same cache line wrecked multi-core scaling until the counters were padded apart.
• The stack. Moving to 64×64 coding units first crashed with SIGBUS: about 440 KB of stack, against the 512 KB a thread gets.

The road to Livsho's codec

The goal is for fdv to sit next to H.264 and HEVC in the Livsho app, and one day replace them: the seller's phone encodes it, every viewer's phone decodes it, and our clip pipeline draws from it. Beating H.264 is the first milestone, and HEVC is the next. Here's the plan:

• Tune for shows. Build a Livsho scene into the test set, then tune on real recordings: cards, figures, hands, studio lights.
• Know where the product is. Spend bits on the region the seller is holding up, and keep the room on SKIP.
• A new product means a fresh picture. Detect the swap and send a key frame right then, instead of dragging the old product's motion along.
• Phone decoders. A fast fdv decoder inside the app on iOS and Android, so viewers spend less battery than on H.264 or HEVC.
• Servers too. AVX2 for x86, so our recording and clip boxes can read fdv as fast as phones do.
• Mobile networks. Recover cleanly from lost packets without waiting for the next key frame.

It's an insane experiment. Every time we measured it, it got a little less insane.

All the code, the scenes and the measurements are on GitHub. Read it, run it, and tell us where we're wrong.

Read the code on GitHub

Get Livsho