How we taught the app to film itself

You're watching a live show. The seller pulls the card everyone was waiting for, the chat explodes, someone outbids you in the last second. You tap Clip. About twenty seconds later the moment is sitting in My clips, exactly the way you saw it. This is how that button works, all the way down to the bytes.

By the numbers

Clips went live on 27 September. Projecting from the first weeks of data, here's the first year:

• 1,000,000 videos drawn.
• 7,000+ hours of finished video.
• 35,000+ shows recorded end to end, every stream of every one.
• 60 TB of raw recordings written, each kept for a year.
• 4,000+ GPU-hours of drawing.
• 3 seconds to draw a 10-second clip, our next target, and about 20 seconds from tapping Clip to the video in your hands.

Version zero: a borrowed camera

Our first clips came from an off-the-shelf cloud recorder. It worked. It was also expensive, and what came out looked like a screen recording of a video call: no bids, no hearts floating up, none of the room. The moment was there. The feeling wasn't.

So we did the slightly unreasonable thing and built the whole pipeline ourselves. The entire thing is a single C program of about 4,000 lines: the RTMP server, the recorder, the uploader, the renderer host, the encoder and the audio mixer. One file, one binary.

Part one: the recorder

Every stream on a live show (the host, each co-host, each guest on stage) is forwarded to our recorder. The recorder never re-encodes a single frame and has no idea what a clip is. Its whole job is to write things down.

We wrote our own RTMP server

Phones publish over RTMP, so the recorder is an RTMP server, written by hand and publish-only. There's no playback path, which means there's nothing to attack there either.

• The handshake is the classic 1,536-byte C1/S1 exchange. Incoming chunks start at 128 bytes, and we ask for 4,096 going out.
• Each connection gets its own thread, with a 30-second receive timeout and TCP_NODELAY on.
• Every stream has its own key in its URL, so a URL that leaks only opens that one stream.

FLV tags straight into libav

RTMP carries FLV tags. We turn each one straight into an AVPacket by hand, with no parser in between: dts is the tag's timestamp, and pts is dts plus the composition time. H.264, HEVC and AAC all go through.

HEVC was a late addition. Streams from OBS send it as FLV codec id 12, which we weren't reading, so those shows were recorded as audio only and their clips came out black. Then 32-bit timestamps wrap around, so we unwrap them. And we hold up to 512 packets (or 2 seconds) until both codec configs have arrived, so a segment never starts without knowing what's in it.

Four-second segments, zero re-encoding

Packets go into libavformat's MPEG-TS muxer, and a new segment starts at the first video keyframe at least 4 seconds in. Each finished segment adds one line to the stream's manifest:

# seq  start_ms       duration_ms  has_video
000041 1791262803120  4000         1
000042 1791262807120  4033         1

start_ms is where it gets interesting. Our first version used the time each segment arrived, and arrival times jitter: boundaries moved by 50–160 ms on average and up to 1.8 s. Now we take the stream's own clock and add the smallest arrival delay seen over the last 30 seconds. The lag barely moves, so the segments line up end to end.

Uploads without an SDK

Four upload threads push segments to object storage. Within one stream they always go in order, so the manifest never names a segment that isn't there yet. Failed uploads retry with 2ⁿ-second backoff, capped at 64 s.

We didn't pull in a cloud SDK, because we already had everything it would give us. libavutil already does SHA-256 and HMAC, so we sign the requests (SigV4) by hand. FFmpeg's own https protocol, running on mbedTLS, already knows how to send a PUT.

Statically linked, very pinned

FFmpeg 9.0.2 is configured with --disable-everything and then just the handful of muxers, decoders and encoders we use. It's linked statically together with mbedTLS 3.6.7, built with zig cc against glibc 2.39, and comes out as one 7 MB binary. That's the same binary that later draws the clips. Two recorders run side by side on separate machines, and readers merge the two copies, so if one hiccups its twin fills the gap.

Part two: the app draws its own clip

This is the trick we're proudest of. A clip shouldn't look like the app. It should be the app.

On a GPU machine we embed our app's rendering engine headless: no window and no phone, just Vulkan 1.1. It's loaded with dlopen, and our real live-screen code is compiled ahead of time and runs inside it, with deterministic rendering on and the debugging machinery off. The layout is drawn at the phone's own 402-point width and scaled up to 1080 pixels wide.

Frames in

libavcodec decodes the recording. Each frame is cropped to the clip's shape and converted to RGBA with sws_scale at its original size, then handed to the app over a platform channel as an 8-byte header followed by the pixels. The GPU does the upscaling. A decoder thread stays 3 frames ahead, so the next picture is ready before the current one has finished drawing.

A virtual clock

The app thinks it's live. Frame k happens at k × 1000 / fps milliseconds, and every timer in the app runs on that virtual clock instead of the wall clock. The show's event log (bids, likes, joins, the auction timer) is replayed against it, so everything lands on exactly the frame where it happened, at whatever speed the GPU can manage.

Frames out

Each finished frame is copied out of a Vulkan image (we cycle through 3) into host-cached memory, then into a ring of 4 slots that an encoder thread drains. NVENC takes RGBA directly, so the colour conversion happens on the encoder chip and not on our CPU. The clip and its 1080×1920 Stories version are encoded in the same pass:

clip    hevc_nvenc  1080x2340  preset p5, tune hq, VBR
        3.5 Mbps avg / 5 Mbps max, lookahead 32, 3 B-frames
        spatial + temporal AQ, keyframe every 4 s, hvc1, faststart
story   h264_nvenc  1080x1920  preset p4, 4 / 6 Mbps, 2 s GOP
colour  BT.601, limited range (tagged, so players agree)

Sound, levelled

Audio follows whoever is on screen, at 48 kHz in float stereo. Switches get 10 ms crossfades so they don't click. Then we level it: we measure loudness in 400 ms blocks, gate out the quiet ones, and bring the whole clip to about −16 dBFS, with gain capped at 5.6×. A limiter with 5 ms of look-ahead and an 80 ms release catches the peaks. It goes out as 96 kbps AAC.

The nice side effect of all this: when we redesign the live screen, clips get the new look for free. The less nice one: when the live screen breaks, clips break with it. Early on, every piece of text in a clip came out with a bright yellow double underline, which is the engine's polite way of saying you forgot something.

Part three: faster than the moment

Every step here was measured, and the clip was the benchmark:

• A test harness → the embedded engine: a 10 s clip went from 13.0 to 8.5 s.
• Interpreted → ahead-of-time compiled: 8.4 to 6.1 s.
• Going to full 1080×2340 resolution pushed the cost from 13 s to 31 s. We clawed it back by letting the GPU do the scaling (image conversion 22.8 → 8.4 s) and by feeding NVENC RGBA directly (encoding 14.3 → 7.1 s).
• Decode-ahead, prefetching the next picture, and the Stories version in the same pass: a 10 s clip plus its Stories version went from 13.9 to 6.2 s, and a 30 s clip from 32.3 to 12.4 s. Against the old build the output was 43–46 dB PSNR, so it looks identical, just faster.

Next stop: 3 seconds for a 10-second clip.

Most of what's left is the encoder. We picked HEVC because it makes small, sharp files, and it also happens to be the slowest codec we could have picked. A faster encode path is next on the list, and this post will get a new number when it lands.

The GPU machines (NVIDIA T4s) handle 8 clips or 16 invoices at a time, up to 4 machines. The backend starts them when work is waiting and stops them once they've been idle for a while. A cold machine claims its first clip about 20 seconds after it's started. Nobody should be paying for a graphics card that's staring at the ceiling.

Everything a machine runs ships as one pinned release: the binary, the worker, the engine, the compiled app and its fonts. The engine itself is built from source in release mode, which takes about an hour on a throwaway 64-core machine, and our build reproduced the production engine byte for byte.

Bonus: the steadiest frame

The recording paid off somewhere we didn't expect: auction pictures. When an auction opens, we look at the next 2.5 seconds of the seller's stream and score every frame as detail ÷ (1 + motion). Detail is the variance of a Laplacian over the luma channel, and motion is the average change from the previous frame, sampled at every 8th pixel. The winner is the sharpest frame where the seller's hand wasn't moving, so you get the item and not a blur.

Bonus: receipts, but cinematic

Once we had a machine that could film any moment, we pointed it at the best one: winning. Win an auction and you get a video invoice, the last seconds before the hammer with your winning bid on screen. It's the only receipt people watch twice.

Room to grow

The curve above is our projection for the first year, made with the optimism of people who just watched their pipeline work, so hold us to the shape rather than the decimals. It ends at around 5,500 videos a day.

The best part is that it barely needs more machines. One GPU machine draws 16 invoices at a time at about 15 seconds each, which is roughly 90,000 a day on paper. Four of them cover the forecast many times over. The real work will be keeping them asleep.

What we took away

• Record dumb, draw smart. The recorder never thinks. All the cleverness happens later, once we know what someone wants.
• Reuse the real thing. Drawing clips with the app itself saved us from building, and babysitting, a second copy of the live screen.
• Measure every step. Each speed-up above started as a number we didn't like.
• Make it fast, then let it sleep. Speed made clips feel instant. Sleeping machines made them affordable.

Next time you tap Clip: somewhere, a graphics card just woke up, replayed your moment on a clock of its own, and went back to bed.

Get Livsho