Going live on Kino: our own WebRTC stack in C

A live show is the hardest thing the Livsho app does. A seller's phone, on whatever network the shop has, sends video to viewers on networks just as bad, while bids land in real time and a late picture loses an auction. Until now we handed all of it to a third-party SDK that owned the camera, the microphone and the battery. Now Kino, the C engine that already plays and re-encodes every video in the app, can carry a show end to end over a WebRTC stack we wrote ourselves. This post walks the path a frame takes from the seller's camera to a viewer's screen, and what happens to it on the way.

Why our own stack

The usual answer to "live video on phones" is libwebrtc, a few million lines of somebody else's code, or a hosted SDK on top of it. Both work until they don't, and when they don't you're left reading logs from a black box while a seller's show stutters. We wanted every byte from the camera to the viewer's decoder to pass through code we wrote or can read: the transport, the congestion control, the jitter buffer, the server. That is about 3,500 lines of C on the phone and one C binary on the server, built on small, well-known libraries: mbedTLS for DTLS, libsrtp2 for SRTP, libopus for sound. The phones lend only their hardware encoders and decoders.

Signalling is WHIP to publish and WHEP to watch: one HTTP POST with an SDP offer, the answer back, a DELETE to leave. After that it is standard WebRTC media. ICE finds the path, DTLS 1.2 makes the keys, SRTP protects every packet, and RTP carries H.265 (or H.264 on phones without a hardware H.265 encoder) and Opus. The server is ICE-lite with a public address, so the phone only needs its own candidates, and there is no TURN relay in between.

The client is one file, built around one rule: nothing in it owns a thread, a socket or a clock. The host hands it every datagram that arrives and calls its tick at the deadline it asks for, passing the time in on every call. That makes the hardest code in the app deterministic in tests: a test drives a whole session, loss and all, with time standing still between steps. Publishing and watching are the same session type with one flag flipped, so ICE, DTLS, SRTP and consent checks are written once.

Sending: one encoder, paced

On the phone, the camera frames the app already makes go straight into the hardware encoder: VideoToolbox on iOS, and on Android a GL copy that draws each upright camera frame into MediaCodec's input surface. That means real time, no B-frames, a keyframe every two seconds or whenever a viewer asks, and parameter sets in front of every keyframe. Sound is 20 ms Opus frames from libopus with in-band FEC on, because neither phone's own Opus encoder can turn FEC on. Its expected-loss setting follows what the estimator measures, so the encoder spends bits on redundancy only when the path is actually losing packets.

Each access unit is cut into RTP payloads of at most 1,200 bytes, with every fragment of a NAL the same size, so the server can send a whole frame to a viewer in one GSO call. Packets go into a send history and then into the pacer, which lets them out at 2.5 times the current estimate. A keyframe never leaves as one burst that overflows a cellular buffer. Each packet gets its transport-wide sequence number at the moment it leaves, not when it was made, because congestion feedback is about departures. Sound skips the pacer: 20 ms of Opus waiting behind a keyframe hurts more than a keyframe arriving 1 ms later.

Every frame carries its camera capture time (the absolute-capture-time header extension), taken from the camera's own timestamp rather than the moment the encoder finished. The server forwards it, so a viewer can measure camera-to-screen delay continuously on every frame instead of guessing.

The estimator

The server sends transport-wide feedback: which packets arrived and when. Kino turns that into a bandwidth estimate in the shape of Google's congestion control. Packets that left within 5 ms of each other form a group. The change in delay between groups, smoothed into a trend, says whether queues on the path are filling. The threshold for "overuse" adapts to the trend (slowly upward, faster downward), so a jittery link doesn't look congested all the time. Loss adds a second, independent limit. The smaller of the two reaches the encoder as a bitrate, at most every 100 ms. If feedback stops altogether while we're sending, the estimate halves once a second until it comes back.

Two lessons came from real phones. First, an overuse verdict now cuts the estimate by at most half. It used to drop to 0.85 of what had just been acknowledged, and an encoder that was already under-sending (a still scene, an Android encoder slow to follow) had just had very little acknowledged. One burst pinned the estimate to its floor, and it took half a minute to climb back. Second, encoders overshoot. When an iPhone at 1080p60 kept sending twice its target, the excess sat in the network as delay. So the client tracks the video it has handed out against what the estimate says the path drains, and when that bucket runs 0.4 s ahead, the next camera frame is skipped before it is encoded. Dropping a frame after encoding would break every frame that references it.

/* The video made, drained at what the path takes (the estimate). */
static void vb_drain(kino_whip *w) {
    double dt = w->vbucket_at ? (double)(w->now - w->vbucket_at) / 1e6 : 0;
    w->vbucket_at = w->now;
    w->vbucket -= dt * w->bwe.rate / 8;
    if (w->vbucket < 0) w->vbucket = 0;
}

int kino_whip_admit(kino_whip *w, int64_t now_us) {
    kw_clock(w, now_us);
    if (w->state != ST_LIVE) return 1;
    vb_drain(w);
    return w->vbucket <= w->bwe.rate / 8 * 0.4;
}

Lost packets come back on NACK from the history, as RTX when the session negotiated it. PLI and FIR from the server become a keyframe request to the encoder.

Watching: repair, then the speaker clock

On the viewer's side, video packets wait in a reorder ring keyed by extended sequence number. A frame is handed on once every packet of it, and of every frame before it, is in. A hole is NACKed at once and again every round trip, up to ten times. When a later frame is complete and the hole has waited 2.5 round trips plus 30 ms (at least 80 ms: two repair rounds), it is given up. Everything up to the next keyframe is dropped, and a PLI asks for one, rate-limited so a bad second doesn't flood the publisher. Sound is repaired differently: a lost Opus packet is rebuilt from the next packet's in-band FEC if that arrives within the reorder wait, and otherwise concealed by libopus for up to 120 ms. A longer hole is jumped.

Then the stream goes into Kino's player engine, the one that plays every replay, in live mode. Its clock is the sound itself, measured at the speaker: media time is the number of real samples that have reached the speaker, so picture and sound can't drift apart, because there is no second clock to drift between. The server's sender reports map each stream's RTP time to wall time, so pictures land on the sound's timeline exactly as they do in a replay. A live buffer still needs a size, though, and that size is the latency. Kino aims at 30 ms plus three times the measured jitter, plus one round trip while packets are being lost (time for one NACK to come back), capped at 280 ms. Each stall adds 40 ms back, and 10 s without one takes 10 ms away. When the queue's low point over the last second sits 40 ms above the target, quiet 10 ms stretches of sound are skipped to catch up, so nobody hears it; 150 ms above it, any stretch is. Pictures follow, because they follow the clock.

A live player has failure modes a replay never sees. A resolution change mid-stream reopens the decoder. On Android, pictures still queued for the screen held buffers of the old MediaCodec, and releasing one after the decoder was freed crashed inside FFmpeg's logging, so they are now let go first. And when a session has to be replaced, the old player keeps its last picture on screen until the new one draws its first. A viewer never sees black.

Networks that fight back

The protocol's textbook assumes UDP goes through. In the Gulf, it doesn't always: one large cellular carrier's firewall lets the HTTPS offer through and silently drops every UDP check that follows. So the server's answer also carries a passive ICE-TCP candidate on port 443. When Kino's UDP checks have had no answer for 1.2 s, it connects there, and from then on every datagram in both directions rides that connection behind a 2-byte length (RFC 4571). ICE, DTLS, SRTP and consent see only datagrams and never know the difference. A TCP connection that can't keep up drops whole datagrams past 1 MB queued, the way UDP would lose them, rather than building up seconds of delay.

Phones also move. A seller walking out of a shop goes from Wi-Fi to cellular mid-show, and the old answer was to notice the dead path, tear the session down and build a new one: a new offer, new ICE checks, a new DTLS handshake, a wait for a keyframe. On a real phone that was 9 seconds with no picture. Now the session moves instead. Kino opens a fresh socket on whatever the default route is now and sends checks at once, and the server follows the address its USE-CANDIDATE checks come from: same session, same DTLS and SRTP keys, no handshake. Lost packets come back by NACK and the picture by a keyframe. If UDP doesn't answer on the new network, TCP follows 0.5 s later. A move starts on whichever comes first: a send that says the network is gone, 0.6 s of silence from the server plus 0.4 s of unanswered checks, or the platform itself saying the network changed. When the platform's signal fires, a move loses no frames at all; a path that simply goes dead costs about 1.1 s. As a bonus, a stream that fled to cellular moves back when Wi-Fi returns.

Settings move the same way. A seller changing frame rate or bitrate mid-show used to restart the stream, a 4 to 5 second gap each time. Now the estimator's range moves on the live session and the encoder is told at once. A new resolution remakes only the encoder: its first frame is a keyframe with new parameter sets, which the players already reopen on.

The other end

The server, called the node, is also C, and also ours. It runs one worker per core, each owning an io_uring ring, its own UDP port and every session on it, so nothing on the packet path takes a lock. Receives are one multishot RECVMSG drawing from a provided buffer ring; sends leave as one GSO call per viewer per loop pass; timers live in a wheel whose next deadline is the ring's wait timeout. A publisher's packets land in a ring sized for the last keyframe interval and the NACK window, and every worker with viewers of that stream encrypts and sends for its own. A new viewer starts with the packets since the last keyframe, so the first picture is ready about 45 ms after the handshake instead of waiting up to two seconds. NACKs are answered from the ring, and a thousand viewers missing the same packet cost the publisher nothing: their keyframe requests are coalesced to one a second. The node also records every show into the same segments the replay player already reads, so scrubbing, clips and replays work on a live show the moment it ends.

Where the milliseconds go

Glass to glass is camera, encode, the network up, the network down, the viewer's buffer, decode and display. The network legs are physics: about 90 ms together on a typical path. Everything else is on the phones, and that is where the work went, taking the total to roughly 150 ms from the seller's camera to a viewer's screen: capture times taken from the camera itself, low-latency rate control on the encoder, the admit gate in place of queueing, and a buffer that sits at 30 ms over the jitter instead of a fixed 80.

How we know it works

The live path is tested on Linux against the real server. Each scenario runs in its own network namespace with netem shaping the link, and every recording is checked frame by frame against numbers burned into the test video. An independent publisher written on a different stack (Pion, in Go) runs through the same scenarios, so a bug can't hide by being symmetric between our client and our server.

• Shaped networks: 1, 5, 10 and 20% loss, 300 ms round trips, ±50 ms jitter, bandwidth stepping from 6 to 1.5 Mb/s and back, a 3 s outage, a publisher replaced mid-show, and 1, 10 and 50 viewers with no frame missing.
• Hostile paths: UDP blocked so the session has to find TCP, a path cut mid-show, a path that goes quiet, and a network move on a timer.
• On phones: shows between an iPhone and a Galaxy S26 Ultra, on Wi-Fi, on cellular, and walking from one to the other.
• Sanitizers and fuzzers: AddressSanitizer and UndefinedBehaviorSanitizer on the server's unit tests, with RFC test vectors where they exist, and libFuzzer on every parser that faces the network: STUN, SDP, RTP, RTCP, H.265, HTTP and tokens.

A live client is a parser facing packets from the open internet, and so is the server. The fuzzers are the tests we care about most.

What it adds up to

A seller can go live from a shop, walk out onto cellular and keep streaming without viewers noticing. A network that blocks UDP gets the show over TCP. A viewer joins and sees a picture almost at once, then watches about a sixth of a second behind the camera, with sound and picture locked to one clock. A bad second costs a few repaired packets, not a frozen screen. And when something does go wrong, the code that went wrong is ours to read and fix, from the camera to the decoder.