The fkra blog
Engineering · 2 min

Sixteen years, one bug, one very patient engineer

Tailscale tracked down a SQLite WAL-reset bug that had been sitting there since 2010. Finding it took weeks of patient work.

Listen to this article

2:34
0:00

AI may have been used to generate this audio.

Press Space or K to play or pause, and J or L to go back or forward 15 seconds.

In August, Tailscale published a write-up of a bug they tracked down in SQLite's WAL reset path. The bug had been there since roughly 2010.

SQLite is, by most measures, the most thoroughly tested piece of software in general use. It ships with hundreds of times more test code than library code and runs on billions of devices, and this bug sat inside it for sixteen years.

I find that comforting.

How it was found

The write-up describes the work step by step: a rare corruption report, an inability to reproduce, instrumentation, a narrower repro, reading the WAL format specification closely, and then the slow closing of the gap between what the code says and what the format requires.

That is weeks of unglamorous, uncertain work on a problem that might have turned out to be nothing.

Work like that goes unassigned and gets done by someone who cannot let it go.

What the bug's age tells you

How long it survived shows the limit of testing.

Sixteen years of enormous test coverage did not find it, because tests encode the failure modes you have already imagined. This one lived in the narrow intersection of a rare state, a specific sequence, and a code path that almost never runs. You reach that only by understanding the system deeply enough to ask a question nobody had asked.

That understanding does not show up in any metric your organisation collects.

The part about your team

Every organisation says it wants deep expertise. Very few build the conditions for it.

Deep expertise comes from staying with one system long enough to develop intuitions about it, and most engineering careers are structured to prevent it: rotate for growth, ship for visibility, move before the interesting problem surfaces. The person who finds a sixteen-year-old bug has been in that neighbourhood long enough to notice the smell.

If everyone on your team has been on their system for eleven months, you have a team of competent people and nobody who can find this class of problem. That is a staffing choice, and it is usually made without anyone noticing they made it.

The write-up is excellent and technical. Read it, then look at how long your longest-tenured engineer has been on their system.

6
170 views
CraftDebuggingSQLiteTailscaleDatabases
Mosab Alrasheed
Get the next entry

One email when we publish. Research, product decisions, and what teams report back.

Now playingSixteen years, one bug, one very patient engineer