Sixteen years, one bug, one very patient engineer

In August, Tailscale published a write-up of a bug they tracked down in SQLite's WAL reset path. The bug had been there since roughly 2010.

Sixteen years. In SQLite — which is, by most measures, the most thoroughly tested piece of software in general use. It ships with hundreds of times more test code than library code. It runs on billions of devices. And this sat inside it for sixteen years.

I find that comforting rather than alarming, and I think it is worth saying why.

It was not found by being clever

Read the write-up and notice what the work actually was. It is not a flash of insight. It is: a rare corruption report, an inability to reproduce, instrumentation, a narrower repro, reading the WAL format specification closely, and then the slow closing of the gap between what the code says and what the format requires.

That is weeks of unglamorous, uncertain work on a problem that might have turned out to be nothing.

Nobody assigns that. It gets done by someone who cannot let it go.

What the bug's age tells you

The interesting fact is not that a bug existed. It is what its survival implies about the shape of testing.

Sixteen years of enormous test coverage did not find it, because tests encode the failure modes you have already imagined. This one lived in the narrow intersection of a rare state, a specific sequence, and a code path that almost never runs. No amount of coverage percentage reaches that. You get there by understanding the system deeply enough to ask a question nobody had asked.

Which is exactly the thing that does not show up in any metric your organisation collects.

The part about your team

Every organisation says it wants deep expertise. Very few build the conditions for it.

Deep expertise comes from staying with one system long enough to develop intuitions about it, and most engineering careers are structured to prevent exactly that — rotate for growth, ship for visibility, move before the interesting problem surfaces. The person who finds a sixteen-year-old bug has been in that neighbourhood long enough to notice the smell.

If everyone on your team has been on their system for eleven months, you have a team of competent people and nobody who can find this class of problem. That is a staffing choice, and it is usually made without anyone noticing they made it.

The write-up is excellent and technical. Read it, then look at how long your longest-tenured engineer has been on their system.