Listen to this article
AI may have been used to generate this audio.
Press Space or K to play or pause, and J or L to go back or forward 15 seconds.GitHub went down yesterday and we found out the way most teams find out: Something that normally takes one click quietly stopped happening, and we spent a while working out whether the problem was ours.
The minutes between "something is odd" and "it is not us" deserve the attention. Everything after them is waiting.
During an outage, the useful question is what your team can still do. You have no say in when it will be fixed, and refreshing a status page is not work.
And that is a question about who knows what.
An outage is an audit nobody scheduled
Who can deploy by hand? Who knows what each automated job was doing, beyond its name? Who can tell a broken build from a runner that never started?
In most teams, that is one person. And that person is the entire reason the outage felt survivable.
That is a single point of failure with a salary, an inbox, and annual leave.
We hit our own tell the same week. A workflow that "fails" in a few seconds having run zero steps means the runner never started, and in our case the Actions budget was gone. The code was fine.
Read that as a red build and you send someone debugging a thing that was never wrong. The misread is expensive because it is so plausible: red cross in the UI, "failed" in the notification, and every instinct says look at the diff.
Why your postmortem will miss this
The standard template asks what broke, when, and what we will change. They are fine questions, and they produce a document that is mostly useful to people who were already in the room.
It almost never captures the diagnostic path: the three things that looked like the cause and were not, and how each was ruled out.
That is the expensive knowledge. Reconstructing a timeline from logs is easy months later. Reconstructing why a competent person spent forty minutes on the wrong hypothesis is impossible, because nobody wrote down that they had it.
So the notes record the route once it was known, and lose the search that found it.
Two questions worth adding
After the next incident, ask these alongside the timeline:
What looked broken and was not? And what was the tell that separated them?
What did we do by hand that is normally automated? Write the commands down. Next time the person who knows them may be on a plane.
Then, once a quarter, hand the notes to someone who was not there and ask them to follow along. If they cannot, the notes are a souvenir. That test takes an afternoon and it is the only one that tells you whether the knowledge left the building with a person or stayed.
Google's SRE practice has argued something close to this for years, and this is a good short version of the culture it takes:
The incident itself is on GitHub's status page.
One email when we publish. Research, product decisions, and what teams report back.
