Your whole analytics database now fits on one laptop

Someone on the team asked me a few days ago: We are a learning company, what do we care about database speed?

Fair question. My answer was that database speed decides who is allowed to ask.

DuckDB shipped a preview of its 2.0 release this week. It is an analytical database that runs in-process (no server, no cluster, no ticket you open so somebody can provision something for you), and it answers a question about a few hundred million rows before your attention runs out.

The response time is the feature I care about here, ahead of the benchmark numbers and the SQL dialect coverage.

The corner shop and the wholesale market

Think about the shop at the end of your street.

You walk down there for one onion. You do not plan it, you do not make a list, you do not ask yourself whether one onion justifies the trip. Two minutes and you are back.

The wholesale market is cheaper and bigger and has everything, but you do not drive there for an onion. You go when you have accumulated a full list that justifies the forty minutes.

The corner shop lets you buy what you need right now. The wholesale market lets you buy what you can justify.

Queries work exactly the same way.

When a query takes a day, the only person who runs it is the person whose job the query is. They batch their questions because each one is expensive, and they ask the questions they can justify to somebody else.

When it takes ten seconds, the person who has the question runs it themselves. And then they run the follow-up.

The follow-up is almost always the useful one, because the first question is rarely the real one. It is the one you knew how to ask before you saw any data.

The threshold nobody designs for

Interface researchers have been describing this for forty years, and organisations keep rediscovering it.

Below about a second, the system feels like an extension of your thinking. Around ten seconds, you stay on task but you stop exploring. Past a minute you switch context, and the question you were holding in your head evaporates along with the rest of your working memory.

At each step you lose the chain of questions, the third and fourth ones that only exist because the second one surprised you. Nobody schedules a chain of exploratory queries. It only happens while the answers keep coming fast.

So a tool that moves a workload from four minutes to four seconds makes analysis 60x faster, and it also makes possible a category of analysis that previously did not happen at all. Only that second effect shows up as an organisational capability.

(Illustrative figures. Exploration collapses long before anyone would call the system slow.)

What this has to do with company memory

We spend most of our time thinking about how organisations hold on to what they know and how they pass it on. So we have seen this pattern before, wearing different clothes.

In our experience, when teams fail to learn, the data is nearly always there: in the warehouse, in the logs, in one spreadsheet somebody maintains heroically. They fail because asking costs enough that nobody asks casually.

And the casual question is where most real understanding comes from.

The formal channels (the dashboard, the weekly report, the quarterly review) answer questions somebody anticipated in advance. That is their job and they do it well. But an organisation's understanding of itself is assembled out of thousands of small unplanned questions: Is that normal? Has it always done that? What happened the week before?

Those questions have no owner, no ticket and no budget. They only get asked when asking is nearly free.

That is why we are sceptical of capability programmes that begin by building more reporting. Another dashboard adds answers to anticipated questions. It does nothing for the unanticipated ones, which is where most learning happens.

But the governance bill arrives

All of this has a price.

Local-first analytics pushes governance out to the edges. Once anyone can pull a dataset onto a laptop and query it in-process, the questions of what is sensitive, what is derived and what is authoritative are still there, and slow, centralised access no longer enforces them as a side effect.

A great deal of data governance in large organisations is accidental. It works because the warehouse is the only door, and the door has a lock on it.

Remove that chokepoint and you inherit three problems at once:

• Provenance: Two people answer the same question differently because they pulled the extract on different days, and neither number is wrong.

• Sensitivity: Data that was governed by virtue of being hard to reach is now sitting in a downloads folder.

• Authority: "The number" stops existing once anyone can produce one, and meetings turn into arguments about extracts.

All three are solvable if you decide to solve them deliberately. The failure mode is discovering them eighteen months in and responding by reintroducing the chokepoint, which restores governance by destroying the thing that made the tool valuable in the first place.

What we would do

Treat response time as a capability metric. Nobody reports it that way, which is why it stays invisible.

Take the five questions your team asks most often about its own work. Time how long it takes a competent person who is not a data specialist to answer each one from a standing start.

That number decides whether your organisation gets to be curious about itself, far more than your warehouse's p99 does.

If the answer is measured in days, you have a learning problem, and response time is where it shows.

If you want the design rationale from the people who built the thing, this talk is the clearest version of it:

Source: A Preview of DuckDB v2.0.