The distance between a demo and a system

Notebook principle 01: follow the experiment past the demo.

I want each build log to explain its failure paths as clearly as its happy path. A promising demonstration earns a follow-up question, not an automatic claim of reliability.

For this notebook, I want to look at frontier AI through the things it helps build. A striking response is a good starting point. The harder questions arrive when the input is incomplete, a service is unavailable, or a result looks convincing but is wrong.

Start with the task, not the leaderboard

Before choosing a model, I want each experiment here to name its smallest useful outcome. For a writing assistant, that might be a sourced draft whose quotations can be checked. For a hardware companion, it might be a visible, accurate connection state. The outcome tells you what to test.

Keep a failure notebook

My build notes should record the input, the expected result, the observed result, and what the surrounding software did. “It worked” is difficult to learn from. “It returned an answer after the source timed out, without saying so” gives you something to fix.

The next experiment

The next experiment I describe here should take one promising workflow and remove one assumption. Disconnect a service. Supply an ambiguous instruction. Restart halfway through. Then ask whether the user can tell what happened and recover without guessing.

That is the kind of AI progress I want this notebook to follow: capability that survives contact with an ordinary day.