I work in AI research and data science, with 12+ years of building data pipelines, experiments and machine learning systems that run in production. These days I focus on LLM agents and how to evaluate them.
The thread through my work: make the numbers answer the question a team actually has. Not just "is it significant?" but "what should we do, and what does it cost us if we're wrong?"
Small, free Python tools for building and checking AI systems:
- evalsize: how many eval examples you need before a difference between prompts or models means anything. Calculator
- evalnoise: where the noise in an eval score comes from (questions, judges, prompt wordings, reruns), and where to spend the next API calls. Calculator
- rolloutwatch: whether a prompt or model change made quality worse, checking as often as you like without false alarms. Calculator
- costplan: what an AI agent will cost and what budget cap to set, from a handful of pilot runs. Calculator
- sturdytools: retries, timeouts, rate limits and circuit breakers for the tools an AI agent calls.
- Decide before looking. The decision rule is written down before the results are in.
- Test the method, not just the model. I check that a method behaves on the data I actually have before I trust what it says.
- Say what the data can't tell you. A clear "we don't know yet" beats a confident guess.
I've published on speech synthesis, LLM efficiency, time-series prediction and semantic search, and I write about AI on Substack.