Skip to content
View ai-with-md's full-sized avatar

Block or report ai-with-md

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
ai-with-md/README.md

Hi, I am MD

I work in AI research and data science, with 12+ years of building data pipelines, experiments and machine learning systems that run in production. These days I focus on LLM agents and how to evaluate them.

The thread through my work: make the numbers answer the question a team actually has. Not just "is it significant?" but "what should we do, and what does it cost us if we're wrong?"

Tools

Small, free Python tools for building and checking AI systems:

  • evalsize: how many eval examples you need before a difference between prompts or models means anything. Calculator
  • evalnoise: where the noise in an eval score comes from (questions, judges, prompt wordings, reruns), and where to spend the next API calls. Calculator
  • rolloutwatch: whether a prompt or model change made quality worse, checking as often as you like without false alarms. Calculator
  • costplan: what an AI agent will cost and what budget cap to set, from a handful of pilot runs. Calculator
  • sturdytools: retries, timeouts, rate limits and circuit breakers for the tools an AI agent calls.

How I work

  • Decide before looking. The decision rule is written down before the results are in.
  • Test the method, not just the model. I check that a method behaves on the data I actually have before I trust what it says.
  • Say what the data can't tell you. A clear "we don't know yet" beats a confident guess.

Elsewhere

I've published on speech synthesis, LLM efficiency, time-series prediction and semantic search, and I write about AI on Substack.

LinkedIn · Substack · X

Pinned Loading

  1. experimentation-toolkit experimentation-toolkit Public

    Is your A/B test telling you the truth? Decision-first A/B testing in Python: A/A calibration, power by simulation, sequential testing, uplift and causal inference on real data.

    Python

  2. sturdytools sturdytools Public

    Retries, timeouts, rate limits and circuit breakers for AI agent tools. One decorator, works with MCP and LangChain.

    Python

  3. evalsize evalsize Public

    How many eval examples do I need? Sample size and power planning for LLM evals, with a browser calculator

    Python

  4. costplan costplan Public

    How much will my AI agent cost? Plan budgets, caps and spend from a handful of pilot runs, with a browser calculator

    Python

  5. evalnoise evalnoise Public

    Where does the noise in my eval score come from? Split it across questions, judges, prompt wordings and runs, and plan where to spend the next API calls

    Python

  6. rolloutwatch rolloutwatch Public

    Did my prompt or model change make things worse? Watch LLM quality after a change, checking daily without false alarms

    Python