Skip to content

Let a bench run an agent defined outside lemans - #9

Open
bruno-costanzo wants to merge 1 commit into
rails:mainfrom
bruno-costanzo:bring-your-own-agent
Open

bruno-costanzo wants to merge 1 commit into
rails:mainfrom
bruno-costanzo:bring-your-own-agent

Conversation

@bruno-costanzo

@bruno-costanzo bruno-costanzo commented Sep 29, 2026 •

Copy link
Copy Markdown

We run our own coding agent (headless Claude Code driven by our app) and wanted to measure it on the Fizzy tickets with lemans, graded exactly like the published runs. Today the only way is patching the frozen Agents::REGISTRY, so this adds a small hook:

  • Lemans::Agents.register(name, agent_class): --agent and bench.yml accept the name; built-in names can't be taken.
  • lemans run --require FILE (repeatable) loads the file that defines and registers the agent.
  • A short "Bring your own agent" section in the README, and a CHANGELOG line.

If you'd rather not grow a public extension point, or want a different shape (e.g. a command-line agent that just runs a binary), happy to change it or close this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant