Skip to content

Add OpenEval adapter for AutoGen evals - #8009

Draft
DresdenGman wants to merge 1 commit into
microsoft:mainfrom
DresdenGman:agent/openeval-adapter
Draft

Add OpenEval adapter for AutoGen evals#8009
DresdenGman wants to merge 1 commit into
microsoft:mainfrom
DresdenGman:agent/openeval-adapter

Conversation

@DresdenGman

Copy link
Copy Markdown

Summary

Add a small OpenEval adapter in autogenstudio/eval so AutoGen eval data can be exported to and imported from the OpenEval format.

Why

Issue #8005 proposed native import/export support. The maintainer guidance pointed to a new module boundary and a minimal to_openeval() / from_openeval() surface, so this keeps the integration narrow and localized.

What changed

  • Added autogenstudio.eval.openeval with to_openeval() and from_openeval() helpers.
  • Re-exported the adapter from autogenstudio.eval.
  • Added openeval-sdk as a runtime dependency for the package.
  • Added a regression test covering a round-trip between AutoGen tasks and OpenEval payloads.

Validation

  • python3 -m py_compile on the new module and test file.
  • Dependency-free smoke test that loaded the module with stubs and verified the conversion behavior.

References

Fixes #8005.

@DresdenGman

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

@adhabnr-ux adhabnr-ux left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this PR, Dresden! I'm the OpenEval spec author — this looks great. A few observations:

What works well:

  • Clean module boundary in autogenstudio/eval/openeval.py
  • to_openeval() / from_openeval() surface is exactly what we proposed
  • Round-trip test is solid — covers single and multi-input tasks, expected_tools, empty outputs
  • Graceful ImportError fallback when SDK isn't installed
  • expected_tools mapping from metadata is a nice touch — that's exactly how our agent eval profile works

Minor suggestions (non-blocking):

  1. The gr_output_match grader is hardcoded to exact_match. For agent evals, you might want to also support llm_judge as an option — users could configure which grader type to use via metadata. But this is fine for v1.
  2. The expected_outputs field (plural) in the test case payload isn't in the OpenEval spec — only expected_output (singular) is standard. The extra field is harmless (unknown fields are ignored per our forward-compatibility rules), but if you want strict compliance, you could drop it.
  3. OPENEVAL_VERSION fallback to "unknown" — the spec requires semver format. If the SDK isn't installed, consider defaulting to "1.0.0" instead.

Overall: This is exactly the kind of integration we hoped for. The code is clean, tested, and follows the spec correctly. Happy to see openeval-sdk as a runtime dependency.

If the AutoGen maintainers want to wait for wider adoption before merging (as @dragonstyle mentioned on the Inspect AI issue), this PR could also live as a standalone package (autogen-openeval-adapter) on PyPI. Either way works.

Great work! 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Proposal] OpenEval Import/Export Support

2 participants