Add OpenEval adapter for AutoGen evals - #8009
Draft
DresdenGman wants to merge 1 commit into
Draft
Conversation
Author
|
@microsoft-github-policy-service agree |
adhabnr-ux
reviewed
Jul 31, 2026
adhabnr-ux
left a comment
There was a problem hiding this comment.
Thanks for this PR, Dresden! I'm the OpenEval spec author — this looks great. A few observations:
What works well:
- Clean module boundary in
autogenstudio/eval/openeval.py to_openeval()/from_openeval()surface is exactly what we proposed- Round-trip test is solid — covers single and multi-input tasks, expected_tools, empty outputs
- Graceful ImportError fallback when SDK isn't installed
expected_toolsmapping from metadata is a nice touch — that's exactly how our agent eval profile works
Minor suggestions (non-blocking):
- The
gr_output_matchgrader is hardcoded toexact_match. For agent evals, you might want to also supportllm_judgeas an option — users could configure which grader type to use via metadata. But this is fine for v1. - The
expected_outputsfield (plural) in the test case payload isn't in the OpenEval spec — onlyexpected_output(singular) is standard. The extra field is harmless (unknown fields are ignored per our forward-compatibility rules), but if you want strict compliance, you could drop it. OPENEVAL_VERSIONfallback to"unknown"— the spec requires semver format. If the SDK isn't installed, consider defaulting to"1.0.0"instead.
Overall: This is exactly the kind of integration we hoped for. The code is clean, tested, and follows the spec correctly. Happy to see openeval-sdk as a runtime dependency.
If the AutoGen maintainers want to wait for wider adoption before merging (as @dragonstyle mentioned on the Inspect AI issue), this PR could also live as a standalone package (autogen-openeval-adapter) on PyPI. Either way works.
Great work! 🎉
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a small OpenEval adapter in
autogenstudio/evalso AutoGen eval data can be exported to and imported from the OpenEval format.Why
Issue #8005 proposed native import/export support. The maintainer guidance pointed to a new module boundary and a minimal
to_openeval()/from_openeval()surface, so this keeps the integration narrow and localized.What changed
autogenstudio.eval.openevalwithto_openeval()andfrom_openeval()helpers.autogenstudio.eval.openeval-sdkas a runtime dependency for the package.Validation
python3 -m py_compileon the new module and test file.References
Fixes #8005.