feat(e2e-tests): stacked e2e after split metrics

[davidberenstein1957]

Summary

Final integration/e2e PR for the split VLM metrics stack.

Keeps e2e and integration-focused coverage isolated from per-metric implementation PRs.

Stack Position

• Base: PR feat(vision-metrics): split img_edit_score #651 (feat/vlm-pr-4c-img-edit-score)
• This is the stack tip / final integration PR
• Canonical umbrella reference: PR feat(evaluation): add VLMMetrics #545 (feat/metrics-vlm-support)

Full Stack Order

1. feat(vendor): add LLM2Vec embedding model #637 vendor
2. feat(infrastructure): add VLM base classes and utilities #638 infrastructure
3. feat(text-metrics): split qa_accuracy #645 qa_accuracy
4. feat(text-metrics): split oneig_alignment #646 oneig_alignment
5. feat(text-metrics): split text_score pair #647 text_score pair
6. feat(text-metrics): split oneig_reasoning #648 oneig_reasoning
7. feat(vision-metrics): split vqa #649 vqa
8. feat(vision-metrics): split vie_score #650 vie_score
9. feat(vision-metrics): split img_edit_score #651 img_edit_score
10. feat(e2e-tests): stacked e2e after split metrics #641 e2e tests (this PR)

Files

• tests/evaluation/test_vlm_e2e.py
• tests/evaluation/test_task.py
• tests/data/test_datamodule.py
• tests/evaluation/_vlm_batch_snapshot_helpers.py

Test Plan

uv run pytest tests/evaluation/test_vlm_e2e.py tests/evaluation/test_task.py tests/data/test_datamodule.py

Review Focus

• End-to-end benchmark execution coverage
• Task/datamodule integration paths
• Split-stack parity with umbrella behavior

Review Flow (Order)

Review the stack in this exact order:

1. feat(vendor): add LLM2Vec embedding model #637 vendor
2. feat(infrastructure): add VLM base classes and utilities #638 infrastructure
3. feat(text-metrics): split qa_accuracy #645 qa_accuracy
4. feat(text-metrics): split oneig_alignment #646 oneig_alignment
5. feat(text-metrics): split text_score pair #647 text_score pair
6. feat(text-metrics): split oneig_reasoning #648 oneig_reasoning
7. feat(vision-metrics): split vqa #649 vqa
8. feat(vision-metrics): split vie_score #650 vie_score
9. feat(vision-metrics): split img_edit_score #651 img_edit_score
10. feat(e2e-tests): stacked e2e after split metrics #641 e2e tests

This PR in the flow (10/10)

• Review after PR feat(vision-metrics): split img_edit_score #651.
• This is the final PR in the stack.
• Confirm this PR's tests and scope before continuing.

[cursor]

Cursor Bugbot has reviewed your changes and found 4 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 7f24f9d. Configure here.}

[cursor]

Package realesrgan moved from optional to core dependency

High Severity

realesrgan was moved from the [upscale] optional dependency group into core dependencies, and the [upscale] extra was deleted entirely. This forces every user to install realesrgan and its heavy transitive dependencies (basicsr, facexlib, gfpgan, etc.) even if they never use upscaling. This PR is about VLM e2e tests and has no reason to change this. Likely an accidental inclusion from a rebase or merge.

Additional Locations (1)

• pyproject.toml#L188-L198

 

^{Reviewed by Cursor Bugbot for commit 7f24f9d. Configure here.}

[cursor]

Version downgraded and Python 3.13 support dropped

High Severity

version was downgraded from "0.3.3" to "0.3.2" and requires-python was tightened from ">=3.10,<3.14" to ">=3.10,<3.13", dropping Python 3.13 support. The PR description says "pyproject.toml — Already updated in PR-2", suggesting these regressions were accidentally included during a rebase or merge conflict resolution.

Additional Locations (1)

• pyproject.toml#L102-L103

 

^{Reviewed by Cursor Bugbot for commit 7f24f9d. Configure here.}

[cursor]

evaluation extra silently drops lmharness and rapidata

Medium Severity

The [evaluation] optional extra was redefined from ["pruna[rapidata]", "pruna[lmharness]"] to ["outlines>1.2.0,<2.0.0", "litellm>=1.0.0"]. Users running pip install pruna[evaluation] will no longer get lm-eval or rapidata. The [rapidata] extra was also completely removed. This is a silent backward-incompatible change to the package's public install interface.

 

^{Reviewed by Cursor Bugbot for commit 7f24f9d. Configure here.}

[cursor]

DrawBench/GenAIBench collate change alters return type

Medium Severity

DrawBench and GenAIBench collate functions changed from prompt_collate (returns (prompts, None)) to prompt_with_auxiliaries_collate (returns (prompts, list[dict])). Any existing code consuming these datasets and expecting gt=None (e.g., model inference handlers, metric update calls that check for None ground truth) will now receive a list of dicts, potentially causing unexpected behavior.

Additional Locations (1)

• src/pruna/data/__init__.py#L117-L118

 

^{Reviewed by Cursor Bugbot for commit 7f24f9d. Configure here.}

[github-actions]

This PR has been inactive for 10 days and is now marked as stale.

[davidberenstein1957]

Stack integration (#641)

• GenEval: Task.from_benchmark("GenEval") uses qa_accuracy(aggregation="all_or_nothing") (+ clip_score); covered by test_task.py.
• OneIG loaders: subset base_datasets entries + Multilingualism Q_D (multilingualism_zh.json) in prompt.py.
• pyproject: restored 0.3.3, requires-python <3.14, realesrgan under [upscale], full [evaluation] extra + VLM deps.
• DrawBench collate: intentional aux dicts for VLM metrics.

Full review decision log: .mine/vlm-review-decisions.md (local).

Stack branches force-pushed; CI should run on feature-branch PRs via #638 workflow change.

[github-actions]

This PR has been inactive for 10 days and is now marked as stale. It will be closed in 7 days if there is no further activity.