Skip to content

bench: rebuild suite into zero-dep guard tier + cross-framework comparative tier#332

Merged
lesnik512 merged 16 commits into
mainfrom
bench/suite-rebuild
Jul 15, 2026
Merged

bench: rebuild suite into zero-dep guard tier + cross-framework comparative tier#332
lesnik512 merged 16 commits into
mainfrom
bench/suite-rebuild

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Rebuilds the benchmark suite around two tiers sharing one scenario vocabulary. Full rationale and design in the change file: planning/changes/2026-07-15.05-benchmark-suite-rebuild.md.

What changed

  • Guard tier (benchmarks/, zero external deps) — 7 scenarios (G1–G7) decomposing modern-di's resolve hot path: transient, cached, depth-6 chain, wide fan-out, cross-scope, child-container build, and the full request lifecycle with a sync-init/async-finalizer teardown. Replaces the four stale fix#1–4 micro-benchmarks (which compared code to monkey-patched reimplementations) and RESULTS.md.
  • Comparative tier (benchmarks/comparative/, isolated standalone uv project) — C1–C4 for modern-di + dishka 1.10.1 + that-depends 4.0.2 + dependency-injector 4.49.1 + wireup 2.12.0, same graph shape, each in its own idiomatic setup. Local-only via just bench-compare; competitor deps live only in the nested pyproject.toml, so the root stays zero-runtime-dependency.
  • CI (.github/workflows/benchmarks.yml) — runs the guard tier on push/PR and posts a non-gating PR comment with the per-scenario delta vs a rolling main baseline (cached via actions/cache, no gh-pages). Never fails the build.
  • Docs/recipesbenchmarks/README.md (living methodology + honest per-framework mapping), just bench (guard) / just bench-compare (comparative).

Notes for review

  • Two design decisions were made during implementation (recorded in the change file): the root pyproject.toml gains a lint-only exclude of benchmarks/comparative (ruff + [tool.ty.src]) so the root toolchain doesn't try to type-check competitor imports — no [project] dependency added; and the CI baseline uses a unique-key + restore-keys cache pattern (Actions cache entries are immutable, so a static key would freeze the baseline).
  • Honest asymmetry, documented: modern-di is the only one of the five that resolves synchronously under an async finalizer — the other four force an awaited resolve — so C4 measures whole-request-lifecycle wall-clock, not sync-vs-sync. C1–C3 are true sync resolves everywhere.
  • Known guidance-tier caveat (accepted, not a bug): dependency-injector's C4 rebuilds its container per round because a fresh container is its per-request idiom, so its C4 number carries construction overhead the others don't. Numbers are optimization guidance, not published claims.

Verification

  • just test-ci: 392 passed, 100% coverage (no library code changed).
  • just bench: 7 passed. just bench-compare: 20 passed (5 frameworks × C1–C4), each asserting the resolved graph (C4/G7 assert the async finalizer ran).
  • just lint-ci and just check-planning: pass.

🤖 Generated with Claude Code

@lesnik512 lesnik512 merged commit 00e16bd into main Jul 15, 2026
8 checks passed
@lesnik512 lesnik512 deleted the bench/suite-rebuild branch July 15, 2026 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant