Skip to content

test: evaluate static shell optimization - #97787

Merged
aurorascharff merged 4 commits into
codex/optimize-static-shell-guidefrom
codex/optimize-static-shell-eval
Aug 27, 2026
Merged

aurorascharff merged 4 commits into
codex/optimize-static-shell-guidefrom
codex/optimize-static-shell-eval

Conversation

@aurorascharff

@aurorascharff aurorascharff commented Aug 24, 2026 •

Copy link
Copy Markdown
Contributor

What

Adds an agent eval for the static-shell optimization workflow in #97737. The fixture uses a release-operations route with a request-specific viewer, reusable checklist, live rollout data, and both direct and client navigation.

Why

This tests whether agents can transfer the guide and optimizer skill to a difficult refactor that is similar to, but not copied from, the guide example.

Verification

  • Claude Sonnet 4.6: baseline 2/7, bundled docs 2/7, optimizer skill 6/7
  • The skill treatment passed every final-code criterion; its only miss was executing the browser RED-to-GREEN loop because the sandbox lacked Chromium libraries
  • Browser-dependent fixtures now provision those libraries during setup; a follow-up attempt passed setup but stalled before producing a transcript

@github-actions

github-actions Bot commented Aug 24, 2026 •

Copy link
Copy Markdown
Contributor

Stats from current PR

🟢 1 improvement

Metric Canary PR Change Trend
node_modules Size 553 MB 552 MB 🟢 132 kB (0%) █████
📊 All Metrics
📖 Metrics Glossary

Dev Server Metrics:

  • Listen = TCP port starts accepting connections
  • First Request = HTTP server returns successful response
  • Cold = Fresh build (no cache)
  • Warm = With cached build artifacts

Build Metrics:

  • Fresh = Clean build (no .next directory)
  • Cached = With existing .next directory

Change Thresholds:

  • Time: Changes < 50ms AND < 10%, OR < 2% are insignificant
  • Size: Changes < 1KB AND < 1% are insignificant
  • All other changes are flagged to catch regressions

⚡ Dev Server

Metric Canary PR Change Trend
Cold (Listen) 865ms 812ms ✓ ▁▁▁▁▁
Cold (Ready in log) 827ms 809ms ✓ ▂▂▃▃▁
Cold (First Request) 1.334s 1.327s ✓ ▂▁▃▄▁
Warm (Listen) 864ms 814ms ✓ ▁▁▁▁▁
Warm (Ready in log) 824ms 820ms ✓ ▃▂▄▄▁
Warm (First Request) 1.307s 1.308s ✓ ▅▂▅▆▁
📦 Dev Server (Webpack) (Legacy)

📦 Dev Server (Webpack)

Metric Canary PR Change Trend
Cold (Listen) 810ms 811ms ✓ ▄▁▅▅▅
Cold (Ready in log) 775ms 781ms ✓ ▂▁▅▄▅
Cold (First Request) 3.247s 3.248s ✓ ▃▁▄▂▄
Warm (Listen) 809ms 811ms ✓ ▅▁▅▅▅
Warm (Ready in log) 774ms 780ms ✓ ▃▁▆▄▆
Warm (First Request) 3.253s 3.282s ✓ ▃▁▄▂▇

⚡ Production Builds

Metric Canary PR Change Trend
Fresh Build 6.094s 6.247s ✓ ▁▁▄▄▁
Cached Build 3.068s 3.032s ✓ ▁▁▃▄▁
📦 Production Builds (Webpack) (Legacy)

📦 Production Builds (Webpack)

Metric Canary PR Change Trend
Fresh Build 23.406s 23.566s ✓ ▃▁▇▄█
Cached Build 23.482s 23.690s ✓ ▁▁▆▄▅
node_modules Size 553 MB 552 MB 🟢 132 kB (0%) █████
📦 Bundle Sizes

Bundle Sizes

⚡ Turbopack

Client

Main Bundles
Canary PR Change
017_4moyv5m34.js gzip 117 B 117 B ✓
05jocc85-b73_.js gzip 341 B N/A -
0cz1d0mv5g_q7.js gzip 39.4 kB 39.4 kB ✓
0dj4voo287tzi.js gzip 71.5 kB N/A -
0e5lgz-xcf6ln.js gzip 273 B 273 B ✓
0enhddgqs2dl-.js gzip 265 B 265 B ✓
0h34mejwa_7pd.js gzip 354 B 354 B ✓
0kpi7r3ik4wam.js gzip 152 B N/A -
0w853zxni3ro0.js gzip 1.36 kB N/A -
13lq2aw3_0bgw.js gzip 3.52 kB N/A -
1elt1qium-r2m.css gzip 115 B 115 B ✓
1m5vdnlf0deiv.js gzip 411 B 411 B ✓
1nigpy_mr0xvj.js gzip 339 B N/A -
1p0l8mn2wz5f2.js gzip 1.97 kB N/A -
1s-3t9k2_uhom.js gzip 117 B N/A -
1sqoyy39jpgkq.js gzip 121 B 121 B ✓
1tf1phijqlx9j.js gzip 220 B 220 B ✓
2b05_424u1zq-.js gzip 120 B N/A -
2ca40xittx95r.js gzip 8.75 kB N/A -
2hgz42b8sldyc.js gzip 5.85 kB N/A -
2kci0vnt-49o-.js gzip 113 B 113 B ✓
2sxgl_rcrrn_p.js gzip 399 B N/A -
2u_p__ndkooh-.js gzip 5.66 kB N/A -
2vst0p6zpfrr_.js gzip 7.53 kB N/A -
2vt62i17bhbdu.js gzip 2.35 kB N/A -
342di-crrg9kq.js gzip 119 B N/A -
36bhyc0i1hpg3.js gzip 46.7 kB N/A -
36hfc2_979aw2.js gzip 119 B N/A -
38wfp6jjrrdfm.js gzip 131 B 131 B ✓
3dl1ilk37-30g.js gzip 117 B N/A -
3h0us7yy7bd_o.js gzip 13.1 kB N/A -
3kvxagoobts7h.js gzip 117 B N/A -
3p99hd9vwe6ae.js gzip 13 kB N/A -
3r0475panhk6m.js gzip 123 B N/A -
3s0i-weykgzrt.js gzip 125 B N/A -
3shl91vd9c4h2.js gzip 65.5 kB N/A -
3wijk2ywqgsv9.js gzip 2.52 kB N/A -
41u5s3oe2-erp.js gzip 2.29 kB N/A -
turbopack-0d..9ejy.js gzip 3.74 kB 3.74 kB ✓
02zc7sqlpqxn3.js gzip N/A 119 B -
0at_009gd70pg.js gzip N/A 153 B -
0e28eh3gbwmp0.js gzip N/A 3.51 kB -
0jwho9fkrb_t3.js gzip N/A 2.29 kB -
0longconn5mlm.js gzip N/A 123 B -
0p4l3ozbez2cz.js gzip N/A 2.35 kB -
0sbh_15ejop4o.js gzip N/A 13 kB -
0skdm-9rpid8g.js gzip N/A 117 B -
0wv-h56sz83wn.js gzip N/A 120 B -
137xup2t69ovc.js gzip N/A 1.97 kB -
1fk10s3m1oecx.js gzip N/A 400 B -
20qhkyyj2_tla.js gzip N/A 116 B -
25lb_468wpv8t.js gzip N/A 117 B -
25t26px0lbw0i.js gzip N/A 71.5 kB -
2imyufe18bsb7.js gzip N/A 1.36 kB -
2kfge3wavhjzk.js gzip N/A 65.5 kB -
2mr-6hunyw7uo.js gzip N/A 7.53 kB -
2p4n3-8-xixh_.js gzip N/A 341 B -
2vqdfieehz5do.js gzip N/A 8.75 kB -
2xn85ykzw2v1e.js gzip N/A 13.1 kB -
33gdcu1dj0nl9.js gzip N/A 46.7 kB -
34knadd6wu9qg.js gzip N/A 119 B -
37-g1kp39exux.js gzip N/A 5.85 kB -
39jq2b6t6z6up.js gzip N/A 339 B -
3a9a9tvkag_-j.js gzip N/A 2.52 kB -
3hk-fj9wj-l1s.js gzip N/A 5.66 kB -
3t121tj-h0kbm.js gzip N/A 125 B -
Total 299 kB 299 kB ⚠️ +1 B

Server

Middleware
Canary PR Change
middleware-b..fest.js gzip 1.02 kB 1.02 kB ✓
Total 1.02 kB 1.02 kB ⚠️ +2 B
Build Details
Build Manifests
Canary PR Change
_buildManifest.js gzip 846 B 845 B ✓
Total 846 B 845 B ✅ -1 B
Build Cache
Canary PR Change
00000001.sst gzip 16.4 MB 12.7 MB 🟢 3.72 MB (-23%)
00000002.sst gzip 10.7 MB 14.7 MB 🔴 +3.96 MB (+37%)
00000003.sst gzip 13.9 MB 14.2 MB 🔴 +286 kB (+2%)
00000004.sst gzip 13.7 MB 13.4 MB 🟢 288 kB (-2%)
00000005.sst gzip 10 MB 10.2 MB 🔴 +115 kB (+1%)
00000006.sst gzip 2.75 MB 2.79 MB 🔴 +39.7 kB (+1%)
00000007.sst gzip 59 B 59 B ✓
00000008.meta gzip 89 B 89 B ✓
00000009.meta gzip 297 kB 298 kB ✓
00000010.meta gzip 297 kB 298 kB ✓
00000011.meta gzip 297 kB 298 kB ✓
00000012.sst gzip 52.9 kB 53.4 kB ✓
00000013.sst gzip 1.75 MB 1.77 MB 🔴 +29.1 kB (+2%)
00000014.sst gzip 59 B 59 B ✓
00000015.meta gzip 116 B 116 B ✓
00000016.meta gzip 327 kB 327 kB ✓
00000017.meta gzip 404 kB 406 kB ✓
00000018.sst gzip 53.9 kB 54.4 kB ✓
00000019.sst gzip 1.37 MB 1.38 MB 🔴 +14.4 kB (+1%)
00000020.sst gzip 59 B 59 B ✓
00000021.meta gzip 116 B 116 B ✓
00000022.meta gzip 327 kB 327 kB ✓
00000023.meta gzip 372 kB 373 kB ✓
00000024.sst gzip 53.9 kB 54.5 kB 🔴 +604 B (+1%)
00000025.sst gzip 1.37 MB 1.38 MB 🔴 +14.4 kB (+1%)
00000026.sst gzip 59 B 59 B ✓
00000027.meta gzip 116 B 116 B ✓
00000028.meta gzip 327 kB 327 kB ✓
00000029.meta gzip 372 kB 373 kB ✓
00000030.sst gzip 53.1 kB 53.6 kB ✓
00000031.sst gzip 1.37 MB 1.38 MB 🔴 +14.5 kB (+1%)
00000032.sst gzip 59 B 59 B ✓
00000033.meta gzip 116 B 116 B ✓
00000034.meta gzip 327 kB 327 kB ✓
00000035.meta gzip 372 kB 373 kB ✓
00000036.sst gzip 53.8 kB 54.3 kB 🔴 +574 B (+1%)
00000037.sst gzip 1.37 MB 1.38 MB 🔴 +14.5 kB (+1%)
00000038.sst gzip 59 B 59 B ✓
00000039.meta gzip 116 B 116 B ✓
00000040.meta gzip 327 kB 327 kB ✓
00000041.meta gzip 372 kB 373 kB ✓
CURRENT gzip 94 B 94 B ✓
LOG gzip 679 B 663 B 🟢 16 B (-2%)
Total 79.5 MB 80 MB ⚠️ +485 kB

📦 Webpack

Client

Main Bundles
Canary PR Change
3322-HASH.js gzip 65.6 kB N/A -
4191.HASH.js gzip 169 B N/A -
9784-HASH.js gzip 5.57 kB N/A -
b1ad9f4c-HASH.js gzip 63.5 kB N/A -
framework-HASH.js gzip 59.7 kB 59.7 kB ✓
main-app-HASH.js gzip 253 B 252 B ✓
main-HASH.js gzip 39.8 kB 39.7 kB ✓
webpack-HASH.js gzip 1.68 kB 1.68 kB ✓
3577.HASH.js gzip N/A 168 B -
578-HASH.js gzip N/A 65.8 kB -
8590-HASH.js gzip N/A 5.53 kB -
a8984546-HASH.js gzip N/A 63.5 kB -
Total 236 kB 236 kB ⚠️ +98 B
Polyfills
Canary PR Change
polyfills-HASH.js gzip 39.4 kB 39.4 kB ✓
Total 39.4 kB 39.4 kB ✓
Pages
Canary PR Change
_app-HASH.js gzip 194 B 193 B ✓
_error-HASH.js gzip 181 B 182 B ✓
css-HASH.js gzip 334 B 331 B ✓
dynamic-HASH.js gzip 1.81 kB 1.81 kB ✓
edge-ssr-HASH.js gzip 255 B 253 B ✓
head-HASH.js gzip 349 B 351 B ✓
hooks-HASH.js gzip 382 B 384 B ✓
image-HASH.js gzip 5.02 kB 5.01 kB ✓
index-HASH.js gzip 260 B 259 B ✓
link-HASH.js gzip 2.48 kB 2.48 kB ✓
routerDirect..HASH.js gzip 317 B 318 B ✓
script-HASH.js gzip 384 B 386 B ✓
withRouter-HASH.js gzip 316 B 315 B ✓
1afbb74e6ecf..834.css gzip 106 B 106 B ✓
Total 12.4 kB 12.4 kB ✓

Server

Edge SSR
Canary PR Change
edge-ssr.js gzip 128 kB 128 kB ✓
page.js gzip 292 kB 292 kB ✓
Total 420 kB 420 kB ✅ -602 B
Middleware
Canary PR Change
middleware-b..fest.js gzip 600 B 600 B ✓
middleware-r..fest.js gzip 156 B 156 B ✓
middleware.js gzip 45.7 kB 45.2 kB 🟢 534 B (-1%)
edge-runtime..pack.js gzip 842 B 842 B ✓
Total 47.3 kB 46.8 kB ✅ -534 B
Build Details
Build Manifests
Canary PR Change
_buildManifest.js gzip 698 B 698 B ✓
Total 698 B 698 B ✓
Build Cache
Canary PR Change
0.pack gzip 4.79 MB 4.78 MB 🟢 8.03 kB (0%)
index.pack gzip 124 kB 124 kB ✓
index.pack.old gzip 124 kB 123 kB 🟢 1.35 kB (-1%)
Total 5.04 MB 5.03 MB ✅ -9.01 kB

🔄 Shared (bundler-independent)

Runtimes
Canary PR Change
app-page-exp...dev.js gzip 373 kB 373 kB ✓
app-page-exp..prod.js gzip 206 kB 206 kB ✓
app-page-tur...dev.js gzip 373 kB 373 kB ✓
app-page-tur..prod.js gzip 206 kB 206 kB ✓
app-page-tur...dev.js gzip 369 kB 369 kB ✓
app-page-tur..prod.js gzip 203 kB 203 kB ✓
app-page.run...dev.js gzip 370 kB 370 kB ✓
app-page.run..prod.js gzip 203 kB 203 kB ✓
app-route-ex...dev.js gzip 82.5 kB 82.5 kB ✓
app-route-ex..prod.js gzip 55.6 kB 55.6 kB ✓
app-route-tu...dev.js gzip 82.5 kB 82.5 kB ✓
app-route-tu..prod.js gzip 55.6 kB 55.6 kB ✓
app-route-tu...dev.js gzip 82.1 kB 82.1 kB ✓
app-route-tu..prod.js gzip 55.4 kB 55.4 kB ✓
app-route.ru...dev.js gzip 82.1 kB 82.1 kB ✓
app-route.ru..prod.js gzip 55.4 kB 55.4 kB ✓
dev-validati...dev.js gzip 134 kB 134 kB ✓
dev-validati...dev.js gzip 134 kB 134 kB ✓
dev-validati...dev.js gzip 132 kB 132 kB ✓
dev-validati...dev.js gzip 132 kB 132 kB ✓
dist_client_...dev.js gzip 324 B 324 B ✓
dist_client_...dev.js gzip 326 B 326 B ✓
dist_client_...dev.js gzip 318 B 318 B ✓
dist_client_...dev.js gzip 317 B 317 B ✓
pages-api-tu...dev.js gzip 46.2 kB 46.2 kB ✓
pages-api-tu..prod.js gzip 34.3 kB 34.3 kB ✓
pages-api.ru...dev.js gzip 46.2 kB 46.2 kB ✓
pages-api.ru..prod.js gzip 34.2 kB 34.2 kB ✓
pages-turbo....dev.js gzip 54.6 kB 54.6 kB ✓
pages-turbo...prod.js gzip 39.6 kB 39.6 kB ✓
pages.runtim...dev.js gzip 54.6 kB 54.6 kB ✓
pages.runtim..prod.js gzip 39.6 kB 39.6 kB ✓
server.runti..prod.js gzip 65.8 kB 65.8 kB ✓
use-cache-pr...dev.js gzip 72.5 kB 72.5 kB ✓
use-cache-pr...dev.js gzip 72.5 kB 72.5 kB ✓
use-cache-pr...dev.js gzip 70.7 kB 70.7 kB ✓
use-cache-pr...dev.js gzip 70.7 kB 70.7 kB ✓
Total 4.09 MB 4.09 MB ✅ -2 B
📝 Changed Files (1 file)

Files with changes:

  • server.runtime.prod.js
View diffs
server.runtime.prod.js

Diff too large to display

📎 Tarball URL
https://vercel-packages.vercel.app/next/commits/7c1007e92ca2569f0d895be65317a26de483999c/next

Commit: 7c1007e

@github-actions

github-actions Bot commented Aug 24, 2026 •

Copy link
Copy Markdown
Contributor

Failing test suites

Commit: 7c1007e | About building and testing Next.js

pnpm test-dev-turbo test/development/acceptance-app/app-hmr-changes.test.ts (turbopack) (job)

  • Error overlay - RSC build errors > Skipped in webpack > should handle successive HMR changes with errors correctly (DD)
Expand output

● Error overlay - RSC build errors › Skipped in webpack › should handle successive HMR changes with errors correctly

Application is in inconsistent state: timeout.

  114 |             }
  115 |             if (status !== 'pending') {
> 116 |               throw new Error(
      |                     ^
  117 |                 `Application is in inconsistent state: ${status}.`
  118 |               )
  119 |             }

  at Object.patch (lib/development-sandbox.ts:116:21)
  at Object.<anonymous> (development/acceptance-app/app-hmr-changes.test.ts:58:11)

pnpm test-start-turbo test/production/create-next-app/examples.test.ts (turbopack) (job)

  • create-next-app --example > should create with GitHub URL trailing slash (DD)
Expand output

● create-next-app --example › should create with GitHub URL trailing slash

Command failed with exit code 1 (EPERM): node /home/runner/work/next.js/next.js/packages/create-next-app/dist/index.js github-url-trailing-slash --example https://github.com/vercel/nextjs-portfolio-starter/

  82 |     await useTempDir(async (cwd) => {
  83 |       const projectName = 'github-url-trailing-slash'
> 84 |       const res = await run(
     |                   ^
  85 |         [
  86 |           projectName,
  87 |           '--example',

  at makeError (../node_modules/.pnpm/execa@2.0.3/node_modules/execa/lib/error.js:58:11)
  at handlePromise (../node_modules/.pnpm/execa@2.0.3/node_modules/execa/index.js:112:26)
  at production/create-next-app/examples.test.ts:84:19
  at useTempDir (lib/use-temp-dir.ts:24:5)
  at Object.<anonymous> (production/create-next-app/examples.test.ts:82:5)

Other failing CI jobs

@aurorascharff
aurorascharff changed the base branch from canary to codex/optimize-static-shell-guide August 24, 2026 11:08
@aurorascharff aurorascharff changed the title Codex/optimize static shell eval test: evaluate static shell optimization Aug 24, 2026
@aurorascharff
aurorascharff marked this pull request as ready for review August 27, 2026 17:04
@aurorascharff
aurorascharff merged commit 25d1ba3 into codex/optimize-static-shell-guide Aug 27, 2026
128 of 132 checks passed
@aurorascharff
aurorascharff deleted the codex/optimize-static-shell-eval branch August 27, 2026 17:04
aurorascharff added a commit that referenced this pull request Sep 8, 2026
Adds an agent eval for the static-shell optimization workflow in #97737.
The fixture uses a release-operations route with a request-specific
viewer, reusable checklist, live rollout data, and both direct and
client navigation.

This tests whether agents can transfer the guide and optimizer skill to
a difficult refactor that is similar to, but not copied from, the guide
example.

- Claude Sonnet 4.6: baseline 2/7, bundled docs 2/7, optimizer skill 6/7
- The skill treatment passed every final-code criterion; its only miss
was executing the browser RED-to-GREEN loop because the sandbox lacked
Chromium libraries
- Browser-dependent fixtures now provision those libraries during setup;
a follow-up attempt passed setup but stalled before producing a
transcript
aurorascharff added a commit that referenced this pull request Sep 10, 2026
Adds an agent eval for the static-shell optimization workflow in #97737.
The fixture uses a release-operations route with a request-specific
viewer, reusable checklist, live rollout data, and both direct and
client navigation.

This tests whether agents can transfer the guide and optimizer skill to
a difficult refactor that is similar to, but not copied from, the guide
example.

- Claude Sonnet 4.6: baseline 2/7, bundled docs 2/7, optimizer skill 6/7
- The skill treatment passed every final-code criterion; its only miss
was executing the browser RED-to-GREEN loop because the sandbox lacked
Chromium libraries
- Browser-dependent fixtures now provision those libraries during setup;
a follow-up attempt passed setup but stalled before producing a
transcript
aurorascharff added a commit that referenced this pull request Sep 10, 2026
Adds an agent eval for the static-shell optimization workflow in #97737.
The fixture uses a release-operations route with a request-specific
viewer, reusable checklist, live rollout data, and both direct and
client navigation.

This tests whether agents can transfer the guide and optimizer skill to
a difficult refactor that is similar to, but not copied from, the guide
example.

- Claude Sonnet 4.6: baseline 2/7, bundled docs 2/7, optimizer skill 6/7
- The skill treatment passed every final-code criterion; its only miss
was executing the browser RED-to-GREEN loop because the sandbox lacked
Chromium libraries
- Browser-dependent fixtures now provision those libraries during setup;
a follow-up attempt passed setup but stalled before producing a
transcript
aurorascharff added a commit that referenced this pull request Sep 10, 2026
Adds an agent eval for the static-shell optimization workflow in #97737.
The fixture uses a release-operations route with a request-specific
viewer, reusable checklist, live rollout data, and both direct and
client navigation.

This tests whether agents can transfer the guide and optimizer skill to
a difficult refactor that is similar to, but not copied from, the guide
example.

- Claude Sonnet 4.6: baseline 2/7, bundled docs 2/7, optimizer skill 6/7
- The skill treatment passed every final-code criterion; its only miss
was executing the browser RED-to-GREEN loop because the sandbox lacked
Chromium libraries
- Browser-dependent fixtures now provision those libraries during setup;
a follow-up attempt passed setup but stalled before producing a
transcript
@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Sep 11, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants