Repository navigation
Conversation
|
Ready for review. This extends the existing per-model manifest selection to the eight-B300 runner while preserving the four-GB300 path. Local selector regressions, manifest routing, pre-commit checks and Actionlint passed; GitHub lint also passed. No runtime or kernel changes. Companion sgl-project/sgl-nvidia-mega#73 remains draft. Its B300 E2E run depends on this workflow reaching main because the cross-repository trigger dispatches main. Native B300 validation has not run yet; we will run that after this routing change merges. |
|
/tag-and-rerun-ci |
| fail-fast: false | ||
| matrix: ${{ fromJSON(needs.select.outputs.matrix) }} | ||
| name: ${{ matrix.models }} (${{ matrix.runner }}) | ||
| runs-on: ${{ matrix.runner }} |
There was a problem hiding this comment.
This workflow is workflow_dispatch only, so this PR's checks never run it, and Mega's run-ci-sglang always dispatches main. As it stands, the first real run of this matrix and the B300 job would come from a Mega PR's label. We may need to dispatch it once from a branch in sgl-project/sglang with sgl-project/sgl-nvidia-mega#73's head (-f models=kimi_k3), to make sure an issue here does not break the integration runs of Mega PRs.
There was a problem hiding this comment.
Agreed; running this before merge. Created the isolated upstream testing branch ci/leejnau-kimi-b300-premerge-51429 at f229a6f and dispatched with models=kimi_k3 against companion #73 head 9fba07645fc25fa5777e674964b619a631037292. Run: https://github.com/sgl-project/sglang/actions/runs/44642111731 . CPU selection passed and selected only kimi_k3 (8-gpu-b300); the GPU job is queued. This is not yet a B300 PASS, and #73 remains draft.
There was a problem hiding this comment.
The first dispatch completed selection, installation and the stock arm, then failed during plugin startup on the relocated SGLang context-helper import. Companion #73 fixes that in 82bcae6 without changing kernels or acceptance criteria. Retrying the exact updated pair (framework 926415d / plugin 82bcae6): https://github.com/sgl-project/sglang/actions/runs/44775081714 . Selection passed and the eight-B300 job is running; matched GPU validation is still pending. The failed run and its artifacts remain available.
There was a problem hiding this comment.
The pre-merge eight-B300 run now passes: https://github.com/sgl-project/sglang/actions/runs/44815092291
It passed the selector, five native binding tests plus four subtests, and both matched TP8/q6 E2E tests (captured/eager transitions, 32K input, GSM8K-200 and C1/C4 timing). GSM8K was 151/200 stock vs. 150/200 plugin, within the unchanged tolerance.
The successful framework revision includes a runtime contract repair discovered during this testing. I submitted that separately as #51977; it is not added to this CI-only PR. This PR can merge independently, while the companion Kimi fixture needs both changes in SGLang main. Exact tested revisions and the dependency are now in the description.
| env: | ||
| TESTS: ${{ steps.select.outputs.tests }} | ||
| TESTS: ${{ matrix.tests }} | ||
| KIMI_CI_OUTPUT_DIR: ${{ runner.temp }}/kimi-ci-${{ github.run_id }}-${{ github.run_attempt }} |
There was a problem hiding this comment.
nit: KIMI_CI_OUTPUT_DIR here and the artifact name kimi-ci-<runner> below tie a workflow shared by every Mega model to one of them. Would it make sense to use a model-neutral name, for example MEGA_E2E_OUTPUT_DIR and megakernel-e2e-<runner>, and have each Mega test write under it?
There was a problem hiding this comment.
Agreed. Framework commit b5f4825 switches the shared workflow to MEGA_E2E_OUTPUT_DIR and megakernel-e2e- artifacts. Companion #73 commit 9fba076 writes Kimi evidence under the kimi_k3 subdirectory of that shared root, avoiding cross-model filename collisions. Local artifact-contract checks and applicable pre-commit checks pass.
| MODELS: ${{ inputs.models }} | ||
| run: | | ||
| python3 -m pip install PyYAML | ||
| python3 -m unittest discover -s scripts/ci -p test_megakernel_matrix.py |
There was a problem hiding this comment.
nit: the selector test runs only at dispatch time, so a regression in megakernel_matrix.py would first show up as a failed Mega PR run. ci-model-inventory.yml runs test_list_stage_models.py on a pull_request path filter. Would it make sense to do the same here?
There was a problem hiding this comment.
Added in f229a6f: CI Megakernel Matrix runs the existing selector regression on ubuntu-latest for pull requests and main pushes touching the selector, its test, or either workflow. It needs only Python/PyYAML and does not use a GPU or install SGLang. The automatic PR run passed: https://github.com/sgl-project/sglang/actions/runs/44642063835
nvpohanh
left a comment
There was a problem hiding this comment.
[by Claude Code] This review is done by agents. Please feel free to disagree if they do not make sense.
| ): | ||
| (root / name).touch() | ||
| self.assertEqual( | ||
| build_matrix(root, [])["include"][0]["runner"], "4-gpu-gb300" |
There was a problem hiding this comment.
[by Claude Code] Severity: functional | Confidence: High
Issue
This test calls build_matrix() on fixture manifests, and build_matrix() prints ::warning:: and ::notice:: lines to stdout. GitHub turns them into real annotations. The select job in pr-test-megakernel.yml (line 79) runs this test on every Mega dispatch, so every run summary shows warnings about fixture files that are not in the Mega PR.
Example
The select job of run 44642111731 shows "test_kimi.py is not listed in ci_models.yaml", "test_qwen.py is not listed in ci_models.yaml", and "qwen requires 8 GB300 GPUs; no configured runner". None of these come from the Mega manifest.
Fix
Wrap the build_matrix() calls in contextlib.redirect_stdout(io.StringIO()) and assert the expected lines. That also tests the skip notice. Or drop the unittest from the select step, since ci-megakernel-matrix.yml now runs it on PRs.
There was a problem hiding this comment.
Addressed in 02f52d8. The existing selector test captures stdout and asserts the expected fixture warnings and unavailable-runner notice, so they no longer become annotations on a real dispatch. The explicit Kimi selection also asserts that it emits no warning/notice. The selector regression and applicable pre-commit checks pass.
| parser.add_argument("e2e", type=Path) | ||
| parser.add_argument("--models", nargs="*", default=[]) | ||
| args = parser.parse_args() | ||
| matrix = json.dumps(build_matrix(args.e2e, args.models)) |
There was a problem hiding this comment.
[by Claude Code] Severity: nit | Confidence: High
Issue
build_matrix() reports bad input by raising ValueError, and __main__ does not catch it. The run then fails with a traceback in the select log, and the run summary shows only "Process completed with exit code 1". The old inline selector printed ::error:: lines, so the reason showed on the summary page that the Mega status links to.
Fix
Catch ValueError here, print ::error::{e}, and exit with status 1.
There was a problem hiding this comment.
Addressed in 926415d. The CLI catches ValueError, emits an ::error:: annotation and exits with status 1. A subprocess regression asserts the annotation, nonzero status and absence of a traceback. Local selector and formatting checks pass.
Motivation
Megakernel E2E CI currently uses a fixed four-GB300 runner. Kimi K3 needs eight GPUs; add per-model routing to the existing eight-B300 runner while preserving the DeepSeek four-GB300 path.
Modifications
Companion sgl-nvidia-mega #73 adds the matched Kimi tests and manifest entry. No production runtime or kernel changes.
Accuracy Tests
Local selector regression, real-manifest routing, applicable pre-commit checks and Actionlint 1.7.7 passed. Automatic selector CPU CI passes; fixture annotations are asserted and invalid selections emit explicit error annotations.
Pre-merge B300 run 44815092291 passed: selector, five native binding tests plus four subtests, and both matched accuracy/performance E2E tests. TP8/q6, DCP off: captured B1/B2, eager B3/B4, return to B1, and a 32,768-token passkey input. GSM8K-200: stock 151/200, plugin 150/200, zero request errors; unchanged accuracy/logprob bounds passed.
The exact tested pair is framework
0da13fa8c67001fc88d7d9ed454cd6a2b2f41b70/ pluginc5d2baead02d801e6a14b7d81239c02198855190. The framework includes this CI work plus the separate runtime repair now submitted as #51977. Earlier attempts exposed import, phase-backend admission and missing q6 binding fields; those failed records remain intact. This CI-only PR can merge independently, but the matched Kimi fixture needs #51977 as well. The successful run is not evidence that the unmodified CI-only branch supplies that runtime fix.Speed Tests and Profiling
No inference-speed change. The companion tests collect matched quality and repeated serving performance. The existing plugin trigger dispatches SGLang
main, so this workflow must be available there before using that trigger for B300.CI States
Latest PR Test (Base): ✅ Run #44775010828
Latest PR Test (Extra): ⊘ Skipped -- add the
run-ci-nv-extralabel to opt in.Latest PR Test (AMD ROCm 10): ✅ Run #44775010114
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: