Skip to content

[CI] megakernel: Select per-model GB300 and B300 runners - #51429

Open
leejnau wants to merge 5 commits into
sgl-project:mainfrom
leejnau:ci/kimi-k3-b300-e2e
Open

leejnau wants to merge 5 commits into
sgl-project:mainfrom
leejnau:ci/kimi-k3-b300-e2e

Conversation

@leejnau

@leejnau leejnau commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Megakernel E2E CI currently uses a fixed four-GB300 runner. Kimi K3 needs eight GPUs; add per-model routing to the existing eight-B300 runner while preserving the DeepSeek four-GB300 path.

Modifications

  • Read the pinned plugin manifest in a CPU selection job, without executing plugin code. Runner labels come from a framework-owned allowlist.
  • Validate model names, GPU counts and test paths; fail explicit requests for unavailable configurations. Retain legacy manifest-free selection.
  • Run independent runner groups and retain model-specific comparison artifacts on success or failure. Workflow inputs and the cross-repository trigger protocol stay unchanged.

Companion sgl-nvidia-mega #73 adds the matched Kimi tests and manifest entry. No production runtime or kernel changes.

Accuracy Tests

Local selector regression, real-manifest routing, applicable pre-commit checks and Actionlint 1.7.7 passed. Automatic selector CPU CI passes; fixture annotations are asserted and invalid selections emit explicit error annotations.

Pre-merge B300 run 44815092291 passed: selector, five native binding tests plus four subtests, and both matched accuracy/performance E2E tests. TP8/q6, DCP off: captured B1/B2, eager B3/B4, return to B1, and a 32,768-token passkey input. GSM8K-200: stock 151/200, plugin 150/200, zero request errors; unchanged accuracy/logprob bounds passed.

The exact tested pair is framework 0da13fa8c67001fc88d7d9ed454cd6a2b2f41b70 / plugin c5d2baead02d801e6a14b7d81239c02198855190. The framework includes this CI work plus the separate runtime repair now submitted as #51977. Earlier attempts exposed import, phase-backend admission and missing q6 binding fields; those failed records remain intact. This CI-only PR can merge independently, but the matched Kimi fixture needs #51977 as well. The successful run is not evidence that the unmodified CI-only branch supplies that runtime fix.

Speed Tests and Profiling

No inference-speed change. The companion tests collect matched quality and repeated serving performance. The existing plugin trigger dispatches SGLang main, so this workflow must be available there before using that trigger for B300.


CI States

Latest PR Test (Base): ✅ Run #44775010828
Latest PR Test (Extra): ⊘ Skipped -- add the run-ci-nv-extra label to opt in.
Latest PR Test (AMD ROCm 10): ✅ Run #44775010114

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@leejnau
leejnau requested a review from YAMY1234 November 19, 2026 23:03
@leejnau

leejnau commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

Ready for review. This extends the existing per-model manifest selection to the eight-B300 runner while preserving the four-GB300 path. Local selector regressions, manifest routing, pre-commit checks and Actionlint passed; GitHub lint also passed. No runtime or kernel changes.

Companion sgl-project/sgl-nvidia-mega#73 remains draft. Its B300 E2E run depends on this workflow reaching main because the cross-repository trigger dispatches main. Native B300 validation has not run yet; we will run that after this routing change merges.

@leejnau

leejnau commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Nov 19, 2026
fail-fast: false
matrix: ${{ fromJSON(needs.select.outputs.matrix) }}
name: ${{ matrix.models }} (${{ matrix.runner }})
runs-on: ${{ matrix.runner }}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This workflow is workflow_dispatch only, so this PR's checks never run it, and Mega's run-ci-sglang always dispatches main. As it stands, the first real run of this matrix and the B300 job would come from a Mega PR's label. We may need to dispatch it once from a branch in sgl-project/sglang with sgl-project/sgl-nvidia-mega#73's head (-f models=kimi_k3), to make sure an issue here does not break the integration runs of Mega PRs.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed; running this before merge. Created the isolated upstream testing branch ci/leejnau-kimi-b300-premerge-51429 at f229a6f and dispatched with models=kimi_k3 against companion #73 head 9fba07645fc25fa5777e674964b619a631037292. Run: https://github.com/sgl-project/sglang/actions/runs/44642111731 . CPU selection passed and selected only kimi_k3 (8-gpu-b300); the GPU job is queued. This is not yet a B300 PASS, and #73 remains draft.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first dispatch completed selection, installation and the stock arm, then failed during plugin startup on the relocated SGLang context-helper import. Companion #73 fixes that in 82bcae6 without changing kernels or acceptance criteria. Retrying the exact updated pair (framework 926415d / plugin 82bcae6): https://github.com/sgl-project/sglang/actions/runs/44775081714 . Selection passed and the eight-B300 job is running; matched GPU validation is still pending. The failed run and its artifacts remain available.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pre-merge eight-B300 run now passes: https://github.com/sgl-project/sglang/actions/runs/44815092291

It passed the selector, five native binding tests plus four subtests, and both matched TP8/q6 E2E tests (captured/eager transitions, 32K input, GSM8K-200 and C1/C4 timing). GSM8K was 151/200 stock vs. 150/200 plugin, within the unchanged tolerance.

The successful framework revision includes a runtime contract repair discovered during this testing. I submitted that separately as #51977; it is not added to this CI-only PR. This PR can merge independently, while the companion Kimi fixture needs both changes in SGLang main. Exact tested revisions and the dependency are now in the description.

env:
TESTS: ${{ steps.select.outputs.tests }}
TESTS: ${{ matrix.tests }}
KIMI_CI_OUTPUT_DIR: ${{ runner.temp }}/kimi-ci-${{ github.run_id }}-${{ github.run_attempt }}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: KIMI_CI_OUTPUT_DIR here and the artifact name kimi-ci-<runner> below tie a workflow shared by every Mega model to one of them. Would it make sense to use a model-neutral name, for example MEGA_E2E_OUTPUT_DIR and megakernel-e2e-<runner>, and have each Mega test write under it?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Framework commit b5f4825 switches the shared workflow to MEGA_E2E_OUTPUT_DIR and megakernel-e2e- artifacts. Companion #73 commit 9fba076 writes Kimi evidence under the kimi_k3 subdirectory of that shared root, avoiding cross-model filename collisions. Local artifact-contract checks and applicable pre-commit checks pass.

MODELS: ${{ inputs.models }}
run: |
python3 -m pip install PyYAML
python3 -m unittest discover -s scripts/ci -p test_megakernel_matrix.py

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the selector test runs only at dispatch time, so a regression in megakernel_matrix.py would first show up as a failed Mega PR run. ci-model-inventory.yml runs test_list_stage_models.py on a pull_request path filter. Would it make sense to do the same here?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in f229a6f: CI Megakernel Matrix runs the existing selector regression on ubuntu-latest for pull requests and main pushes touching the selector, its test, or either workflow. It needs only Python/PyYAML and does not use a GPU or install SGLang. The automatic PR run passed: https://github.com/sgl-project/sglang/actions/runs/44642063835

@nvpohanh nvpohanh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Claude Code] This review is done by agents. Please feel free to disagree if they do not make sense.

):
(root / name).touch()
self.assertEqual(
build_matrix(root, [])["include"][0]["runner"], "4-gpu-gb300"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Claude Code] Severity: functional | Confidence: High

Issue

This test calls build_matrix() on fixture manifests, and build_matrix() prints ::warning:: and ::notice:: lines to stdout. GitHub turns them into real annotations. The select job in pr-test-megakernel.yml (line 79) runs this test on every Mega dispatch, so every run summary shows warnings about fixture files that are not in the Mega PR.

Example

The select job of run 44642111731 shows "test_kimi.py is not listed in ci_models.yaml", "test_qwen.py is not listed in ci_models.yaml", and "qwen requires 8 GB300 GPUs; no configured runner". None of these come from the Mega manifest.

Fix

Wrap the build_matrix() calls in contextlib.redirect_stdout(io.StringIO()) and assert the expected lines. That also tests the skip notice. Or drop the unittest from the select step, since ci-megakernel-matrix.yml now runs it on PRs.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 02f52d8. The existing selector test captures stdout and asserts the expected fixture warnings and unavailable-runner notice, so they no longer become annotations on a real dispatch. The explicit Kimi selection also asserts that it emits no warning/notice. The selector regression and applicable pre-commit checks pass.

Comment thread scripts/ci/megakernel_matrix.py Outdated
parser.add_argument("e2e", type=Path)
parser.add_argument("--models", nargs="*", default=[])
args = parser.parse_args()
matrix = json.dumps(build_matrix(args.e2e, args.models))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[by Claude Code] Severity: nit | Confidence: High

Issue

build_matrix() reports bad input by raising ValueError, and __main__ does not catch it. The run then fails with a traceback in the select log, and the run summary shows only "Process completed with exit code 1". The old inline selector printed ::error:: lines, so the reason showed on the summary page that the Mega status links to.

Fix

Catch ValueError here, print ::error::{e}, and exit with status 1.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 926415d. The CLI catches ValueError, emits an ::error:: annotation and exits with status 1. A subprocess regression asserts the annotation, nonzero status and absence of a traceback. Local selector and formatting checks pass.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants