refactor(runtime)!: adopt the modern Infini stack - #506
Draft
voltjia wants to merge 23 commits into
Draft
Conversation
voltjia
force-pushed
the
refactor/adopt-modern-infini-stack
branch
2 times, most recently
from
August 11, 2026 22:09
5c24dc2 to
7c36e2f
Compare
BREAKING CHANGE: legacy InfiniCore runtime and inference APIs are removed in favor of InfiniRT, InfiniOps, and the migrated InfiniLM runtime.
voltjia
force-pushed
the
refactor/adopt-modern-infini-stack
branch
from
August 13, 2026 15:26
7c36e2f to
077867b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary - Move the runtime context, tensor layer, operator adapters, graph bridge, distributed wrappers, and Python bindings into InfiniLM so the inference repository owns its integration layer. - Build InfiniRT, a focused canonical InfiniOps closure, and InfiniCCL from the revisions pinned by InfiniCore. - Select exact InfiniOps implementation slots from a consumer-owned
scripts/configs/infiniops_ops.json: native slot 0 where appropriate, generated Torch slot 8 for Argmax, linked slot 16 for FlashAttention and FlashInfer sampling. - Replace production*Infinilmoperator APIs with canonical APIs and restrict the compiled modern model surface to supported dense non-quantized Qwen3. - Rebase the complete 20-commit migration onto currentmain58b8c0cb. ## Motivation The inference stack previously depended on runtime and operator implementations duplicated across repository boundaries. This change makes InfiniRT, InfiniOps, and InfiniCCL authoritative while keeping inference-specific integration code with InfiniLM. Related to InfiniTensor/InfiniCore#1373. ## Component Contract - InfiniRT0cdbb16967e15f2e055dea1ec9641617bf3b6cf6- InfiniOps94965ca433ea44b9cd4b8520a0f1d9687f81cfa9- InfiniCCLe8ccc0cb23ca5b1b1d63be29ba61ea031de72807InfiniTensor/InfiniOps#931 provides unified exact-slot selection throughops.json; stacked InfiniTensor/InfiniOps#930 provides TVM-FFI transport and FlashInferTopKTopPSamplingFromLogitsslot 16. Native sampling PR InfiniTensor/InfiniOps#922 is not in the selected InfiniOps ancestry. The FlashInfer linked DSOs remain external runtime dependencies under the existing linked-provider contract. The build environment must provideflashinfer-jit-cache>=0.6.7,<0.7andapache-tvm-ffi==0.1.10. ## Validation ### Existing migration baseline The prior exact component combination passed a clean A100 wheel build, import, static and paged attention, eager and graph execution, batch 1/2, greedy/non-greedy sampling, TP1/TP2, PP2/TP1, and TP2+PP2. ### Final FlashInfer provider - InfiniOps Release/NDEBUG build passed without provider warnings. - Linked resolver tests passed 33/33. - FlashInfer slot 16 GPU tests passed 10/10, including both filter orders, host/CUDA indices, int32/int64 output, seed/offset, and a non-default-stream nonblocking sentinel. - InfiniLM range-diff confirms all 20 migration commits survived the rebase unchanged;git diff --checkpasses. - The final integrated InfiniLM wheel/Qwen smoke against the new component SHA is in progress; this PR remains draft until that run is recorded here. ## Supported Boundary - Validated platform: NVIDIA A100. - Compiled model: dense non-quantized Qwen3. - Unsupported quantization and linear-bias configurations are rejected before worker startup. - Paged attention uses the default 256-token block size; custom block sizes remain unvalidated. - Other model families and platforms remain outside this migration validation. ## Type of Change - [x] refactor - [x] fix - [x] test - [x] docs - [x] build / CI - [x] breaking change ## Landing Order 1. InfiniTensor/InfiniOps#931. 2. InfiniTensor/InfiniOps#930, rebased/retargeted after InfiniTensor/InfiniOps#931 merges. 3. InfiniTensor/InfiniCCL#57, InfiniTensor/InfiniCCL#58, and InfiniTensor/InfiniCCL#59. 4. InfiniTensor/InfiniCore#1406. 5. This PR.