Skip to content

refactor(runtime)!: adopt the modern Infini stack - #506

Draft
voltjia wants to merge 23 commits into
mainfrom
refactor/adopt-modern-infini-stack
Draft

refactor(runtime)!: adopt the modern Infini stack#506
voltjia wants to merge 23 commits into
mainfrom
refactor/adopt-modern-infini-stack

Conversation

@voltjia

@voltjia voltjia commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Summary - Move the runtime context, tensor layer, operator adapters, graph bridge, distributed wrappers, and Python bindings into InfiniLM so the inference repository owns its integration layer. - Build InfiniRT, a focused canonical InfiniOps closure, and InfiniCCL from the revisions pinned by InfiniCore. - Select exact InfiniOps implementation slots from a consumer-owned scripts/configs/infiniops_ops.json: native slot 0 where appropriate, generated Torch slot 8 for Argmax, linked slot 16 for FlashAttention and FlashInfer sampling. - Replace production *Infinilm operator APIs with canonical APIs and restrict the compiled modern model surface to supported dense non-quantized Qwen3. - Rebase the complete 20-commit migration onto current main 58b8c0cb. ## Motivation The inference stack previously depended on runtime and operator implementations duplicated across repository boundaries. This change makes InfiniRT, InfiniOps, and InfiniCCL authoritative while keeping inference-specific integration code with InfiniLM. Related to InfiniTensor/InfiniCore#1373. ## Component Contract - InfiniRT 0cdbb16967e15f2e055dea1ec9641617bf3b6cf6 - InfiniOps 94965ca433ea44b9cd4b8520a0f1d9687f81cfa9 - InfiniCCL e8ccc0cb23ca5b1b1d63be29ba61ea031de72807 InfiniTensor/InfiniOps#931 provides unified exact-slot selection through ops.json; stacked InfiniTensor/InfiniOps#930 provides TVM-FFI transport and FlashInfer TopKTopPSamplingFromLogits slot 16. Native sampling PR InfiniTensor/InfiniOps#922 is not in the selected InfiniOps ancestry. The FlashInfer linked DSOs remain external runtime dependencies under the existing linked-provider contract. The build environment must provide flashinfer-jit-cache>=0.6.7,<0.7 and apache-tvm-ffi==0.1.10. ## Validation ### Existing migration baseline The prior exact component combination passed a clean A100 wheel build, import, static and paged attention, eager and graph execution, batch 1/2, greedy/non-greedy sampling, TP1/TP2, PP2/TP1, and TP2+PP2. ### Final FlashInfer provider - InfiniOps Release/NDEBUG build passed without provider warnings. - Linked resolver tests passed 33/33. - FlashInfer slot 16 GPU tests passed 10/10, including both filter orders, host/CUDA indices, int32/int64 output, seed/offset, and a non-default-stream nonblocking sentinel. - InfiniLM range-diff confirms all 20 migration commits survived the rebase unchanged; git diff --check passes. - The final integrated InfiniLM wheel/Qwen smoke against the new component SHA is in progress; this PR remains draft until that run is recorded here. ## Supported Boundary - Validated platform: NVIDIA A100. - Compiled model: dense non-quantized Qwen3. - Unsupported quantization and linear-bias configurations are rejected before worker startup. - Paged attention uses the default 256-token block size; custom block sizes remain unvalidated. - Other model families and platforms remain outside this migration validation. ## Type of Change - [x] refactor - [x] fix - [x] test - [x] docs - [x] build / CI - [x] breaking change ## Landing Order 1. InfiniTensor/InfiniOps#931. 2. InfiniTensor/InfiniOps#930, rebased/retargeted after InfiniTensor/InfiniOps#931 merges. 3. InfiniTensor/InfiniCCL#57, InfiniTensor/InfiniCCL#58, and InfiniTensor/InfiniCCL#59. 4. InfiniTensor/InfiniCore#1406. 5. This PR.

@voltjia
voltjia force-pushed the refactor/adopt-modern-infini-stack branch 2 times, most recently from 5c24dc2 to 7c36e2f Compare August 11, 2026 22:09
@voltjia
voltjia force-pushed the refactor/adopt-modern-infini-stack branch from 7c36e2f to 077867b Compare August 13, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant