Shared PTW and other VM improvements - #392
Open
ATWeatherly wants to merge 14 commits into
Open
Conversation
TFifo::empty() reports only entries that have emerged from the pipe's latency window (SimChannel::queue_empty ignores pending_count_), so processFlush()'s drain guard could not see a replayed store pushed into pipe_req_ earlier in the same tick. The walk then swept the store's set while its line was still clean and finished before the write applied, losing the store on an end-of-kernel flush. Observed with vm_stress on simx/XLEN=32 once the VM MMU model added translation latency, but the window is generic to any replay that lands as the flush starts; size() counts in-flight entries and closes it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TxArbiter and TxRxArbiter sized their per-output request group with log2ceil(num_inputs / num_outputs); when the input count is not a multiple of the output count that rounds down and the trailing inputs are never polled (VX_stream_arb uses CDIV). Callers had to pad input counts up to a power-of-two multiple to work around it. Also adds the CHANGELOG entry for the cache flush-walk fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…BARE cases A stray `.` in VX_mmu's VX_mmu_tlb instantiation broke every VM RTL build with profiling enabled, which is why rtlsim VM never made it into CI. The MMU and PTW took a 32-bit `satp`, truncating the CSR on XLEN=64 and testing bit 31 where Sv39 keeps its mode field in bits 63:60; the port is now XLEN wide and the translate gate follows both encodings. VX_mmu_tlb hardcoded 5 index bits, so VX_CFG_TLB_SIZE silently mis-indexed the CAM for any value other than 32; the width is derived from the config. ci/testcases/vm.yaml isa-6..10 and tests/regression/basic/start.S still used the retired VX_CFG_VM_ADDR_MODE / VX_CFG_PAGE_TABLE_BASE_ADDR names. VX_VM_ADDR_MODE is a resolved VX_types contract fixed at configure time and gen_config drops unknown -D keys, so those BARE cases only re-ran the default mode; they are removed rather than left claiming coverage, and start.S uses the live VX_VM_* / VX_MEM_* names. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W3o1M5sa5KmNwsbU9wptw4
…Bs; Sv39 The per-MMU blocking Sv32 walker could not terminate on the superpage leaves the runtime installs for identity-mapped regions, so rtlsim VM was broken and absent from CI. It is replaced by one device-level walker (hw/rtl/mem/VX_mmu_ptw.sv): generic over the Sv32/Sv39 page-table geometry, VX_CFG_PTW_NUM_WALKERS concurrent walk slots, PTE fetches on a dedicated L3 requestor port (L3_PTW_IDX), and two direct-mapped page-walk caches (VX_mmu_pwc.sv, VX_CFG_PTW_WALK_CACHE_SIZE entries) so a warm walk starts one (Sv32) or two (Sv39) levels below the root. Leaf, validity, and superpage-alignment checks per the privileged spec report a fault on the fill; the TLB then replays the access untranslated, mirroring the CP walker's defensive pass-through. TLB misses ride a new VX_ptw_bus_if through one VX_ptw_arb per hierarchy level (banks -> core -> socket -> cluster -> device), each folding its source index into the tag, so every bank owns one outstanding walk and no per-requestor serialization is needed. SATP stays a per-core CSR; the root PPN travels with each walk request, which also keys the walk caches. Per-core TLBs are banked (VX_CFG_TLB_NUM_BANKS, low VPN bits select the bank; the iTLB stays single-banked) so a miss parked in one bank no longer blocks hits in the others, and fills record their page level so superpage entries match and translate at their real size. The TLBs are invalidated by a one-cycle pulse at the start of the DCR cache flush — the flush request itself is translated, so holding the invalidate at the level of the pending flush would livelock its own walk — and the device walker drops its walk caches on the same event. Walker and walk-cache counters move to sysmem_perf (device totals, same value on every core, like the L3 counters), with new PWC1/PWC2 MPM ids; TLB counters stay per core. DBG_TRACE_MMU traces the TLB banks and the walker. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W3o1M5sa5KmNwsbU9wptw4
Mirrors the RTL topology so the two stay comparable: the per-core Mmu keeps a banked TLB with one outstanding walk per bank and superpage-aware, level-carrying entries, and hands misses over PtwReq/PtwRsp channels to one device-level Ptw SimObject with concurrent walk slots and PWC1/PWC2, fetching PTEs on the dedicated L3 requestor port. TLB counters surface per core and the walker counters as device totals under the MEM MPM class — previously the SimX MMU counters were never surfaced at all. The TLBs and walk caches are invalidated on the cache-flush sweep, and a satp write flushes only when the value changes. The L3 bypass arbiter serves its inputs in power-of-two groups per memory port, so the walker's odd extra port would never be polled; the L3 input count is padded up with idle ports instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W3o1M5sa5KmNwsbU9wptw4
… VA overlap The vm catalog now runs the compute regressions on rtlsim as well as simx at both XLENs, adds an xrt (RTL command processor) case, a vm_stress TLB-pressure test — strided page touches per task plus a VX_MEM_PHYS buffer — and full-tier single-bank and multi-cluster variants. vm_stress exposed that the VA allocator hands out addresses from the same base the pinned identity-mapped slab occupies, so a VX_MEM_PHYS buffer created after any ordinary allocation collided in the page table; the slab's range is now reserved out of the VA space at VM bring-up. The VM design doc and vm.md are updated to the shared-walker topology. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W3o1M5sa5KmNwsbU9wptw4
VM previously worked on simx/rtlsim/gem5 and silently no-oped on xrt/opae: the runtime discovers VM support from DEV_CAPS bit 24, which the RTL regfile never set, so no page tables were built while a VM-enabled kernel still wrote satp and walked an empty table. The RTL CP now matches the software CP model: DEV_CAPS publishes VM_ENABLED, CP_SATP_LO/HI (0x028/0x02C) hold the page-table root, and the DMA engine translates its device-side operand once per chunk (chunks never cross a page) through a new VX_cp_mmu walker that reads PTEs over the DMA's own device AXI channel while the transfer FSM is parked. F_MEM_PHYSICAL skips translation; a faulted walk passes the address through untranslated, mirroring cp_translate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W3o1M5sa5KmNwsbU9wptw4
needs_translation() and the two full_addr wires in VX_mmu were hardcoded to 32 bits. On XLEN=64 the 48-bit byte address is truncated, and Verilator flags WIDTHTRUNC; the AFU sims (xrtsim/opaesim) run without -Wno-fatal, so every -DVX_CFG_VM_ENABLE build64 of libxrtsim.so failed. The value is unused today (only the SATP mode gates translation), so this is lint-only, but it blocked the vm:xrt-cp case at XLEN=64. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The banked MMU model popped a missing packet off ReqIn and parked it in a BankMiss slot. That slot is not a SimChannel, so a store waiting on a page walk was invisible to SimChannelBase::inflight_count(), and the processor could declare the kernel done and flush the dcache while the store was still walking: vm_stress on simx/XLEN=32 lost exactly one store this way (the flush-wb swept the line one cycle before the MSHR replay landed). Popping early also let later packets on the same lane overtake the parked one, which the RTL's per-lane elastic buffer never does. Leave the packet at the head of its port and pop it only when the replay is sent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ches VX_cp_axil_regfile gained a satp output and VX_cp_dma a satp input for the CP DMA walker, but the block-level harnesses under hw/unittest/ still instantiated the old port lists; Verilator's PINMISSING is fatal there, so 'make -C hw/unittest' (the unittest:hw CI case) failed to build. Expose satp on both wrapper tops so a future VM-aware DMA test can drive it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With translation on, SimX ran 16-36% fewer cycles than rtlsim on hit-heavy kernels (sgemm, diverge) while the VM-off baseline agreed within 2%: the model translated a TLB hit in the same tick with unlimited throughput, whereas every RTL access crosses the per-lane elastic buffer, the lane->bank crossbar, the single-ported bank CAM, and the gather crossbar. Charge that path: one lookup per bank per cycle with round-robin port arbitration, a TRANSLATE_LATENCY (5, calibrated) on translated and replayed requests, and skip the lookup when the output port is stalled so a retry does not double-count the TLB perf counters. Calibrated RV32: sgemm 1,298,133 vs rtlsim 1,295,022 (+0.2%), diverge 1,149,964 vs 1,107,733 (+3.8%); RV64: sgemm +0.1%, diverge +4.1%, vm_stress +3.0%. Gate it with model_parity-sgemm / model_parity-diverge in the vm catalog (RV32-pinned like every check gate, full tier). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cache, share VM geometry VX_cp_mmu took satp[43:0] as the root PPN in both modes, so under Sv32 the MODE bit and ASID field folded into the table address (masked on xrt only because the AFU truncates the CP address to 32 bits). Its 2-entry translation cache was consulted even in BARE mode and cleared only by hardware reset, so a translation cached under one page table survived a CP_SATP rewrite, the runtime's CACHE_FLUSH, or a later process reusing the VA; the SW CP model walks every command, so simx could not see it. And it re-derived the Sv32/Sv39 geometry and PTE decode by hand, without the superpage alignment check VX_mmu_ptw enforces. The walker now imports VX_gpu_pkg (VM_* params, vm_pte_valid/is_leaf), slices the root PPN like VX_mmu, bypasses the cache in BARE mode, drops it on a CACHE_FLUSH DCR read or a SATP change (a walk straddling the invalidation is answered but not cached), and faults misaligned superpages. VX_cp_core derives the flush pulse from the DCR proxy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A TLB flush only suppressed an install in the one-cycle flush pulse, and the walker had no flush term at all, so a walk already in flight when the CACHE_FLUSH DCR arrived installed a translation resolved against the old page table into the freshly flushed TLB and page-walk caches. The TLB bank now latches flush_pending for a walk that was outstanding at the flush, discards that fill and re-walks; the PTW marks active slots stale and keeps their PTEs out of the PWCs. The SimX Mmu/Ptw mirror both (stale bank -> re-walk, stale slot -> no PWC fill). Ptw::on_reset now also zeroes its perf counters like the caches do. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- The shared PTW's client count and the mmu_flush loop used VX_CFG_NUM_CORES, but a cluster instantiates NUM_SOCKETS * SOCKET_SIZE cores; a non-divisible shape left trailing cores' walker ports unbound (first TLB miss hung the run). - Drop the L3 request-count padding that worked around the arbiter grouping bug (fixed separately). - Identity maps installed inside the pinned slab re-reserved VA space the slab reservation already held, printing an allocator overlap error per VX_MEM_PHYS buffer; VMManager now owns the slab range and skips those. - Tlb perf counters reset per launch (Mmu::on_reset), and the RTL's power-of-two TLB_NUM_BANKS constraint is static_asserted in SimX. - CHANGELOG entries for the VM fixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
|
sorry for the monster PR. Let me know if I should make better attempt to break individual components down |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(
hw/rtl/mem/VX_mmu_ptw.sv,VX_CFG_PTW_NUM_WALKERSconcurrent walks, two direct-mapped page-walk caches,PTE fetches on a dedicated L3 requestor port) reached through a new
VX_ptw_bus_if/VX_ptw_arbhierarchy.VX_CFG_TLB_NUM_BANKS), superpage leaves installed at their real level, walk faultsreported on the fill. SimX timing twin (
sim/simx/mem/ptw.{h,cpp}, bankedMmu) with the TLB/PTW/PWC MPMcounters under the MEM class.
CP_SATP_LO/HI,DEV_CAPS.VM_ENABLED, and a per-chunkVX_cp_mmuwalker in the DMA engine (F_MEM_PHYSICALbypasses).tests/regression/vm_stress;ci/testcases/vm.yamlnow runs simx and rtlsim at both XLENs, anxrtcase through the CP + DMA walker, single-bank and multi-cluster L2/L3 variants, and
model_paritycases(sgemm, diverge) that hold the SimX MMU model to rtlsim.
[Unreleased]/Fixed): SimX lost stores around a walk;SimX VM timing parity (was +36%/+16% rtlsim-vs-simx cycles on sgemm/diverge, now +0.2%/+3.8%); CP walker Sv32
root-PPN slice, translation-cache invalidation (CACHE_FLUSH/SATP), BARE bypass, superpage alignment, shared
VX_gpu_pkggeometry; stale installs from walks straddling a flush (RTL + SimX); PTW client sizing byNUM_SOCKETS*SOCKET_SIZE; pinned-slab identity maps no longer re-reserve VA; perf counters reset per launch;VX_mmubypass-path width on XLEN=64; CP block testbenches carry the new ports.NO REAL FPGA TEST PERFORMED. Everything else is solid though. Waiting for Rogues Gallery admin memory configuration of U50 FPGAs due to 3.0 xrt runtime changes (something like sudo bash -c 'source /opt/xilinx/xrt/setup.sh && xrt-smi configure --host-mem -d 0000:04:00.1 --size 1G enable')