#250 - Surface 802.3x pause state and counters by chloecrozier · Pull Request #252 · NVIDIA/daqiri

chloecrozier · 2026-08-20T05:36:57Z

Closes #250.

Found while chasing the ResNet benchmark's 308 Gb/s cap: 802.3x pause was eating 21.7% of
400 GbE line rate with every drop counter at zero, so it read as a slow transmitter. Two
things should have caught it and neither did.

tune_system.py had no pause check. Added --check pause: reports the pause parameters
per NIC, and separately whether the pause counters are non-zero, since pause that never fires
costs nothing. Also Step 10 in the system configuration guide and a warning in the raw
Ethernet benchmarking guide.

Pause counters never reached the xstats dump. They are not in rte_eth_xstats_get on
mlx5 at all, so the dump could not print them. src/net_pause.{h,cpp} reads them from the
kernel netdev instead (ETHTOOL_GPAUSEPARAM / ETHTOOL_GSTATS), and both raw engines call
it: warn at init when pause is enabled, print the counters with the stats dump. Counters are
per-run deltas against an init baseline, because the raw values are cumulative since boot and
would call a run throttled when pause was off for it.

Test plan

400 GbE loopback, pause on: warns at init, reports the frames exchanged during the run
No pause activity: reports 0, no false "throttled" warning despite ~132M lifetime frames
Port with no netdev (software loopback): silent no-op
--check pause parsing: both-off, both-on, rx-only, tx-only, RX/TX negotiated lines
Build clean, clang-format clean, check_doc_refs.py and mkdocs anchors pass

Pause is enabled by default on many mlx5 ports and caps raw-Ethernet throughput without incrementing any drop counter, so it reads as a slow transmitter rather than a misconfiguration. On a 400 GbE loopback it cost 21.7% of line rate while rx_discards_phy and rx_out_of_buffer stayed at 0. --check pause reports the ethtool pause parameters per NIC and, separately, whether the pause counters are non-zero, since pause that is enabled but never asserted costs nothing. It deliberately makes no per-direction claim: ethtool and systemd-networkd document their rx/tx pause naming with opposite senses, so the advice is to disable both. This covers the tuning-script half of #250. DAQIRI's own xstats dump still cannot see these counters, which is tracked in the same issue. Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>

The tuning-script check added in the previous commit only helps if it is run beforehand. It says nothing about whether pause fired during a run, and the engines could not tell you either: mlx5 does not expose the pause counters through rte_eth_xstats_get, so a throttled run left no trace in the shutdown dump even though it prints every non-zero xstat. Across an entire benchmark campaign not one application log mentioned pause while ethtool showed nine non-zero pause counters on the same ports. Read them from the kernel netdev instead, which works alongside a bound PMD because mlx5 is bifurcated and keeps its netdev. src/net_pause.{h,cpp} wraps ETHTOOL_GPAUSEPARAM and ETHTOOL_GSTATS with no DPDK or ibverbs dependency, so it lives in daqiri_common and both raw engines use it: a warning at init when pause is enabled, and the counters alongside the existing stats dump. The counters are cumulative since boot, so the totals say nothing about a single run -- a link with pause disabled still reports millions from earlier traffic, which would claim a run was throttled when it was not. init records a baseline and the dump reports the delta, matching the surrounding xstats, which are also per-run. No per-direction claim is made anywhere: ethtool and systemd-networkd document their rx/tx pause naming with opposite senses, so the advice is to disable both. Verified on a 400 GbE loopback: warns with pause enabled and reports the frames exchanged during the run, stays silent when the delta is zero, and no-ops on a port with no netdev (software loopback) rather than printing zeros. Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>

greptile-apps · 2026-08-20T05:40:04Z

Greptile Summary

The PR adds 802.3x pause-state diagnostics and per-run pause-counter reporting for the DPDK and raw ibverbs engines, plus a corresponding host-tuning check.

Reads pause configuration and mlx5 counters through kernel ethtool ioctls.
Captures initialization baselines so runtime statistics report per-run deltas.
Documents pause-related throughput diagnosis and updates DGX Spark benchmark results.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains in the eligible follow-up review scope.

No blocking failure remains.

Important Files Changed

Filename	Overview
src/net_pause.cpp	Implements best-effort netdev resolution, ethtool pause-state reads, baseline capture, and per-run counter logging.
src/net_pause.h	Defines the internal pause telemetry state and helper interfaces shared by the raw engines.
src/engines/dpdk/daqiri_dpdk_engine.cpp	Adds initialization warnings and pause-counter output alongside DPDK port statistics.
src/engines/ibverbs/daqiri_ibverbs_engine.cpp	Adds equivalent pause initialization and statistics reporting to the raw ibverbs engine.
python/tune_system.py	Adds a pause-focused readiness check using ethtool configuration and lifetime counters.
src/CMakeLists.txt	Compiles the shared pause telemetry implementation into the common library.

_{Reviews (4): Last reviewed commit: "#250 - Suggest pacing over pause and dro..." | Re-trigger Greptile}

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>

…ration Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>

cliffburdick · 2026-08-25T18:04:08Z

+
+        On a lossless RoCE or PFC fabric, flow control is deliberate and required. Only disable pause on links dedicated to raw Ethernet where the application tolerates loss, and never on a shared fabric where your pause frames affect other tenants.
+
+        Pause frames on the wire are also not evidence of a misconfiguration by themselves. A peer that cannot buffer a line-rate burst — an FPGA is the common case, and it will pause often — asserts pause by design, and that is the mechanism working. Disabling pause on such a link does not recover the throughput; it converts the throttling into drops on the peer. The counters say which end asserted it: `rx_pause_ctrl_phy` counts frames **received** (the peer throttling this port's transmit) and `tx_pause_ctrl_phy` frames **sent** (this port's receive path throttling the peer). Non-zero `rx_pause_ctrl_phy` against a device known to be shallow-buffered is expected; non-zero `tx_pause_ctrl_phy` points back at this host.


Consider adding something about using pacing rather than pause frames. Something like "If the receiver normally requires pause frames, packet pacing can instead be enabled to smooth out the packet rate to prevent buffer overruns".

cliffburdick · 2026-08-25T18:04:59Z

+        ??? abstract "See an example output"
+
+            ```
+            2026-08-19 09:43:00 - WARNING - Interface eth0 has 802.3x pause enabled (RX: on, TX: on). Link-level flow control can idle the link and cost over 20% of line rate with no drop counter to reveal it. Disable both directions with `ethtool -A eth0 rx off tx off` for raw-Ethernet benchmarking; keep it on lossless RoCE/PFC fabrics.


I would remove the "Over 20%" since it's ambiguous and very system-dependent. Instead just say "can idle the link and prevent achieving higher rates"

cliffburdick

Couple small comments. Good PR overall

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>

chloecrozier · 2026-08-25T20:47:41Z

@cliffburdick thanks for reviewing, I just pushed the changes

chloecrozier added 2 commits August 19, 2026 10:06

chloecrozier requested a review from RamyaGuru August 20, 2026 05:36

#250 - Document raw-engine pause reporting

675527f

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>

RamyaGuru requested a review from cliffburdick August 20, 2026 14:21

#250 - Report which end asserted pause instead of assuming misconfigu…

d907dfc

…ration Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>

cliffburdick reviewed Aug 25, 2026

View reviewed changes

cliffburdick requested changes Aug 25, 2026

View reviewed changes

chloecrozier added 3 commits August 25, 2026 12:52

#251 - update DGX Spark raw and RoCE numbers to cross-host measurements

996e15f

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>

#251 - publish five-model ResNet inference results for DGX Spark

21b28ac

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>

#250 - Suggest pacing over pause and drop the 20% line-rate claim

86973c9

Signed-off-by: Chloe Crozier <chloecrozier@gmail.com>

chloecrozier requested a review from cliffburdick August 25, 2026 20:47

Navigation Menu

Uh oh!

#250 - Surface 802.3x pause state and counters - #252

#250 - Surface 802.3x pause state and counters#252
chloecrozier wants to merge 7 commits into
mainfrom
250-surface-pause-counters

chloecrozier commented Aug 20, 2026

Uh oh!

greptile-apps Bot commented Aug 20, 2026 •

edited

Loading

Greptile Summary

Uh oh!

cliffburdick Aug 25, 2026

Uh oh!

cliffburdick Aug 25, 2026

Uh oh!

cliffburdick left a comment

Uh oh!

chloecrozier commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		On a lossless RoCE or PFC fabric, flow control is deliberate and required. Only disable pause on links dedicated to raw Ethernet where the application tolerates loss, and never on a shared fabric where your pause frames affect other tenants.

		Pause frames on the wire are also not evidence of a misconfiguration by themselves. A peer that cannot buffer a line-rate burst — an FPGA is the common case, and it will pause often — asserts pause by design, and that is the mechanism working. Disabling pause on such a link does not recover the throughput; it converts the throttling into drops on the peer. The counters say which end asserted it: `rx_pause_ctrl_phy` counts frames received (the peer throttling this port's transmit) and `tx_pause_ctrl_phy` frames sent (this port's receive path throttling the peer). Non-zero `rx_pause_ctrl_phy` against a device known to be shallow-buffered is expected; non-zero `tx_pause_ctrl_phy` points back at this host.

Uh oh!

Conversation

chloecrozier commented Aug 20, 2026

Test plan

Uh oh!

greptile-apps Bot commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

cliffburdick Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

cliffburdick Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

cliffburdick left a comment

Choose a reason for hiding this comment

Uh oh!

chloecrozier commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

greptile-apps Bot commented Aug 20, 2026 •

edited

Loading