#283 - Gate ibverbs real-time RX timestamps

[dleshchev]

Fixes #283.

Summary

• Request mlx5 real-time CQ timestamps only when mlx5dv_query_device() advertises MLX5DV_CONTEXT_FLAGS_REAL_TIME_TS.
• Retain the default device-clock CQ format on unsupported devices and convert those timestamps through the existing mlx5dv_ts_to_ns() path.
• Preserve the valid ibverbs improvements #281 RX WQ UMEM access flags, adapter-relative page-size encoding, and reduced 64-stride region geometry.

Failure and expected behavior

After #281, raw ibverbs RX fails inside daqiri_init() before any traffic measurement:

CREATE_RQ (DevX) failed: Remote I/O error (syndrome 0x43d98f)

The identical configuration initializes with a pre-#281 image. Raw RX should also initialize on devices whose firmware does not support real-time CQ timestamps; those devices should use the free-running device clock and DAQIRI's existing clock conversion.

Root cause and isolation

#281 unconditionally set MLX5_RQC_TIMESTAMP_FORMAT_REAL_TIME in the DevX RQ context. The affected device does not advertise the corresponding mlx5dv capability, and firmware rejects the unsupported RQ context during CREATE_RQ.

The RX-side changes were restored independently in diagnostic container builds:

Diagnostic change	Result
Unmodified main	CREATE_RQ fails, syndrome 0x43d98f
Restore pre-#281 512-stride region geometry	Same failure
Restore pre-#281 RX WQ UMEM access flags	Same failure
Restore both geometry and UMEM flags	Same failure
Omit only the unsupported real-time timestamp request	Initialization succeeds
Capability-gated timestamp request with #281 UMEM flags and 64-stride geometry	Initialization succeeds

Increasing the RX buffer count to restore the old effective queue depth likewise does not change the failure.

The old setup succeeds because it leaves the RQ in the default free-running timestamp format. The corrected setup makes the same choice only when required by device capability; capable devices retain the real-time format and its direct UTC decode.

log_wq_pg_sz remains encoded relative to the mlx5 adapter's fixed 4-KiB page, rather than the operating-system page used to align and register the UMEM. This is why the adapter-relative value remains valid on ARM64 kernels using larger pages. The currently reachable validation systems report 4-KiB kernel pages, so a live 64-KiB-kernel confirmation remains explicitly outstanding while this PR is a draft.

Validation

Build

The final commit was built and installed successfully with the project container:

IMAGE_TAG=daqiri:ibverbs-rx-init-fix-final \
BASE_TARGET=dpdk \
DAQIRI_ENGINE="dpdk ibverbs" \
scripts/build-container.sh

git-clang-format --diff origin/main reports no changes, and git diff --check passes.

Initialization correctness proof

An RX-only raw ibverbs run on the affected hardware created the CQ and 64-stride DevX RQ, installed the flow, started the worker, and exited cleanly after five seconds:

Striding RQ: stride=8192B strides/WQE=64 region=524288B num_wqe=512
real-time CQ timestamps are unsupported; using the device clock format
DevX striding RQ ready
ibverbs backend initialized with 1 RX queue(s), 0 TX queue(s)
RX complete: packets=0 bytes=0 bursts=0

The zero-packet RX-only run proves initialization only; it is not a throughput result.

Physical cross-host completion and throughput proof

A subsequent physical cross-host raw ibverbs TX/RX run used the final fixed image on RX and an unchanged existing ibverbs image on TX. No benchmark configuration or harness semantics were changed.

• Active TX interval: 10.0131 seconds
• Application TX: 14,860,288 packets, 119,833,362,432 bytes
• Application RX: 14,860,288 packets, 119,833,362,432 bytes
• Application payload throughput over the active interval: 95.741 Gb/s TX and RX
• PHY TX delta: 14,860,292 packets, 119,892,804,548 bytes
• PHY RX delta: 14,860,292 packets, 119,892,804,548 bytes
• The four extra PHY packets and 964 bytes are background control traffic; the data-packet PHY byte count is exactly the application packet count multiplied by the 8,068-byte wire frame.
• Stable one-second physical samples were approximately 97.0-97.4 Gb/s TX and 96.6-97.3 Gb/s RX.
• RX out_of_buffer remained flat; CQ errors and application-ring-full drops were zero.

This physical run proves that the corrected RX path receives and completes cross-host traffic at line-rate-class load. It is separate from the initialization proof and does not update or replace the published performance tables.

Environment

• ARM64 NVIDIA DGX Spark
• Ubuntu 24.04.4 LTS
• Kernel 6.17.0-1014-nvidia
• NVIDIA ConnectX-7 MT2910 family (15b3:1021)
• Firmware 28.45.4028
• rdma-core/libibverbs/providers 2407mlnx52-1.2407061

Regression coverage

The failure occurs in device firmware during DevX object creation and there is no hardware-backed unit-test framework in the repository. The initialization and physical cross-host procedures above provide the regression check for this draft. A live run on a 64-KiB ARM64 kernel remains the only outstanding validation item.

Scope

This PR changes only the ibverbs raw-RX timestamp-format selection and matching decode path. It does not alter TCP/socket behavior, benchmark documentation, published performance tables, or cross-host harness semantics.

[greptile-apps]

Greptile Summary

The PR capability-gates mlx5 real-time receive timestamps and retains device-clock conversion on unsupported hardware.

• Queries the mlx5 real-time timestamp capability while creating each RX queue.
• Selects timestamp decoding according to the configured queue format.
• Updates the public timestamp documentation to describe the fallback conversion.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
src/engines/ibverbs/daqiri_ibverbs_engine.cpp	Gates the real-time RQ timestamp format on the advertised mlx5 capability and converts default-format timestamps through the existing device-clock path.
src/engines/ibverbs/daqiri_ibverbs_engine.h	Records the selected timestamp format in per-queue state for matching decode behavior.
include/daqiri/common.h	Clarifies that the public timestamp accessor returns epoch nanoseconds after any required engine-side conversion.
docs/api-reference/configuration.md	Documents the capability-gated raw-ibverbs timestamp behavior.
docs/api-reference/cpp.md	Documents device-clock conversion for C++ API consumers.
docs/api-reference/python.md	Documents device-clock conversion for Python API consumers.
docs/concepts.md	Explains the two mlx5 RX timestamp representations and their application-transparent handling.

_{Reviews (3): Last reviewed commit: "#283 - Raise ibverbs timestamp fallback ..." | Re-trigger Greptile}

[cliffburdick]

I would make this critical since we don't want to support it. We should find out why this is failing on Spark, though.

[dleshchev]

Done in 959ec9e (#283 - Raise ibverbs timestamp fallback log level). I changed the unsupported real-time CQ timestamp fallback from DAQIRI_LOG_WARN to DAQIRI_LOG_CRITICAL and re-ran the IGX hardware-loopback validation; it now emits the critical log, creates the DevX striding RQ, transfers traffic, and exits with cqe_errors=0 / app_ring_full_drops=0.