#257 - Add external memory region bindings

[cliffburdick]

No description provided.

[greptile-apps]

Greptile Summary

This update adds application-owned memory-region bindings across the C++ and Python APIs and propagates allocation context metadata through the DPDK and ibverbs engines.

• Adds external-region types, initialization overloads, and effective memory-requirement queries.
• Validates bound addresses, capacities, memory kinds, alignment, GPU affinity, and CUDA contexts.
• Updates GPU reorder initialization, execution, stream validation, event handling, and cleanup to run in the selected allocation context.
• Documents the requirements-and-bindings workflow and ownership contract.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains.

No blocking failure remains.

Important Files Changed

Filename	Overview
src/engine.cpp	Validates external bindings and integrates borrowed allocations into common memory-region setup while retaining caller ownership.
src/engines/dpdk/daqiri_dpdk_engine.cpp	Validates both reorder regions against one CUDA device and context and activates that context around CUDA operations.
src/engines/ibverbs/daqiri_ibverbs_engine.cpp	Applies matching CUDA device/context checks and context guards to ibverbs reorder setup, processing, and cleanup.
src/common.cpp	Adds binding-aware initialization overloads and computes engine-adjusted memory-region requirements.
python/daqiri_common_pybind.cpp	Exposes external-memory bindings and requirement queries to Python while preserving GIL release during initialization.

_{Reviews (6): Last reviewed commit: "#257 - Add external memory region bindin..." | Re-trigger Greptile}

[dleshchev]

Validation gap from the benchmark pass: the existing loopback runs still only exercise DAQIRI-owned memory. They build and run cleanly, but they do not call get_memory_region_requirements(...), pass MemoryRegionBindings into daqiri_init(...), use owned=false regions, or cover the caller-owned CUDA context / capacity / alignment paths added here.

Since this PR adds a new public C++/Python API plus engine registration behavior for external regions, I think it needs a minimal direct smoke path before merge. A small C++ example or benchmark flag would be enough: parse the software-loopback config, query requirements, cudaMalloc caller-owned TX/RX buffers, initialize with bindings, run briefly, shutdown, then free the buffers. Without that, this new API could regress while the current benchmark suite still passes.

[dleshchev]

see comment above, otherwise good to merge

[tmartin-gh]

General comment: Tested this PR out locally on my setup with an MPS application, it was functional and performant. No other concerns at this time.

[tmartin-gh]

We have successfully used a combination of rte_extmem_register(), rte_dev_dma_map(), and mlock() to use externally allocated memory with DPDK.