#257 - Add external memory region bindings

docs/api-reference/configuration.md

@@ -60,6 +60,12 @@ These settings apply globally to both TX and RX:
 `memory_regions:` List of regions where packet buffers are stored. The number of regions
 and their `kind` determines the receive mode (CPU-only, header-data split, or batched GPU).
 
+YAML describes region semantics but never contains process-local pointers. C++ and Python callers
+can attach application-owned allocations by name at initialization; see
+[Application-owned memory regions](cpp.md#application-owned-memory-regions). A runtime binding
+replaces DAQIRI's allocation for that region. The legacy `owned: false` form requires a matching
+runtime binding.
+
 - **`name`**: Memory region name. Referenced by queue configurations.
   - type: `string`
 - **`kind`**: Memory type.

docs/api-reference/cpp.md

@@ -56,6 +56,44 @@ Only one engine may be active in a process. Calling `daqiri_init()` again before
 After `shutdown()` completes, a later `daqiri_init()` creates a fresh engine instance and
 allocates/registers its resources again.
 
+### Application-owned memory regions
+
+Applications that need allocations in a specific CUDA context (including MPS applications) can
+bind their own storage to configured memory-region names. Query the engine-adjusted requirements
+first; the returned capacity includes packet headroom, slot alignment, and any DPDK buffer-count
+safety adjustment.
+
+```cpp
+daqiri::NetworkConfig config;
+if (daqiri::parse_network_config("config.yaml", config) != daqiri::Status::SUCCESS) {
+  throw std::runtime_error("invalid DAQIRI config");
+}
+
+daqiri::MemoryRegionRequirements requirements;
+daqiri::get_memory_region_requirements(config, requirements);
+const auto& rx = requirements.at("RX_GPU");
+
+void* rx_gpu = nullptr;
+cudaMalloc(&rx_gpu, rx.capacity);  // Uses the application's current context.
+daqiri::MemoryRegionBindings bindings{{"RX_GPU", {rx_gpu, rx.capacity}}};
+
+if (daqiri::daqiri_init(config, bindings) != daqiri::Status::SUCCESS) {
+  cudaFree(rx_gpu);
+  throw std::runtime_error("DAQIRI initialization failed");
+}
+
+// Return every outstanding burst before shutdown.
+daqiri::shutdown();
+cudaFree(rx_gpu);
+```
+
+Bindings may cover only some regions; DAQIRI allocates the rest. Bound memory is borrowed, so its
+allocation and CUDA context must remain alive until `shutdown()` completes. DAQIRI deregisters it
+from the NIC but never frees, unpins, or unmaps it. Direct TCP/UDP sockets reject bindings because
+their configured regions are not packet pools. DPDK also rejects externally bound `huge` regions:
+EAL assumes ownership of pre-existing hugepage mappings during initialization and unmaps them at
+cleanup. External `huge` regions remain supported by ibverbs/RDMA.
+
 If GPU RX `reorder_configs` are configured for Raw Ethernet (`stream_type: "raw"`), set
 one CUDA stream per GPU reorder plan before pulling reordered bursts. CPU reorder configs do not use a
 CUDA stream. See the [Configuration YAML Reference](configuration.md#rx-reorder-configs)
@@ -647,10 +685,12 @@ workflow sections above show the common call order and ownership rules.
 | `version_year()` / `version_month()` / `version_patch()` | Return the CalVer components. |
 | `abi_version()` | Return the DAQIRI shared-library ABI version. |
 | `daqiri_init(NetworkConfig &config)` | Initialize DAQIRI from an already-populated config object. |
+| `daqiri_init(config, bindings)` | Initialize with non-owning external memory bindings. |
 | `daqiri_init(const std::string &yaml_string_or_path)` | Initialize from a YAML string or YAML file path. |
 | `daqiri_init_from_yaml_string(const std::string &yaml_string)` | Initialize from YAML content. |
 | `daqiri_init_from_yaml_file(const std::string &yaml_path)` | Initialize from a YAML file path. |
 | `parse_network_config(...)` | Parse YAML into `NetworkConfig` without starting the engine. |
+| `get_memory_region_requirements(config, requirements)` | Return effective slot size, count, capacity, and alignment. |
 | `get_engine_type()` | Return the active engine type after initialization. |
 | `get_engine_type(config)` | Return the engine type selected by a config object. |
 | `shutdown()` | Stop DAQIRI and release engine-owned resources. |

docs/api-reference/python.md

@@ -144,6 +144,30 @@ config = {
 status = daqiri.daqiri_init(config)
 ```
 
+Application-owned CUDA buffers use the same requirements-and-bindings workflow as C++. Addresses
+are integers, and the Python allocation owner must stay alive until after `shutdown()`:
+
+```python
+status, parsed = daqiri.parse_network_config("config.yaml")
+status, requirements = daqiri.get_memory_region_requirements(parsed)
+rx_req = requirements["RX_GPU"]
+
+# For example, gpu_owner may be a CuPy allocation made in the desired context.
+bindings = {
+    "RX_GPU": daqiri.ExternalMemoryRegion(int(gpu_owner.data.ptr), rx_req.capacity),
+}
+status = daqiri.daqiri_init(parsed, bindings)
+
+# Return outstanding bursts first.
+daqiri.shutdown()
+del gpu_owner
+```
+
+Bindings may cover only a subset of regions. They are supported by DPDK, raw ibverbs, and
+RoCE/RDMA, but not direct TCP/UDP sockets. DPDK rejects external `huge` regions because EAL cannot
+preserve ownership of hugepage mappings that predate initialization; use `host`, `host_pinned`, or
+ibverbs/RDMA for caller-owned CPU hugepages.
+
 Parse without starting the engine:
 
 ```python
@@ -513,6 +537,8 @@ The workflow sections above show the common call order and ownership rules.
 | Function | Returns |
 | --- | --- |
 | `daqiri_init(config)` | `Status` |
+| `daqiri_init(config, bindings)` | `Status` |
+| `get_memory_region_requirements(config)` | `(Status, dict[str, MemoryRegionRequirement])` |
 | `daqiri_init_from_yaml_string(yaml_string)` | `Status` |
 | `daqiri_init_from_yaml_file(yaml_path)` | `Status` |
 | `parse_network_config(yaml_string_or_path)` | `(Status, NetworkConfig)` |
@@ -709,6 +735,8 @@ names that mostly omit the trailing underscore from the C++ member name (e.g.
 | `RxQueueConfig` | RX queue wrapper with common queue fields, timeout, and `QueuePollMode`. |
 | `TxQueueConfig` | TX queue wrapper with common queue fields and `QueuePollMode`. |
 | `MemoryRegionConfig` | Memory region kind, affinity, access flags, sizes, counts, and ownership. |
+| `ExternalMemoryRegion` | Non-owning integer address and byte capacity. |
+| `MemoryRegionRequirement` | Effective kind, slot size, buffer count, capacity, and alignment. |
 | `VlanActionConfig` | VLAN push parameters: VLAN ID, priority, DEI, and ethertype. |
 | `TunnelConfig` | VXLAN, GRE, or NVGRE tunnel template fields for hardware encap/decap actions. |
 | `FlowAction` | Flow action type, scalar queue target `id`, queue-list target `ids`, optional VLAN config, and optional tunnel config. |

docs/concepts.md

@@ -402,6 +402,12 @@ buffer pools and can lead to `NO_FREE_BURST_BUFFERS`,
 When to call each `free_*` function is documented in the
 [C++ API Usage page](api-reference/cpp.md#rx-step-3-free-buffers).
 
+The backing allocation may be DAQIRI-owned or application-owned. For application-owned regions,
+the caller binds an address and capacity to a configured region name during initialization.
+DAQIRI owns the temporary NIC registration and packet-pool bookkeeping; the caller retains the
+allocation and its CUDA context until shutdown finishes. DAQIRI never frees externally bound
+memory.
+
 ## RX Packet Aggregation and Reorder
 
 DAQIRI can perform GPU- or CPU-side packet aggregation and reordering

include/daqiri/common.h

@@ -107,9 +107,19 @@ inline int EnabledDirections(const std::string &dir) {
  *    INTERNAL_ERROR: Internal error
  */
 Status daqiri_init(NetworkConfig &config);
+Status daqiri_init(NetworkConfig &config, const MemoryRegionBindings &bindings);
 Status daqiri_init(const std::string &yaml_string_or_path);
+Status daqiri_init(const std::string &yaml_string_or_path,
+                   const MemoryRegionBindings &bindings);
 Status daqiri_init_from_yaml_string(const std::string &yaml_string);
+Status daqiri_init_from_yaml_string(const std::string &yaml_string,
+                                    const MemoryRegionBindings &bindings);
 Status daqiri_init_from_yaml_file(const std::string &yaml_path);
+Status daqiri_init_from_yaml_file(const std::string &yaml_path,
+                                  const MemoryRegionBindings &bindings);
+
+Status get_memory_region_requirements(const NetworkConfig &config,
+                                      MemoryRegionRequirements &requirements);
 
 Status parse_network_config(const std::string &yaml_string_or_path,
                             NetworkConfig &config);

include/daqiri/types.h

@@ -166,6 +166,23 @@ struct UDPIPV4Pkt {
 
 enum class MemoryKind { HOST, HOST_PINNED, HUGE, DEVICE, INVALID };
 
+struct ExternalMemoryRegion {
+  void* data = nullptr;
+  size_t capacity = 0;
+};
+
+using MemoryRegionBindings = std::unordered_map<std::string, ExternalMemoryRegion>;
+
+struct MemoryRegionRequirement {
+  MemoryKind kind = MemoryKind::INVALID;
+  size_t slot_size = 0;
+  size_t num_bufs = 0;
+  size_t capacity = 0;
+  size_t alignment = 0;
+};
+
+using MemoryRegionRequirements = std::unordered_map<std::string, MemoryRegionRequirement>;
+
 enum MemoryAccess {
   MEM_ACCESS_LOCAL = 1U,
   MEM_ACCESS_RDMA_WRITE = 1U << 1,

python/daqiri_common_pybind.cpp

@@ -230,13 +230,13 @@ py::tuple segment_packet_bytes_impl(BurstParams *burst, int seg, int idx,
   return py::make_tuple(status, py::bytes(out));
 }
 
-Status daqiri_init_from_python(py::object config_obj) {
+Status daqiri_init_from_python(py::object config_obj, const MemoryRegionBindings& bindings) {
   try {
     if (py::isinstance<py::str>(config_obj) ||
         py::isinstance<py::bytes>(config_obj)) {
       const auto yaml_string_or_path = py::cast<std::string>(config_obj);
       py::gil_scoped_release release;
-      return daqiri_init(yaml_string_or_path);
+      return daqiri_init(yaml_string_or_path, bindings);
     }
 
     if (py::isinstance<py::dict>(config_obj)) {
@@ -245,21 +245,21 @@ Status daqiri_init_from_python(py::object config_obj) {
           config_obj, py::arg("default_flow_style") = false);
       const auto yaml_str = py::cast<std::string>(py_yaml_str);
       py::gil_scoped_release release;
-      return daqiri_init_from_yaml_string(yaml_str);
+      return daqiri_init_from_yaml_string(yaml_str, bindings);
     }
 
     if (py::isinstance<NetworkConfig>(config_obj)) {
       auto &config = config_obj.cast<NetworkConfig &>();
       py::gil_scoped_release release;
-      return daqiri_init(config);
+      return daqiri_init(config, bindings);
     }
 
     if (py::hasattr(config_obj, "value")) {
       const auto yaml_string_or_path_obj = py::str(config_obj.attr("value"));
       const auto yaml_string_or_path =
           py::cast<std::string>(yaml_string_or_path_obj);
       py::gil_scoped_release release;
-      return daqiri_init(yaml_string_or_path);
+      return daqiri_init(yaml_string_or_path, bindings);
     }
 
     if (py::hasattr(config_obj, "as_dict")) {
@@ -268,14 +268,14 @@ Status daqiri_init_from_python(py::object config_obj) {
           config_obj.attr("as_dict")(), py::arg("default_flow_style") = false);
       const auto yaml_str = py::cast<std::string>(py_yaml_str);
       py::gil_scoped_release release;
-      return daqiri_init_from_yaml_string(yaml_str);
+      return daqiri_init_from_yaml_string(yaml_str, bindings);
     }
 
     const auto yaml_string_or_path_obj = py::str(config_obj);
     const auto yaml_string_or_path =
         py::cast<std::string>(yaml_string_or_path_obj);
     py::gil_scoped_release release;
-    return daqiri_init(yaml_string_or_path);
+    return daqiri_init(yaml_string_or_path, bindings);
   } catch (const py::error_already_set &e) {
     DAQIRI_LOG_ERROR("Python config conversion failed: {}", e.what());
     return Status::INTERNAL_ERROR;
@@ -743,6 +743,29 @@ void bind_config_types(py::module_ &m) {
       .def_readwrite("num_bufs", &MemoryRegionConfig::num_bufs_)
       .def_readwrite("owned", &MemoryRegionConfig::owned_);
 
+  py::class_<ExternalMemoryRegion>(m, "ExternalMemoryRegion")
+      .def(py::init([](uintptr_t address, size_t capacity) {
+             return ExternalMemoryRegion{reinterpret_cast<void*>(address), capacity};
+           }),
+           "address"_a, "capacity"_a)
+      .def_property(
+          "address",
+          [](const ExternalMemoryRegion& region) {
+            return reinterpret_cast<uintptr_t>(region.data);
+          },
+          [](ExternalMemoryRegion& region, uintptr_t address) {
+            region.data = reinterpret_cast<void*>(address);
+          })
+      .def_readwrite("capacity", &ExternalMemoryRegion::capacity);
+
+  py::class_<MemoryRegionRequirement>(m, "MemoryRegionRequirement")
+      .def(py::init<>())
+      .def_readonly("kind", &MemoryRegionRequirement::kind)
+      .def_readonly("slot_size", &MemoryRegionRequirement::slot_size)
+      .def_readonly("num_bufs", &MemoryRegionRequirement::num_bufs)
+      .def_readonly("capacity", &MemoryRegionRequirement::capacity)
+      .def_readonly("alignment", &MemoryRegionRequirement::alignment);
+
   py::class_<RxQueueConfig>(m, "RxQueueConfig")
       .def(py::init<>())
       .def_readwrite("common", &RxQueueConfig::common_)
@@ -985,13 +1008,31 @@ PYBIND11_MODULE(_daqiri, m) {
   bind_enums(m);
   bind_config_types(m);
 
-  m.def("daqiri_init", &daqiri_init_from_python, "config"_a,
+  m.def("daqiri_init", &daqiri_init_from_python, "config"_a, "bindings"_a = MemoryRegionBindings{},
         "Initialize DAQIRI from a YAML path, YAML string, dict, or config-like "
         "object");
-  m.def("daqiri_init_from_yaml_string", &daqiri_init_from_yaml_string,
-        "yaml_string"_a, py::call_guard<py::gil_scoped_release>());
-  m.def("daqiri_init_from_yaml_file", &daqiri_init_from_yaml_file,
-        "yaml_path"_a, py::call_guard<py::gil_scoped_release>());
+  m.def(
+      "daqiri_init_from_yaml_string",
+      [](const std::string &yaml, const MemoryRegionBindings &bindings) {
+        return daqiri_init_from_yaml_string(yaml, bindings);
+      },
+      "yaml_string"_a, "bindings"_a = MemoryRegionBindings{},
+      py::call_guard<py::gil_scoped_release>());
+  m.def(
+      "daqiri_init_from_yaml_file",
+      [](const std::string& path, const MemoryRegionBindings& bindings) {
+        return daqiri_init_from_yaml_file(path, bindings);
+      },
+      "yaml_path"_a, "bindings"_a = MemoryRegionBindings{},
+      py::call_guard<py::gil_scoped_release>());
+  m.def(
+      "get_memory_region_requirements",
+      [](const NetworkConfig &config) {
+        MemoryRegionRequirements requirements;
+        const Status status = get_memory_region_requirements(config, requirements);
+        return py::make_tuple(status, requirements);
+      },
+      "config"_a);
   m.def(
       "parse_network_config",
       [](const std::string &yaml_string_or_path) {