[multicast] DDM multicast exchange: V4 protocol, MRIB sync by zeeshanlakhani · Pull Request #696 · oxidecomputer/maghemite

zeeshanlakhani · 2026-04-02T11:42:34Z

Adds multicast group subscription distribution to the DDM exchange protocol with a V4 version.

Key changes/additions:

V4 exchange protocol with multicast support, negotiated via an MCAST_CAPABLE discovery flag (RFC 5492 style). The wire version byte stays at the V2 floor, so V2/V3 peers are unaffected and mixed-version topologies interoperate cleanly.
Two validated newtypes in the recently merged client-common sub-crate (the cycle-free leaf crate, so the API-types crates Omicron consumes avoid an omicron_common cycle):
- OverlayMulticast (announced group is multicast)
- UnderlayMulticastIpv6 (admin-local mapping within ff04::/64)
  Both validate at construction and on deserialization, rejecting invalid origins at every ingress.
Membership half/lower: ddmd programs DPD replication-group members in-process (ddm::mcast) from imported routes plus the peer table via a triggered sweep with periodic reconciliation. This matches how ddmd installs imported unicast underlay routes (sys::add_underlay_routes).
Periodic exchange resync in the peer state machine to repair drift from missed pulls
Learned multicast state is exposed via DDM admin API for Omicron consumption
update_imported_mcast is atomic within the Db (single lock for import/delete/diff, which differs from the tunnel work)
Collapsed send_update dispatch, with exchange responses now checked for HTTP success (require_success, unicast included)
Shared pull handler functions (collect_underlay_tunnel, collect_multicast), and a shared tfport name parser in mg-common (port_link_from_ifname), fixing link derivations
Admin API binds synchronously and consistently on ::, with an --api-only flag (admin API without state machines, analogous to mgd --no-bgp-dispatcher) for test fixtures and the Linux build.
More serde round-trip and validation tests, a version negotiation contract test pinning mixed-version behavior, plus trio coverage of multicast advertise, router restart, and withdraw.
CI: softnpu/sidecar-lite/dendrite pins bumped to multicast-capable builds and the trio/quartet jobs have been moved to helios-3.0 to match dendrite's build target. The test harness now fails fast if dendrite or tfport lands in SMF maintenance instead of timing out on peering.

References

Stacked on zl/mrib (MRIB: Multicast RIB implementation, #675).

Omicron's oxidecomputer/omicron#10381 introduces a stubbed `ddmd` admin endpoint because spawning a real `ddmd` in a generic test toolchain is not viable: the routing state machine (discovery, exchange, route synchronization) depends on illumos networking facilities the toolchain does not provide. Consumers of the stub, e.g., Nexus RPW (multicast members), sled-agent's DDM reconciler, and anything that resolves the DDM internal-DNS service name, cannot exercise the real admin surface from Omicron's test harness. This work adds an opt-in `--no-state-machine` flag to `ddmd` that runs only the admin API server and skips the state machine entirely, allowing the fixture to spawn the real binary. This is analogous to `mgd --no-bgp-dispatcher`, which Omicron's `MgdInstance` already uses for the same purpose. To make the fixture path usable on Linux, `ddmd` itself must build on Linux. The previous code pulled the illumos-only crates `libnet`, `dpd-client`, `opte-ioctl`, and `oxide-vpc` unconditionally through `ddm`, which failed to link on Linux (`-lzfs`, `-ldlpi`). This change introduces an `illumos` feature in both `ddm` and `ddmd` (default-on, mirroring `mgd`'s `mg-lower` pattern) that marks those four crates optional. The buildomat `linux.sh` job now builds `ddmd` and `ddmadm`, with `ddmd` invoked as `cargo build --bin ddmd --no-default-features`. The illumos-only halves of `ddm` are isolated by the feature gate: - The routing state machine implementation moves from `sm.rs` into `sm/state.rs`. - The exchange runtime (HTTP push/pull and route programming) moves from `exchange.rs` into `exchange/runtime.rs`. - The discovery runtime (UDPv6 solicitation/advertisement loops) moves from `discovery.rs` into `discovery/runtime.rs`. Each parent `mod.rs` keeps the platform-agnostic types and re-exports the runtime surface so existing call sites resolve unchanged on illumos. The runtime submodules are gated as a unit by `#[cfg(all(feature = "illumos", target_os = "illumos"))]`. We also remove the single-function `ddm/src/util.rs`, inlining the function into `discovery/runtime.rs`, where its sole caller lives. The SIGTERM cleanup handler is installed regardless of the flag, so Ctrl-C still exits cleanly in `--no-state-machine` mode. The imported route sets are empty in that mode, so the cleanup itself is a noop. Passing `--addr` alongside `--no-state-machine` is harmless but ignored, with a warning logged.

Fills out the testing work for `ddmd` in Omicron. Includes: - Adds a `PUT /peer` admin endpoint, gated at v2.0.0 (MULTICAST_SUPPORT), that injects a `PeerInfo` directly into the in-memory peer table at a supplied interface index. Intended for fixtures running ddmd with `--no-state-machine`. Note that in typical operations the discovery handler will overwrite any directly-injected entry the next time a peer is observed on that interface. - Extracts the `get_peers` and `put_peer` impl bodies into `do_get_peers` / `do_put_peer` free functions in `ddm/src/admin.rs`, mirroring `mgd::bgp_admin::do_bgp_apply`, so the endpoints can be exercised in-process. - Adds `tests::put_peer_round_trips` over a tempdir-backed setup covering the round-trip and same interface-index overwrite invariant. - Adds a `put_peer` synthetic injection into the `run_trio_tests` smoke test that exercises the full HTTP path through the generated client against a running ddmd.

Brings in main, which carries #689 (PEER_DURATIONS at DDM API v2) from zl/mrib. MULTICAST_SUPPORT shifts from v2 to v3. PEER_DURATIONS landed first and owns v2 so the PeerInfo conversion chain is v3→v2→v1. Peer state now comes from #689's per-interface InterfaceState (FSM- driven) rather than the manually-populated `Db.peers` cache. GET /peers reads identity and `if_name` directly off the per-interface state. We drop the PUT /peer endpoint as omicron #10346 drives its DDM tests through an in-process dropshot sim (DdmInstance with a set_peers back-door), not a real ddmd, so the endpoint had no consumer.

The tfport datalink naming logic, parsing names like tfportrear0_0 and tfportqsfp0_0 into a kind, port, and link, was duplicated in ddm's sys module and mg-lower's dendrite module. Hoist it into a single mg_common::tfport module so both consumers share one parser and one set of error types. No behavior change.

…rite mod The call sites that resolve a tfport interface name to a DPD (PortId, LinkId) tuple repeated the same inline parse-and-construct sequence. This work factors it into a single `port_link_from_ifname` helper function so that the resolution lives in one place. Note: no behavior change.

…ted routes This work splits DDM underlay multicast (RFD 488) into two directional halves. Multicast origination advertises locally originated MRIB groups outward (MRIB to the DDM admin API to the underlay). Membership, in turn, consumes the routes other sleds originate and programs the local switch's replication members in DPD directly. This commit implements the membership half and refactors origination to match the unicast lower-half's shape and approach. Membership updates run in `ddmd` because both of its inputs (the set of DDM-imported multicast routes and the peer table) are already owned in-process by the DDM state machine. A dedicated sweep thread (`ddm::mcast::run`) reconciles each affected underlay group's DPD members. This mechanism is trigger-driven over an mpsc channel from the import, withdraw, peer-expiry, and peer-link-resolution paths, with a 10s backstop and a startup seed from DPD's member-bearing groups so orphans left by a withdraw during downtime are properly drained. Each DPD member GET/PUT is bounded at 3s (DPD_REQUEST_TIMEOUT, kept so a group's fetch-write pair stays under the chosen backstop interval) and stalls are classified as typed `TimedOut` outcomes. Therefore, an unresponsive DPD retains the group for the next pass rather than dropping it and leaking any replication state. mg-lower's mrib module is now origination-only, watch-driven with a 1s periodic resync backstop, mirroring the unicast lower-half loop. The DPD and DDM clients take an optional address so integration harnesses can target a listener on a dynamically assigned port, exposed as the --dendrite-addr and --ddm-addr mgd flags respectively. Supporting changes attached to this work: - db: factor the imported-route tree scan into scan_origin_tree and return removed unicast and multicast next hops together via `RemovedNexthopRoutes`. - ddmd: wire the sweep's notify channel and start it unconditionally, including --api-only, where it owns the receiver with a known empty peer set. - admin: collect get_peers via filter_map rather than a manual loop.

Merge main and land DDM underlay multicast support, where DPD is updated directly from maghemite. Here, discovery gains a capability-negotiated V4, as nodes still advertise the V2 floor in the version byte, and (now) signal support with a new `MCAST_CAPABLE` flag, so deployed V2/V3 peers ignore the unknown bit (RFC 5492) while two capable nodes negotiate up to V4. The multicast wire types land in a frozen `ddm-protocol::v4` module post-merge, with validated counterparts in API version 3 (MULTICAST_SUPPORT). The correct-by-construction newtypes are consolidated into the cycle-free `client-common` leaf, including `Vni`, `UnderlayMulticastIpv6`, and `OverlayMulticast`.

Reword the peer-context and --api-only doc comments across the admin handler context, the multicast sweep, the discovery module, and ddmd startup to state that the peer set is empty when no state machines run. Start the admin Dropshot server synchronously so the API is bound before handler() returns, and log the bound address, which reflects the assigned port when 0 is requested.

…lict mapping, port-file admin binding

…e resync The membership sweep moves from a dedicated thread with a std mpsc channel to a task on the daemon's runtime with a bounded tokio channel. Triggers are wake hints only, so a try_send dropped on a full channel costs at most one reconcile interval, and the sweep selects between a trigger and a periodic tick that defers rather than bursts after a long pass. Exchange state gains a periodic resync pull. The initial pull is one-shot, so routes a neighbor originates after we pull it, late multicast group memberships in particular, would otherwise never be imported absent a push from that neighbor. Periodic pulls import without redistributing, and each router runs its own resync, so a transit re-flooding its peers on every periodic pull would churn in steady state for no benefit. The smf manifest default for admin_host moves from ::1 to :: so the admin API is reachable over the underlay rather than only from the local host, matching mgd's existing default.

zeeshanlakhani force-pushed the zl/ddm-mcast branch 3 times, most recently from 339f250 to 1b5996d Compare April 2, 2026 11:49

zeeshanlakhani changed the title ~~ddm-mcast~~ [multicast] DDM multicast exchange: V4 protocol, MRIB sync, M2P hooks Apr 2, 2026

zeeshanlakhani force-pushed the zl/ddm-mcast branch from 1b5996d to 0671b1f Compare April 2, 2026 15:38

zeeshanlakhani requested review from rcgoodfellow and taspelund April 2, 2026 15:43

zeeshanlakhani marked this pull request as ready for review April 2, 2026 15:43