clusterer_controller: zero-config HA clusterer control module with automatic sharing tag management via encrypted UDP multicast - #4074
Conversation
|
Thank you very much for the contribution, I really like the idea behind it. I do see though that it is marked as a Draft - is it still work in progress, or has it reached to its final state? Let us know when it is ready to review. |
|
Hi, |
2cf5b36 to
34f1042
Compare
|
Now it's ready for review, thanks! |
f93b82d to
4348580
Compare
|
Follow-up commit Since this PR is already open for review, I added this as a distinct follow-up commit rather than squashing it into the main one, so the incremental change is easy to review and the existing review isn't disrupted by a force-push of the main commit. It enforces consistency between
Hybrid environments are unaffected. |
82e123e to
b7a173e
Compare
|
Config API for controller-managed clusters is now the per-cluster Same The controller-managed ids and the Verified locally (the build links wolfSSL, so the controller runs without a node): the new syntax loads, a managed id with no controller config aborts, a controller config for an unmanaged id aborts, and matching config starts. Docs (admin guide, tests appendix, README) and the PR description are updated to this form. |
|
Follow-up Previously the clusterer-side integration (the
So clusterer_controller can be completely omitted — a build without it produces the stock upstream clusterer module: no Verified with |
Crypto is libsodium-only (see ce8e805)Worth stressing for reviewers: as of ce8e805 the module's cryptography settled on libsodium, and the earlier wolfSSL / OpenSSL-based paths were dropped entirely — there is no TLS-library fallback anymore. clusterer_controller now uses XChaCha20-Poly1305 + Argon2id for the shared-secret / at-rest key material and a Noise_NNpsk0 (Curve25519 / ChaCha20-Poly1305 / SHA-256) handshake for the join, all on libsodium primitives. Why the switch:
libsodium is therefore a hard build dependency of the module now; there is no |
Why this PR also touches
|
aad3a24 to
5dc976d
Compare
Follow-up already implemented: a consumer messaging API over the controller's encrypted plane(Updated — this comment originally claimed a script could migrate from clusterer's generic messaging "by renaming the call". That is true of the function names and false of the delivery guarantees; the correction and what was done about it are at the end.) A heads-up on where this module goes next — the work is implemented and tested on a development branch, and will be proposed once this PR lands, since it extends the packet layer introduced here. The controller's encrypted UDP plane (XChaCha20-Poly1305 under the rotating session key, Noise_NNpsk0 join) is currently private to the module. The follow-up opens it to consumers as a small API, at two levels. Module tier, bound via
Payloads up to 1300 bytes, channel names up to 31 chars; sends are IPC-marshalled to the controller worker so any process can send. Consumer packets carry a cleartext magic so the pre-decrypt rate limiter classifies them against their own budget instead of the deliberately tight join budget — without that separation, the join limiter silently dropped one consumer packet in ten under load. Script tier — the same three functions and two events clusterer offers, plus a list send: Addressing a subset is not the same as filtering one
A broadcast is the opposite case and gets the opposite treatment. There are no unaddressed nodes, so it stays one multicast; sending it as N unicasts would cost roughly twice the packets (2(N−1) against one multicast plus N−1 acknowledgements) for no benefit, which matters at the 256 nodes this module is designed for. Delivery guarantees, opt-inDefault is best-effort, unordered, at-most-once. A reliable broadcast stays a multicast and repairs by unicast to the nodes that did not answer. Repair rather than rebroadcast is the important part: a duplicate that asked to be acknowledged is acknowledged again, so resending the multicast to fix two missing nodes would have every other member answer a second time. The subtle case is a lost acknowledgement, not a lost message. The sender resends identical bytes with the same sequence number and the receiver's replay check correctly drops the duplicate — so without more, a receiver whose ACK was lost swallows every retransmission in silence while the sender spends its budget and concludes failure. Hence the re-ACK: the replay check does the deduplication, the re-ACK closes the loop, and delivery stays at most once. The correctionThe original text said a script moves between clusterer's messaging and this one "by renaming the call". The surface is deliberately identical, but the guarantees are not: Two related fixes came out of the same review. Consumer traffic now has its own sequence space: it shared one monotonic counter per sender with the control plane, so on any multipath network a reordered consumer packet could make a One implementation note for reviewers: raising a scriptable event from the controller worker requires the process to be declared with |
The complete script, MI and event surface — including what the follow-up adds(Updated: adds the list send, the opt-in reliable flag, the per-cluster consumer settings, and a note on how counts are returned.) Everything this module exposes today, plus what the consumer messaging follow-up adds, so the script- and operator-facing surface can be reviewed as a whole. Script functionsCurrent:
Added by the follow-up — deliberately clusterer's generic-message surface, so the shapes are familiar (the delivery guarantees differ; see the previous comment):
On return values. The count comes back in an output variable and the function itself is simply true or false — which is why the call goes inside the What the count is, and is not. It is how many nodes the message was sent to, which is known immediately. How many acknowledged it cannot be known at that point — the answers arrive afterwards — so for a reliable send that result currently goes to the log as it completes, reported against the configured budget: Surfacing that back to the script as a completion event is the remaining piece of this work. (An earlier revision of this comment documented padding past unused arguments — Events
Clusterer's exact parameter set, so existing event routes port unchanged. The module reserves the Pseudo-variables
MI commands
Module parametersExisting: Added for the consumer plane, each a global default that a cluster may override in its own
The per-cluster form follows what Module API (for other modules)Bound with A hardening change worth flagging to reviewersGiving consumer traffic its own rate budget raised what a single source address can push through to the cipher from 20 packets a second to 1000 — the right budget for a real peer, a poor one for a stranger, since the packets fail to decrypt but the node still performs the failed decryption, and the sender needs to know nothing but the port and two cleartext magic bytes. Consumer packets are therefore now required to come from an address the cluster already knows. That is safe to require because consumer traffic only ever passes between joined members — control traffic, where a stranger legitimately appears (a It does not fix address spoofing and does not claim to — a forged source copying a member's address still reaches the limiter. What it removes is the far easier attack of pointing a flood at the port from anywhere. Verified on a three-node cluster: 399,000 forged consumer packets in three seconds from a non-member address, after which membership was intact, cross-node traffic still worked, and the rate limiter had not logged once — the packets never reached it. |
|
Pushed a small follow-up fix:
|
Hi @razvancrainea Razvan, Best regards, |
|
Found and fixed a real bug while testing this module's actual multi-node discovery for the first time in production (every deployment so far had been single-node, where peer discovery never needed to cross the wire).
On two production hosts with the Checked three other multi-node clusters already running elsewhere - same Fix: bind the join to Pushed to this branch: 0b3af68 |
Restructured into 3 logical commits (28 → 3) — no code changeThe old history was 28 commits of my own iteration, which made this look like a
Ordering is dependency-first: the clusterer API lands before the module that binds The tree is byte-for-byte identical to the previous head Commits referenced in the comments aboveOlder comments in this thread cite specific hashes. They now map as:
Every other commit in the old history touched only Happy to split it further, or to break out the |
dd0f515 to
22cb916
Compare
tm_init_cluster() read this node cluster id once, at mod_init, and baked it into the ";cid=<id>" Via parameter (and into a cached tm_node_id used to decide whether an anycast reply is ours). That assumes the id is known and stable by mod_init, which holds for a statically configured clusterer but not for a controller-managed one: there the id is assigned at runtime, after the node joins, and can change on re-election. So mod_init froze the still-unassigned id (-1, printed as its unsigned form 18446744073709551615) into every outgoing Via, and every node compared incoming cids against -1 - t_anycast_replicate() could never route a reply to the owning node. Read the id live instead: pre-build only the fixed ";<param>=" prefix at init, and have tm_via_cid() append the current get_my_id() per request (returning no parameter while the node still has no id); compare incoming cids against the live get_my_id() too. cl_get_my_id() already returns the runtime id, so this needs nothing from the caller and self-corrects across re-elections.
A control plane for the clusterer module: nodes discover each other over
encrypted UDP multicast, elect a master, and are assigned their cluster identity
at runtime, so an HA cluster needs no per-node id in the config and no database
behind it. Native, controller-managed and hybrid topologies coexist - a cluster
is controller-managed only if the clusterer declares it so via cluster_options.
Crypto is libsodium-only, no fallback: XChaCha20-Poly1305 for the payload AEAD,
Argon2id as the bootstrap KDF, and Noise_NNpsk0 for the join handshake, with
X25519 / HKDF-SHA256 / RNG underneath. libsodium is linked dynamically (distro
static libs are usually non-PIC), so target hosts need the runtime package. The
module is added to exclude_modules in Makefile.conf.template, like the other
modules with external-lib dependencies - enable it via include_modules - and
libsodium-dev joins the CI apt requirements.
Membership and liveness:
- master election, with the invariant "MASTER_ALIVE keepalive armed <=> I am
the elected master", which is what stops a node demoted purely by election
from continuing to broadcast and flapping between two masters;
- master-mediated ALIVE, so liveness costs O(N) messages per round rather than
every node pinging every other (O(N^2));
- a membership digest carried in MASTER_ALIVE, with a RESYNC repair path when
a node's view diverges from the master's;
- the join handshake is ACKed and retransmitted under a bounded budget, and
both the KEY_GRANT and the join-time state snapshot are unicast to the
joining node instead of broadcast to everyone.
Several controller clusters can share one BIN socket - each has its own
multicast group, shared-port unicast is routed by cluster_id, and multicast is
used as the fallback when two clusters collide on a port.
Hardening: the join path is flood-DoS rate-limited and its inputs are validated
and bounds-checked (including the Noise msg2 decrypt output and the zero-length
cipherstate passthrough). Decrypt failures are classified rather than logged
uniformly - a bootstrap-key failure means a wrong password, a foreign cluster or
tampering and warns, while a transient session-key mismatch during a rekey or a
split-brain heal is expected and stays at DBG. The split-brain defer budget
resets on a fresh higher-IP JOIN_REQ, so a lower-IP node waits for a live
higher-IP peer instead of self-promoting, bounded by CL_CTR_JOIN_DEFER_HARDMAX.
Multicast is sent and joined on this node's own interface rather than
INADDR_ANY, which is what makes discovery work on a multi-homed host where the
default route does not carry the cluster network.
Validated end-to-end on a 3-node cluster: controller / native / hybrid modes,
master failover and election stability (staggered and simultaneous starts both
converge to a single stable master), multiple controller clusters on one BIN
socket, the MI surface and its error paths, buffer overrun/underrun fuzzing,
wrong-password rejection and config mismatch. Ships with the admin guide, the HA
test appendix, the generated README and a join-rejection test.
The controller already owns an authenticated, encrypted, rate-limited UDP
plane between the nodes of a cluster, plus the membership that says who is on
it. Any other module wanting to exchange a message between nodes had to build
that transport again from scratch. This exports it instead.
A consumer registers a named channel pre-fork and sends on it:
clctr_api_t clctr;
if (load_clctr_api(&clctr) < 0) ... /* mod_init */
clctr.register_channel(&ch, my_cb); /* mod_init - PRE-FORK */
clctr.send_mcast(cluster_id, &ch, &payload, 0);
clctr.send_ucast(cluster_id, node_id, &ch, &payload, 0);
and inherits the XChaCha20-Poly1305 group session key and its rotation, the
per-packet receive gauntlet (magic gate, cluster_id filter, size bound,
per-source rate limiting) and the membership view, with no transport code of
its own.
The delivery contract is the part worth reading, because it is what a consumer
gets wrong:
- the receive callback runs in the CONTROLLER's worker process for that
cluster, not in the process that called send. A consumer needing to wake a
different process brings its own mechanism (shm + eventfd, ipc_send_rpc).
Callbacks run on the cluster's receive path, so they must stay short.
- sends are marshalled to that same worker over IPC, which is what keeps
ordering per node and leaves the anti-replay sequence space single-writer.
- CLCTR_SEND_TO_SELF dispatches locally rather than listening for our own
packet, and a unicast to our own node id degenerates to exactly that, with
nothing on the wire.
- src_node_id is 0 when the sender had not been assigned an id yet.
CLCTR_SEND_RELIABLE asks for acknowledgement and resend while unacknowledged.
It is opt-in per send rather than per channel on purpose: it costs one ACK per
recipient, so a reliable broadcast turns a single packet into N-1 packets back.
Most consumer traffic is better served by being idempotent and retried by its
own logic.
Payload sizing has a deliberate split: CLCTR_MAX_PAYLOAD is a compile-time
lower bound for consumers that size local buffers statically, while the real
runtime limit is cc_max_payload, derived from the interface MTU at mod_init and
larger on jumbo-frame links.
Ships with tests for the consumer sequence, channel filtering, reliable
broadcast, and the script-facing messaging surface.
cl_ctr_handle_member_list() refreshed last_seen for every IP in the list, via cl_ctr_upsert_peer_locked(). A MEMBER_LIST is a membership announcement: the master lists a node until it prunes it, whether that node is reachable or not. Treating it as evidence of liveness makes every receiver believe it has just heard from peers it has never heard from. That is what made a dead member immortal. cl_ctr_alive_bitmap() is built from last_seen, so a backup holding refreshed timestamps for a node that is down asserts to the whole cluster that it is up the moment that backup becomes master - and from then on nobody can age it out. It survives its own funeral. Seen in production: 10.22.20.241 stayed in cluster 243 for about a day, across a master change, after being rolled back to a build with no controller at all. Nothing was listening on its BIN port, and the two surviving nodes spent ~3,500 failed connects a day each trying to reach it - roughly 10,668 of one node's 10,801 daily ERROR lines, which buried every other error on the box. The member-list path now inserts peers it did not know about and leaves last_seen alone for peers it already tracks. Liveness continues to come only from direct evidence: a packet sent BY the peer, or the master's alive bitmap, which the master derives from packets it received itself. A newly learned peer is still seeded with last_seen = now, so it gets one purge window to prove itself instead of being dropped on the next tick. This terminates rather than oscillating: once the master prunes a dead node it stops listing it, and every receiver ages it out one window later.
A second controller process on the same host takes the same ip:port without
complaint, because the socket needs SO_REUSEADDR and INADDR_ANY to receive
multicast at all. The kernel then treats the two traffic types differently:
a multicast datagram is delivered to EVERY co-bound socket, but a unicast
is delivered to exactly ONE, chosen without reference to the destination
address. Measured on 5.4, 200/200 trials, the winner being the most
recently bound socket; SO_REUSEPORT does not help, it only changes which
socket wins.
Every 1:1 leg therefore lands in the wrong process. The join KEY_GRANT is
unicast, so the joining node never authenticates and dies with
cannot authenticate ... (wrong password, or a foreign cluster on
cluster_id N). Shutting down.
which sends the operator hunting a credential problem that does not exist.
Reversing the start order moves the victim to the other instance, which is
how the mechanism was confirmed: it follows bind order, not address.
This does not make that topology work - one controller instance per host
per port is what is supported, and production is unaffected because a node
runs one. It makes the diagnosis available:
- cl_ctr_setup_socket() probes the port WITHOUT SO_REUSEADDR before
taking it. That bind fails EADDRINUSE precisely when another process
already holds it, and succeeds when the port is free; a probe WITH
SO_REUSEADDR would succeed either way, which is why the real bind
cannot tell. One warning naming the actual constraint. Advisory, never
fatal - a restart can briefly race the outgoing process, and a spurious
warning is cheaper than refusing to start.
- cl_ctr_maybe_forward()'s "no local cluster for this packet" drop was
LM_DBG. L_DBG is 4 and deployments run 3, so a whole class of
"the cluster does not converge" had no trace at any log level anyone
uses. It is now LM_WARN, rate-limited to one per 30s carrying the
suppressed count - a misdelivering peer can produce one per packet, and
a warning that fires at line rate is its own outage.
The in-process case is unchanged: several clusters in ONE process sharing a
port are still recovered by cl_ctr_maybe_forward() through the shared
cluster array, and that path never reaches the new warning.
678bc4c stopped MEMBER_LIST from refreshing last_seen, on the grounds that a membership announcement says who BELONGS to the cluster, not who is reachable. NODE_ASSIGN carries exactly the same kind of claim and was still calling cl_ctr_upsert_peer_locked() on the IP in its payload, so the bug survived the fix. This half was worse, because it is a SELF-loop rather than a peer-to-peer echo. IP_MULTICAST_LOOP means the master receives its own NODE_ASSIGN - the handler's own doc comment says 'all nodes (including master via loopback) apply the assignment'. So every roster announcement refreshed last_seen for every node named in it, including a dead one, and since the master is the node that runs cl_ctr_prune_stale(), the corpse kept itself alive. Measured on the 3-node staging RGS cluster before this commit: a member whose controller plane was isolated with iptables, and which was confirmed silent by tcpdump on the master, was STILL a member on both survivors after four minutes against a 30 s purge deadline. Behaviour was identical with and without 678bc4c, which is what showed the earlier fix was only half of it. Liveness still has two sound sources and both are untouched: the master learns it from the unicast ALIVE a settled non-master sends it, and backups learn it from the master's MASTER_ALIVE bitmap, which cl_ctr_apply_alive_bitmap() uses to set last_seen. A node the master no longer believes in is simply absent from that bitmap, so receivers age it out instead of being told to keep it. cl_ctr_rejoin_superior_master() was audited at the same time and deliberately left alone: it upserts sender_ip from a beacon actually sent by that node, which is direct evidence. (cherry picked from commit 9bd7d10)
Removing a node from the topology did not touch the transport. clusterer sends with msg_send(send_sock, proto, &node->addr) and the core keeps the TCP connection in its own table keyed by destination, so nothing in clusterer_ctrl_remove_node() - delete_neighbour, remove_node_list, CLUSTER_NODE_DOWN, report_node_state - ever closed the socket. It survived until the TCP layer timed it out or the peer closed first. That is not academic. A node can be dead to the control plane and still perfectly reachable on BIN: hung, half-open, or partitioned on one plane only. Measured on a 3-node staging cluster, with the victim's control plane isolated and BIN deliberately left reachable: clusterer_controller purged the node at its 30 s deadline and both survivors still held ESTABLISHED connections to it, in both directions. The close lives here rather than in clusterer_controller on purpose. The BIN transport belongs to clusterer, and the controller drives it through this API instead of reaching into the core's TCP layer itself. It is exposed as close_node_conn() for the case where only the transport should be dropped; remove_node() now does it for you. Two ordering constraints, both load-bearing: - the url is copied out BEFORE remove_node_list(), which frees the node; - the close happens AFTER cl_list_lock is released and after the capability event_cb callbacks have run, because a callback may still send a final BIN packet to the departing node and pulling the socket first would only turn that into an error. get_node_by_id() walks node_list, which does not contain current_node, so this can never close our own listener. (cherry picked from commit d1d12a5)
…at happened Two defects in the previous commit, both found by measuring on a live cluster rather than by reading the code back. 1. It closed at most ONE connection. There are normally two per peer - the one we dialled out to its BIN port and the one it dialled in to ours - and both are reachable by the same address, because the core registers an alias for the peer's advertised port on an accepted connection as well. That aliasing is what lets an inbound connection be reused for outbound sends. tcp_close_connection() closes exactly one per call and flags it F_CONN_FORCE_CLOSED, which makes the next lookup skip it, so the fix is to loop until it reports nothing left. That terminates by construction: each pass removes one connection from the candidate set. The cap is paranoia against a future core change that stops setting the flag. 2. The logging announced intent, not outcome. It printed 'dropping BIN connection' before calling, and only said anything afterwards on a negative return - but tcp_close_connection() returns 1 for closed and 0 for nothing found, so the interesting case was silent and the log could not distinguish 'closed it' from 'there was nothing there'. On the first live test that made a no-op look like a success. It now reports the count it actually closed, and an outright failure is an error rather than a debug line. (cherry picked from commit e5838d3)
The master's MEMBER_LIST and NODE_ASSIGN are multicast, and IP_MULTICAST_LOOP delivers them back to the master itself. Both handlers ran the learn path on the payload IPs, so the master re-inserted every node its own announcement named - including a peer it had just purged. The re-seeded copy kept the roster naming the dead node, every receiver re-learned it one window later, and the purge could never converge. Observed live on a 3-node staging cluster: 'new peer <ip>' every 35 s, in lockstep with the purge cycle, for a node whose control plane was provably blocked the whole time. An announcement we authored was built FROM this table; it cannot teach us anything. Receivers other than the author still learn membership from these packets exactly as before - that is the legitimate propagation path, and a node the master stops listing now ages out everywhere one window later, which restores the termination argument cl_ctr_learn_peer_locked() was written around. This also explains why a resurrected peer escaped removal forever: a MEMBER_LIST entry carries only IP + is_master, so the re-learned copy had node_id 0, and cl_ctr_prune_stale() only propagates removal to clusterer for node_id > 0. 'removed node_id' fired exactly once per incident and never again. (cherry picked from commit bbfc93cf57742fc89f11d531299d326f4dac3376)
Membership authority: in a controller-managed cluster the only way in is the controller's JOIN handshake, after which the controller calls clctl.add_node(). Wire self-discovery is the zero-config mode's mechanism and must not apply - but cl_db_mode() deliberately reads 0 for controller-managed clusters (so the controller's runtime add/remove works), and that same predicate gated the learn paths, which put controller-managed clusters on exactly the self-discovery behaviour they must not have. The consequence, measured on a 3-node staging cluster: a node the controller had expelled talked its way straight back in within one ping interval - PING from unknown -> UNKNOWN_ID -> NODE_DESCRIPTION -> add_node() - and the BIN connections that had just been closed were re-dialled (2 established grew to 4). A node dead to the control plane but alive on BIN is exactly the hung / one-plane-partitioned failure the purge exists for. New cl_ctr_owns_membership() (0 in non-controller builds, so default builds are unchanged) now gates all three wire-learn sites: - handle_full_top_update's two unknown-node learns, alongside cl_db_mode; - handle_internal_msg_unknown's NODE_DESCRIPTION add_node - which also no longer gossips the stranger's description onward via flood_message. The UNKNOWN_ID reply to a stranger's ping is kept: telling the node we do not know it is what prompts its controller to re-join properly. The ignore is logged at INFO, rate-limited to one line per 30 s, because a live expelled node re-announces on every ping cycle. (cherry picked from commit f2903b1)
Split-brain resolution had two merge paths that behaved differently. A master that learned of a superior via MASTER_BEACON went through cl_ctr_rejoin_superior_master() - demote, adopt the master, and send a JOIN_REQ. A master that learned of it via MASTER_ALIVE merely yielded: it recorded the winner and stopped asserting mastership, and that was all. Yielding alone is not a merge. The winner learned the yielding node only from that packet's sender-upsert - a peer entry with node_id 0 - and the ONLY place a node_id is ever assigned is handle_join_req(). Without a JOIN_REQ the yielded node sits in the winner's table as id 0 forever: it is never named in a NODE_ASSIGN, so it is never added to clusterer on any node, and its membership digest can never match the master's MASTER_ALIVE - which turns into a RESYNC-per-second livelock as the master keeps 're-broadcasting full state' that structurally cannot contain the missing node. Observed live on the 3-node staging cluster after a partition heal: the controller admitted the returning node as a member everywhere, but the master held it at node_id=0, clusterer never learned it, and RESYNC fired once a second indefinitely. This had been masked before the membership-authority change: clusterer's wire self-discovery quietly re-added the node at the BIN level, hiding the controller-level livelock. The yield path now calls cl_ctr_rejoin_superior_master() - identical to the beacon merge - which demotes, arms the dead watchdog, and sends the JOIN_REQ (join_pending-guarded, so an exchange already in flight is not stomped). The function takes the discovery vector as a string so the merge logs say which path found the superior. (cherry picked from commit 5084afa)
55e5429 to
fc5d332
Compare
The flag's note read like a cost trade-off - acks are not free, so opt in when you want them - and recommended idempotent-and-self-retrying consumers as the better default. Both true, and both beside the point: on a channel with concurrent traffic the flag does not work at all. Every packet carries a sequence number and the anti-replay guard drops anything not strictly greater than the last accepted from that peer. A retransmit re-sends cached bytes, so it carries its original seq. Serialised 1:1 - the join handshake this was built for - is safe, because nothing can overtake it. The moment a second message can, the retransmit lands behind a higher seq and is discarded, the receiver re-acks it so the sender stops trying, and neither end says anything above debug level. Measured on the cross-node cache pull channel: the flag made delivery WORSE, 61% -> 49% under 40% reply loss at two to three times the packets, with zero retransmitted payloads ever delivered. That change was reverted; this note is so the next consumer does not repeat it. Comment only - no functional change. (cherry picked from commit 321f8cf)
…ates them
cc_max_payload is derived from the interface MTU at mod_init, but the two
buffers it gates were sized from the COMPILE-TIME CLCTR_MAX_PAYLOAD:
cl_ctr_script_send() char buf[CLCTR_MAX_PAYLOAD]; /* 1300 */
guard: 2 + taglen + gen_msg->len > cc_max_payload
cl_ctr_rpc_consumer_send() char pkt[CL_CTR_CONSUMER_PKT_MAX]; /* 83+1300 */
guard: cl_ctr_consumer_submit(), payload_len > cc_max_payload
On any link above MTU 1411 the gate admits more than the buffer holds. This is
not a corner case: build host 222 runs enp6s18 at MTU 9000, so cc_max_payload
is 8889 against a 1300-byte stack array, and the prod gateways log
"interface ens18 MTU=1500, max consumer payload=1389". A 1350-byte script
message passes the gate and overruns by 52 bytes; on the 9000 link the ceiling
is 7589.
Fixed by making the buffers honour the bound, NOT by capping the bound. An
upper clamp on cc_max_payload would have been a smaller diff and would have
made the MTU derivation dead code - the capability is wanted, and the host
this was found on is itself on a jumbo link. Both buffers are now pkg_malloc'd
in mod_init from cc_max_payload, after the MTU probe has settled it. mod_init
is pre-fork, so every worker inherits its own copy-on-write copy: no sharing
between processes, and no per-call allocation on a path cachedb_perf's pull
replies use for every message. Neither send function can re-enter itself - a
script function runs to completion inside one route execution, and the
consumer send runs in the cluster worker's own loop.
CL_CTR_PKT_OVERHEAD is split out of CL_CTR_CONSUMER_PKT_MAX so the packet size
can be computed at runtime; the macro stays for the compile-time contract in
api.h.
One real cap added, which does not touch jumbo frames: a datagram still has to
fit a UDP payload, and loopback reports MTU 65536, which computed a payload one
byte past what sendto() accepts. Clamped to CL_CTR_UDP_PAYLOAD_MAX minus the
overhead. At MTU 9000 the bound is unchanged at 8889.
Both sites also now bound against their own buffer as well as against
cc_max_payload - the sibling cmd_cl_ctr_send_req_list() has always guarded with
sizeof(buf), and the gap between the two forms is what this was. The consumer
path drops with an error if the two ever disagree again.
PROVEN fail-then-pass with /dn/task86b, one node, both prefixes built with
-fstack-protector-all as a DETECTOR (a 52-byte overrun of a stack array
otherwise may corrupt nothing observable and the run would prove nothing):
before 1350 bytes -> *** stack smashing detected ***: terminated,
workers alive 0
after 1350 bytes -> 200, X-T86B: SENT, workers alive 17
5000 bytes -> 200, X-T86B: SENT, workers alive 17
(the jumbo case: proof the headroom is usable, not
merely safe)
The rig needs one node because the overrun is in the memcpy that builds the
frame, before anything is sent - no cluster has to form and no peer has to
exist.
Not exposed on the fleet today: none of .241/.242/.243 call cl_ctr_send_req,
cl_ctr_broadcast_req or cl_ctr_send_rpl in their configs. The consumer path is
reachable by any API consumer following the documented cc_max_payload contract
in api.h:98-103.
Full 131-module build, 0 errors, 0 stamp mismatches.
(cherry picked from commit b3227013d03ddb06322ebed77706d705998a07be)
…_send
The flag does not work on a channel carrying concurrent messages, and fails
silently when it does not. cl_ctr_check_and_update_seq() requires
pkt_seq > last_seq, and a retransmit reuses its original seq. On the serialised
1:1 exchange this was built for - the join handshake, KEY_GRANT - nothing else
from that peer is in flight, so the retransmit still carries the highest seq
and is accepted. On a concurrent channel, later messages have already advanced
last_consumer_seq by the time the retransmit arrives, so it is dropped as
out-of-order AND re-ACKed, which stops the sender retransmitting and leaves it
believing it delivered.
Measured, not argued: on the cross-node cache pull channel under 40% reply
loss, three runs each, plain delivered 61.2% and RELIABLE 49.4% - worse, at two
to three times the packets, with not one retransmitted payload ever delivered
and pulls_late_stored at zero across all six runs.
So the default now DROPS the flag rather than refusing the send: a plain send
measurably out-delivers a reliable one here, making this an improvement rather
than a consolation prize. One rate-limited warning per process per minute says
so, and names the condition to check.
Gated by a modparam rather than removed, because the question is answerable by
exactly one party. The only callers that can reach the flag today are the
script functions cmd_cl_ctr_send_req / cmd_cl_ctr_broadcast_req, and they all
share one channel (cl_ctr_script_chan) - concurrent by definition if two routes
fire at once. Whether that happens is a property of the deployment's own
routes: the admin can answer it, a module author cannot be asked via config.
modparam("clusterer_controller", "enable_reliable_send", 1)
One gate, at cl_ctr_consumer_submit(), because both the script functions and
the consumer API funnel through it - the flag cannot survive by another route.
api.h now documents that the flag is gated, and why.
This is MITIGATION, NOT REPAIR. It stops the broken path being reached by
accident; it does not make RELIABLE reliable. Task OpenSIPS#71 stays open for the real
fix - a fresh seq per retransmit with message identity carried separately, or a
small per-peer window instead of a single high-water mark - or for deleting the
flag, which is defensible since the internal control traffic that genuinely
depends on serialised delivery does not go through it.
Verified on a live node, both directions: with the modparam absent a
cl_ctr_send_req(..., reliable=1) logs the refusal once and sends plain; with
enable_reliable_send=1 it is honoured and nothing is logged. The clctr send
path is otherwise unchanged - the /dn/task86b rig still passes at 1350 and
5000 bytes with 17 workers alive. Full 131-module build, 0 stamp mismatches.
(cherry picked from commit ab7851614c50214d48f95298d22ac7486b9eb25a)
… length Both consumer enqueue sites cached and re-sent plain_len for a packet that cl_ctr_seal_and_send() had encrypted in place. What goes on the wire is CL_CTR_WIRE_HDR_SZ + plain_len + CL_CTR_TAG_SZ, so every repair left 44 bytes short and every receiver discarded it as "short packet, dropping" before it could be decrypted. Consumer retransmission had never delivered a single byte. The control plane's own enqueue (KEY_GRANT) always passed the sealed length, which is exactly why the join handshake's ARQ worked and this did not. Measured on a two-node rig under 40% loss: the receiver logged 113 x "short packet (25 bytes)", 25 being precisely the plain_len of the messages being repaired. (cherry picked from commit 7e69636bfe6ae352cabda7b6496b1176e4dfae2d)
cl_ctr_check_and_update_seq() was a bare high-water mark, so it could not tell "already delivered" from "delivered out of order" and had to reject both. Two consequences: a retransmit carries its ORIGINAL seq, so once any later message had been accepted the repair was discarded - and then re-ACKed, which stopped the sender retransmitting and left it believing it had delivered; and ordinary reordering, which any multipath network produces, silently dropped consumer payloads that were never duplicated at all. Replace it with the standard construction (IPsec RFC 4303 A.2, DTLS RFC 6347 4.1.2.6): the mark plus a 1024-slot circular bitmap of what has already been accepted at or below it. A seq below the mark whose bit is clear was never delivered, so its repair is accepted; a seq whose bit is set is a true duplicate, re-ACKed but not delivered twice. No seq is ever accepted twice, so replay protection is unchanged. Below the window the receiver cannot know whether it ever had the message, so it drops it and deliberately does NOT acknowledge it - an acknowledgement it cannot justify is the silent failure this is meant to remove. mod_init reports the per-peer message rate the window spans, since both terms of that bound (consumer_retries x consumer_retry_ms) are configurable. (cherry picked from commit f3dded5b6d0dba7709ade6902fa47b1980c02bf6)
cl_ctr_retx_enqueue_bcast() armed the shared timerfd unconditionally to now + retry_ms on every send. timerfd_settime() REPLACES a pending expiry, so a steady stream of reliable sends pushed the deadline forward faster than it could arrive: at fifty messages a second with a 40 ms gap the timer was re-armed every 20 ms to fire 40 ms later, and never fired at all while traffic continued. Measured on the rig: 45 lost messages produced 2 retransmits. Two related defects in the same timer. The sweep armed and re-armed to a flat CL_CTR_RETX_INTERVAL_US (250 ms) rather than to the earliest pending deadline, so a consumer entry due in 40 ms waited 250; and it reset next_due_us to now + 250 ms on every retry, discarding the per-cluster cadence that cl_ctr_retx_enqueue_consumer() had just set. Give each entry its own retry_ivl_us and route every arm through cl_ctr_retx_rearm(), which arms for the earliest deadline in the queue and disarms when it is empty. Never arm 0 us - that is timerfd's disarm, not "fire immediately". (cherry picked from commit c1028110d8dcf7790257f502900f32bafb9e4271)
The cache was one inline CL_CTR_NODE_ASSIGN_MAX_SZ array per entry (~580 bytes), sized for the largest control-plane packet. Any reliable consumer send above roughly 540 bytes was therefore never queued for repair at all - well under CLCTR_MAX_PAYLOAD, the size api.h tells consumers they may rely on - and it degraded in silence, on an LM_DBG. Sizing it to the compile-time CLCTR_MAX_PAYLOAD instead would cap repair at 1300 bytes on a jumbo-frame link, a seventh of what that link and cc_max_payload happily carry: the same trap the transmit buffers had before they were sized from the MTU. So the bound is cl_ctr_retx_pkt_max, derived in mod_init from cc_max_payload beside cl_ctr_script_buf_sz, and each entry allocates for the packet it actually holds - the cost then tracks outstanding reliable messages instead of reserving queue x MTU up front, which at a 9000 MTU would be 1.1 MB per cluster sitting idle. cl_ctr_retx_release() becomes the only way an entry is cleared, so a buffer cannot be dropped by a path that merely clears `used`; the flush path releases each entry rather than memset-ing the array over live pointers. A send too large to cache still goes out once, but now says so with a rate-limited warning instead of a debug line. (cherry picked from commit dd2befdaf0efa0d0fb0c6d34c126ed70327e01c9)
…uffer It built its payload in a CLCTR_MAX_PAYLOAD stack array while its two siblings, cl_ctr_send_req() and cl_ctr_broadcast_req(), use the cc_max_payload-sized cl_ctr_script_buf. On a jumbo-frame link a script could therefore broadcast an 8 KB message but could not send that same message to a list of nodes: the list form stopped at a seventh of what the other two carried, and blamed "the cluster plane" rather than its own buffer. Share the buffer and the guard. (cherry picked from commit 25eafda958d0ee8dbaa1c7950dfec3a67baab49d)
…contract The switch was added hours earlier to gate a mechanism that could not work: a retransmit reused its seq, the receiver's anti-replay guard was a bare high-water mark, and the repair was sent truncated to its plaintext length - so on any channel it was rejected, and on a concurrent one silently re-ACKed. Measured on the cross-node cache pull channel under 40% reply loss: plain 61.2% delivered, reliable 49.4%, at two to three times the packets. With the sealed length, the replay window and the deadline-driven retransmit timer fixed, the flag does what it says. Interleaved A/B on a two-node rig, 40% loss on the consumer data leg, 300 messages at 50/s, three runs each: 57.0/59.0/60.0% before, 93.3/94.0/92.7% after - both matching closed form, for no working ARQ (1-p) and for two retries (1-p^3). Packet counts corroborate it independently: before, exactly 300 packets went out for 300 messages, so not one retransmit ever reached the wire. Separately, with a clean data leg and 35% ACK loss, 128 duplicates arrived and none was delivered twice. So the default becomes 1. The switch stays as a kill switch, because ARQ has a cost an operator may not want: a reliable broadcast draws one ACK back from every member, so on a large cluster one message becomes an event. Turning it off degrades a reliable send to a plain one - the payload still goes out - and warns. api.h and the admin guide are rewritten accordingly: the at-most-once contract, the fact that consumers are not asked to deduplicate, and the one real limit - the window spans messages rather than time, so a peer that outruns it gets a repair that is dropped and deliberately not acknowledged. (cherry picked from commit 93b1aaef0d08f939adb225fa3ef96325310d46e9)
The replay window compared sequence numbers with plain unsigned arithmetic, so when a peer's 32-bit counter wrapped, every packet after the wrap read as 2^32-ish behind and was rejected until the next key rotation reset both sides. Fail-closed, but silent - and the rotation it waits for may never come: a fresh master_salt is generated only in cl_ctr_on_became_master(), so a cluster with a stable master never rekeys on its own. Nor is the wrap remote. Every ACK bumps my_seq, so the control plane advances at the rate of the consumer traffic it acknowledges, and at the window's own 12,800 msg/s ceiling 2^32 is under four days. Compare with serial-number arithmetic instead. On its own that would trade the outage for something worse - a packet captured 2^31 or more messages ago has a difference that reads back as a large POSITIVE and would be accepted as fresh - so the forward step is bounded too: CL_CTR_REPLAY_MAX_JUMP (2^24) or more is refused as implausible rather than believed, with its own verdict and log line so a real partition is not mistaken for an attack. A peer that genuinely exceeds it is recovered by the window reset that already runs when it rejoins or reappears in a MEMBER_LIST. Note the old code accepted that 2^31-behind packet as well - unsigned comparison called it a forward jump - so the bound closes a hole rather than opening one. What remains is inherent to a 32-bit counter: a packet held across a full 2^32 cycle lands back inside the window. Proven by /dn/task89, which extracts the window logic from a given revision so it always tests the real code: against the parent commit the four wrap cases and the 2^31 replay fail; against this one all fourteen pass. (cherry picked from commit 09740850b0de9b4dff171de71d0ad5dc34513a44)
…to CLCTR_MAX_PAYLOAD mod_init derived the maximum consumer payload from the interface MTU and then raised it back to CLCTR_MAX_PAYLOAD whenever the link was smaller than that. The effect was the exact inverse of why the MTU is consulted at all: on any interface under about 1411 - a VPN, a GRE or IPIP tunnel, PPPoE, the usual 1400/1420/1436 - the module advertised a bound ABOVE what the path carries. Every full-size consumer datagram then IP-fragmented, and a fragmented datagram is lost entirely if any one fragment is, so the padding manufactured precisely the losses the retransmit machinery then had to repair. Honour the link and say so instead. CLCTR_MAX_PAYLOAD stays what a consumer sizes a compile-time buffer with, but it stops being a promise the network cannot keep, and a consumer that cares about the real bound now asks for it. (cherry picked from commit f0ff9a2dfb, clusterer_controller portion only - the cachedb_perf half belongs to the cachedb_perf PR)
A consumer needs the MTU-derived bound, not the compile-time constant, and the only way to reach it was to declare `extern int cc_max_payload`. That is defined in clusterer_controller.c, and OpenSIPS modules are dlopen'd, so the reference leaves an undefined symbol in the consumer's .so that resolves only if clusterer_controller happens to have been loaded first. It does not fail at build time - it fails at startup with cachedb_perf.so: undefined symbol: cc_max_payload and takes the whole config down with it. Every other cross-module call here goes through the bound API for exactly this reason, so add get_max_payload() alongside them and say plainly in api.h not to declare the extern. (cherry picked from commit bdb52a3ba3, clusterer_controller portion only - the cachedb_perf half belongs to the cachedb_perf PR)
The MTU is read from the interface the plane runs on and kept in cc_mtu (what
we joined at) and cc_mtu_now (the live reading). There is deliberately no
modparam for it: a configured value is a second source of truth that can
disagree with the kernel, so the config says 1500 while the link says 9000 and
the node refuses to start while being perfectly consistent with its own
segment. That is the same defect class as the CLCTR_MAX_PAYLOAD floor.
Because the MTU now decides whether a node may join at all, two places that
used to shrug become fatal:
- auto-detection that cannot name the interface owning the resolved IP used
to warn and carry on with the compile-time default. Both explicit modes
(my_ip, interface) always yield a name, so this is only reachable from the
default-route probe, and naming either modparam is the fix.
- a failed SIOCGIFMTU leaves the node unable to establish whether it belongs
in the cluster, which is not a state it can serve SIP from.
The value is exported through cl_ctr_list_members, cl_ctr_node_info and
cl_ctr_list_config, because a log line is not something an operator can alert
on - and one gateway in this fleet has no journal at all.
(cherry picked from commit ccfdc3e40f6da9283e3e80ea5372f969b7358eba)
…ter's JOIN_REQ and ALIVE now carry the sender's MTU, and the master admits only nodes whose value equals its own. That single rule is the whole mechanism: by induction every member carries the master's MTU, so the backup that eventually becomes master necessarily carries it too and a handover cannot change it. Nothing has to be inherited, agreed or carried in MEMBER_LIST - membership is itself the proof of agreement. A partition inherits the property on both sides, and a master that drifts removes itself while the backup keeps the original value, so a drifting node cannot redefine the cluster's MTU on its way out. The check has to happen here because the failure is undetectable later: an oversized datagram is dropped by the small-MTU node's NIC with no ICMP, since every node is on one L2 segment and there is no router to generate one, and multicast has no path-MTU discovery in any topology. The receiver therefore can never observe the packet it failed to receive. At join both sides are still exchanging small handshake packets that do arrive. It refuses BEFORE KEY_GRANT for key custody, not speed: the joiner holds no session key until then, so a refused node never obtains the group key at all, where admit-then-eject would have handed it multicast decryption and needed a rotation to take it back. The refusal also drops the peer from the table - while still NEW a node upserts every joiner it hears, so refusing alone would leave a node we just turned away being counted as a member. Unconditional, unlike the config gate: there is no policy knob, because a node that cannot participate must not serve SIP. Without shared usrloc, dialog replication and shtag coordination it answers REGISTERs into a local store and routes on a partial view while the load balancer keeps feeding it calls. A joiner that advertises no MTU at all is an older build. It is admitted with a warning rather than refused, or a rolling upgrade could never complete: every not-yet-upgraded node would be turned away by the first upgraded master. Master election is by highest IP and completely MTU-blind, so on a simultaneous cold start a single misconfigured host that wins it will refuse an otherwise healthy fleet. cl_ctr_mtu_note_reject() says so out loud once the master has turned away several distinct peers while holding no members of its own. It only warns - a master that stepped down because OTHERS disagreed would break the rule that a node acts on its own condition alone, and would hand anyone holding the bootstrap key a way to force re-elections. A refused node self-terminates, naming both MTUs and the interface, but only after checking the claim against its own kernel reading: a reject quoting our own MTU back at us contradicts itself and is ignored. It acts whenever an admission request is outstanding, not merely while NEW - on a simultaneous cold start both nodes reach the join deadline, both self-promote, and the loser merges into the winner, so it is ACTIVE rather than NEW when its JOIN_REQ is refused. A settled member with no request outstanding still ignores rejects, which is what stops one member from evicting another. (cherry picked from commit ab56c508c22292681bf64582985f107774725563)
Nothing notified the module when its link changed - the MTU was read once at mod_init and never revisited - so a node could go on believing it matched a cluster it no longer did. The ALIVE timer now re-reads it. Poll rather than netlink RTM_NEWLINK: netlink drops events on ENOBUFS, so a reconcile poll would have to exist anyway and a missed MTU change is exactly the silent failure this removes, and it fires on transients a poll harmlessly misses - a bond failover, a driver reset, a VLAN parent bounce. Acting on the first event would kill a healthy node over a blip, hence the strike count as well. One ioctl per query_time costs nothing. Own drift terminates the node rather than warning, because shrinking an MTU breaks RECEIVE, not send: the kernel still fragments on egress, so the node keeps emitting heartbeats and looks healthy to every peer while silently no longer receiving anything larger than its new MTU. MEMBER_LIST on a real cluster is well over 1500, so it would sit on stale membership and an incomplete BIN mesh, invisible to its peers, while the LB kept handing it calls. This is the node acting on its OWN reading, which is what makes acting safe at all - a switch-side change does not move /sys/class/net/X/mtu. Peer drift only warns, and must: terminating because someone else changed would turn one `ip link` command into a fleet outage. That peer detects its own change and removes itself. What is advertised is the live reading, not the value we joined at. Announcing the joined-at value instead made the peer warning unreachable - a node whose link changed kept announcing the old number, so no peer could ever observe the change, and the only account of it was in the log of the node that was about to disappear. (cherry picked from commit 5d7f0130257d6d98c7438db0b3e56231c08d1bfc)
The drift poll had two outcomes where it needed three. cl_ctr_read_iface_mtu() returns -1 when the ioctl fails, and that fell into the same branch as "the link agrees again", which did two wrong things at once. It cleared the strike count, so a read failing at roughly the confirmation cadence could hold a genuinely drifted node alive indefinitely - the exact silent failure the poll exists to remove. And it logged that the MTU was "back to" a value it had never read: a measurement that did not happen, which is worse than no line at all, because an operator reading it concludes the link recovered. A failed read is neither drift nor recovery. It is an absence of evidence, so it now leaves the strike count exactly where it stood and says so - on the first failure and rarely after, since a poll that has gone blind must not be able to look like a quiet healthy one. It is deliberately not fatal, even though mod_init refuses to START without an MTU. The asymmetry is the point: at startup the node holds no membership and loses nothing by refusing, while here it is an established member carrying calls, and an ioctl we could not complete is evidence about our own visibility rather than about the link. The drift this guards against - a host-local `ip link` change - leaves the interface perfectly readable; a read that fails means it was renamed or removed, which announces itself far more loudly elsewhere. The decision moves into cl_ctr_mtu_step() so it can be exercised directly: the sequences that matter (a failure landing mid-drift, a true transient, a drift that persists) are awkward to stage against a real interface and trivial to enumerate. /dn/task97 extracts the function straight from this file and runs both the old and new logic over the same inputs - the old one never confirms a drift once a failed read interrupts it, including when failures alternate with drift readings forever. (cherry picked from commit 984e55502008f08759a11af58769b53ae4d3babc)
The fragmentation note claimed "the DF bit is not set so fragmentation occurs transparently where the network allows it". That was never measured and is false. The socket sets no IP_MTU_DISCOVER at all, so Linux's default applies - IP_PMTUDISC_WANT - and that default DOES set DF, on unicast and on multicast alike, confirmed on the wire at 2000, 4000 and 8900 bytes. What actually carries a 4395-byte MEMBER_LIST across a 1500-byte link is that it EXCEEDS THE LOCAL MTU: a datagram the kernel must fragment locally cannot also be doing path-MTU discovery, so those fragments leave with DF clear. The old wording described the right outcome by the wrong mechanism, and the mechanism matters - a datagram that FITS the local MTU goes out with DF SET, so meeting a smaller-MTU hop it is dropped with an ICMP "fragmentation needed" that this topology routinely filters, rather than being fragmented and delivered. The behaviour is left exactly as it is. Clearing DF would let a jumbo node reach a peer across a narrow routed hop, but this module's contract is the interface it runs on, not how the operator carries multicast between segments: enforcing one MTU across the cluster is what makes the local bound mean anything, and moving packets between segments is the network's job. (cherry picked from commit 3c65e429ef88e6613f52ee0008c9f23ecef6225e)
… guard and what DF actually does
2 participants
clusterer_controller — self-forming HA coordination for OpenSIPS
clusterer_controllersits on top of theclusterermodule and removes thestatic
my_node_info/neighbor_node_infowiring: nodes sharing a multicastgroup, a
cluster_idand a password self-organise — they discover eachother, elect a deterministic master (highest IP, sticky), assign
clusterernode-ids, hold sharing-tags on exactly one node, and fail over automatically —
with an encrypted control plane and zero per-node topology config.
This revision lands the unicast / scalability series (steady-state and
join cost cut from O(N²) → O(N), with anti-entropy repair and reliable
handshakes) and a module-wide
cc_→cl_ctr_rename so every symbolmatches the public
cl_ctr_*MI/pvar/function surface.Control plane at a glance
Encrypted UDP on the cluster's multicast group (default
239.0.10.x:3333).Only a small cleartext header is visible on the wire —
magic (2B)+cluster_id (2B)+nonce— everything else is AEAD-sealed(XChaCha20-Poly1305 + Argon2id, key agreement via Noise_NNpsk0).
Two key tiers, told apart by the magic byte:
0xCC01JOIN_REQ,KEY_GRANT,ACK,JOIN_REJECT,MASTER_BEACON0xCC00MASTER_ALIVE(+liveness bitmap +membership digest),ALIVE,NODE_ASSIGN,MEMBER_LIST,RESYNC,GOODBYE,KEY_HANDOFFLegend for the diagrams below:
-->>unicast (1:1),-)multicast (group).1. Cluster formation (cold start)
Nodes start together, discover over multicast, and converge on the highest-IP
node as master with no split brain — lower-IP nodes defer self-promotion while
a higher-IP peer is still joining.
sequenceDiagram autonumber participant N1 as .191 participant N2 as .192 participant N3 as .193 highest IP Note over N1,N3: simultaneous cold start, no master yet N1-)N3: JOIN_REQ (mcast, Noise msg1) N2-)N3: JOIN_REQ (mcast, Noise msg1) N3-)N1: JOIN_REQ (mcast, Noise msg1) Note over N1,N2: higher-IP peer still joining -> defer self-promotion Note over N3: highest IP -> self-promote, mint session key N3-->>N2: KEY_GRANT (unicast, Noise msg2 + salt) N2-->>N3: ACK N3-->>N1: KEY_GRANT (unicast) N1-->>N3: ACK N3-->>N2: NODE_ASSIGN + MEMBER_LIST (unicast) N3-->>N1: NODE_ASSIGN + MEMBER_LIST (unicast) Note over N1,N3: converged - .193 master, .192 backup, .191 member2. Member join into a running cluster
The join is unicast to the joiner: only the newcomer needs the full peer
list, so the master sends it 1:1 instead of multicasting N packets that every
member would decrypt and discard. Only the newcomer's own assignment is
multicast, so existing members learn it. A lost
KEY_GRANTis recovered byACK + retransmit (§5), a lost snapshot by the
JOIN_REQretry.sequenceDiagram autonumber participant J as .191 joiner participant E as .192 member participant M as .193 master J-)M: JOIN_REQ (mcast, Noise msg1 + BIN socket) M-->>J: KEY_GRANT (unicast, session key) J-->>M: ACK (unicast) M-)E: NODE_ASSIGN newcomer .191=id3 (mcast -> all learn it) M-->>J: NODE_ASSIGN .193=id1, .192=id2 (unicast, peers) M-->>J: MEMBER_LIST 3 members (unicast, snapshot) J-)M: ALIVE (joiner now participates) Note over J,M: per join the group sees ONE packet, not N3. Steady-state liveness — master-mediated (O(N²) → O(N))
Previously every node multicast an
ALIVEeveryquery_timeand every node ranan election off every one — O(N²) packets and cluster-wide decrypts. Now a settled
member unicasts its ALIVE to the master; the master folds all peers' liveness
into a per-node-id bitmap on its
MASTER_ALIVE, and members trust that bitmapto keep their election windows populated.
sequenceDiagram autonumber participant A as .191 member participant B as .192 backup participant M as .193 master A-->>M: ALIVE (unicast, ~query_time) B-->>M: ALIVE (unicast, ~query_time) M-)A: MASTER_ALIVE (mcast, ~1/s) + liveness bitmap + membership digest M-)B: MASTER_ALIVE (mcast) + bitmap + digest Note over A,B: refresh last_seen from the master's bitmap - no peer-to-peer ALIVE4. Anti-entropy — membership digest → RESYNC
Every
MASTER_ALIVEcarries a digest = active-peer count + order-independentXOR of
FNV-1a(node_id, ip). A member whose computed digest differs has missed aNODE_ASSIGN(whole peer, or just a node-id) and pulls a rate-limited RESYNC;the master coalesces a burst into one re-broadcast. This is the multicast
counterpart to the 1:1 ACK path.
sequenceDiagram autonumber participant Mem as .192 member participant M as .193 master M-)Mem: MASTER_ALIVE + digest = count + XOR hash Note over Mem: local digest != master's, missed a NODE_ASSIGN Mem-->>M: RESYNC (unicast, rate-limited) M-)Mem: NODE_ASSIGN (all peers) + MEMBER_LIST (re-broadcast) Note over Mem: digests match again5. Reliable handshake — ACK + bounded retransmit
The 1:1 handshake rides unreliable UDP. A lost
KEY_GRANTused to cost the joinerits whole join window; now the master retransmits until ACKed (bounded), so a
single loss heals in milliseconds.
sequenceDiagram autonumber participant J as .191 joiner participant M as .193 master J-)M: JOIN_REQ M-->>J: KEY_GRANT (unicast) x lost Note over M: no ACK within retransmit timeout M-->>J: KEY_GRANT (retransmit) J-->>M: ACK Note over J,M: recovered in ms, not a full join window6. Graceful leave (GOODBYE)
A departing node multicasts
GOODBYE. Survivors re-elect locally off the sameevent — no MEMBER_LIST re-broadcast (that O(N), fragmenting packet was dropped from
the failover path);
MASTER_ALIVEre-asserts and the digest reconciles stragglers.sequenceDiagram autonumber participant D as .191 leaving participant B as .192 backup participant M as .193 master D-)M: GOODBYE (mcast) D-)B: GOODBYE (mcast) Note over B,M: local re-election off the same GOODBYE - no re-broadcast Note over M: still master, 2 members - no role change M-)B: MASTER_ALIVE + digest (count now 2)7. Master failover (crash)
The master goes silent; members detect the missed
MASTER_ALIVEkeepalives and runthe same deterministic election locally — the backup (highest-IP survivor)
promotes and asserts via
MASTER_ALIVE.sequenceDiagram autonumber participant Me as .191 member participant B as .192 backup participant M as .193 master Note over M: master crashes x Note over Me,B: MASTER_ALIVE keepalives stop -> timeout Note over Me,B: identical local re-election -> highest-IP survivor wins Note over B: .192 promotes to master B-)Me: MASTER_ALIVE (new master asserts) + digest Me-->>B: ALIVE (unicast to new master) Note over Me,B: converged - .192 master, .191 member8. Graceful master handoff (KEY_HANDOFF)
On a planned master shutdown the outgoing master hands the session key to its
successor sealed to the successor's long-lived X25519 pubkey (
crypto_box_seal,learned from ALIVE) — so the successor takes over without a rejoin.
sequenceDiagram autonumber participant S as .192 successor participant M as .193 master leaving Note over M: planned shutdown M-->>S: KEY_HANDOFF (unicast, session key sealed to S's pubkey) M-)S: GOODBYE (mcast) Note over S: already holds the key -> promotes with no rejoin S-)S: MASTER_ALIVE (asserts as master)9. Wrong-password rejection
A node with the wrong password cannot produce a bootstrap-valid
JOIN_REQ. Afterrepeated failures from one source the master emits an authenticated
JOIN_REJECT;independently, a genuinely-wrong-password joiner self-terminates after its defer
budget rather than forming a lone split-brain master.
sequenceDiagram autonumber participant R as .200 wrong password participant M as .193 master R-)M: JOIN_REQ (bootstrap AEAD fails to authenticate) Note over M: repeated bootstrap-decrypt failures from .200 M-->>R: JOIN_REJECT (unicast, GCM-authenticated) Note over R: in NODE_NEW state -> exit(-1), no split brain10. Split-brain merge (MASTER_BEACON)
If a partition heals and two masters exist, each periodically emits a
bootstrap-keyed
MASTER_BEACON(readable across different session keys) carryingits
member_count. The inferior master (smaller count, IP tiebreak) yields andrejoins the superior one.
sequenceDiagram autonumber participant A as .191 master, 1 member participant B as .193 master, 3 members Note over A,B: partition heals - two masters B-)A: MASTER_BEACON (bootstrap-keyed, member_count=3) A-)B: MASTER_BEACON (member_count=1) Note over A: inferior (fewer members) -> demote A-)B: JOIN_REQ (rejoin the superior master) B-->>A: KEY_GRANT + NODE_ASSIGN + MEMBER_LIST (unicast) Note over A,B: single cluster, one master11. Shared-port unicast demux (multi-cluster on one node)
When several controller clusters on one node share a multicast port (distinct
groups), the kernel demuxes multicast by group but unicast by port alone — a 1:1
reply can land on the wrong sibling's socket. The receiver recovers it: on a
cluster_idmismatch it forwards the still-encrypted datagram to the correctlocal cluster's worker via
ipc_send_rpc(), which decrypts it with its own key.sequenceDiagram autonumber participant P as peer (cluster 2) participant W1 as worker cluster 1 participant W2 as worker cluster 2 P-->>W1: unicast reply for cluster 2 (arrives on wrong socket) Note over W1: cleartext cluster_id=2 != 1 W1->>W2: ipc_send_rpc(still-encrypted datagram) Note over W2: decrypt with cluster-2 key, process normally Note over W1,W2: forwarded once, never re-forwarded - no loopComplexity impact
Module parameters
clusterid=(required),multicast=A.B.C.D:PORT(required),password=(optional, overrides the globalpassword),bin_socket=bin:IP:PORT(optional, required only when multiple clusters are defined),manage_shtags=0|1(optional, overrides the globalmanage_shtags). Repeat for multiple simultaneous clusters.my_ipinterface.interfacemy_ipis set.query_time5passwordcluster)3eCrEt*5629(change in production)manage_shtagscluster)1$shtag()changes on managed clusters). When 0, sharing tags behave exactly as stockclustererand are left to scripts/MI.master_stickinesscluster)11(sticky): a live master keeps its role when a higher-IP node joins — only the backup slot changes.0: pure highest-IP election — a higher-IP node takes over as soon as it appears.on_config_mismatchreject|warn|adoptrejectmanage_shtags/master_stickiness/query_timediffer from the running cluster's:rejectrefuses the join (JOIN_REJECT),warnlogs once and allows it,adoptmakes the joiner take the cluster's values.Compatibility & deployment
RESYNCtype, ALIVE/MASTER_ALIVElayout, unicast routing). All nodes of a cluster must run this build — old and
new cannot interoperate; deploy with a coordinated cutover. A node that doesn't
understand the digest simply advertises none and is never asked to resync.
cc_→cl_ctr_/CC_→CL_CTR_rename (functions, types,macros); the KDF label and bootstrap salt were renamed too, which changes the
derived keys — another reason for the coordinated cutover. Wire magic bytes
(
0xCC…) are unchanged.Testing
Validated on netns rigs and a live 6-node test cluster (2 controller clusters):
cold start → deterministic single master; single / chained / rapid-double failover;
higher-IP join → sticky backup; two clusters sharing port 3333 → both converge with
misdelivered unicast correctly re-routed; wrong-password node rejected without split
brain; staged stop→rejoin captured on the wire confirming the unicast join path.