Skip to content

feat(server): zero-downtime restart via socket handoff - #415

Merged
gg582 merged 1 commit into
devfrom
feat/socket-handoff
Oct 18, 2026
Merged

gg582 merged 1 commit into
devfrom
feat/socket-handoff

Conversation

@gg582

@gg582 gg582 commented Oct 18, 2026

Copy link
Copy Markdown
Member

Closes #405

Summary

Zero-downtime restart via SO_REUSEPORT socket handoff: the new process binds the same port while the old one still serves, the old process is then signalled (SIGTERM/SIGINT) into the existing graceful-shutdown drain, and clients never see the port refuse a connection.

The kernel capability was already there — cwist_make_socket_ipv4() set SO_REUSEPORT unconditionally — but it was an undocumented side effect of the multi-worker fork design. This PR makes it an explicit, documented contract:

  • CWIST_REUSEPORT knob (default on; 0/false disables) now gates SO_REUSEPORT at every listen-socket bind site: TCP (cwist_make_socket_ipv4), the C1M accept path (async_server.c), and both HTTP/3 UDP binds (single-port and multiport). Disabling restores bind()-fails-on-conflict as a guard against accidental double starts; since worker processes re-bind after fork, CWIST_REUSEPORT=0 requires CWIST_WORKERS=1 (documented).
  • tests/test_socket_handoff.c: the full handoff sequence as a test (see below).
  • docs/deployment/zero-downtime-restart.md (+ Korean mirror, runtime-config rows EN/KO): handoff sequence, kernel constraints, systemd integration, HTTP/3 behavior.

Design notes

  • Why SO_REUSEPORT over fd passing: no handoff channel, no supervisor protocol, no trust between generations; it composes with the existing forked-worker model, which already relies on reuseport rebinds after fork.
  • Handoff protocol: start new process (bind succeeds via reuseport; kernel hashes new flows across both) → SIGTERM the old one (CWIST closes its listen sockets at once, so new flows now hash to the new process only) → old finishes in-flight requests up to CWIST_DRAIN_TIMEOUT and exits; idle keep-alive connections get a graceful FIN, and clients with standard stale-keep-alive retry land on the new process with no failed request.
  • Kernel constraint: Linux ≥ 3.9 for TCP SO_REUSEPORT; BSDs/macOS also have it. Where it doesn't exist the code compiles it out and a second bind fails as before.
  • UDP/HTTP-3 decision: the QUIC UDP socket also binds with reuseport, so the overlap works, but QUIC connection state (connection-ID keys in lsquic) cannot migrate between processes. Datagrams for the draining generation may arrive at the new one and are undecryptable there; clients recover via QUIC retransmission or a fresh connection (TLS session resumption works against the new process). TCP handoffs are lossless; HTTP/3 handoffs are lossless at the request level with a small share of connection resets. Documented in the deployment guide.
  • systemd: documented both patterns — overlap restart (Type=simple + deploy script; not systemctl restart, which reopens the downtime window) and socket activation (.socket unit holds the listener across restarts and crashes) as an alternative.

Test plan

tests/test_socket_handoff.c is a scaled-down local version of the C10k acceptance scenario (256 held keep-alive connections instead of 10k — full C10k is impractical in make test; stated here per the issue's allowance):

  1. Server process A serves; 256 keep-alive connections are opened and verified against A.
  2. Server process B binds the same port while A holds it (the overlap — fails outright without reuseport).
  3. SIGTERM to A; every held connection is then driven with another request: each is either answered by draining A or cleanly closed, in which case the client reconnects and must reach B. 0 requests failed (1 answered by draining A, 255 reconnected to B).
  4. After A exits 0, 64 fresh connections all reach B; B shuts down cleanly.

Results:

  • make test_socket_handoff — pass
  • make SANITIZE=address,undefined test_socket_handoff — pass, leak/UB-clean (children _exit; parent closes all fds)
  • Regression smoke, all pass with WERROR=1: test_listen_ex, test_shutdown, test_listen_signal_restore, test_classic_shutdown, test_multiport, test_https
  • make env-check — pass (new knob documented EN+KO)
  • make api-check — pass (no public API change; the knob helper lives in a private header)
  • python3 scripts/ci/check_test_wiring.py — pass

Possible CI note: the test uses 256+ fds and forked servers on an ephemeral port; it follows the same pattern as test_listen_ex.c/test_classic_shutdown.c, so environments that already run those should run this one.

CWIST listen sockets already set SO_REUSEPORT; make that an explicit,
documented contract for zero-downtime restarts:

- New CWIST_REUSEPORT knob (default on; 0/false disables) gates
  SO_REUSEPORT on the TCP listen socket (cwist_make_socket_ipv4), the
  C1M accept path, and both HTTP/3 UDP bind sites. Disabling restores
  bind() failure on port conflicts and requires CWIST_WORKERS=1.
- tests/test_socket_handoff.c: process B binds the port under 256 held
  keep-alive connections to A, A is SIGTERMed to drain, every request
  still gets a response (draining A, or B after reconnect), and all
  fresh connections reach B once A exits.
- docs/deployment/zero-downtime-restart.md (with Korean mirror): the
  handoff sequence, kernel >= 3.9 constraint, systemd socket activation
  as an alternative, and the UDP/HTTP-3 decision (QUIC connection state
  cannot migrate; clients recover via retransmit/retry).

Closes #405
@gg582
gg582 merged commit 535e0ff into dev Oct 18, 2026
26 checks passed
gg582 added a commit that referenced this pull request Oct 19, 2026
…415)

CWIST listen sockets already set SO_REUSEPORT; make that an explicit,
documented contract for zero-downtime restarts:

- New CWIST_REUSEPORT knob (default on; 0/false disables) gates
  SO_REUSEPORT on the TCP listen socket (cwist_make_socket_ipv4), the
  C1M accept path, and both HTTP/3 UDP bind sites. Disabling restores
  bind() failure on port conflicts and requires CWIST_WORKERS=1.
- tests/test_socket_handoff.c: process B binds the port under 256 held
  keep-alive connections to A, A is SIGTERMed to drain, every request
  still gets a response (draining A, or B after reconnect), and all
  fresh connections reach B once A exits.
- docs/deployment/zero-downtime-restart.md (with Korean mirror): the
  handoff sequence, kernel >= 3.9 constraint, systemd socket activation
  as an alternative, and the UDP/HTTP-3 decision (QUIC connection state
  cannot migrate; clients recover via retransmit/retry).

Closes #405
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant