Skip to content

fix(https): deferred-conn UAF and full-GC parking starvation (#344) - #346

Merged
gg582 merged 3 commits into
devfrom
fix/344-https-deferred-and-park
Oct 10, 2026
Merged

gg582 merged 3 commits into
devfrom
fix/344-https-deferred-and-park

Conversation

@gg582

@gg582 gg582 commented Oct 10, 2026 •

Copy link
Copy Markdown
Member

Fixes the fly.board crashes and the intermittent client timeouts that remained after f594b6c8 (#344). Each of the two commits addresses one problem, and each was reproduced before it was fixed.

1. Use-after-free on deferred HTTPS connections (04500d8f)

https_thread_handler reads conn->deferred after the handler returns. Once the dispatch ack is published, the async completion owns the conn (see the comment in async.c: "Foreign completion may free a after ack"). By the time the worker reads the flag, the completion has either:

  • closed the conn, so the worker reads freed memory, or
  • reset deferred and resubmitted the conn to the pool for keep-alive, so the worker closes a conn that is still queued and the next worker picks up freed memory. That second worker then crashes in SSL_shutdown, cwist_http_header_get, or https_connection_teardown, which are the crash sites seen in production.

If the completion lands before the ack, for example a cached response sent right after cwist_async_defer, it runs inside the ack on the worker itself. In that case the bug is deterministic, not a rare race.

Fix: cwist_async_defer() sets a thread-local handoff flag, and the pool worker checks that flag instead of the conn.

Evidence:

  • ASan run of fly.board under load: heap-use-after-free in https_pool_worker (8-byte write at offset 64 of a freed conn) and in https_thread_handler (read of deferred at offset 42 of an 88-byte conn that another pool thread had freed in https_connection_teardown).
  • New test test_https_async_defer_completes_before_ack. It fails without the fix: the keep-alive follow-up SSL_write fails because the worker closed the resubmitted conn.

2. No parking under full GC, so idle keep-alive starves the pool (2c2e3177)

https_park_enabled() returned false whenever full GC was on. An idle keep-alive connection then held its pool thread for the whole idle budget: 30 s for HTTP/1.1, 300 s for h2. oborona.zip has 1 core, 4 workers, and 1 pool thread per worker, so a few idle browser connections were enough to block every new connection.

Reproduced on an ASan fly.board with CWIST_WORKER_THREADS=1 and 8 idle TLS clients:

idle clients new connection, before after
8 × HTTP/1.1 2 of 5 timed out at 15 s, 1 took 12 s 5 of 5 under 7 ms
8 × h2 2 of 5 timed out at 15 s, 1 took 12 s 5 of 5 under 5 ms

What changed:

  • Idle HTTP/1.1: the conn holds only itself and its read buffer. Both are disowned to the connection registry at wrap time, so the conn can move to another thread unchanged.
  • Idle h2: before parking, h2_session_disown() removes the session's allocations from the parking thread's GC tracking: the session struct, the batch and CONTINUATION buffers, the HPACK table, and the hook context. Anything allocated after resume is disowned again at the next park. h2_can_park() already guarantees there are no streams, deferred frames, or partial header blocks.
  • Park-owned teardown (idle expiry and pool shutdown) untracks the conn from the connection registry first, so the tracking thread's exit sweep does not close it a second time.

Test: test_https_park_full_gc reruns test_https_park.c with full GC on, including the h2 cases. It fails without this change (the second connection cannot complete the TLS handshake).

3. h2 conns leaked after a deferred stream (a7524819, from review)

The handoff flag must be set only when the HTTP/1.1 completion takes the conn. An h2 stream's completion goes through h2_queue, and the h2 loop keeps the conn. Before this fix, if a session ended in the same run that deferred a stream, its fd, SSL, and conn were never closed. The old conn->deferred check had the same leak. Test: test_http2_conn_closed_after_deferred_stream.

Verification (all under ASan + UBSan)

  • Tests: test_https, test_https_park, test_https_park_full_gc, test_https_full_gc, test_conn_registry, test_http2, test_http2_prebuffer, test_grpc, test_websocket, test_https_metrics, test_shutdown, test_full_gc_sweep, test_full_gc_ownership_handoff, test_io_queue_full_gc, test_metrics. All pass.
  • fly.board (production env: CWIST_C1M_MODE=1, 4 workers, TLS + h2 + h3, full GC): 2 rounds of mixed wrk, h2load, ab, curl HEAD, and idle clients that park and resume. 0 ASan reports.
  • Graceful SIGTERM with parked h1 and h2 connections: clean shutdown, 0 ASan reports.

#345 candidates

gg582 added 3 commits October 10, 2026 23:27
https_thread_handler read conn->deferred after the handler returned, but
once the dispatch ack is published the async completion owns the conn and
may already have closed it (use-after-free) or reset deferred and
resubmitted it to the pool, so the worker closed a conn that was queued
for keep-alive and the next worker used freed memory. A completion that
lands before the ack (fly.board's cached responses) runs inside the ack
on the worker itself, which makes this deterministic, not a rare race.

cwist_async_defer() now records the handoff in a thread-local flag that
the pool worker checks instead of the conn. Caught by ASan on fly.board
(heap-use-after-free in https_pool_worker / https_thread_handler); the
new test_https case reproduces it.
Parking was disabled whenever full GC was on, so an idle keep-alive
connection held its pool thread for the whole idle budget (30 s HTTP/1.1,
300 s h2). On a small box (oborona.zip: 1 core, 4 workers x 1 pool
thread) a handful of idle browser connections starved every new
connection, which clients saw as ERR_TIMED_OUT.

- An idle HTTP/1.1 conn holds only the conn and its read buffer, both
  disowned to the connection registry at wrap time, so it can move
  threads as is.
- An idle h2 session disowns its graph (session, batch and CONTINUATION
  buffers, HPACK table, hook context) from the parking thread before it
  parks; allocations made after resume are disowned again at the next
  park.
- Park-owned teardown (idle expiry, pool shutdown) untracks the conn from
  the connection registry first, so the tracking thread's exit sweep does
  not close it a second time.

test_https_park_full_gc reruns test_https_park.c with full GC on; it fails
without this change (the second connection cannot even handshake).
The handoff flag was set for any deferred request on an HTTPS conn,
including h2 streams, whose completion goes through h2_queue while the h2
loop keeps the conn. When the session ended in the same pool-thread run
that deferred a stream (e.g. GOAWAY right behind the request), the worker
skipped the close and leaked the fd, SSL and conn. The old conn->deferred
check leaked the same way. Only an HTTP/1.1 completion takes the conn, so
only it sets the flag now.

test_http2_conn_closed_after_deferred_stream fails without this (the
server never closes the connection).

@gg582 gg582 left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 04500d8f / 2c2e3177: one problem found.

h2 connections whose stream was deferred are never closed. cwist_async_defer() sets the new thread-local flag whenever req->https_conn is set, and h2 streams set it too (http2.c:3101). However, an h2 stream's completion goes through h2_queue, and the h2 loop still owns the conn. cwist_http2_serve_connection_ex frees only the session when it returns, so https_thread_handler is the only place the conn gets closed. If the session ends in the same pool-thread run that deferred a stream (for example, a client GOAWAY right behind the request), the worker sees deferred == true, skips the close, and the fd, SSL, and conn leak.

The old !conn->deferred check had the same leak (h2 never resets conn->deferred), so this is not a regression. It is still a real leak, and this PR touches exactly that line.

Reproduction: an h2 request to a deferred route followed immediately by GOAWAY. The server never closes the TCP connection, and wait_closed times out.

Suggested fix: set the flag only for HTTP/1.1, i.e. if (!req->h2_queue), since only that completion takes the conn. Add a regression case to test_https_park.c; it runs both with and without full GC.

@gg582

gg582 commented Oct 10, 2026

Copy link
Copy Markdown
Member Author

Agreed. I confirmed it with the reproduction above before fixing it.

Fixed in the latest commit (fix(https): close h2 conns whose stream was deferred):

  • cwist_async_defer() now sets the handoff flag only when !req->h2_queue. The header doc is updated to match.
  • New case test_http2_conn_closed_after_deferred_stream sends a deferred h2 request with GOAWAY right behind it, then asserts that the server closes the connection. It fails without the fix and passes with it.

Re-run under ASan + UBSan: test_https_park, test_https_park_full_gc (which now includes the new case under full GC), test_https, test_http2. All pass.

No other findings, so this is ready to merge once CI is green.

@gg582
gg582 merged commit 8e4e159 into dev Oct 10, 2026
18 checks passed
@gg582

gg582 commented Oct 10, 2026

Copy link
Copy Markdown
Member Author

That GC was hard. Even I made that GC it's stupid(and quite fast)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant