Repository navigation
fix(https): deferred-conn UAF and full-GC parking starvation (#344) - #346
Conversation
https_thread_handler read conn->deferred after the handler returned, but once the dispatch ack is published the async completion owns the conn and may already have closed it (use-after-free) or reset deferred and resubmitted it to the pool, so the worker closed a conn that was queued for keep-alive and the next worker used freed memory. A completion that lands before the ack (fly.board's cached responses) runs inside the ack on the worker itself, which makes this deterministic, not a rare race. cwist_async_defer() now records the handoff in a thread-local flag that the pool worker checks instead of the conn. Caught by ASan on fly.board (heap-use-after-free in https_pool_worker / https_thread_handler); the new test_https case reproduces it.
Parking was disabled whenever full GC was on, so an idle keep-alive connection held its pool thread for the whole idle budget (30 s HTTP/1.1, 300 s h2). On a small box (oborona.zip: 1 core, 4 workers x 1 pool thread) a handful of idle browser connections starved every new connection, which clients saw as ERR_TIMED_OUT. - An idle HTTP/1.1 conn holds only the conn and its read buffer, both disowned to the connection registry at wrap time, so it can move threads as is. - An idle h2 session disowns its graph (session, batch and CONTINUATION buffers, HPACK table, hook context) from the parking thread before it parks; allocations made after resume are disowned again at the next park. - Park-owned teardown (idle expiry, pool shutdown) untracks the conn from the connection registry first, so the tracking thread's exit sweep does not close it a second time. test_https_park_full_gc reruns test_https_park.c with full GC on; it fails without this change (the second connection cannot even handshake).
The handoff flag was set for any deferred request on an HTTPS conn, including h2 streams, whose completion goes through h2_queue while the h2 loop keeps the conn. When the session ended in the same pool-thread run that deferred a stream (e.g. GOAWAY right behind the request), the worker skipped the close and leaked the fd, SSL and conn. The old conn->deferred check leaked the same way. Only an HTTP/1.1 completion takes the conn, so only it sets the flag now. test_http2_conn_closed_after_deferred_stream fails without this (the server never closes the connection).
gg582
left a comment
There was a problem hiding this comment.
Review of 04500d8f / 2c2e3177: one problem found.
h2 connections whose stream was deferred are never closed. cwist_async_defer() sets the new thread-local flag whenever req->https_conn is set, and h2 streams set it too (http2.c:3101). However, an h2 stream's completion goes through h2_queue, and the h2 loop still owns the conn. cwist_http2_serve_connection_ex frees only the session when it returns, so https_thread_handler is the only place the conn gets closed. If the session ends in the same pool-thread run that deferred a stream (for example, a client GOAWAY right behind the request), the worker sees deferred == true, skips the close, and the fd, SSL, and conn leak.
The old !conn->deferred check had the same leak (h2 never resets conn->deferred), so this is not a regression. It is still a real leak, and this PR touches exactly that line.
Reproduction: an h2 request to a deferred route followed immediately by GOAWAY. The server never closes the TCP connection, and wait_closed times out.
Suggested fix: set the flag only for HTTP/1.1, i.e. if (!req->h2_queue), since only that completion takes the conn. Add a regression case to test_https_park.c; it runs both with and without full GC.
|
Agreed. I confirmed it with the reproduction above before fixing it. Fixed in the latest commit (
Re-run under ASan + UBSan: No other findings, so this is ready to merge once CI is green. |
|
That GC was hard. Even I made that GC it's stupid(and quite fast) |
Fixes the fly.board crashes and the intermittent client timeouts that remained after
f594b6c8(#344). Each of the two commits addresses one problem, and each was reproduced before it was fixed.1. Use-after-free on deferred HTTPS connections (
04500d8f)https_thread_handlerreadsconn->deferredafter the handler returns. Once the dispatch ack is published, the async completion owns the conn (see the comment inasync.c: "Foreign completion may free a after ack"). By the time the worker reads the flag, the completion has either:deferredand resubmitted the conn to the pool for keep-alive, so the worker closes a conn that is still queued and the next worker picks up freed memory. That second worker then crashes inSSL_shutdown,cwist_http_header_get, orhttps_connection_teardown, which are the crash sites seen in production.If the completion lands before the ack, for example a cached response sent right after
cwist_async_defer, it runs inside the ack on the worker itself. In that case the bug is deterministic, not a rare race.Fix:
cwist_async_defer()sets a thread-local handoff flag, and the pool worker checks that flag instead of the conn.Evidence:
heap-use-after-freeinhttps_pool_worker(8-byte write at offset 64 of a freed conn) and inhttps_thread_handler(read ofdeferredat offset 42 of an 88-byte conn that another pool thread had freed inhttps_connection_teardown).test_https_async_defer_completes_before_ack. It fails without the fix: the keep-alive follow-upSSL_writefails because the worker closed the resubmitted conn.2. No parking under full GC, so idle keep-alive starves the pool (
2c2e3177)https_park_enabled()returned false whenever full GC was on. An idle keep-alive connection then held its pool thread for the whole idle budget: 30 s for HTTP/1.1, 300 s for h2. oborona.zip has 1 core, 4 workers, and 1 pool thread per worker, so a few idle browser connections were enough to block every new connection.Reproduced on an ASan fly.board with
CWIST_WORKER_THREADS=1and 8 idle TLS clients:What changed:
h2_session_disown()removes the session's allocations from the parking thread's GC tracking: the session struct, the batch and CONTINUATION buffers, the HPACK table, and the hook context. Anything allocated after resume is disowned again at the next park.h2_can_park()already guarantees there are no streams, deferred frames, or partial header blocks.Test:
test_https_park_full_gcrerunstest_https_park.cwith full GC on, including the h2 cases. It fails without this change (the second connection cannot complete the TLS handshake).3. h2 conns leaked after a deferred stream (
a7524819, from review)The handoff flag must be set only when the HTTP/1.1 completion takes the conn. An h2 stream's completion goes through
h2_queue, and the h2 loop keeps the conn. Before this fix, if a session ended in the same run that deferred a stream, its fd, SSL, and conn were never closed. The oldconn->deferredcheck had the same leak. Test:test_http2_conn_closed_after_deferred_stream.Verification (all under ASan + UBSan)
test_https,test_https_park,test_https_park_full_gc,test_https_full_gc,test_conn_registry,test_http2,test_http2_prebuffer,test_grpc,test_websocket,test_https_metrics,test_shutdown,test_full_gc_sweep,test_full_gc_ownership_handoff,test_io_queue_full_gc,test_metrics. All pass.CWIST_C1M_MODE=1, 4 workers, TLS + h2 + h3, full GC): 2 rounds of mixed wrk, h2load, ab, curl HEAD, and idle clients that park and resume. 0 ASan reports.#345 candidates