feat(Pool): acquire timeout, borrow validation and connection age bounds - #575
Open
Yaraslaut wants to merge 5 commits into
Open
feat(Pool): acquire timeout, borrow validation and connection age bounds#575Yaraslaut wants to merge 5 commits into
Yaraslaut wants to merge 5 commits into
Conversation
… bounds (#562) The pool had a sophisticated waiter model but none of the operational controls that keep a pool healthy in production: an exhausted BoundedWait pool parked the calling thread forever with no diagnostic, and a connection killed by a firewall, a failover or a server restart was handed to the next caller as-is. Four controls, all configured through PoolConfig: - Acquire(timeout) returns std::expected<PooledDataMapper, PoolError>. A timed-out acquirer de-registers itself from the waiter FIFO, so a connection returned afterwards goes to the next real waiter instead of being lost. wait_for evaluates its predicate under the mutex, so a false result proves the node is still parked and the de-registration is race-free. The overload also exists on the non-waiting strategies, where it always succeeds, so call sites need not know the strategy. - validateOnBorrow (default: Yes) checks SqlConnection::IsAlive() before handing a connection out, discarding a dead one and serving the caller from the next idle or a fresh connection. - maxIdleTimeMs retires a connection that sat idle too long. This covers what validation cannot: IsAlive() reads the driver-local SQL_ATTR_CONNECTION_DEAD attribute, and drivers commonly only set it after an operation has already failed, so a half-open socket left by a firewall dropping the flow still passes the check. - maxLifetimeMs retires a connection by total age, counted from creation and surviving checkout. This retires connections that are alive but no longer appropriate — after a failover a pooled connection stays bound to the old node and nothing else would ever move it. Both bounds default to disabled; a default recycle window would silently change behaviour for existing deployments. They are millisecond counts rather than std::chrono::duration because PoolConfig is a non-type template parameter and duration is not a structural type. Retirement is lazy — checked when a connection leaves or enters the idle set — so no thread is added and the destructor contract is unchanged. A direct hand-off to a parked BoundedWait waiter deliberately bypasses both the bounds and the liveness check: retiring there would strand a waiter that only a hand-off can wake, and building a replacement would mean a throwing connect on a noexcept path. Pooled connections now carry creation and idle timestamps, so the pool stores an Entry rather than a bare unique_ptr<DataMapper>. Retired entries are destroyed after the pool mutex is released, keeping ODBC disconnects off the lock. Also fixes a pre-existing capacity leak: BoundedWait incremented its checked-out count before constructing the DataMapper, so a failing connect permanently consumed a slot. PoolConfig gains defaulted members only, so existing configurations continue to compile unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sed connection Two CI failures from the pool health change. docs/connection-pool.md was never added to the Doxygen input list, so the links to it from README.md and docs/async.md could not be resolved and the documentation-coverage job failed with LIGHTWEIGHT_DOCS_WARN_AS_ERROR. The bigger one was a segfault under valgrind, surfaced by the new pool tests: SqlConnection::Close() frees the DBC handle, and the driver manager invalidates every statement allocated from that connection at the same moment. ~SqlStatement then called SQLFreeHandle on the dangling statement handle, reading released driver memory. The new validation-on-borrow test provoked this by closing a pooled connection and letting the pool retire (and so destroy) the DataMapper. This is a pre-existing hazard rather than a regression -- any code closing a connection while a statement object built from it is still in scope hit it -- but the retirement paths added for #562 make it far easier to reach, since the pool now destroys connections on its own. ~SqlStatement now skips the free when its connection is already closed, matching the null-check that SqlStatement::IsAlive() already performs on the same pointer. Normal destruction is unaffected: DataMapper declares _connection before _stmt, so the statement is always destroyed while its connection is still open. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…ence Applying /code-review findings on this branch. AcquireAsync claimed the BoundedWait capacity slot before calling MakeEntry(), so a throwing connect leaked that slot permanently. The synchronous AcquireReadyLocked was deliberately fixed in this same diff; the async path kept the inverted order. Reorder to match. The new _clock data member was guarded by BUILD_TESTS, which is PRIVATE to LightweightTest, while Pool<DefaultPoolConfig> is instantiated inside the library itself via GlobalDataMapperPool(). That gives the class two different layouts in one program -- latent only because no test currently touches the global pool, and an out-of-bounds read the moment one does. Make _clock and SetClock unconditional as the time-injection seam AGENT.md calls for; IdleCount/WaiterCount stay gated as before. The connection-pool doc's headline example for the timeout overload did not compile: mapper->Query<User>() resolves through the std::expected to PooledDataMapper::Query and needs one more dereference. Also add the <ranges> includes that Pool.hpp and PoolHealthTests.cpp were getting only transitively for std::views::iota, and refresh the ReturnLocked comment claiming a synchronous waiter is never abandoned -- the new Acquire(timeout) abandons one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…c creation Three lines Codecov flagged, each a path the existing tests structurally avoid: - Every time-based test injects a clock, so NowIfTracking()'s real-clock branch -- the one every deployment takes -- was never executed. A pool configured with a lifetime far longer than the test now runs without an injected clock. - Lifetime retirement was only tested for the Bounded* strategies. UnboundedGrow otherwise keeps every returned connection unconditionally, which makes it the strategy where the lifetime bound is the only thing that can retire one, so the retire-outside-the-lock arm needed its own case. - AcquireAsync's own creation path: every async test starts from a pool with pre-created entries, so the coroutine always found one idle and never built one itself. initialSize = 0 forces it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The async fresh-connection test used BoundedOverflow, which does no capacity accounting, leaving the `++_checkedOut` that BoundedWait needs untaken. The new case runs the same path under BoundedWait and then checks the accounting really balanced, by acquiring up to capacity afterwards without blocking -- a slot leaked here would permanently shrink the pool, which a smoke test would not show. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #562.
What the issue asked for
DataMapper/Pool.hpphad a more sophisticated waiter model than its peers but none of the operational controls that keep a pool healthy in production: no acquire timeout, no validation on borrow, no idle eviction, no max lifetime, and no retry when a broken connection is checked out. The consequences called out were a connection killed by a firewall or a failover being handed to the next caller, and an exhaustedBoundedWaitpool blocking the calling thread forever with no diagnostic.What changed
All four controls live in
PoolConfig, which is a non-type template parameter, so they are resolved at compile time. Every added member is defaulted — existing configurations compile unchanged.Acquire timeout.
Acquire(std::chrono::milliseconds)returnsstd::expected<PooledDataMapper, PoolError>. A timed-out acquirer removes itself from the waiter FIFO, so a connection returned afterwards reaches the next real waiter rather than being handed into a dead node and lost.wait_forevaluates its predicate under the mutex and reports the final value, so afalseresult proves the node is stillParkedand no hand-off can be in flight — the de-registration is race-free rather than merely unlikely to race. The overload also exists onBoundedOverflowandUnboundedGrow, where it always succeeds, so generic code can take a timeout without knowing the strategy.AcquireAsyncdeliberately has no timeout overload: it suspends a coroutine rather than occupying a thread, and there is no timer infrastructure inAsync/to expire it against. That overlaps #553.Validation on borrow (
validateOnBorrow, defaultYes). ChecksSqlConnection::IsAlive()before a connection leaves the pool, discarding a dead one and serving the caller from the next idle or a fresh connection — this is also the "retry on broken connection" item.maxIdleTimeMsretires a connection that sat idle past the bound. This is not redundant with validation:IsAlive()reads the driver-localSQL_ATTR_CONNECTION_DEADattribute, and drivers commonly only set it once an operation has already failed, so the half-open socket left behind when a firewall drops a flow withoutFINorRSTstill passes the check. An idle bound set below the firewall's timeout removes that failure mode instead of trying to detect it.maxLifetimeMsretires by total age, counted from creation and surviving checkout. This targets connections that are alive but no longer appropriate — after a failover or a rolling restart a pooled connection stays bound to the old node, reports itself healthy, and nothing else in the pool would ever move it.Both bounds default to disabled. A default recycle window would silently change behaviour for every existing deployment, and the right value depends on the infrastructure the connections traverse. They are millisecond counts rather than
std::chrono::durationbecausestd::chrono::durationkeeps its representation private and so is not a structural type;PoolConfig::MaxIdleTime()/MaxLifetime()read them back as durations.Design notes
Retirement is lazy. Expired connections are dropped as they leave or enter the idle set. No housekeeping thread is added, so the destructor contract ("the pool must outlive every acquirer") is untouched and no
Pool— including the process-wideGlobalDataMapperPooland every short-lived test pool — grows a thread. The trade-off, documented indocs/connection-pool.md: a pool that goes completely idle keeps its sockets until the nextAcquire. Correctness holds regardless, since an expired connection is discarded rather than handed out, but the pool is not a mechanism for releasing connections during quiet periods.One deliberate bypass. A direct hand-off to a parked
BoundedWaitwaiter skips both the bounds and the liveness check. A waiter blocks on a predicate that only a hand-off satisfies, so retiring the connection there would strand it, and building a replacement means aDataMapperconstruction that can throw inside anoexceptReturn. Such a connection was in active use moments earlier and is checked normally the next time it comes out of the idle set.Structural change. Pooled connections now carry creation and idle timestamps, so the pool stores an
Entryrather than a bareunique_ptr<DataMapper>; this threads throughPooledDataMapper,WaiterNodeand the async awaitable.PooledDataMapper's public surface (operator->,Get()) is unchanged. Retired entries are moved out and destroyed after the pool mutex is released, so ODBC disconnects never run under the lock.Incidental fix.
BoundedWaitincremented_checkedOutbefore constructing theDataMapper, so a failing connect permanently consumed a slot. The increment now happens only once the connection stands up.Risk assessment
validateOnBorrowdefaults toYes, so every pool now performs oneSQLGetConnectAttrper borrow. It is driver-local with no round trip, and it can only cause a dead connection to be replaced rather than handed out. Set it toValidateOnBorrow::Noto opt out.SqlQueryFormatterinvolvement and no SQL emitted.SQL_ATTR_CONNECTION_DEADis core ODBC.wait_foron the existing per-waiter CV.SqlLogger(that would have broken downstream implementors), so retirement reuses the existing hooks.TracksTime == false) no clock is ever read — the timestamp fields are written as default-constructed values and never compared.if constexprcompiles out every disabled bound.Testing
src/tests/DataMapper/PoolHealthTests.cpp, 10 cases. The time-based ones use an injected clock (SetClock, underBUILD_TESTS, alongside the existingIdleCount/WaiterCountseams) rather than sleeping.The health tests avoid weak assertions about object identity: they close a pooled connection so it is observably dead, disable validation, and then assert on liveness of what comes back — a live connection proves the bound retired the dead one and built a replacement. One test asserts the opposite direction (validation disabled really does hand the broken connection back), pinning the baseline the others rely on.
Covered: timeout on an exhausted pool; no ghost waiter left behind after a timeout; timeout satisfied by a concurrent return; timeout always succeeding on non-waiting strategies; dead connection replaced with validation on; dead connection returned with validation off; idle bound; lifetime bound retiring on return; lifetime measured as total age across repeated churn rather than reset per return;
BoundedWaitreleasing capacity when it retires.Results
Skips are pre-existing
UNSUPPORTED_DATABASEcases.[Pool](28 cases, including the pre-existing pool tests) was additionally run 40× on sqlite3 and 20× on postgres to check for flakiness: zero failures.Compilers. Built and run under
clang-debug(ASan + UBSan,-Werror), clean.gcc-releaseis a Linux-only preset and this machine'sg++is Apple clang, so a full GCC build was not possible; instead GCC 15 was used for a front-end check ofPool.hppinstantiating all three growth strategies against sixPoolConfigcombinations — clean under-Wall -Wextra -Wconversion. That covers the GCC-specific risks that matter here (structural-type acceptance of the extendedPoolConfigas an NTTP, two-phase lookup in the new templates) but not codegen differences, so CI's GCC release legs are still the real check.clang-tidy did not run — the
clang-debugpreset reported[clang-tidy] Not found.on this machine.Not touched:
SqlBackup/ConnectionPool.cpphas the same unboundedcv.wait. The issue cites it as evidence but scopes the fix toPoolConfig, and it is an internal detail of the backup pipeline, so it is left for a follow-up.🤖 Generated with Claude Code