Skip to content

feat(Pool): acquire timeout, borrow validation and connection age bounds - #575

Open
Yaraslaut wants to merge 5 commits into
masterfrom
feat/562-pool-health
Open

feat(Pool): acquire timeout, borrow validation and connection age bounds#575
Yaraslaut wants to merge 5 commits into
masterfrom
feat/562-pool-health

Conversation

@Yaraslaut

Copy link
Copy Markdown
Member

Closes #562.

What the issue asked for

DataMapper/Pool.hpp had a more sophisticated waiter model than its peers but none of the operational controls that keep a pool healthy in production: no acquire timeout, no validation on borrow, no idle eviction, no max lifetime, and no retry when a broken connection is checked out. The consequences called out were a connection killed by a firewall or a failover being handed to the next caller, and an exhausted BoundedWait pool blocking the calling thread forever with no diagnostic.

What changed

All four controls live in PoolConfig, which is a non-type template parameter, so they are resolved at compile time. Every added member is defaulted — existing configurations compile unchanged.

Acquire timeout. Acquire(std::chrono::milliseconds) returns std::expected<PooledDataMapper, PoolError>. A timed-out acquirer removes itself from the waiter FIFO, so a connection returned afterwards reaches the next real waiter rather than being handed into a dead node and lost. wait_for evaluates its predicate under the mutex and reports the final value, so a false result proves the node is still Parked and no hand-off can be in flight — the de-registration is race-free rather than merely unlikely to race. The overload also exists on BoundedOverflow and UnboundedGrow, where it always succeeds, so generic code can take a timeout without knowing the strategy.

AcquireAsync deliberately has no timeout overload: it suspends a coroutine rather than occupying a thread, and there is no timer infrastructure in Async/ to expire it against. That overlaps #553.

Validation on borrow (validateOnBorrow, default Yes). Checks SqlConnection::IsAlive() before a connection leaves the pool, discarding a dead one and serving the caller from the next idle or a fresh connection — this is also the "retry on broken connection" item.

maxIdleTimeMs retires a connection that sat idle past the bound. This is not redundant with validation: IsAlive() reads the driver-local SQL_ATTR_CONNECTION_DEAD attribute, and drivers commonly only set it once an operation has already failed, so the half-open socket left behind when a firewall drops a flow without FIN or RST still passes the check. An idle bound set below the firewall's timeout removes that failure mode instead of trying to detect it.

maxLifetimeMs retires by total age, counted from creation and surviving checkout. This targets connections that are alive but no longer appropriate — after a failover or a rolling restart a pooled connection stays bound to the old node, reports itself healthy, and nothing else in the pool would ever move it.

Both bounds default to disabled. A default recycle window would silently change behaviour for every existing deployment, and the right value depends on the infrastructure the connections traverse. They are millisecond counts rather than std::chrono::duration because std::chrono::duration keeps its representation private and so is not a structural type; PoolConfig::MaxIdleTime() / MaxLifetime() read them back as durations.

Design notes

Retirement is lazy. Expired connections are dropped as they leave or enter the idle set. No housekeeping thread is added, so the destructor contract ("the pool must outlive every acquirer") is untouched and no Pool — including the process-wide GlobalDataMapperPool and every short-lived test pool — grows a thread. The trade-off, documented in docs/connection-pool.md: a pool that goes completely idle keeps its sockets until the next Acquire. Correctness holds regardless, since an expired connection is discarded rather than handed out, but the pool is not a mechanism for releasing connections during quiet periods.

One deliberate bypass. A direct hand-off to a parked BoundedWait waiter skips both the bounds and the liveness check. A waiter blocks on a predicate that only a hand-off satisfies, so retiring the connection there would strand it, and building a replacement means a DataMapper construction that can throw inside a noexcept Return. Such a connection was in active use moments earlier and is checked normally the next time it comes out of the idle set.

Structural change. Pooled connections now carry creation and idle timestamps, so the pool stores an Entry rather than a bare unique_ptr<DataMapper>; this threads through PooledDataMapper, WaiterNode and the async awaitable. PooledDataMapper's public surface (operator->, Get()) is unchanged. Retired entries are moved out and destroyed after the pool mutex is released, so ODBC disconnects never run under the lock.

Incidental fix. BoundedWait incremented _checkedOut before constructing the DataMapper, so a failing connect permanently consumed a slot. The increment now happens only once the connection stands up.

Risk assessment

  • Behaviour change: validateOnBorrow defaults to Yes, so every pool now performs one SQLGetConnectAttr per borrow. It is driver-local with no round trip, and it can only cause a dead connection to be replaced rather than handed out. Set it to ValidateOnBorrow::No to opt out.
  • Per-DBMS: the logic is dialect-independent — no SqlQueryFormatter involvement and no SQL emitted. SQL_ATTR_CONNECTION_DEAD is core ODBC.
  • Threading: the locking protocol is unchanged; the new state is per-entry timestamps read under the same mutex. The one new wait is wait_for on the existing per-waiter CV.
  • ABI/API: additive. No new virtual was added to SqlLogger (that would have broken downstream implementors), so retirement reuses the existing hooks.
  • Performance: with the default config (TracksTime == false) no clock is ever read — the timestamp fields are written as default-constructed values and never compared. if constexpr compiles out every disabled bound.

Testing

src/tests/DataMapper/PoolHealthTests.cpp, 10 cases. The time-based ones use an injected clock (SetClock, under BUILD_TESTS, alongside the existing IdleCount/WaiterCount seams) rather than sleeping.

The health tests avoid weak assertions about object identity: they close a pooled connection so it is observably dead, disable validation, and then assert on liveness of what comes back — a live connection proves the bound retired the dead one and built a replacement. One test asserts the opposite direction (validation disabled really does hand the broken connection back), pinning the baseline the others rely on.

Covered: timeout on an exhausted pool; no ghost waiter left behind after a timeout; timeout satisfied by a concurrent return; timeout always succeeding on non-waiting strategies; dead connection replaced with validation on; dead connection returned with validation off; idle bound; lifetime bound retiring on return; lifetime measured as total age across repeated churn rather than reset per return; BoundedWait releasing capacity when it retires.

Results

Full suite dbtool suite
sqlite3 1411 passed, 1 skipped pass
mssql2022 (Docker) 1409 passed, 3 skipped pass
postgres (Docker 16) 1410 passed, 2 skipped pass

Skips are pre-existing UNSUPPORTED_DATABASE cases. [Pool] (28 cases, including the pre-existing pool tests) was additionally run 40× on sqlite3 and 20× on postgres to check for flakiness: zero failures.

Compilers. Built and run under clang-debug (ASan + UBSan, -Werror), clean. gcc-release is a Linux-only preset and this machine's g++ is Apple clang, so a full GCC build was not possible; instead GCC 15 was used for a front-end check of Pool.hpp instantiating all three growth strategies against six PoolConfig combinations — clean under -Wall -Wextra -Wconversion. That covers the GCC-specific risks that matter here (structural-type acceptance of the extended PoolConfig as an NTTP, two-phase lookup in the new templates) but not codegen differences, so CI's GCC release legs are still the real check.

clang-tidy did not run — the clang-debug preset reported [clang-tidy] Not found. on this machine.

Not touched: SqlBackup/ConnectionPool.cpp has the same unbounded cv.wait. The issue cites it as evidence but scopes the fix to PoolConfig, and it is an internal detail of the backup pipeline, so it is left for a follow-up.

🤖 Generated with Claude Code

… bounds (#562)

The pool had a sophisticated waiter model but none of the operational
controls that keep a pool healthy in production: an exhausted BoundedWait
pool parked the calling thread forever with no diagnostic, and a connection
killed by a firewall, a failover or a server restart was handed to the next
caller as-is.

Four controls, all configured through PoolConfig:

- Acquire(timeout) returns std::expected<PooledDataMapper, PoolError>.
  A timed-out acquirer de-registers itself from the waiter FIFO, so a
  connection returned afterwards goes to the next real waiter instead of
  being lost. wait_for evaluates its predicate under the mutex, so a false
  result proves the node is still parked and the de-registration is
  race-free. The overload also exists on the non-waiting strategies, where
  it always succeeds, so call sites need not know the strategy.

- validateOnBorrow (default: Yes) checks SqlConnection::IsAlive() before
  handing a connection out, discarding a dead one and serving the caller
  from the next idle or a fresh connection.

- maxIdleTimeMs retires a connection that sat idle too long. This covers
  what validation cannot: IsAlive() reads the driver-local
  SQL_ATTR_CONNECTION_DEAD attribute, and drivers commonly only set it
  after an operation has already failed, so a half-open socket left by a
  firewall dropping the flow still passes the check.

- maxLifetimeMs retires a connection by total age, counted from creation
  and surviving checkout. This retires connections that are alive but no
  longer appropriate — after a failover a pooled connection stays bound to
  the old node and nothing else would ever move it.

Both bounds default to disabled; a default recycle window would silently
change behaviour for existing deployments. They are millisecond counts
rather than std::chrono::duration because PoolConfig is a non-type template
parameter and duration is not a structural type.

Retirement is lazy — checked when a connection leaves or enters the idle
set — so no thread is added and the destructor contract is unchanged. A
direct hand-off to a parked BoundedWait waiter deliberately bypasses both
the bounds and the liveness check: retiring there would strand a waiter
that only a hand-off can wake, and building a replacement would mean a
throwing connect on a noexcept path.

Pooled connections now carry creation and idle timestamps, so the pool
stores an Entry rather than a bare unique_ptr<DataMapper>. Retired entries
are destroyed after the pool mutex is released, keeping ODBC disconnects
off the lock.

Also fixes a pre-existing capacity leak: BoundedWait incremented its
checked-out count before constructing the DataMapper, so a failing connect
permanently consumed a slot.

PoolConfig gains defaulted members only, so existing configurations
continue to compile unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Yaraslaut
Yaraslaut requested a review from a team as a code owner August 18, 2026 16:15
@github-actions github-actions Bot added documentation Improvements or additions to documentation Data Mapper tests labels Aug 18, 2026
…sed connection

Two CI failures from the pool health change.

docs/connection-pool.md was never added to the Doxygen input list, so the
links to it from README.md and docs/async.md could not be resolved and the
documentation-coverage job failed with LIGHTWEIGHT_DOCS_WARN_AS_ERROR.

The bigger one was a segfault under valgrind, surfaced by the new pool
tests: SqlConnection::Close() frees the DBC handle, and the driver manager
invalidates every statement allocated from that connection at the same
moment. ~SqlStatement then called SQLFreeHandle on the dangling statement
handle, reading released driver memory. The new validation-on-borrow test
provoked this by closing a pooled connection and letting the pool retire
(and so destroy) the DataMapper.

This is a pre-existing hazard rather than a regression -- any code closing a
connection while a statement object built from it is still in scope hit it --
but the retirement paths added for #562 make it far easier to reach, since
the pool now destroys connections on its own. ~SqlStatement now skips the
free when its connection is already closed, matching the null-check that
SqlStatement::IsAlive() already performs on the same pointer. Normal
destruction is unaffected: DataMapper declares _connection before _stmt, so
the statement is always destroyed while its connection is still open.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Aug 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Yaraslaut and others added 3 commits August 19, 2026 19:31
…ence

Applying /code-review findings on this branch.

AcquireAsync claimed the BoundedWait capacity slot before calling
MakeEntry(), so a throwing connect leaked that slot permanently. The
synchronous AcquireReadyLocked was deliberately fixed in this same diff;
the async path kept the inverted order. Reorder to match.

The new _clock data member was guarded by BUILD_TESTS, which is PRIVATE to
LightweightTest, while Pool<DefaultPoolConfig> is instantiated inside the
library itself via GlobalDataMapperPool(). That gives the class two
different layouts in one program -- latent only because no test currently
touches the global pool, and an out-of-bounds read the moment one does.
Make _clock and SetClock unconditional as the time-injection seam AGENT.md
calls for; IdleCount/WaiterCount stay gated as before.

The connection-pool doc's headline example for the timeout overload did not
compile: mapper->Query<User>() resolves through the std::expected to
PooledDataMapper::Query and needs one more dereference.

Also add the <ranges> includes that Pool.hpp and PoolHealthTests.cpp were
getting only transitively for std::views::iota, and refresh the ReturnLocked
comment claiming a synchronous waiter is never abandoned -- the new
Acquire(timeout) abandons one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…c creation

Three lines Codecov flagged, each a path the existing tests structurally avoid:

- Every time-based test injects a clock, so NowIfTracking()'s real-clock branch
  -- the one every deployment takes -- was never executed. A pool configured
  with a lifetime far longer than the test now runs without an injected clock.
- Lifetime retirement was only tested for the Bounded* strategies.
  UnboundedGrow otherwise keeps every returned connection unconditionally, which
  makes it the strategy where the lifetime bound is the only thing that can
  retire one, so the retire-outside-the-lock arm needed its own case.
- AcquireAsync's own creation path: every async test starts from a pool with
  pre-created entries, so the coroutine always found one idle and never built
  one itself. initialSize = 0 forces it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The async fresh-connection test used BoundedOverflow, which does no capacity
accounting, leaving the `++_checkedOut` that BoundedWait needs untaken. The new
case runs the same path under BoundedWait and then checks the accounting really
balanced, by acquiring up to capacity afterwards without blocking -- a slot
leaked here would permanently shrink the pool, which a smoke test would not show.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Core API Data Mapper documentation Improvements or additions to documentation tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Connection pool has no acquire timeout, liveness check, idle eviction or max lifetime

1 participant