Implement RTN23 idle-transport detection and the recovery spec points around it

[AndyTWF]

Implements RTN23 — dead-transport detection — and the recovery-path spec points that
auditing the constant it had been misusing turned up. Detection latency drops from 120s
to ~25s.

To keep the change set bounded, this is deliberately not a general spec update. Nothing outside the connection-recovery paths above is touched, and the protocol version is unchanged — protocol stays at 6 upstream and this SDK still sends v=2.

Where a clause on those paths has since been superseded, I've brought that one clause forward rather than implement the version it replaced. That is the only reason RTN14h appears here: the pre-6.1.0 resume semantics it replaces were the direct cause of several of the bugs above, so preserving them would have meant knowingly implementing behaviour the spec has already retired. Fast-forwarding is scoped to the clauses these fixes already have their hands on, not pursued for its own sake.

PUB-3824

Implemented (not previously present)

• RTN23a — disconnect a transport idle longer than maxIdleInterval + realtimeRequestTimeout. The monitor was measuring against connectionStateTtl.
• RTN23b — send heartbeats=true so protocol HEARTBEATs are contractual rather than an undocumented server default.
• CD2h — maxIdleInterval parsed from connectionDetails; previously never read.
• TO3l11 — realtimeRequestTimeout is now a public, validated ClientOptions property; it was internal.

Corrected

• RTN15c6 / RTN15c7 / RTN16f — continuation is now detected positively; a connection cleared per RTN15g reconnects with a new connectionId and no error, which the old test read as a continuation.
• RTN19a1 / RTN19a2 — serials kept on a successful resume, renumbered on a failed one, and publish order preserved across the requeue.
• RTN7b / RTN7d / RTN7e — RTL6c2 queue failed alongside RTL6c1; failures now report the reason for the state change.
• RTN24 — a connection update no longer churns channels (was 4 spurious protocol messages per reauth).
• RTL3d / RTL3d1 — reattach unconditional on entering CONNECTED, and ordered before external listeners.
• RTL3b / RTL3c / RTN11b / RTP5a — channels detach when the connection goes away, so presence members no longer survive a close.
• RTL5l — a channel detaches immediately whenever the connection is not CONNECTED.
• RTL11 — queued presence messages fail with an ErrorInfo rather than a bare exception.
• RTL15b2 — channelSerial no longer cleared on SUSPENDED.
• RTN14d — retryIn reports the delay actually waited.
• RTN14e — the suspend deadline is evaluated on every path into DISCONNECTED, including the token/auth paths that never suspended.
• RTB1 / RTB1a / RTB1b — backoff coefficients, jitter and the connectionStateTtl clamp.
• RTN15a / RTN15h3 — immediate reconnect for a non-token DISCONNECTED.
• RTN17 / RTN17j — which attempts consult the fallback domains: now all of them, timer-driven ones included. Previously host selection was skipped for those, so once the immediate retry budget was spent every attempt was pinned to the primary and a client whose primary datacenter was down could never reach a fallback at all. The immediate retry is also bounded (was ~65 attempts/sec) and connectivity is checked once per attempt rather than twice.
• RTN17j — the connectivity probe is now scoped to what the clause covers, "the use of an alternative host", and taken only when the chosen host is a fallback. It used to gate the immediate retry itself, so a reconnect that was going to stay on the primary waited out a probe nothing asked for, and a probe failing while the realtime endpoint was fine cancelled the reconnect outright — up to disconnectedRetryTimeout of downtime where RTN15h3 says reconnect now. The token-renewal path took no probe at all and could open a transport against another datacenter on the strength of an earlier failure; both paths now share one gated decision. ably-js draws the line in the same place.
• RTN17i — separately, what counts as a fallback-worthy failure has narrowed. A transport dropping out of CONNECTED is no longer grounds to move off the primary domain, so the first reconnect after a transient network blip returns to the primary instead of a fallback datacenter — RTN17i requires the primary be preferred "even if a previous connection attempt to that endpoint has failed", and RTN17f admits an exception only through RSC15l1 ("host unresolvable or unreachable"), which describes an attempt that never landed. A drop out of CONNECTED says the opposite: the host answered a moment ago. The fallbacks are deferred by one attempt, not lost — if the datacenter really has gone, that reconnect fails at connect time and the attempt after it moves off the primary. A server-sent DISCONNECTED carrying a 500–504 status is untouched by this, since RTN17f1 makes that fallback-worthy however it arrives.
• RTN11d — full reinitialisation on connect() from CLOSED, FAILED or CLOSING.
• RTN12b — close timeout is realtimeRequestTimeout, not a hardcoded 1s.
• RTN8d / RTN9d — key and id cleared only in the terminal states the clauses name, cleared before the state change is emitted, and cleared even when a transition throws.
• RTN21 — connectionDetails overrides only the attributes it carries.
• RTN24 / RTN19a — an UPDATE (a reauth, typically) no longer redrives the message queues. RTN19a is scoped to "when a transport is disconnected for any reason", and an update disconnects nothing: the transport that will ACK the in-flight messages is the one they went out on, so resending put a duplicate of each on the wire for Ably to discard by msgSerial. Invisible to callers, but this PR reworks exactly that path.
• RSA4c / RSA4c1 — the auth callback invocation is bounded, and cause is set per the clause.

Adopted from specification 6.1.0

• RTN14h, replacing RTN15g — the client always attempts a resume and lets the server decide whether continuity survives, rather than discarding its connection state once connectionStateTtl has passed. Verified against the live sandbox: a resume is still honoured well beyond the advertised TTL, so the old gate was throwing away resumes the server would have accepted.
• RTN27c — a reconnect from DISCONNECTED is now genuinely a resume attempt; clearing the key on every failed attempt had made that false from the second attempt onwards.
• RTN15g1 / RTN15g2 / RTN15g3 and RTL4j / RTL4j1 / RTL4j2 — deleted at 6.1.0. The TTL freshness check and ATTACH_RESUME go with them; channelSerial (RTL4c1, kept across a suspend by RTL15b2) already carries the continuity signal.
• RTN16g3, replacing RTN16g2 — createRecoveryKey() now returns a key while SUSPENDED, where it previously returned empty. RTN8d/RTN9d keep the connectionKey through SUSPENDED because RTN14h always attempts a resume, so the connection is still recoverable there and the key has to be available to hand over. RTN16g3 comes from ably/specification#511, which tombstones RTN16g2 and drops SUSPENDED from the states where createRecoveryKey() returns null; that merged while this branch was in review. Written before it merged because it is the direct consequence of RTN14h above, and splitting the two would mean two behaviour changes for callers instead of one. ably-js has behaved this way since 2.27.0.

A clause the 6.1.0 edit had missed, since corrected

• RTN27d used to call SUSPENDED a state whose next connect attempt is "a clean connection (not a resume attempt)", which RTN14h, RTN8d, RTN9d and DF1a contradict between them — spec 6.1.0 moved resumability to the server without touching RTN27. ably/specification#511 has since amended the clause in place and dropped that sentence, so this PR follows RTN27d rather than deviating from it. Amending in place is the right shape here: RTN27 declares its states "mutually exclusive and exhaustive", so tombstoning the clause would have left SUSPENDED undescribed.

Behaviour changes

No public API changed, but these are visible to callers:

• authCallback / authUrl are now bounded by realtimeRequestTimeout (default 10s) — previously unbounded. A callback that took longer and still succeeded will now fail the auth attempt (RSA4c). The only item here that can break a working deployment.
• HeartbeatMonitorDelay rejects values below 1ms — previously accepted, but 0 hot-looped and negatives silently killed idle detection, so no working configuration is affected.
• Connection.Key and Id read as null during a CLOSED/FAILED state-change callback — they are now cleared before the event is emitted (RTN8d/RTN9d), and retained through SUSPENDED.
• connectionId no longer changes after a long disconnection — the client always attempts a resume now (RTN14h). Use ChannelStateChangedEventArgs.Resumed to detect a fresh connection.
• createRecoveryKey() returns a key while SUSPENDED, where it previously returned empty (RTN16g3).
• Some ErrorInfo messages and codes changed — affects code that matches on them.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added configurable realtime request timeout validation.
  • Added server-provided idle intervals and protocol heartbeats, with caller overrides preserved.
  • Improved connection recovery, channel reattachment, serial continuity, and suspended-state recovery.
  • Added immediate reconnect handling and fallback connectivity checks.

• Bug Fixes

  • Auth callbacks now respect request timeouts and report underlying failure details.
  • Queued callbacks continue processing after an application callback error.
  • Retry and close timers now reflect configured timeout values accurately.
  • Improved connection and channel state handling during failures and transitions.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: ddfec4aa-e457-4ef4-acd1-77a242e0dc09

📥 Commits

Reviewing files that changed from the base of the PR and between e5ea962 and 2120a9e.

📒 Files selected for processing (7)

• src/IO.Ably.Shared/Realtime/Workflows/RealtimeState.cs
• src/IO.Ably.Shared/Realtime/Workflows/RealtimeWorkflow.cs
• src/IO.Ably.Shared/Transport/AttemptFailedState.cs
• src/IO.Ably.Shared/Transport/ConnectionAttemptsInfo.cs
• src/IO.Ably.Shared/Transport/ConnectionManager.cs
• src/IO.Ably.Tests.Shared/Realtime/ConnectionSpecs/ConnectionFallbackSpecs.cs
• src/IO.Ably.Tests.Shared/Realtime/RealtimeWorkflowSpecs.cs

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

Walkthrough

The change adds server-provided idle limits, timeout validation, bounded authentication callbacks, revised reconnect handling, connection continuity, safer queue cleanup, and updated channel transitions.

Changes

Realtime lifecycle changes

Layer / File(s)	Summary
Idle, timeout, and authentication contracts src/IO.Ably.Shared/..., src/IO.Ably.Shared.MsgPack/..., src/IO.Ably.Tests.Shared/...	ConnectionDetails and ConnectionInfo carry MaxIdleInterval. RealtimeRequestTimeout enforces timer-safe bounds. Auth callbacks use the configured timeout. Protocol heartbeats are requested by default and remain caller-overridable.
Connection retry and state workflow src/IO.Ably.Shared/Realtime/Workflows/..., src/IO.Ably.Shared/Transport/..., src/IO.Ably.Tests.Shared/Realtime/...	Immediate retries are counted and bounded. Retry delays report actual timer values and respect state TTLs. Connection continuity, message queues, state transitions, and teardown behavior are updated.
Channel and presence lifecycle src/IO.Ably.Shared/Realtime/..., src/IO.Ably.Tests.Shared/Realtime/...	Suspended channels retain serials and recovery keys. Channel transitions handle reconnect, close, suspend, and detach states. Presence callbacks no longer stop queue cleanup.
Lifecycle regression coverage src/IO.Ably.Tests.Shared/Realtime/..., src/IO.Ably.Tests.Shared/Infrastructure/...	Tests cover serialization, timeout limits, retry timing, fallback checks, recovery keys, idle monitoring, channel transitions, connection continuity, and fake transport creation.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant RealtimeWorkflow
  participant ConnectionState
  participant Transport
  Client->>RealtimeWorkflow: queue heartbeat monitor tick
  RealtimeWorkflow->>ConnectionState: read connection and idle state
  RealtimeWorkflow->>Transport: request disconnect after idle threshold
  Transport->>RealtimeWorkflow: report disconnected state
  RealtimeWorkflow->>RealtimeWorkflow: schedule bounded reconnect

Loading

Suggested reviewers: sacoo7

Merge Risk: 🔵 Low · up to 2120a

Heartbeat monitoring may still arm for a transport that did not request protocol heartbeats, risking false idle disconnects.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 12.57% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 175 functions across 36 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies RTN23 idle-transport detection and the related connection-recovery changes, which are the main objectives of the pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/rtn23-spec-research-ca5ed0

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{A rabbit checks the heartbeat glow
Keys stay safe through suspended nights
Queues clear when callbacks fail
Reconnect paths follow the state
Tests guard each changing route
The workflow hops back into line}

_{Comment @coderabbitai help to get the list of available commands.}

[sacOO7]

LGTM