Skip to content

fix(realtime): handle channel errors and rejoins like realtime-js - #1433

Merged
jan-tennert merged 4 commits into
supabase:masterfrom
sproctor:realtime-channel-errors
Oct 27, 2026
Merged

jan-tennert merged 4 commits into
supabase:masterfrom
sproctor:realtime-channel-errors

Conversation

@sproctor

@sproctor sproctor commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

What changed

When the server refuses a postgres_changes subscription (for example, the table isn't in the supabase_realtime publication), it replies OK to the join and then sends a system message with status: "error". supabase-kt treated every message with status: "error" as a channel error and rejoined: leave, wait for UNSUBSCRIBED, join. That rejoin ran inline on the websocket message loop, which is the only thing that reads the reply to the leave, so the loop hung until a heartbeat timeout forced a reconnect. Then it joined, was refused, and hung again. The refusal itself never reached systemFlow(), because RSystemEvent only handled status: "ok".

This handles channel errors and rejoins the way realtime-js does (through Phoenix's JS client):

  1. A system error is reported, not rejoined. It's logged and forwarded to systemFlow(), and the channel stays joined. Only phx_error and a rejected join (a phx_reply with status: "error" while joining) rejoin, as in Phoenix. A reply error to some other push is logged.
  2. Rejoins are scheduled. scheduleRejoin() launches the rejoin in the realtime scope after rejoinDelay instead of running it in the caller's coroutine (Phoenix's rejoinTimer). It replaces a pending rejoin, and subscribe(), unsubscribe(), teardown and disconnect() cancel it. scheduleRejoin() (@SupabaseInternal) is no longer suspend.
  3. A rejoin sends the join again instead of leaving first. The server replaces the channel's current join, as with Phoenix's rejoin(). Enabling presence on a subscribed channel (presenceChangeFlow()) rejoins the same way, so it no longer goes through UNSUBSCRIBED.
  4. Messages for an earlier join_ref are dropped, as in Phoenix's isMember: for example, the server's close for the join a rejoin replaced, which would otherwise remove the channel.

Unchanged: maxAttempts and rejoinDelay keep their meaning. Phoenix retries forever with backoff; this keeps supabase-kt's fixed delay and limit, and a successful join still resets the count.

Behavior change on token expiry. When a channel's access token expires on the server, Realtime sends a system error ("Token has expired …") and closes the channel. Before, the error made the channel rejoin, which deadlocked as above, and the heartbeat timeout's reconnect then rejoined with a fresh token. Now the error goes to systemFlow() and the server's close removes the channel, as in realtime-js. supabase-kt already sends refreshed tokens to joined channels, so this only matters when a refresh doesn't reach the server in time.

Resolving each message to exactly one event (second commit)

RealtimeEvent.resolveEvent took the first event in EVENTS whose condition matched, so overlapping conditions were settled by the set's iteration order. Two already overlapped: a successful join reply listing postgres changes matched both RPostgresServerChangesEvent and RSystemReplyEvent, and a system message about an expired access token matched both RSystemEvent and RTokenExpiredEvent. The second commit makes the conditions mutually exclusive (each broader event leaves out what a more specific one handles), resolves with singleOrNull, and leaves EVENTS in its original order.

It's in this PR because this PR widens one of those overlaps: RSystemEvent now also handles system messages with status: "error", so whether an expired-token message reached RTokenExpiredEvent would have depended on the order of EVENTS, which I'd otherwise have had to reorder. If you'd rather review it on its own, I'm happy to split it into a separate PR; this one would then only keep RSystemEvent from claiming RTokenExpiredEvent's messages.

Why

We hit this in production: an app release subscribed to a table that wasn't in the publication yet. Every device's channel deadlocked until a heartbeat timeout, reconnected about every 37 seconds, and never received changes for any table, while status kept flipping back to SUBSCRIBED, so the app saw nothing wrong.

Testing

  • RealtimeChannelRejoinTest, against a scripted socket (ScriptedWebsocket, test-only): a refused subscription reaches systemFlow() while the channel stays subscribed and nothing more is sent; a rejected join is retried with a join, not a leave; rejected joins give up after maxAttempts; a channel error rejoins without leaving; the server's close for the replaced join is ignored; a successful join resets the attempt count. All six fail on master. Without the earlier-join_ref check, the close test fails; with system errors still rejoining, the refusal test fails.
  • RealtimeEventTest: a system error resolves to RSystemEvent, a rejected reply to RErrorEvent, a join reply listing postgres changes to RPostgresServerChangesEvent (and a rejected one to RErrorEvent), and an expired-token system error to RTokenExpiredEvent. testEachMessageHasOneEvent checks that every kind of message matches exactly one event, which resolveEvent relies on; it fails with the previous conditions.
  • testResubscribeOnPresenceChange now expects the presence rejoin to join again without leaving.
  • ./gradlew :realtime-kt:jvmTest :realtime-kt:detekt pass.
  • An earlier version of this fix was checked against a local Supabase stack with a table removed from the publication: before, the deadlock and reconnect loop above.

Not changed here: RealtimeImpl.onMessage takes any message whose ref equals the idle heartbeat ref (0) for a heartbeat reply, which swallows the server's messages for a socket's first join. That's #1434, which builds on this.

🤖 Generated with Claude Code

When the server refuses a postgres_changes subscription (e.g. the table isn't
in the supabase_realtime publication), it sends a system message with status
"error". Every message with status "error" made the channel rejoin: leave,
wait for UNSUBSCRIBED, join. The rejoin ran inline on the websocket message
loop, which is the only thing that reads the reply to the leave, so the loop
hung until a heartbeat timeout forced a reconnect, and the refusal never
reached systemFlow().

Handle these the way realtime-js (Phoenix's client) does:
- A system "error" is logged and forwarded to systemFlow(); the channel stays
  joined. Only phx_error and a rejected join rejoin; a reply error to another
  push is logged.
- scheduleRejoin() schedules the rejoin in the realtime scope after
  rejoinDelay instead of running it in the caller's coroutine, replacing a
  pending one; subscribe(), unsubscribe(), teardown and disconnect() cancel
  it. It is no longer suspend (@SupabaseInternal).
- A rejoin sends the join again instead of leaving first; the server
  replaces the channel's current join. Enabling presence on a subscribed
  channel rejoins the same way.
- Messages for an earlier join_ref are dropped, such as the server's close
  for the join a rejoin replaced.

maxAttempts and rejoinDelay keep their meaning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sproctor
sproctor force-pushed the realtime-channel-errors branch from eabed79 to d7c78d1 Compare October 23, 2026 02:40
@sproctor sproctor changed the title fix(realtime): channel errors deadlock the message loop, retry forever and never reach systemFlow fix(realtime): handle channel errors and rejoins like realtime-js Oct 23, 2026
resolveEvent took the first event in EVENTS whose condition matched, so
overlapping conditions were settled by the set's iteration order: a
successful join reply listing postgres changes matched both
RPostgresServerChangesEvent and RSystemReplyEvent, and a system message about
an expired access token both RSystemEvent and RTokenExpiredEvent. Now that
RSystemEvent also handles system errors, the second overlap covers the
"error" status too.

Make the conditions mutually exclusive, each broader event leaving out what a
more specific one handles, and resolve with singleOrNull, so the order of
EVENTS doesn't matter. EVENTS is back in its original order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Cancel a pending rejoin in updateStatus() when the channel becomes
  UNSUBSCRIBING or UNSUBSCRIBED, instead of in teardown(), unsubscribe() and
  disconnect() separately. disconnect() no longer casts to
  RealtimeChannelImpl, and cancelRejoin() is private.
- Resolve events with firstOrNull again; testEachMessageHasOneEvent keeps the
  conditions from overlapping.
- Shorten or remove comments.
- RealtimeChannelRejoinTest uses Turbine for the sent messages, status and
  systemFlow(), and collects its postgresChangeFlow. ScriptedWebsocket no
  longer uses the delicate isClosedForReceive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jan-tennert
jan-tennert merged commit 4b19d09 into supabase:master Oct 27, 2026
13 checks passed
sproctor added a commit to sproctor/supabase-kt that referenced this pull request Oct 27, 2026
Brings in supabase#1433, which this branch was stacked on, and supabase#1423, which also
resets the heartbeat state, on connect. Conflict and resolution:

- supabase#1423's reset on connect stored 0, which on this branch is a real ref, so
  every new socket started with heartbeat 0 pending and its first tick timed
  out. It now stores NO_HEARTBEAT, and this branch's reset in disconnect() is
  dropped as redundant: every socket goes through connect().
- Both sides added RealtimeHeartbeatTest; the tests are combined, this
  branch's reading sent messages with Turbine as ScriptedWebsocket now does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sproctor
sproctor deleted the realtime-channel-errors branch October 28, 2026 14:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants