Skip to content

fix(kvstore): Retry Bigtable mutate errors raised while reading the stream - #132103

Open
sentry-junior[bot] wants to merge 1 commit into
masterfrom
kcons/bigtable-retry-stream-errors
Open

sentry-junior[bot] wants to merge 1 commit into
masterfrom
kcons/bigtable-retry-stream-errors

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

When a contextual Bigtable timeout is set (as save_event does), _mutate_rows gives the operation a 25s retry budget and caps each attempt at 5s. In practice, an attempt that hits its 5s cap fails the whole operation on the first try (SENTRY-60R4).

In google-cloud-bigtable 2.30.1, _RetryableMutateRowsWorker._do_mutate_retryable_rows only turns RETRYABLE_MUTATION_ERRORS into _BigtableRetryableError when they're raised by the initial mutate_rows call. Errors raised while iterating the response stream, such as DeadlineExceeded("Stream removed (Deadline Exceeded)"), reach api_core's retry as-is. DEFAULT_RETRY only matches _BigtableRetryableError, so api_core treats them as non-retryable and re-raises them right away. The SENTRY-60R4 event shows this: error_list has one entry with a 25s deadline.

This change passes a retry whose predicate also matches Aborted, DeadlineExceeded and ServiceUnavailable. Those are the client's own RETRYABLE_MUTATION_ERRORS, minus InternalServerError, which the client only retries for RST_STREAM. The worker keeps per-row statuses between attempts, so only rows that haven't succeeded are sent again. If the budget runs out, the client still swallows the resulting RetryError and returns None statuses, so callers keep getting BigtableError("Timed out mutating row").

The change only applies when a contextual timeout is set. The no-context path is unchanged.

Notes for review:

  • This imports the client's private _BigtableRetryableError. It's meant as a stopgap until ref(nodestore): Use the Bigtable data client for KV storage #131562 moves us to the data client. Its unary mutate_row doesn't have this gap, and its bulk_mutate_rows retries stream errors.
  • set() still retries once on ServiceUnavailable / InternalServerError. For errors raised while reading the stream, that outer retry now mostly won't fire, because the inner retry absorbs them and reports a timeout instead.

via @kcons.

--

View Session [Sentry]

…tream

The legacy Bigtable client only retries errors raised when a MutateRows
call starts. A DeadlineExceeded raised while reading the response stream,
such as an attempt hitting its per-attempt timeout, fails the operation on
the first attempt even when the overall retry budget has time left.

Use a retry predicate that also covers those errors so the contextual
timeout budget is spent retrying.

Co-Authored-By: Kyle Consalus <6990351+kcons@users.noreply.github.com>
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Nov 21, 2026
@kcons
kcons requested a review from a team November 21, 2026 01:24
@kcons
kcons marked this pull request as ready for review November 21, 2026 01:39
@kcons
kcons requested review from a team as code owners November 21, 2026 01:39
@sentry-junior
sentry-junior Bot requested a review from kcons November 21, 2026 01:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant