Repository navigation
fix(kvstore): Retry Bigtable mutate errors raised while reading the stream - #132103
Open
sentry-junior[bot] wants to merge 1 commit into
Open
sentry-junior[bot] wants to merge 1 commit into
sentry-junior[bot] wants to merge 1 commit into
Conversation
…tream The legacy Bigtable client only retries errors raised when a MutateRows call starts. A DeadlineExceeded raised while reading the response stream, such as an attempt hitting its per-attempt timeout, fails the operation on the first attempt even when the overall retry budget has time left. Use a retry predicate that also covers those errors so the contextual timeout budget is spent retrying. Co-Authored-By: Kyle Consalus <6990351+kcons@users.noreply.github.com>
kcons
marked this pull request as ready for review
November 21, 2026 01:39
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a contextual Bigtable timeout is set (as
save_eventdoes),_mutate_rowsgives the operation a 25s retry budget and caps each attempt at 5s. In practice, an attempt that hits its 5s cap fails the whole operation on the first try (SENTRY-60R4).In google-cloud-bigtable 2.30.1,
_RetryableMutateRowsWorker._do_mutate_retryable_rowsonly turnsRETRYABLE_MUTATION_ERRORSinto_BigtableRetryableErrorwhen they're raised by the initialmutate_rowscall. Errors raised while iterating the response stream, such asDeadlineExceeded("Stream removed (Deadline Exceeded)"), reach api_core's retry as-is.DEFAULT_RETRYonly matches_BigtableRetryableError, so api_core treats them as non-retryable and re-raises them right away. The SENTRY-60R4 event shows this:error_listhas one entry with a 25s deadline.This change passes a retry whose predicate also matches
Aborted,DeadlineExceededandServiceUnavailable. Those are the client's ownRETRYABLE_MUTATION_ERRORS, minusInternalServerError, which the client only retries for RST_STREAM. The worker keeps per-row statuses between attempts, so only rows that haven't succeeded are sent again. If the budget runs out, the client still swallows the resultingRetryErrorand returnsNonestatuses, so callers keep gettingBigtableError("Timed out mutating row").The change only applies when a contextual timeout is set. The no-context path is unchanged.
Notes for review:
_BigtableRetryableError. It's meant as a stopgap until ref(nodestore): Use the Bigtable data client for KV storage #131562 moves us to the data client. Its unarymutate_rowdoesn't have this gap, and itsbulk_mutate_rowsretries stream errors.set()still retries once onServiceUnavailable/InternalServerError. For errors raised while reading the stream, that outer retry now mostly won't fire, because the inner retry absorbs them and reports a timeout instead.via @kcons.
--
View Session [Sentry]