Repository navigation
Buffer a new workflow task while the run already has one - #1640
Merged
Merged
Conversation
Since #1442, core replies to lang's completion right away while the WFT waits on local activities, so the run has a WFT but no outstanding activation. A legacy query task arriving then isn't buffered: it replaces the run's WFT ("Trying to send a new WFT for a run which already has one!"), and the WFT is never completed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test hands core its tasks; core taking the task after the query means it has handled the query. Restores the longer doc comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
buffer_wft_if_outstanding_work held new work back only while lang had an activation or work was queued. Since #1442, a run waiting on local activities holds its WFT with neither, so a task arriving then replaced the WFT, which was never reported. Checking the WFT itself matches the method's doc comment; the buffered task is applied once the WFT is reported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signal the query handled and stay pending in one stage, and say what the lost workflow task leads to. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # crates/sdk-core/tests/integ_tests/workflow_tests/local_activities.rs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sushisource
approved these changes
Oct 13, 2026
THardy98
marked this pull request as ready for review
October 14, 2026 13:03
THardy98
added a commit
to temporalio/sdk-dotnet
that referenced
this pull request
Oct 14, 2026
Brings in temporalio/sdk-rust#1640. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
THardy98
added a commit
to temporalio/sdk-python
that referenced
this pull request
Oct 14, 2026
Brings in temporalio/sdk-rust#1640. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
THardy98
added a commit
to temporalio/sdk-ruby
that referenced
this pull request
Oct 15, 2026
Brings in temporalio/sdk-rust#1640. Ran core:update's tool and cargo fetch, and regenerated the protos Core's API update changed. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was changed
Core now holds a new workflow task back while the run already has one open, not only while the SDK is busy. One line in
buffer_wft_if_outstanding_work:|| self.wft.is_some(). The method's doc comment already said it buffers while a workflow task is outstanding.The test shows the bug it fixes:
With the fix, the query waits, the workflow task is reported, and then the query is answered.
Why?
Core held a new task back only while the workflow was busy, meaning the SDK was processing an activation or work was queued. Since #1442, core answers the SDK right away while local activities run, so a workflow waiting on a local activity didn't look busy. A query arriving in that gap was handled immediately and replaced core's record of the workflow task.
Core never completed the lost workflow task, so it timed out on the server (10 seconds by default). The retry then often failed with a nondeterminism error,
Invalid transition while handling update response ... in state CompletedImmediatelyCompleteCreated, and the workflow stayed stuck.We hit this in load tests on Temporal Cloud (throughput_stress, 40 concurrent workflows, 1 hour). Per run:
Every nondeterminism error followed an "already has one" in the same workflow run, about 10 seconds earlier. Debug logs from an earlier Python run showed the replacing task was a legacy query, as in this test.
Load test with the fix: .NET on Core
mainagainst Coremainplus this fix, 10 runs each (throughput_stress, 40 concurrent workflows, 20 minutes):main(aefe3059)main+ this fix (74b68df0)Checklist
Closes
How was this tested:
cargo test -p temporalio-sdk-core --test integ_tests --features temporalio-common/serde_serialize,test-utilities,ephemeral-server,temporalio-sdk/experimental,temporalio-sdk-core/otel -- legacy_query_while_waiting_on_la_is_bufferedcargo test -p temporalio-sdk-core --lib), and so do the local activity, query and update integration tests (cargo integ-test local_activities,queries_tests,update: 97, 8 and 44 tests).cargo lintis clean.CHANGELOG.mdandcrates/sdk-core/CHANGELOG.md.Any docs updates needed?
No.
🤖 Generated with Claude Code