Skip to content

Buffer a new workflow task while the run already has one - #1640

Merged
THardy98 merged 8 commits into
mainfrom
test/legacy-query-while-waiting-on-la
Oct 14, 2026
Merged

THardy98 merged 8 commits into
mainfrom
test/legacy-query-while-waiting-on-la

Conversation

@THardy98

@THardy98 THardy98 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

What was changed

Core now holds a new workflow task back while the run already has one open, not only while the SDK is busy. One line in buffer_wft_if_outstanding_work: || self.wft.is_some(). The method's doc comment already said it buffers while a workflow task is outstanding.

The test shows the bug it fixes:

  1. A workflow task arrives, and the workflow starts a local activity.
  2. While the local activity runs, a query arrives as its own task. (This is agnostic to queries; any task arriving then would do the same.)
  3. Core should hold the query until the workflow task is done. Without the fix, it handles the query right away and loses track of the workflow task, failing on core's own check: "Trying to send a new WFT for a run which already has one!"

With the fix, the query waits, the workflow task is reported, and then the query is answered.

Why?

Core held a new task back only while the workflow was busy, meaning the SDK was processing an activation or work was queued. Since #1442, core answers the SDK right away while local activities run, so a workflow waiting on a local activity didn't look busy. A query arriving in that gap was handled immediately and replaced core's record of the workflow task.

Core never completed the lost workflow task, so it timed out on the server (10 seconds by default). The retry then often failed with a nondeterminism error, Invalid transition while handling update response ... in state CompletedImmediatelyCompleteCreated, and the workflow stayed stuck.

We hit this in load tests on Temporal Cloud (throughput_stress, 40 concurrent workflows, 1 hour). Per run:

SDK "already has one" Nondeterminism errors Runs that hung
.NET 92-112 19-33 7 of 7
Python 2-8 0-4 2 of 7
TypeScript 0-2 0-2 0 of 7

Every nondeterminism error followed an "already has one" in the same workflow run, about 10 seconds earlier. Debug logs from an earlier Python run showed the replacing task was a legacy query, as in this test.

Load test with the fix: .NET on Core main against Core main plus this fix, 10 runs each (throughput_stress, 40 concurrent workflows, 20 minutes):

.NET build Runs completed "already has one" per run Nondeterminism errors per run
Core main (aefe3059) 1 of 10 23-51 3-15
Core main + this fix (74b68df0) 10 of 10 0 0

Checklist

  1. Closes

  2. How was this tested:

    • The test fails without the fix and passes with it (30 of 30 runs):
      cargo test -p temporalio-sdk-core --test integ_tests --features temporalio-common/serde_serialize,test-utilities,ephemeral-server,temporalio-sdk/experimental,temporalio-sdk-core/otel -- legacy_query_while_waiting_on_la_is_buffered
    • Core's unit tests pass (cargo test -p temporalio-sdk-core --lib), and so do the local activity, query and update integration tests (cargo integ-test local_activities, queries_tests, update: 97, 8 and 44 tests).
    • cargo lint is clean.
    • Added entries under Unreleased / Fixed in CHANGELOG.md and crates/sdk-core/CHANGELOG.md.
  3. Any docs updates needed?
    No.

🤖 Generated with Claude Code

THardy98 and others added 3 commits October 8, 2026 16:53
Since #1442, core replies to lang's completion right away while the WFT
waits on local activities, so the run has a WFT but no outstanding
activation. A legacy query task arriving then isn't buffered: it replaces
the run's WFT ("Trying to send a new WFT for a run which already has
one!"), and the WFT is never completed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test hands core its tasks; core taking the task after the query
means it has handled the query. Restores the longer doc comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
THardy98 and others added 2 commits October 13, 2026 09:28
buffer_wft_if_outstanding_work held new work back only while lang had an
activation or work was queued. Since #1442, a run waiting on local
activities holds its WFT with neither, so a task arriving then replaced
the WFT, which was never reported. Checking the WFT itself matches the
method's doc comment; the buffered task is applied once the WFT is
reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signal the query handled and stay pending in one stage, and say what
the lost workflow task leads to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@THardy98 THardy98 changed the title Failing test: legacy query while a WFT waits on a local activity Buffer a new workflow task while the run already has one Oct 13, 2026
THardy98 and others added 2 commits October 13, 2026 10:06
# Conflicts:
#	crates/sdk-core/tests/integ_tests/workflow_tests/local_activities.rs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@THardy98
THardy98 marked this pull request as ready for review October 14, 2026 13:03
@THardy98
THardy98 requested a review from a team as a code owner October 14, 2026 13:03
@THardy98
THardy98 merged commit 212e815 into main Oct 14, 2026
42 of 43 checks passed
@THardy98
THardy98 deleted the test/legacy-query-while-waiting-on-la branch October 14, 2026 13:54
THardy98 added a commit to temporalio/sdk-dotnet that referenced this pull request Oct 14, 2026
Brings in temporalio/sdk-rust#1640.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
THardy98 added a commit to temporalio/sdk-python that referenced this pull request Oct 14, 2026
Brings in temporalio/sdk-rust#1640.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
THardy98 added a commit to temporalio/sdk-ruby that referenced this pull request Oct 15, 2026
Brings in temporalio/sdk-rust#1640. Ran core:update's tool and cargo
fetch, and regenerated the protos Core's API update changed.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants