Repository navigation
Conversation
…atch fails When the dispatch of a query fails part way, for example because one fragment instance can't get enough operator memory, ClusterScheduler fails the query and SimpleQueryTerminator cancels the fragment instances that were already dispatched. It waits for a grace period of one second before it does so. Those fragment instances can't finish on their own: their consumer is gone, so they keep the operator memory they reserved in LocalExecutionPlanner.checkMemory() until they are cancelled. If more queries arrive within that second, they find the memory taken, fail their dispatch too and hold more memory for another second. With a small memory pool this cascades. Remember in ClusterScheduler that the dispatch failed, and terminate such a query without the grace period. Queries that ran normally keep the grace period.
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #19085 +/- ##
============================================
+ Coverage 48.11% 48.12% +0.01%
- Complexity 621 625 +4
============================================
Files 5572 5572
Lines 410286 411555 +1269
Branches 54129 54191 +62
============================================
+ Hits 197408 198068 +660
- Misses 212878 213487 +609 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
The failing SonarCloud quality gate is not caused by this change. Its only bug finding is S3078 ("Use an AtomicBoolean") in |
Description
Problem
IoTDBMultiTAGsWithAttributesTableITis flaky. Usually onlyfourTableJoinTestfails (a known issue with the small memory pool of the test DataNode). But in some runs 9 to 15 test cases fail, all with:Once this starts, every later query in the class fails, even ones that need less than 2 KB.
Cause
fourTableJoinTestplans 25 fragment instances, many of them reserving about 1 MB of operator memory, and the operator pool of the test DataNode has about 15 MB. Some instances fail inLocalExecutionPlanner.checkMemory(), so the dispatch fails andClusterSchedulerfails the query.The instances that were dispatched before the failure are cancelled by
SimpleQueryTerminator, but only afterTERMINATION_GRACE_PERIOD_IN_MS(1 s). Depending on timing, some of them have already lost their consumer and can't finish on their own. They keep their reservation until the cancel arrives.With DEBUG logging for
LocalExecutionPlannerenabled, a failing run shows it:fourTableJoinTestfails at 15:46:19,687. Six of its instances (1.6,1.8,1.9,1.12,1.14,1.20) log[ConsumeMemory]but never[ReleaseMemory], together about 4.2 MB.Fix
ClusterSchedulerremembers that the dispatch failed, andstop()then calls the newIQueryTerminator.terminateImmediately(), which cancels the dispatched instances without the grace period.The grace period was added in #5808 so that instances flushing their last results to the consumer aren't cancelled while they finish normally. A query whose dispatch failed has no results to deliver, so it doesn't need it. Queries that were dispatched successfully keep the grace period.
Verification
New
SimpleQueryTerminatorTestchecks the delay of both methods.SimpleQueryTerminatorTest,QueryExecutionRetryTest,QueryOwnershipTestandAsyncPlanNodeSenderTestpass (17 tests).spotless:checkpasses foriotdb-core/datanode.IoTDBMultiTAGsWithAttributesTableITalone, 10 runs each on macOS, on this branch's base and with this change:fourTableJoinTestonly)fourTableJoinTest,timeColumnOuterJoinTest1)fourTableJoinTestitself still fails, because its 25 instances don't fit into the memory pool of the test DataNode. That's a separate issue and unchanged by this PR.This PR has:
Key changed/added classes (or packages if there are too many classes) in this PR
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/ClusterScheduler.javaiotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/IQueryTerminator.javaiotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/SimpleQueryTerminator.javaiotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/scheduler/SimpleQueryTerminatorTest.java🤖 Generated with Claude Code