Repository navigation
Conversation
…AttributesTableIT With the default AUTO extension policy, the number of DataRegionGroups follows the CPU cores of the DataNodes (0.5 per core): 2 on a 4-core CI runner, 6 on a 12-core machine. The number of fragment instances of fourTableJoinTest grows with it, and many of them reserve about 1 MB of operator memory because of the 1 MB sort buffer the test sets. The DataNode heap doesn't grow with it. With 6 regions, the 25 fragment instances don't fit into the about 16 MB of operator memory of a 256 MB heap, and the test fails with "There is not enough memory to execute current fragment instance". Give the test DataNodes 128 MB of heap per expected DataRegion, but never less than the configured heap. On a 4-core machine this is the default 256 MB, so CI runs with the same settings as before.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #19086 +/- ##
===========================================
Coverage 48.11% 48.12%
- Complexity 621 625 +4
===========================================
Files 5572 5572
Lines 410286 411548 +1262
Branches 54129 54190 +61
===========================================
+ Hits 197408 198052 +644
- Misses 212878 213496 +618 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
The failing SonarCloud quality gate is not caused by this change. Its only bug finding is S3078 ("Use an AtomicBoolean") in |
Description
Problem
On machines with many CPU cores,
IoTDBMultiTAGsWithAttributesTableIT.fourTableJoinTestfails every time:It passes on CI.
Cause
The test runs with the default
AUTODataRegionGroup extension policy, so the number of DataRegionGroups follows the CPU cores of the DataNodes (data_region_per_data_node_proportion = 0.5). The fragment instances of the four-table join grow with it, and many of them reserve about 1 MB of operator memory because the test sets a 1 MB sort buffer. The DataNode heap stays at 256 MB, which gives about 16 MB of operator memory:fourTableJoinTestLimiting the DataNode JVMs to 4 cores (
-XX:ActiveProcessorCount=4) on the same machine gives 2 regions, and all 34 test cases pass.Fix
Scale the DataNode heap with the expected number of DataRegions instead of limiting the regions, so that a machine with more cores still runs the test with all of them: 128 MB per region (
availableProcessors() / 2), but never less than the configured heap (DataNodeMaxHeapSize, default 256 MB). The test needs about 3.5 MB of operator memory per region, so this leaves a margin of about 2x.On a 4-core machine the result is the default 256 MB, so CI runs with the same settings as before.
Verification
IoTDBMultiTAGsWithAttributesTableITalone, on macOS with 12 cores:fourTableJoinTestfails in every runTableSimpleIT, 5 runsTableSimpleIT, JVMs limited to 4 coresTableClusterIT(1C3D)spotless:checkpasses forintegration-test.This PR has:
Key changed/added classes (or packages if there are too many classes) in this PR
integration-test/src/test/java/org/apache/iotdb/relational/it/db/it/IoTDBMultiTAGsWithAttributesTableIT.java🤖 Generated with Claude Code