Skip to content

Fix automatic LOAD retries during region recovery - #19098

Open
Caideyipi wants to merge 3 commits into
masterfrom
fix/load-scanner-restart
Open

Caideyipi wants to merge 3 commits into
masterfrom
fix/load-scanner-restart

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

After a cluster restart, automatic LOAD can encounter NO_AVAILABLE_REGION_GROUP (906) while SchemaRegion groups recover. Schema verification wraps the exception and drops its status, so LOAD returns LOAD_FILE_ERROR (1100) and moves a valid TsFile to the failed directory.

Preserve the region-unavailable status through tree schema verification and translate it to LOAD_TEMPORARY_UNAVAILABLE_EXCEPTION (1110) in LOAD analysis. The existing automatic loader can then retain the file and retry. Permanent schema errors keep their existing behavior.

Exposed by the active-scan restart CI failure.

Validation

  • The schema-flush regression fails on the unpatched base with 1100 instead of 1110.
  • All 8 diagnostics tests and 10 existing active-load unit tests passed.
  • Both methods in IoTDBLoadTsFileActiveScanIT passed with 1 ConfigNode and 3 DataNodes: 0 failures, 0 errors, 0 skipped.
  • Spotless passed. A clean JDK 17 reactor build rebuilt the required IoTDB dependencies and test-node templates with the build cache disabled.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Initial independent self-review (one round): reviewed the full diff against master (e6b44b57e49e1e7f0ecf5d6c62fa7e1f69a9a869) and the relevant schema, LOAD analysis, and automatic-loader context.

[P2] Preserve wrapped region-unavailable errors. At the reviewed revision, TreeSchemaAutoCreatorAndVerifier.java:311 only inspected the top-level exception. Template activation can wrap status 906 in SemanticException(IoTDBException), so LOAD still returned 1100 and moved a recoverable file to the failed directory. Fixed by preserving 906 through the cause chain, with transient and permanent-error regression controls.

Validation:

  • Before the repair, the wrapped transient regression failed with expected 1110 versus actual 1100; the permanent-error control passed.
  • After the repair, a clean dependency reactor passed all 20 relevant unit tests: 10 diagnostics, 7 loader, and 3 handler tests, with no failures, errors, or skips. All 482 goals executed. Spotless passed. All newly added tests were run locally.
  • The earlier two-method 1C3D integration run passed at the reviewed SHA; it was not repeated after this repair. The full test suite and Chinese-locale build were not run locally.
  • CI for the final SHA is pending. The earlier Sonar reliability failure points to java:S3078 in unchanged PipeTaskInfo.java:1174; that report is outside this PR's diff and remains unverified here.

Reviewed SHA: ecbf4150d72031399d7336eb66409cac09a603e3
Final local SHA: e5e2b2767b9c1b9483bd2287732e4c62eca24c1f
Confirmed source branch and PR head SHA: e5e2b2767b9c1b9483bd2287732e4c62eca24c1f

The repair was committed and pushed to apache/iotdb:fix/load-scanner-restart after refreshing the remote head. All update steps are complete. The PR remains Draft because this round found a behavioral issue and requires a fresh independent review.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Re-review (one round): reviewed the complete diff against master (e6b44b57e49e1e7f0ecf5d6c62fa7e1f69a9a869) and the relevant schema activation, LOAD analysis, and active-loader paths, including the wrapped status-906 repair.

No actionable findings. The cause-chain handling preserves NO_AVAILABLE_REGION_GROUP through the SemanticException(IoTDBException) path while permanent metadata errors remain non-retryable. The tests cover both wrapped outcomes and the direct LOAD-analysis status mapping.

Validation:

  • The recorded clean dependency-reactor run passed all 20 relevant unit tests (10 diagnostics, 7 loader, 3 handler). All added tests ran locally; Spotless passed.
  • The two 1C3D integration tests passed on the preceding reviewed SHA (ecbf4150d72031399d7336eb66409cac09a603e3) and were not rerun after this repair. The full local test suite and Chinese-locale build were not run locally.
  • Final-SHA CI is incomplete: zh-locale-compile passed, one Simple (17) job failed during Set up job before any test step ran, and other checks were pending when checked. No overall CI pass is claimed.

Reviewed SHA: e5e2b2767b9c1b9483bd2287732e4c62eca24c1f
Final local SHA: e5e2b2767b9c1b9483bd2287732e4c62eca24c1f
Confirmed source branch and PR head SHA: e5e2b2767b9c1b9483bd2287732e4c62eca24c1f

@Caideyipi
Caideyipi marked this pull request as ready for review November 3, 2026 03:49
@codecov

codecov Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 61.90476% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.13%. Comparing base (1b2158f) to head (6ac971c).
⚠️ Report is 9 commits behind head on master.

Files with missing lines Patch % Lines
...ryengine/plan/analyze/load/LoadTsFileAnalyzer.java 11.11% 8 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19098      +/-   ##
============================================
+ Coverage     48.12%   48.13%   +0.01%     
  Complexity      625      625              
============================================
  Files          5572     5572              
  Lines        411488   411604     +116     
  Branches      54166    54211      +45     
============================================
+ Hits         198010   198141     +131     
+ Misses       213478   213463      -15     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment on lines +902 to +911
if (throwable instanceof IoTDBRuntimeException
&& ((IoTDBRuntimeException) throwable).getErrorCode()
== TSStatusCode.NO_AVAILABLE_REGION_GROUP.getStatusCode()) {
analysis.setFinishQueryAfterAnalyze(true);
analysis.setFailStatus(
RpcUtils.getStatus(
TSStatusCode.LOAD_TEMPORARY_UNAVAILABLE_EXCEPTION, throwable.getMessage()));
setRealStatement(analysis);
return true;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can you distinguish a recovering region from a truly missing one?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It cannot be distinguished from status 906 alone. RegionWriteExecutor also returns NO_AVAILABLE_REGION_GROUP when the local region lock is missing or the consensus group does not exist, so the status does not prove that recovery is in progress.

The contract here is to retain the TsFile and retry region-unavailable errors. A genuinely missing region can therefore continue to produce retries and needs operator intervention; this change does not diagnose or repair permanent region loss. Other schema-error statuses keep their existing failure behavior.

I corrected the comments in the schema verifier and LOAD analyzer in 6ac971c to make that boundary explicit. This is a comment-only clarification, and Spotless passed.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

External-comment follow-up at 6ac971c08673bc50c382e64d6b67b790cf9d96ca:

  • Region availability question: clarified that 906 is an availability signal and cannot distinguish recovery from persistent region loss. The file is retained for retry in either case; persistent loss needs operator intervention. The two misleading/incomplete Java comments were clarified, with no behavior change.
  • Sonar report: the reported reliability issue is java:S3078 in PipeTaskInfo.java:1174. That file is unchanged by this PR, so the report is left for a separate fix; no unrelated change is included here.
  • Codecov report: the report shows 61.90476% patch coverage and eight uncovered analyzer lines. The existing diagnostics exercise direct/wrapped schema failures and the single-file analysis retry path, with permanent-error controls. Their earlier passing run is recorded in the preceding validation summaries. This follow-up adds no executable lines, and no coverage improvement or green coverage gate is claimed.

Validation: mvn spotless:apply -P with-integration-tests -pl iotdb-core/datanode passed with exit code 0. This update changes only Java comments; compilation and tests were not repeated, and no tests were added. CI for the new head has not been awaited.

Follow-up starting SHA: e5e2b2767b9c1b9483bd2287732e4c62eca24c1f
Final local SHA: 6ac971c08673bc50c382e64d6b67b790cf9d96ca
Confirmed source branch and PR head SHA: 6ac971c08673bc50c382e64d6b67b790cf9d96ca

The clarification is committed and pushed. All discussions are left unresolved.

@sonarqubecloud

sonarqubecloud Bot commented Nov 3, 2026

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants