Skip to content

Fix deletion recovery and table last-cache correctness - #19081

Merged
jt2594838 merged 1 commit into
masterfrom
fix/deletion-recovery-last-cache
Nov 2, 2026
Merged

jt2594838 merged 1 commit into
masterfrom
fix/deletion-recovery-last-cache

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

WAL deletion replay currently writes mods that can also hide inserts replayed later into the same recovered file. Restrict recovered mods to each flushed device and its on-disk end time while preserving the original FIELD and time ranges, tree wildcard semantics, table isolation, and NULL/empty TAG identities.

Correct table last-cache handling for sparse FIELD values, last_by(TAG/ATTRIBUTE, Y), NULL identities, deleted devices, time predicates, TTL, and grouped/partial aggregation. Query writeback now uses a dedicated entry that updates the device row time only from an explicit TIME result; existing write-side cache updates retain their behavior. Apply the same contract to direct and cross-Region query writeback, and avoid modifying shared cached pairs when filtering query results.

Add focused recovery and last-cache regressions. Unrelated deletion/compaction ITs already in #19051 are excluded. The identical DeletionTestUtils helper is retained because the recovery regressions require it for this PR to compile independently against master.

Validation

  • Independent workspace re-review completed with no findings. The retained production code, unit tests, and recovery/last-cache test logic match the reviewed snapshot; publication only removes the unrelated duplicate test blocks and unused imports.
  • Retained execution evidence: Windows/JDK 17 with TsFile 8f665d113dbb5257b901c528c31c8bc24da2a018, without version mapping; clean DataNode/IT package, 12 cache unit tests, and 9 existing last-cache ITs plus 2 isolated verification-only probes passed. The latest last-cache cluster run used 1 ConfigNode and 3 DataNodes. Recovery ITs also have earlier passing evidence.
  • The 3 added TsFilePlanRedoerTest methods have passing remote evidence and have not been run locally. Two pre-existing ignored tests remain unexecuted. The full IT suite was not run; builds used -Denforcer.skip=true.
  • Publication checks: Spotless formatting, IT discovery/shard selection, scoped diff checks, and exact retained-source comparison. Build/test results above were reused from the reviewed snapshot at 28c5028bb462403315e46319b57e9beb96ffaa69; compilation and executable tests were not repeated after refreshing master and removing duplicate ITs.

This PR has:

  • been self-reviewed.
  • added unit tests or modified existing tests to cover the fixes.
  • added integration tests.
  • been tested in a test IoTDB cluster as described above.

@sonarqubecloud

sonarqubecloud Bot commented Nov 2, 2026

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@codecov

codecov Bot commented Nov 2, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 39.53488% with 52 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.11%. Comparing base (1b2158f) to head (93dec7a).

Files with missing lines Patch % Lines
...urce/relational/LastQueryAggTableScanOperator.java 0.00% 35 Missing ⚠️
...e/plan/planner/DataNodeTableOperatorGenerator.java 0.00% 7 Missing ⚠️
...metadata/fetcher/cache/TableDeviceSchemaCache.java 0.00% 4 Missing ⚠️
.../metadata/fetcher/cache/TableDeviceCacheEntry.java 0.00% 3 Missing ⚠️
.../dataregion/wal/recover/file/TsFilePlanRedoer.java 93.75% 2 Missing ⚠️
...ngine/execution/fragment/DataNodeQueryContext.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19081      +/-   ##
============================================
- Coverage     48.12%   48.11%   -0.01%     
  Complexity      625      625              
============================================
  Files          5572     5572              
  Lines        411488   411537      +49     
  Branches      54166    54185      +19     
============================================
+ Hits         198010   198017       +7     
- Misses       213478   213520      +42     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jt2594838
jt2594838 merged commit 76d573a into master Nov 2, 2026
40 of 42 checks passed
@jt2594838
jt2594838 deleted the fix/deletion-recovery-last-cache branch November 2, 2026 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants