Skip to content

Add opt-in all-version purge to named-blob cleanup - #3305

Draft
zichengl wants to merge 1 commit into
linkedin:masterfrom
zichengl:8845012+zichengl/cleaner-purge-all-versions
Draft

zichengl wants to merge 1 commit into
linkedin:masterfrom
zichengl:8845012+zichengl/cleaner-purge-all-versions

Conversation

@zichengl

@zichengl zichengl commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Summary

  1. Add an opt-in all-version purge mode to the named-blob cleaner, including latest READY and recent IN_PROGRESS versions. Existing constructors keep purge disabled.
  2. Use ascending primary-key order for MySQL purge pages, retaining an inclusive last-name cursor so a single name can span multiple pages.
  3. Add a caller-owned eligibility predicate, bounded first-page dry-run preview, counters, and failure-safe paging. Exclusions remain authoritative and container lifecycle status is unchanged.

Durability risk: high when enabled. Purge intentionally deletes live/latest versions. Callers must explicitly approve each container, fence source reads/writes, and verify that the source data is no longer needed. Eligibility is checked before each page and delete, but revocation cannot cancel an already-issued deletion. Deletion or metadata failures stop purge; subsequent runs restart the sweep. Repeated versions after metadata cleanup also stop the sweep, and purge never skips names after query kills.

This draft proposes only the generic OSS mechanism. Deployment configuration, migration/completeness checks, composite-database delegation, and verification of composite-router delete routing remain integration responsibilities. No rollout is included. Expired/already-deleted mappings retain their existing cleanup paths. Primary-key ordering avoids the mixed-order filesort but does not guarantee bounded latency when many index entries are filtered.

Please review the API/runner policy first, then the MySQL paging behavior. Live SQL validation remains pending.

Testing Done

Using JDK 17:

Command Result
./gradlew --no-daemon --console=plain -PdisableShipkit :ambry-frontend:test --tests com.github.ambry.frontend.NamedBlobsCleanupRunnerTest :ambry-named-mysql:test --tests com.github.ambry.named.MySqlNamedBlobDbTest :ambry-named-mysql:compileIntTestJava Passed: 24 cleaner tests and 19 database unit tests; integration tests compile.
git diff --check Passed.
./gradlew --no-daemon --console=plain -PdisableShipkit :ambry-named-mysql:intTest --tests 'com.github.ambry.named.MySqlNamedBlobDbIntegrationTest.testPurge*' Blocked: local MySQL connection refused. Paging and EXPLAIN assertions are not yet verified against a running database.

Full build/full test suite not run.

Add caller-owned purge eligibility and dry-run preview without changing container lifecycle or enabling purge by default. Scan MySQL purge pages in primary-key order, wait for metadata cleanup before paging, and stop on deletion failures or repeated versions. Keep deployment and migration-specific policy outside the generic cleaner.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@codecov-commenter

codecov-commenter commented Oct 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 67 lines in your changes missing coverage. Please review.
✅ Project coverage is 38.01%. Comparing base (52ba813) to head (e13e1ec).
⚠️ Report is 424 commits behind head on master.

Files with missing lines Patch % Lines
...github/ambry/frontend/NamedBlobsCleanupRunner.java 0.00% 48 Missing ⚠️
.../java/com/github/ambry/named/MySqlNamedBlobDb.java 0.00% 14 Missing ⚠️
.../main/java/com/github/ambry/named/NamedBlobDb.java 0.00% 5 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (52ba813) and HEAD (e13e1ec). Click for more details.

HEAD has 1 upload less than BASE
Flag BASE (52ba813) HEAD (e13e1ec)
3 2
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3305       +/-   ##
=============================================
- Coverage     64.24%   38.01%   -26.24%     
+ Complexity    10398     6486     -3912     
=============================================
  Files           840      939       +99     
  Lines         71755    80881     +9126     
  Branches       8611     9768     +1157     
=============================================
- Hits          46099    30743    -15356     
- Misses        23004    47651    +24647     
+ Partials       2652     2487      -165     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants