Repository navigation
Conversation
Add caller-owned purge eligibility and dry-run preview without changing container lifecycle or enabling purge by default. Scan MySQL purge pages in primary-key order, wait for metadata cleanup before paging, and stop on deletion failures or repeated versions. Keep deployment and migration-specific policy outside the generic cleaner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #3305 +/- ##
=============================================
- Coverage 64.24% 38.01% -26.24%
+ Complexity 10398 6486 -3912
=============================================
Files 840 939 +99
Lines 71755 80881 +9126
Branches 8611 9768 +1157
=============================================
- Hits 46099 30743 -15356
- Misses 23004 47651 +24647
+ Partials 2652 2487 -165 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Durability risk: high when enabled. Purge intentionally deletes live/latest versions. Callers must explicitly approve each container, fence source reads/writes, and verify that the source data is no longer needed. Eligibility is checked before each page and delete, but revocation cannot cancel an already-issued deletion. Deletion or metadata failures stop purge; subsequent runs restart the sweep. Repeated versions after metadata cleanup also stop the sweep, and purge never skips names after query kills.
This draft proposes only the generic OSS mechanism. Deployment configuration, migration/completeness checks, composite-database delegation, and verification of composite-router delete routing remain integration responsibilities. No rollout is included. Expired/already-deleted mappings retain their existing cleanup paths. Primary-key ordering avoids the mixed-order filesort but does not guarantee bounded latency when many index entries are filtered.
Please review the API/runner policy first, then the MySQL paging behavior. Live SQL validation remains pending.
Testing Done
Using JDK 17:
./gradlew --no-daemon --console=plain -PdisableShipkit :ambry-frontend:test --tests com.github.ambry.frontend.NamedBlobsCleanupRunnerTest :ambry-named-mysql:test --tests com.github.ambry.named.MySqlNamedBlobDbTest :ambry-named-mysql:compileIntTestJavagit diff --check./gradlew --no-daemon --console=plain -PdisableShipkit :ambry-named-mysql:intTest --tests 'com.github.ambry.named.MySqlNamedBlobDbIntegrationTest.testPurge*'Full build/full test suite not run.