Repository navigation
Backport release/v6.7: Fix pruning issue in SS causing huge disk spike - #4508
Conversation
* Pruning in SS removes old state to keep a recent window of x blocks. * From Pebble perspective, pruning means iterating all keys and removing the ones older than the retention window (+ keeping some past ones if there were no changes). But deleting does delete right a way. For pebble a deletion is inserting a tombstone value that means that a given key no longer exists. Then Pebble has to "clean that up" removing the old value + the tombstone during compaction and other background tasks. * The bug: When iterating the keys to prune, the iterator was closed after (via defer) after completing the compaction. But Pebble could not compact because the iterator was active, hence "locking" the data that was supposed to be removed. left: fix right: the bug <img width="378" height="339" alt="image" src="https://github.com/user-attachments/assets/6545ea1f-3035-46a5-a424-376926dee7b1" /> (cherry picked from commit 58b7631)
PR SummaryMedium Risk Overview An open Pebble iterator pins Reviewed by Cursor Bugbot for commit d1aa21b. Bugbot is set up for automated code reviews on this repo. Configure here. |
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
There was a problem hiding this comment.
This is a clean backport of #4321 to release/v6.7. pruneDescending now closes its scan iterator before compactPrunedRange runs, so the iterator no longer pins Pebble's readState and stops obsolete sstables from being deleted during a long compaction. The guarded double-close is correct, the deleted-key bounds are cloned so closing the iterator early is safe, and the new seilog logger matches how other sei-db packages create theirs. I found no issues.
Findings: 0 blocking | 0 non-blocking | 0 posted inline
Blockers
- None at the file/PR level.
Non-blocking
- None at the file/PR level.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## release/v6.7 #4508 +/- ##
================================================
- Coverage 61.84% 60.80% -1.04%
================================================
Files 2168 2068 -100
Lines 188739 176790 -11949
================================================
- Hits 116728 107500 -9228
+ Misses 61157 59406 -1751
+ Partials 10854 9884 -970
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
|
@seidroid review |
There was a problem hiding this comment.
Backport of #4321 to release/v6.7. pruneDescending now closes its scan iterator before the range compaction runs, so the iterator's pinned readState no longer keeps obsolete sstables on disk during compaction, and the compaction time is now logged. The change is correct, because the deleted-span keys are cloned before the iterator closes and the guarded close prevents a double close, so I approve; codex's reading found nothing, which matches mine, and it added no findings to keep or drop.
Pre-existing
Already true on the base branch, not introduced here.
- suggestion —
pruneAscendinginsei-db/db_engine/pebbledb/mvcc/db_ascending.go:121still defersitr.Close()until aftercompactPrunedRangeruns (line 232). Nodes on legacy ascending-encoded databases therefore still get the same pinned-sstable disk spike that this change fixes for descending databases. The upstream fix (#4321) and main have the same gap.
1 nit, not posted on the code
sei-db/db_engine/pebbledb/mvcc/db.go:1052— A failed compaction is logged at Info level with only anerrfield. Logging failures at Error/Warn would make them easier to find and alert on.
seidroid review · decision approve · session 7292a75a5c3f4e06a249756ac21ac64d · turn resp_claude_aa68e640ab1b227d8cb80bf89e4283a7 · item a078edaa8fcc56e0abdf738f10363ab9
Findings: 0 blocking | 0 non-blocking | 0 posted inline | 1 pre-existing
|
@seidroid review |
There was a problem hiding this comment.
Nothing material has changed since my last review: the diff is the same db.go change (blob 421fe4dcf8..df65836d87), which closes the pruneDescending scan iterator before range compaction so obsolete sstables stop being pinned, and the updated merge tree reads the same around it. It is still correct and the earlier findings still stand unchanged; codex's reading found nothing, which matches mine, and it added no findings to keep or drop.
Pre-existing
Already true on the base branch, not introduced here.
- suggestion —
pruneAscendinginsei-db/db_engine/pebbledb/mvcc/db_ascending.go:121still defersitr.Close()until aftercompactPrunedRangeruns (line 232). Nodes on legacy ascending-encoded databases therefore still get the same pinned-sstable disk spike that this change fixes for descending databases. The upstream fix (#4321) and main have the same gap.
1 nit, not posted on the code
sei-db/db_engine/pebbledb/mvcc/db.go:1052— A failed compaction is logged at Info level with only anerrfield. Logging failures at Error/Warn would make them easier to find and alert on.
seidroid review · decision approve · session 7292a75a5c3f4e06a249756ac21ac64d · turn resp_claude_a09970af0eb40829d53380487cc36e8d · item 945d399541095a9a953d563a4e01a32b
Findings: 0 blocking | 0 non-blocking | 0 posted inline | 1 pre-existing
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 480fc8c. Configure here.
| // still pinning them. | ||
| if err := closeItr(); err != nil { | ||
| return err | ||
| } |
There was a problem hiding this comment.
Ascending prune still pins read state
High Severity
pruneAscending still closes its full-database iterator only in a defer, so the iterator stays open across compactPrunedRange. That pins Pebble's read state for the whole compaction, so obsolete sstables cannot be deleted and prune tombstones cannot be elided. Legacy databases stay on this path, so the disk spike remains for them.
Triggered by learned rule: sei-db pebbledb: ascending and descending MVCC iterators must stay aligned
Reviewed by Cursor Bugbot for commit 480fc8c. Configure here.
Systematic changelog re-generation for the v6.7.2 patch release. Adds the PRs merged to `release/v6.7` since the v6.7.1 changelog (#4481), as reported by `scripts/generate-changelog.sh release/v6.6 release/v6.7`. New in v6.7.2: - #4520 Backport `release/v6.7`: Export known peer address count as a Prometheus gauge - #4515 Backport `release/v6.7`: Count only connected peers in /net_info n_peers - #4508 Backport `release/v6.7`: Fix pruning issue in SS causing huge disk spike - #4507 Backport `release/v6.7`: Initialize app from last header when a frozen node restarts Shipped in v6.7.1, but merged after #4481 was generated, so the v6.7.1 changelog never listed them: - #4503 Backport `release/v6.7`: Refuse pinned write modes that disagree with persisted migration - #4502 Backport `release/v6.7`: Guard FlatKV rollback floor - #4499 Backport `release/v6.7`: Resolve composite iteration duplicates to memiavl, matching Read - #4489 Backport `release/v6.7`: Remove genesis export - #4486 Backport `release/v6.7`: Re-derive the composite write mode on rollback - #4485 Backport `release/v6.7`: Make the composite store flatKV and write mode atomic - #4474 Backport `release/v6.7`: Validate module names in the memiavl snapshot importer v6.7.1 release bookkeeping: - #4506 Backport `release/v6.7`: Update v6.7 change log in prep to cut v6.7.1 patch - #4482 Bump version to v6.7.1 in prep for patch release Only the `## v6.7` PR list changes, so the `backport release/v6.7` cherry-pick applies cleanly (simulated with `git merge-tree` against `origin/release/v6.7`). Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary - Bump `version.json` from `v6.7.1` to `v6.7.2` for the patch release. - Merge after the `release/v6.7` backport of #4557 (the v6.7.2 changelog), so the release includes it. - New since v6.7.1: #4507, #4508, #4515 and #4520. All four are labeled `non-app-hash-breaking`, so moving from v6.7.1 to v6.7.2 should not need a coordinated validator switch. - Push `v6.7.2` by hand on this PR's merge commit once it has merged; the tagging ruleset stops `uci-release-publish` from creating it. ## Test plan - [x] `git diff --check` Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Masih H. Derkani <m@derkani.org>


Backport of #4321 to
release/v6.7.