Skip to content

Backport release/v6.7: Fix pruning issue in SS causing huge disk spike - #4508

Merged
alexander-sei merged 4 commits into
release/v6.7from
backport-4321-to-release/v6.7
Oct 12, 2026
Merged

alexander-sei merged 4 commits into
release/v6.7from
backport-4321-to-release/v6.7

Conversation

@seidroid

@seidroid seidroid Bot commented Oct 8, 2026

Copy link
Copy Markdown

Backport of #4321 to release/v6.7.

* Pruning in SS removes old state to keep a recent window of x blocks.
* From Pebble perspective, pruning means iterating all keys and removing
the ones older than the retention window (+ keeping some past ones if
there were no changes). But deleting does delete right a way. For pebble
a deletion is inserting a tombstone value that means that a given key no
longer exists. Then Pebble has to "clean that up" removing the old value
+ the tombstone during compaction and other background tasks.
* The bug: When iterating the keys to prune, the iterator was closed
after (via defer) after completing the compaction. But Pebble could not
compact because the iterator was active, hence "locking" the data that
was supposed to be removed.

left: fix
right: the bug
<img width="378" height="339" alt="image"
src="https://github.com/user-attachments/assets/6545ea1f-3035-46a5-a424-376926dee7b1"
/>

(cherry picked from commit 58b7631)
@cursor

cursor Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches MVCC prune lifecycle and Pebble compaction timing; low logic change but affects production disk usage during pruning.

Overview
Fixes a state-store pruning disk spike by closing the full-DB scan iterator before post-prune compaction runs, instead of holding it open until function return.

An open Pebble iterator pins readState, which prevents reclaiming obsolete SSTables while compactPrunedRange runs (often slowly on large deleted spans). The change adds an idempotent closeItr helper (defer still handles errors mid-scan) and logs compaction duration via seilog (pruneDescending: compacted pruned range).

Reviewed by Cursor Bugbot for commit d1aa21b. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedOct 12, 2026, 4:12 PM

@masih
masih marked this pull request as draft October 8, 2026 09:11

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a clean backport of #4321 to release/v6.7. pruneDescending now closes its scan iterator before compactPrunedRange runs, so the iterator no longer pins Pebble's readState and stops obsolete sstables from being deleted during a long compaction. The guarded double-close is correct, the deleted-key bounds are cloned so closing the iterator early is safe, and the new seilog logger matches how other sei-db packages create theirs. I found no issues.

Findings: 0 blocking | 0 non-blocking | 0 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • None at the file/PR level.

@masih masih added the on-hold Ticket is on hold label Oct 8, 2026
@codecov

codecov Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.61538% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.80%. Comparing base (210b31b) to head (d1aa21b).

Files with missing lines Patch % Lines
sei-db/db_engine/pebbledb/mvcc/db.go 84.61% 1 Missing and 1 partial ⚠️
Additional details and impacted files

Impacted file tree graph

@@               Coverage Diff                @@
##           release/v6.7    #4508      +/-   ##
================================================
- Coverage         61.84%   60.80%   -1.04%     
================================================
  Files              2168     2068     -100     
  Lines            188739   176790   -11949     
================================================
- Hits             116728   107500    -9228     
+ Misses            61157    59406    -1751     
+ Partials          10854     9884     -970     
Flag Coverage Δ
sei-chain-pr 64.74% <84.61%> (?)
sei-db 69.80% <ø> (ø)
sei-db-state-db ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sei-db/db_engine/pebbledb/mvcc/db.go 67.43% <84.61%> (+0.21%) ⬆️

... and 100 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@alexander-sei
alexander-sei marked this pull request as ready for review October 9, 2026 12:41
@alexander-sei alexander-sei added non-app-hash-breaking and removed on-hold Ticket is on hold labels Oct 9, 2026
@alexander-sei
alexander-sei self-requested a review October 9, 2026 18:16
@alexander-sei
alexander-sei marked this pull request as draft October 9, 2026 19:47
@alexander-sei
alexander-sei marked this pull request as ready for review October 9, 2026 19:47
@alexander-sei
alexander-sei marked this pull request as draft October 9, 2026 19:47
@alexander-sei
alexander-sei marked this pull request as ready for review October 9, 2026 19:48
@alexander-sei

Copy link
Copy Markdown
Contributor

@seidroid review

@seidroid seidroid Bot left a comment •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backport of #4321 to release/v6.7. pruneDescending now closes its scan iterator before the range compaction runs, so the iterator's pinned readState no longer keeps obsolete sstables on disk during compaction, and the compaction time is now logged. The change is correct, because the deleted-span keys are cloned before the iterator closes and the guarded close prevents a double close, so I approve; codex's reading found nothing, which matches mine, and it added no findings to keep or drop.

Pre-existing

Already true on the base branch, not introduced here.

  • suggestion — pruneAscending in sei-db/db_engine/pebbledb/mvcc/db_ascending.go:121 still defers itr.Close() until after compactPrunedRange runs (line 232). Nodes on legacy ascending-encoded databases therefore still get the same pinned-sstable disk spike that this change fixes for descending databases. The upstream fix (#4321) and main have the same gap.
1 nit, not posted on the code
  • sei-db/db_engine/pebbledb/mvcc/db.go:1052 — A failed compaction is logged at Info level with only an err field. Logging failures at Error/Warn would make them easier to find and alert on.

seidroid review · decision approve · session 7292a75a5c3f4e06a249756ac21ac64d · turn resp_claude_aa68e640ab1b227d8cb80bf89e4283a7 · item a078edaa8fcc56e0abdf738f10363ab9

Findings: 0 blocking | 0 non-blocking | 0 posted inline | 1 pre-existing

@alexander-sei

Copy link
Copy Markdown
Contributor

@seidroid review

@seidroid seidroid Bot left a comment •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing material has changed since my last review: the diff is the same db.go change (blob 421fe4dcf8..df65836d87), which closes the pruneDescending scan iterator before range compaction so obsolete sstables stop being pinned, and the updated merge tree reads the same around it. It is still correct and the earlier findings still stand unchanged; codex's reading found nothing, which matches mine, and it added no findings to keep or drop.

Pre-existing

Already true on the base branch, not introduced here.

  • suggestion — pruneAscending in sei-db/db_engine/pebbledb/mvcc/db_ascending.go:121 still defers itr.Close() until after compactPrunedRange runs (line 232). Nodes on legacy ascending-encoded databases therefore still get the same pinned-sstable disk spike that this change fixes for descending databases. The upstream fix (#4321) and main have the same gap.
1 nit, not posted on the code
  • sei-db/db_engine/pebbledb/mvcc/db.go:1052 — A failed compaction is logged at Info level with only an err field. Logging failures at Error/Warn would make them easier to find and alert on.

seidroid review · decision approve · session 7292a75a5c3f4e06a249756ac21ac64d · turn resp_claude_a09970af0eb40829d53380487cc36e8d · item 945d399541095a9a953d563a4e01a32b

Findings: 0 blocking | 0 non-blocking | 0 posted inline | 1 pre-existing

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 480fc8c. Configure here.

// still pinning them.
if err := closeItr(); err != nil {
return err
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ascending prune still pins read state

High Severity

pruneAscending still closes its full-database iterator only in a defer, so the iterator stays open across compactPrunedRange. That pins Pebble's read state for the whole compaction, so obsolete sstables cannot be deleted and prune tombstones cannot be elided. Legacy databases stay on this path, so the disk spike remains for them.

Fix in Cursor Fix in Web

Triggered by learned rule: sei-db pebbledb: ascending and descending MVCC iterators must stay aligned

Reviewed by Cursor Bugbot for commit 480fc8c. Configure here.

@alexander-sei
alexander-sei merged commit d09acd8 into release/v6.7 Oct 12, 2026
72 checks passed
@alexander-sei
alexander-sei deleted the backport-4321-to-release/v6.7 branch October 12, 2026 20:52
masih pushed a commit that referenced this pull request Oct 16, 2026
Systematic changelog re-generation for the v6.7.2 patch release.

Adds the PRs merged to `release/v6.7` since the v6.7.1 changelog
(#4481), as reported by `scripts/generate-changelog.sh release/v6.6
release/v6.7`.

New in v6.7.2:
- #4520 Backport `release/v6.7`: Export known peer address count as a
Prometheus gauge
- #4515 Backport `release/v6.7`: Count only connected peers in /net_info
n_peers
- #4508 Backport `release/v6.7`: Fix pruning issue in SS causing huge
disk spike
- #4507 Backport `release/v6.7`: Initialize app from last header when a
frozen node restarts

Shipped in v6.7.1, but merged after #4481 was generated, so the v6.7.1
changelog never listed them:
- #4503 Backport `release/v6.7`: Refuse pinned write modes that disagree
with persisted migration
- #4502 Backport `release/v6.7`: Guard FlatKV rollback floor
- #4499 Backport `release/v6.7`: Resolve composite iteration duplicates
to memiavl, matching Read
- #4489 Backport `release/v6.7`: Remove genesis export
- #4486 Backport `release/v6.7`: Re-derive the composite write mode on
rollback
- #4485 Backport `release/v6.7`: Make the composite store flatKV and
write mode atomic
- #4474 Backport `release/v6.7`: Validate module names in the memiavl
snapshot importer

v6.7.1 release bookkeeping:
- #4506 Backport `release/v6.7`: Update v6.7 change log in prep to cut
v6.7.1 patch
- #4482 Bump version to v6.7.1 in prep for patch release

Only the `## v6.7` PR list changes, so the `backport release/v6.7`
cherry-pick applies cleanly (simulated with `git merge-tree` against
`origin/release/v6.7`).

Co-authored-by: Cursor <cursoragent@cursor.com>
masih added a commit that referenced this pull request Oct 16, 2026
## Summary
- Bump `version.json` from `v6.7.1` to `v6.7.2` for the patch release.
- Merge after the `release/v6.7` backport of #4557 (the v6.7.2
changelog), so the release includes it.
- New since v6.7.1: #4507, #4508, #4515 and #4520. All four are labeled
`non-app-hash-breaking`, so moving from v6.7.1 to v6.7.2 should not need
a coordinated validator switch.
- Push `v6.7.2` by hand on this PR's merge commit once it has merged;
the tagging ruleset stops `uci-release-publish` from creating it.

## Test plan
- [x] `git diff --check`

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Masih H. Derkani <m@derkani.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants