Skip to content

remove bbolt integrity check entirely - #115

Open
Trent-TSE wants to merge 2 commits into
mainfrom
devin/1784145521-bbolt-check-sigterm
Open

remove bbolt integrity check entirely#115
Trent-TSE wants to merge 2 commits into
mainfrom
devin/1784145521-bbolt-check-sigterm

Conversation

@Trent-TSE

@Trent-TSE Trent-TSE commented Jul 15, 2026

Copy link
Copy Markdown

Summary

The post-step bbolt integrity check runs under systemd-run --scope -p RuntimeMaxSec=6s. Once cache.db gets large (~96–112 MB for supabase/supabox), the cold-read check exceeds 6s, systemd SIGTERMs it (exit 143), and the unhandled exit code fell through to the corruption branch → Skipping sticky disk commit due to integrity check failure on every run, permanently freezing the builder cache. Observed across 10 repos in a single 3-hour window; zero genuine bbolt corruption reports in fleet logs over the same period.

Per discussion, the check was added to chase the copy-up corruption bug fixed in BLA-3252 and no longer fires legitimately — so this drops it entirely rather than special-casing exit 143:

  • delete checkBoltDbIntegrity and its commit-gating (integrityCheckPassed) in the post step
  • delete reporter.reportIntegrityCheckFailure
  • skip-integrity-check input kept in action.yml (marked deprecated, no effect) so existing workflows don't warn
  • post-step logDatabaseHashes retained, relabeled "before unmount"

dist/ rebuilt with pnpm build. release/v2 counterpart: #116.

Link to Devin session: https://app.devin.ai/sessions/35c21220e53a4e43a8287135eba4351d
Requested by: @Trent-TSE


View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled. (Staging)

… skip

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@adityamaru

Copy link
Copy Markdown
Contributor

@piob-io this seems like a good fix for the symptom but the value of these bbolt checks existing is questionable now. The context here is that we added these to chase down a notorious corruption bug that was fixed in https://linear.app/blacksmith-sh/issue/BLA-3252/stickydisk-copyup-corruption-investigation#comment-b4dffbf7

So we could consider dropping these but after checking they dont fire legitimately for customers anymore

@devin-ai-integration

Copy link
Copy Markdown
Contributor

Data point on whether they still fire legitimately: I checked observability.logs for the last 6 hours across the fleet — every single integrity-check failure warning was a benign one (RuntimeMaxSec SIGTERM / timeout / memory-limit skip). Zero warnings matched the real-corruption path (⚠ <file>: <bbolt output> where bbolt actually reported a mismatch). So at least in that window the checks caught no genuine corruption, only false positives that were blocking commits. Longer-window queries time out via the MCP client, but the same query can be run over 30d to confirm before dropping the checks entirely.

@piob-io

piob-io commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Cleaning this up sounds great to me. Sorry for double-load during the transition, but we need second PR to merge into the release/v2 branch (nothing to re-tag for that one, just targeted branch merge).

@devin-ai-integration

Copy link
Copy Markdown
Contributor

release/v2 counterpart is up: #116 (CI green).

@piob-io

piob-io commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Devin, as discussed, just drop the whole corruption check in both PRs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@linear-code

linear-code Bot commented Jul 15, 2026

Copy link
Copy Markdown

BLA-4444

@devin-ai-integration devin-ai-integration Bot changed the title fix: don't treat RuntimeMaxSec SIGTERM (exit 143) as bbolt corruption remove bbolt integrity check entirely Jul 15, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor

Done — dropped the integrity check entirely in both PRs (this one and #116): removed checkBoltDbIntegrity, its commit-gating in the post step, and reporter.reportIntegrityCheckFailure. Kept the skip-integrity-check input in action.yml as a deprecated no-op so existing workflows don't break, and kept the post-step logDatabaseHashes (relabeled "before unmount"). dist rebuilt.

@piob-io

piob-io commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Update PR descriptions, they look stale.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants