Skip to content

store/badger: a driver on Badger, its own lockstep module - #102

Open
dangra wants to merge 4 commits into
masterfrom
badger-store
Open

dangra wants to merge 4 commits into
masterfrom
badger-store

Conversation

@dangra

@dangra dangra commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

A driver.Store on dgraph-io/badger v4, registered as the badger scheme: store.Open("badger:///var/lib/app/durable"), with sync_writes and memtable as query options.

Layout. Badger is one ordered key space on an LSM tree, so this is the bbolt layout without the page tricks. A namespace byte leads every key (a active rows, c cursor, t terminal, e expiry index, s slots). An operation row is keyed by step and phase and upserted in place, state inside it; the output sits in the terminal row; nothing is kept beside its row, since Badger moves values past its threshold to the value log by itself. The terminality commit writes the terminal row and deletes the run's stage and slot in the same transaction, so there is no staged bucket and no sweep. Reap walks the expiry index. A head is four point reads.

Transactions. Serializable with conflict detection at commit. A commit Badger refuses as a conflict is retried, which is how two CreateRuns racing for one slot resolve: both read it free, the second to commit is refused, the retry finds the occupant (TestConcurrentCreateRunsOneWins). Badger coalesces concurrent commits into one write batch, the group commit the bbolt driver does by hand. Sync writes are on by default, matching bbolt's durability; WithSyncWrites(false) trades the last moments before a power loss for throughput. WithMemTableSize bounds Badger's buffer and with it the largest transaction; the value threshold follows it. Badger's own logs are discarded unless WithLogger routes them to slog. A value-log GC pass runs once a minute.

Its own module. Badger's dependency graph (OpenTelemetry API, flatbuffers, klauspost/compress, ristretto) should enter only the binaries that import the driver, so store/badger has its own go.mod, released in lockstep like the otel contrib: the release script now discovers tagged modules under store/ as well as contrib/, verify passes, and docs/releasing.md says so. CI builds, tests, benchmarks, fuzzes, staticchecks, and vulnerability-scans the module; dependabot holds its runtime dependency to patch updates. Two transitive dependencies with uncalled advisories are pinned at their fixed versions.

Shared contract test. The differential fuzzer moves out of the bbolt test into store/internal/storetest (Seed, Fuzz, Canonicalize); both persistent drivers call it, and the badger seed corpus is bbolt's. Twenty seconds of fuzzing found no divergence from store/mem.

Numbers. BenchmarkCompare runs ten engine workloads over every in-tree store; medians of seven passes on an NVMe disk (ext4 on LUKS), random payloads. The figures first posted here were taken on tmpfs, where fsync is free, and are withdrawn.

workload, runs/sec mem bbolt badger badger, no sync badger before the fixes
Machine start, one at a time 22,744 148 173 3,017 168
Small action, one at a time 61,378 310 494 5,441 496
Machine start burst (200) 30,047 2,152 3,482 5,783 187
Small action burst (1,000) 60,890 14,022 7,673 10,232 376
Retry storm 1,319 293 624 1,055 57
Unwind wave 22,655 848 1,926 3,848 103
Await fan-in 92,460 4,507 8,881 16,081 703

The first real-disk run put Badger with sync writes 5 to 35 times behind bbolt under concurrency. Four fixes, all in this PR:

  • Group commit (batch.go). Badger in sync-writes mode synced its log per commit and shared nothing: 10,047 syncs on the 1,000-run burst where bbolt made 50. Every write now goes through one goroutine that commits whatever is queued in one transaction, the way bbolt's Batch shares an fsync; a write that fails runs alone so its error is its own. Sync writes off plus a shared DB.Sync was rejected: Badger does not sync the log of a memtable it retires, so a power loss could drop a commit reported durable.
  • Snapshot walks. An iterator opened in a write transaction sorts every pending write of the transaction, the whole batch once writes are batched. Terminality and the duplicate-schedule path walk a read-only snapshot instead; with one writer that is what has committed, and the cancel row, the only run row another write in the batch can add, is read and deleted by key. Recovery: 1,918 → 314 ms (bbolt 192).
  • Blob cache. bbolt's cache of in-flight inputs and states moves to store/internal/blobcache and both drivers use it. States get rows of their own beside the operation row, whose value leads with a flag byte saying whether one exists, so a read fetches blobs by point reads, from the cache when it can; Badger's iterator copies every value it passes. A retrying run allocates 327 KB instead of 3.0 MB (bbolt 307 KB).
  • Compact at close. Deleted rows stay until compaction rewrites their tables, and level 0 compacts only once several tables pile up. Close compacts level 0: after reaping everything Badger holds 16 KB after Close instead of 78 MB. While open it still holds 90 MB of dead rows (bbolt 10 MB); the suite now reports disk both open and closed.

bbolt still leads on the small-action burst (14,022 vs 7,673; Badger without syncs reaches 10,232, so it is write-path CPU), status polls (758k vs 402k/s, five point reads per head), recovery, and disk held while open.

TMPDIR=/path/on/the/disk/under/test go test -run xxx -bench Compare -benchtime 1x -count 7 ./store/badger/

🤖 Generated with Claude Code

Badger is one ordered key space on an LSM tree, so the driver is the
bbolt layout without the page tricks: a namespace byte leads every
key, an operation row is keyed by step and phase and upserted in place
with its state inside, the output sits in the terminal row, and the
terminality commit deletes the run's stage and slot in its own
transaction. Transactions are serializable; a commit refused as a
conflict is retried, which is how two CreateRuns racing for one slot
resolve. Sync writes are on by default; WithMemTableSize bounds
Badger's buffer and with it the largest transaction.

It is a module of its own, github.com/dangra/durable/store/badger,
because Badger's dependency graph (OpenTelemetry, flatbuffers,
compress) should enter only the binaries that import the driver. The
release script discovers tagged modules under store/ as well as
contrib/, CI builds, tests, fuzzes, staticchecks, and vulnerability
scans the module, and dependabot holds its runtime dependency to patch
updates.

The differential contract fuzzer moves from the bbolt test into
store/internal/storetest, shared by both persistent drivers.
Copilot AI balanced review requested due to automatic review settings October 7, 2026 17:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A concurrent accepted cancellation can be lost during terminalization, and memtable parsing permits integer overflow.

3 open findings
What changed in this PR

Adds a lockstep-released Badger-backed persistent store, with shared contract fuzzing and full CI/release integration.

Changes:

  • Implements Badger storage, URI options, garbage collection, tests, and benchmarks.
  • Extracts differential fuzzing for reuse by persistent drivers.
  • Integrates the module into documentation, releases, CI, and Dependabot.
File Description
store/​store.go Documents the Badger driver.
store/​internal/​storetest/​storetest.go Adds shared differential fuzzing.
store/​bbolt/​fuzz_test.go Uses the shared fuzzer.
store/​badger/​store.go Implements Badger persistence.
store/​badger/​store_test.go Tests storage semantics and recovery.
store/​badger/​open.go Registers URI configuration.
store/​badger/​open_test.go Tests URI opening and validation.
store/​badger/​fuzz_test.go Applies shared contract fuzzing.
store/​badger/​bench_test.go Adds driver benchmarks.
store/​badger/​go.mod Defines the separate module.
store/​badger/​go.sum Locks module dependencies.
store/​badger/​testdata/​fuzz/​FuzzStoreContract/​9ce1b12254efbcda Adds fuzz corpus input.
store/​badger/​testdata/​fuzz/​FuzzStoreContract/​880fe9fa59e5ecf0 Adds fuzz corpus input.
store/​badger/​testdata/​fuzz/​FuzzStoreContract/​03e3357aa67793e4 Adds fuzz corpus input.
README.md Lists Badger as persistent storage.
doc.go Updates package documentation.
spec/​README.md Adds Badger to the architecture.
spec/​04-engine.md Documents shared driver fuzzing.
docs/​releasing.md Covers store-module releases.
scripts/​release.sh Discovers and tags store modules.
.github/​workflows/​ci.yml Tests, fuzzes, and scans Badger.
.github/​dependabot.yml Configures Badger dependency updates.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread store/badger/store.go Resolved
Comment thread store/badger/open.go Resolved
Comment thread store/badger/store.go Resolved
dangra and others added 3 commits October 7, 2026 14:39
…memtable sizes

Badger's conflict detection sees every point read, a missing key
included, but of a prefix walk only the seek key and the rows it
returned. A RequestCancel committing while a terminality commit was in
flight inserted a row the commit's walk never saw, so the commit went
through, wrote a terminal record without the request, and deleted the
row RequestCancel had reported accepted. The terminality commit now
point-reads the cancel key, so the cancel committing first forces a
retry that folds it in. TestCancelRacingTerminalityIsKept interleaves
the two transactions by hand and fails without the read.

parseBytes rejects counts that overflow int64 once the unit is applied,
and the package doc counts a nonterminal head as five point reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…workloads

BenchmarkCompare runs ten engine workloads over every in-tree store
(and Badger without sync writes, for scale): sequential and burst
machine starts and small actions, a retry storm, an unwind wave, an
await fan-in, status polling, recovery, and reap. Each pass opens a
fresh store and reports per-run throughput, latency, allocations, and
disk use. Payloads are random so Badger's block compression does not
flatter it, and the suite warns when TMPDIR is tmpfs, where fsync is
free. It lives in this module, the only one that can import all three
drivers, and is skipped under -short, which CI's benchmark smoke step
now passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…close

The store comparison found four gaps against bbolt; each has a fix.

Group commit. Badger in sync-writes mode syncs its log per committed
transaction and shares nothing between concurrent ones: a 1,000-run
burst paid about 10,000 syncs where bbolt paid 50. Every write now goes
through one goroutine that commits whatever writes are queued in one
transaction, the way bbolt's Batch shares an fsync. A write that fails
runs alone so its error is its own, and the rest retry as a batch.
Turning sync writes off and sharing DB.Sync instead was rejected:
memtable rotation does not sync the log it retires, so a power loss
could drop a commit that had been reported durable.

Blob cache. The engine reads a run whole each time its worker picks it
up, and Badger's iterator copies every value it passes, so a retrying
run copied its 32 KB input on every attempt. bbolt's cache of in-flight
inputs and states moves to store/internal/blobcache and both drivers
use it. States get a row of their own beside the operation row, whose
value now leads with a flag byte saying whether one is there, so a read
fetches them by point reads, from the cache when it can, and walks only
the operation rows.

Snapshot walks. An iterator opened in a write transaction sorts every
pending write of the transaction, which under batching is the whole
batch; terminality and the CreateRun dedup path walked there, which
made recovery quadratic in the batch. They walk a read-only snapshot
now. With one writer the snapshot is what is committed, and the only
row of a run another write in the same batch can add is a cancel
request, which terminality reads and deletes by key.

Compact at close. Deleted rows stay on disk until compaction rewrites
their tables, and level 0 is compacted only once several tables pile
up, so a quiet store kept what it had reaped. Close compacts level 0.
The comparison now reports disk both while the store is open and after
Close.

Head reads decode rows in place instead of copying them first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants