Conversation
Throughput and memory improvements for the scan_reader (SCAN+DUMP+PTTL+RESTORE) path and the standalone redis_writer. On a ~38M-key standalone source, end-to-end scan-based sync goes from ~90k to ~280k keys/s with flat memory. All new options default to the prior behavior except the scan queue bound (see below), which fixes a latent OOM. - redis_writer: add 'parallel' (default 1) — fan RESTOREs across N target connections draining a shared channel. A single connection was the cap while the target had ample headroom. - scan_reader: add 'dump_parallel' (default 1) — N source connections each run DUMP+PTTL, draining the shared scan queue (the Go channel hands each key to exactly one worker; no key partitioning needed). - scan_reader: batch DUMP/PTTL sends — add client.SendNoFlush(); dump() buffers source commands and flushes every 128 keys or 5ms instead of Send()'s flush-per-command. CPU profiling showed ~64% of time in socket syscalls; this was the single biggest win. - scan_reader: bound the scan->dump queue via 'dump_queue_size' (default 100000, previously hardcoded to 100000000). The old value buffered ~the whole keyspace in RAM (OOM on large DBs) and the large live heap also degraded throughput via GC. Set a large value to restore the old behavior.
|
|
…commands The loader unconditionally collected every Rewrite() command into a slice before deciding RESTORE-vs-individual, so a single huge collection (a 120M-member zset in our data) materialized the whole raw value plus all rewrite commands in memory and OOMed the loader (~54GiB). Cap the raw-value capture at target_redis_proto_max_bulk_len via a small io.Writer: while a value fits the cap, buffer its commands (it may RESTORE); once it overflows, flush the buffer and stream the remaining commands directly, dropping the raw bytes. Peak heap is now bounded by the threshold regardless of object size. Small values still replay as a single RESTORE; oversized ones stream as individual commands (Rewrite's leading del gives replace semantics).
oOraph
force-pushed
the
bound_scan_reader_dump_queue
branch
from
June 26, 2026 18:15
1fb5dc1 to
9e7d7be
Compare
animavitis
added a commit
to animavitis/RedisShake
that referenced
this pull request
Sep 16, 2026
* scan_reader: drop filtered keys before DUMP filter.Filter runs on entries emitted by the reader, which is after DUMP and PTTL have already been issued for every key the scan returned. Splitting a migration into slices with allow_key_prefix therefore re-reads the whole dataset on every pass and costs more than a single run. Move the key check into the scan loop so a key that cannot pass the filter is never queued for DUMP. Verified against an 8.7M key source: a single hex prefix slice copied 543,589 keys while reading only that slice, where the unpatched build had already read 1.97M keys at 35 percent of the scan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * scan_reader: add dump_queue_size to bound the scan to dump handoff The handoff queue is hardcoded at 100000000 entries, so on a large database scan() races ahead of dump() and buffers most of the keyspace in RAM. Two migration runs reached 1.16M queued keys and 3.2 GB resident before finishing. Expose the size as scan_reader.dump_queue_size, default 100000. A bounded queue makes Put() block, backpressuring the scan to the speed of the dump so memory stays flat. Set a large value to restore the previous behaviour. The idea and the default are taken from the upstream proposal in PR tair-opensource#1056, reduced to this one self-contained change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
animavitis
added a commit
to animavitis/RedisShake
that referenced
this pull request
Sep 18, 2026
* scan_reader: drop filtered keys before DUMP filter.Filter runs on entries emitted by the reader, which is after DUMP and PTTL have already been issued for every key the scan returned. Splitting a migration into slices with allow_key_prefix therefore re-reads the whole dataset on every pass and costs more than a single run. Move the key check into the scan loop so a key that cannot pass the filter is never queued for DUMP. Verified against an 8.7M key source: a single hex prefix slice copied 543,589 keys while reading only that slice, where the unpatched build had already read 1.97M keys at 35 percent of the scan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * scan_reader: add dump_queue_size to bound the scan to dump handoff The handoff queue is hardcoded at 100000000 entries, so on a large database scan() races ahead of dump() and buffers most of the keyspace in RAM. Two migration runs reached 1.16M queued keys and 3.2 GB resident before finishing. Expose the size as scan_reader.dump_queue_size, default 100000. A bounded queue makes Put() block, backpressuring the scan to the speed of the dump so memory stays flat. Set a large value to restore the previous behaviour. The idea and the default are taken from the upstream proposal in PR tair-opensource#1056, reduced to this one self-contained change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * build: give the Windows binary its .exe suffix The dist target named every binary redis-shake regardless of target platform, so the Windows archives shipped a file Windows will not execute directly and that a user has to rename before it works. Name it redis-shake.exe for windows targets and remove it after packaging, so it does not sit in bin/ next to the native build that follows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Throughput & memory improvements for the
scan_reader(SCAN + DUMP + PTTL + RESTORE) path and the standaloneredis_writer. On a ~40M-key standalone source, end-to-end scan-based sync went from ~25k to ~280k keys/s with flat memory.Motivation
Migrating a large keyspace (tens of millions of keys) via
scan_reader— e.g. from a server that can't servePSYNC, like Dragonfly — was throughput- and memory-limited:DUMP+PTTLper key caps ~22k keys/s on round-trip latency.client.Send()flushes after every command, so each key costs 2 source send syscalls; CPU profiling showed ~64% of time in socket syscalls.needDumpQueueis hardcoded to100_000_000, so on a large DB it buffers ~the whole keyspace in RAM (OOM), and the large live heap also degrades throughput via GC during scan-fill.Changes
redis_writer.parallel(default1, unchanged behavior) — fan RESTOREs across N target connections draining a shared channel.scan_reader.dump_parallel(default1, unchanged behavior) — N source connections running DUMP+PTTL, draining the shared scan queue (the Go channel hands each key to exactly one worker — no key partitioning).DUMP/PTTLsends — addclient.SendNoFlush();dump()buffers source commands and flushes every 128 keys or 5ms instead of per-command. Amortizes hundreds of commands per syscall — the single biggest win. Internal only (no API change); preserves command/reply ordering.scan_reader.dump_queue_size(default100000, was an effectively-unbounded hardcoded100000000) — bound the scan→dump handoff queue. The old value buffered ~the whole keyspace in RAM (OOM risk on large DBs) and the oversized live heap hurt throughput via GC; the bounded default backpressuresscan()todump()speed and keeps memory flat. Set a large value to restore the previous behavior.Benchmark (Tens of millions of keys, standalone → standalone)
count=50000Memory stays ~50 MiB with the bounded queue (vs many GiB unbounded).
Compatibility
parallelanddump_paralleldefault to1(no change). The one default change isdump_queue_size(100000 vs the old hardcoded 100000000): it caps the in-flight scan buffer — throughput is unaffected or improved (benchmarks above), only peak memory changes (for the better). Setdump_queue_sizelarge to opt back into the old unbounded behavior.