feat(crypt): add encrypt-to-disk option to avoid OOM on large file uploads - #2869
Open
yvfx wants to merge 3 commits into
Open
feat(crypt): add encrypt-to-disk option to avoid OOM on large file uploads#2869yvfx wants to merge 3 commits into
yvfx wants to merge 3 commits into
Conversation
…loads When uploading encrypted files through downstream drivers (e.g. quark_open), crypt.Put() passes cipher.EncryptData()'s io.Reader (non-seekable) directly. Drivers that need random reads must cache the entire stream into HybridCache, which writes through Page Cache and causes OOM kills in memory-constrained containers — a 6GB encrypted file dies in ~30s with a 2GB container limit. Add `encrypt_to_disk` option (default: false) to crypt storage config. When enabled, the encrypted stream is first written to a temp file on disk (via TEMP_DIR), then the resulting *os.File (seekable) is passed to downstream drivers. This avoids HybridCache entirely — memory usage drops from ~fileSize to ~50MB. The temp file is automatically cleaned up via streamOut.Closers when op.Put() completes (success or failure). Tradeoff: encryption and upload become serial instead of overlapping, and TEMP_DIR must have enough space for the largest encrypted file. Signed-off-by: yvfx <121942035+yvfx@users.noreply.github.com>
Signed-off-by: yvfx <121942035+yvfx@users.noreply.github.com>
Signed-off-by: yvfx <121942035+yvfx@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary / 摘要
When uploading encrypted files through downstream drivers (e.g. quark_open),
crypt.Put()passescipher.EncryptData()'sio.Reader(non-seekable) directly. Drivers that need random reads (e.g. for proof generation or chunked upload) must cache the entire stream into HybridCache, which writes through Linux Page Cache. In memory-constrained Docker containers, dirty pages accumulate and cause OOM kills — a 6GB encrypted file dies in ~30s with a 2GB container limit.Add
encrypt_to_diskoption (default:false) to crypt storage config. When enabled, the encrypted stream is first written to a temp file on disk (viaTEMP_DIR), then the resulting*os.File(seekable, implementsmodel.File) is passed to downstream drivers. This avoids HybridCache entirely — memory usage drops from ~fileSize to ~50MB.The temp file is automatically cleaned up via
streamOut.Closerswhenop.Put()completes (success or failure).User-visible: new "Encrypt To Disk" checkbox in crypt storage config
Default
false, no behavior change for existing setupsTradeoff: encryption and upload become serial instead of overlapping, and
TEMP_DIRmust have enough space for the largest encrypted fileThis PR has breaking changes.
This PR changes public API, config, storage format, or migration behavior.
This PR requires corresponding changes in related repositories.
Related Issues / 关联 Issue
None found.
Testing / 测试
go test ./...local → crypt(EncryptToDisk=true) → quark_openEncryptToDisk=false(default) preserves original behaviorChecklist / 检查清单
gofmt,go fmt, orprettierwhere applicable.AI Disclosure / AI 使用声明
Tools used / 使用工具:
Usage scope / 使用范围:
Code generation / 代码生成
Review assistance / 审查辅助
I have reviewed and validated all AI-assisted content included in this PR.
I have ensured that all AI-assisted commits include
Co-Authored-Byattribution.I can reproduce all AI-assisted content included in this PR without any AI tools.