Skip to content

feat(crypt): add encrypt-to-disk option to avoid OOM on large file uploads - #2869

Open
yvfx wants to merge 3 commits into
OpenListTeam:mainfrom
yvfx:main
Open

feat(crypt): add encrypt-to-disk option to avoid OOM on large file uploads#2869
yvfx wants to merge 3 commits into
OpenListTeam:mainfrom
yvfx:main

Conversation

@yvfx

@yvfx yvfx commented Jul 29, 2026

Copy link
Copy Markdown

Summary / 摘要

When uploading encrypted files through downstream drivers (e.g. quark_open), crypt.Put() passes cipher.EncryptData()'s io.Reader (non-seekable) directly. Drivers that need random reads (e.g. for proof generation or chunked upload) must cache the entire stream into HybridCache, which writes through Linux Page Cache. In memory-constrained Docker containers, dirty pages accumulate and cause OOM kills — a 6GB encrypted file dies in ~30s with a 2GB container limit.

Add encrypt_to_disk option (default: false) to crypt storage config. When enabled, the encrypted stream is first written to a temp file on disk (via TEMP_DIR), then the resulting *os.File (seekable, implements model.File) is passed to downstream drivers. This avoids HybridCache entirely — memory usage drops from ~fileSize to ~50MB.

The temp file is automatically cleaned up via streamOut.Closers when op.Put() completes (success or failure).

  • User-visible: new "Encrypt To Disk" checkbox in crypt storage config

  • Default false, no behavior change for existing setups

  • Tradeoff: encryption and upload become serial instead of overlapping, and TEMP_DIR must have enough space for the largest encrypted file

  • This PR has breaking changes.

  • This PR changes public API, config, storage format, or migration behavior.

  • This PR requires corresponding changes in related repositories.

Related Issues / 关联 Issue

None found.

Testing / 测试

  • go test ./...
  • Manual test / 手动测试:
    • Built custom Docker image from this branch
    • Uploaded 6GB file via local → crypt(EncryptToDisk=true) → quark_open
    • Container memory stayed <100MB throughout (previously OOM killed at 30s with 2GB limit)
    • Temp file on disk cleaned up after upload completed
    • Verified EncryptToDisk=false (default) preserves original behavior

Checklist / 检查清单

  • I have read CONTRIBUTING.
  • I confirm this contribution follows the repository license, contribution policy, and code of conduct.
  • I have formatted the changed code with gofmt, go fmt, or prettier where applicable.
  • I have requested review from relevant maintainers or code owners where applicable.

AI Disclosure / AI 使用声明

  • This PR includes AI-assisted content.

Tools used / 使用工具:

  • ChatGPT
  • Codex
  • GitHub Copilot
  • Claude
  • Gemini
  • Other (please specify) / 其他(请注明):

Usage scope / 使用范围:

  • Code generation / 代码生成

  • Review assistance / 审查辅助

  • I have reviewed and validated all AI-assisted content included in this PR.

  • I have ensured that all AI-assisted commits include Co-Authored-By attribution.

  • I can reproduce all AI-assisted content included in this PR without any AI tools.

yvfx added 3 commits July 29, 2026 13:58
…loads

When uploading encrypted files through downstream drivers (e.g. quark_open),
crypt.Put() passes cipher.EncryptData()'s io.Reader (non-seekable) directly.
Drivers that need random reads must cache the entire stream into HybridCache,
which writes through Page Cache and causes OOM kills in memory-constrained
containers — a 6GB encrypted file dies in ~30s with a 2GB container limit.

Add `encrypt_to_disk` option (default: false) to crypt storage config. When
enabled, the encrypted stream is first written to a temp file on disk
(via TEMP_DIR), then the resulting *os.File (seekable) is passed to
downstream drivers. This avoids HybridCache entirely — memory usage drops
from ~fileSize to ~50MB.

The temp file is automatically cleaned up via streamOut.Closers when
op.Put() completes (success or failure).

Tradeoff: encryption and upload become serial instead of overlapping,
and TEMP_DIR must have enough space for the largest encrypted file.

Signed-off-by: yvfx <121942035+yvfx@users.noreply.github.com>
Signed-off-by: yvfx <121942035+yvfx@users.noreply.github.com>
Signed-off-by: yvfx <121942035+yvfx@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant