Skip to content

Open in-memory databases without copying the buffer - #48

Merged
hadashiA merged 2 commits into
mainfrom
claude/open-from-memory
Nov 8, 2026
Merged

hadashiA merged 2 commits into
mainfrom
claude/open-from-memory

Conversation

@hadashiA

@hadashiA hadashiA commented Nov 8, 2026

Copy link
Copy Markdown
Owner

What

  • DefaultStorageFactory now uses MemoryStream.TryGetBuffer to hand the stream's own buffer to InMemoryPageLoader, falling back to ToArray() only when the buffer is not exposable (e.g. new MemoryStream(bytes)).
  • New overload ReadOnlyDatabase.OpenAsync(ReadOnlyMemory<byte> data) opens a database directly from a caller-owned buffer with no Stream and no copy. The catalog parser is fed through a new internal seekable ReadOnlyMemoryStream.
  • InMemoryPageLoader now takes ReadOnlyMemory<byte> instead of Memory<byte> — it never writes to the buffer. Source-compatible for existing callers (byte[] / Memory<byte> convert implicitly).

Why

Opening a fully in-memory database previously always paid a second full-size allocation (ms.ToArray()), doubling peak memory — noticeable for the embedded/game use cases where the whole file is already in memory.

Notes for review

  • Zero-copy paths alias the live buffer, so the stream/memory must not be mutated while the database is open. This is noted on the overload's XML doc and at the TryGetBuffer site.
  • ReadOnlyMemoryStream implements only what the catalog parser needs (Read/ReadAsync/ReadByte/Seek); writes throw NotSupportedException.
  • Tests cover all three paths: the memory overload, the exposable-buffer stream path, and the ToArray() fallback. 98/98 tests pass (net10.0).

🤖 Generated with Claude Code

hadashiA and others added 2 commits November 8, 2026 10:56
DefaultStorageFactory copied the entire MemoryStream contents with
ToArray(), doubling peak memory for fully in-memory databases. Use
TryGetBuffer to alias the stream's own buffer when it is exposable,
falling back to ToArray() otherwise.

Add a ReadOnlyDatabase.OpenAsync(ReadOnlyMemory<byte>) overload that
reads the caller's buffer directly with no Stream or copy involved;
the catalog parser is fed through an internal seekable read-only
Stream view over the memory. InMemoryPageLoader now takes
ReadOnlyMemory<byte> since it never writes to the buffer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hadashiA
hadashiA merged commit af92d9c into main Nov 8, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant