Repository navigation
Open in-memory databases without copying the buffer - #48
Merged
Merged
Conversation
DefaultStorageFactory copied the entire MemoryStream contents with ToArray(), doubling peak memory for fully in-memory databases. Use TryGetBuffer to alias the stream's own buffer when it is exposable, falling back to ToArray() otherwise. Add a ReadOnlyDatabase.OpenAsync(ReadOnlyMemory<byte>) overload that reads the caller's buffer directly with no Stream or copy involved; the catalog parser is fed through an internal seekable read-only Stream view over the memory. InMemoryPageLoader now takes ReadOnlyMemory<byte> since it never writes to the buffer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
DefaultStorageFactorynow usesMemoryStream.TryGetBufferto hand the stream's own buffer toInMemoryPageLoader, falling back toToArray()only when the buffer is not exposable (e.g.new MemoryStream(bytes)).ReadOnlyDatabase.OpenAsync(ReadOnlyMemory<byte> data)opens a database directly from a caller-owned buffer with noStreamand no copy. The catalog parser is fed through a new internal seekableReadOnlyMemoryStream.InMemoryPageLoadernow takesReadOnlyMemory<byte>instead ofMemory<byte>— it never writes to the buffer. Source-compatible for existing callers (byte[]/Memory<byte>convert implicitly).Why
Opening a fully in-memory database previously always paid a second full-size allocation (
ms.ToArray()), doubling peak memory — noticeable for the embedded/game use cases where the whole file is already in memory.Notes for review
TryGetBuffersite.ReadOnlyMemoryStreamimplements only what the catalog parser needs (Read/ReadAsync/ReadByte/Seek); writes throwNotSupportedException.ToArray()fallback. 98/98 tests pass (net10.0).🤖 Generated with Claude Code