feat(sdk): size the manifest for 50 TiB payloads (DSPX-4651) - #1021
Open
dmihalcik-virtru wants to merge 1 commit into
Open
dmihalcik-virtru wants to merge 1 commit into
dmihalcik-virtru wants to merge 1 commit into
Conversation
Closes out item 1 of spec/DSPX-4648-web-sdk-large-files.md: the manifest,
not the payload, is what actually caps TDF size today. At the old 1 MiB
default a 50 TiB source needs 52.4M segments and a ~2.9 GB manifest,
against a 10 MB read-side ceiling.
Four changes, all sized off the pure helpers landed in DSPX-4650:
- Raise MANIFEST_MAX_SIZE to 256 MiB (DEFAULT_MANIFEST_MAX_SIZE) and make
it configurable on both sides -- ZipReader takes { manifestMaxSize },
EncryptConfiguration takes manifestMaxSize. Both default to the same
constant, and a test asserts they cannot drift apart.
- Auto-select the segment size by source length. chooseSegmentSize walks a
1/4/16/64/256 MiB ladder and takes the smallest rung whose segments array
fits 80% of the budget. 50 TiB lands on 16 MiB under both GMAC and HS256;
anything under ~3.6 TiB keeps the historical 1 MiB, so ordinary files are
untouched. Only the OpenTDF path auto-selects; EncryptParamsBuilder still
pins windowSize, so its behaviour is unchanged.
- Reject an over-budget manifest before encrypting rather than after.
sourceSize() reports a length for 'buffer' and 'file-browser' directly and
probes 'remote' with HEAD, falling back to a one-byte Content-Range
request; failures yield undefined rather than throwing, since a size probe
must not be able to fail an otherwise valid encrypt. writeStream then
rejects before getReader() is ever called. The end-of-stream assert stays
authoritative -- the up-front number cannot see the assertions or the root
signature, neither of which exists until the payload is done.
- Replace the Blob-concatenation root signature with ByteAccumulator on both
the write and read paths. WebCrypto has no incremental HMAC and the
concatenation is also the message assertions bind to, so a streaming digest
is not available; this instead writes the digests straight into one
exactly-sized buffer. Byte-identical output, without 3.3M live typed arrays
or the extra Blob copy.
Also teaches the mock server to answer HEAD and emit Content-Range, so the
remote size probe is testable in the browser tiers and not just in Node.
Still open, and called out as deferred in the spec: the segmentInfos and
eager chunks per-segment arrays, measuring the manifest JSON.parse transient
heap, and confirming with go-sdk/java-sdk that a 16 MiB segmentSize needs no
schema coordination.
Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>
|
Warning Review limit reachedNext included review available in 59 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (16)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
dmihalcik-virtru
added this pull request to stack #1033
September 10, 2026 20:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task
DSPX-4651, underDSPX-4648(epicDSPX-4502). Item 1 ofspec/DSPX-4648-web-sdk-large-files.md.Stacked on #1020. Review only the last commit; the base diff belongs to the PR below it.
What and why
The manifest, not the payload, is what caps TDF size today. At the 1 MiB default segment size a
50 TiB source produces 52,428,800 segments and a ~2.9 GB manifest — against a 10 MB read-side
ceiling that, until #1019, was only enforced at decrypt time. Neither a bigger cap nor a bigger
segment fixes that alone; the landing point argued for in the spec is 16 MiB segments under a
256 MiB manifest cap, and this PR implements it.
Sizes are computed for HS256 (56 B/entry) even though the default is GMAC (36 B/entry), so the
numbers hold whichever is in force — see the ADR in #1020.
Changes
1.
MANIFEST_MAX_SIZE→ 256 MiB, and configurable. The value moves toDEFAULT_MANIFEST_MAX_SIZEinscale-limits.ts, next to the arithmetic that derives what fitsunder it.
ZipReadertakes{ manifestMaxSize };EncryptConfigurationtakesmanifestMaxSize.Both default to the same constant, and a test asserts the read and write ceilings cannot drift
apart — otherwise the SDK could write a container it then refuses to open. 256 MiB was chosen so
50 TiB at 16 MiB/HS256 fits with ~28% headroom while a one-shot
JSON.parsestays viable.2. Auto-selected segment size.
chooseSegmentSize()walks a 1/4/16/64/256 MiB ladder andreturns the smallest rung whose
segmentsarray fits 80% of the manifest budget (the rest isheadroom for policy, key access, and assertions, whose size isn't known at that point). 50 TiB
lands on 16 MiB under both algorithms; anything up to ~3.6 TiB keeps the historical 1 MiB.
The tradeoff is documented on
SEGMENT_SIZE_LADDER: larger segments cost seek granularity, alarger integrity-failure blast radius (GCM verification is all-or-nothing per segment), and more
per-segment memory — which is what will constrain item 4's prefetch window.
Scope note: only the
OpenTDFpath auto-selects.EncryptParamsBuilderstill setswindowSize: DEFAULT_SEGMENT_SIZE, so the legacy tdf3 builder's behaviour is byte-for-byteunchanged.
3. Fail fast instead of fail late. #1019 made the write side enforce the cap, but at
end-of-stream — after the whole payload is encrypted and already downstream. This adds the
up-front check for sources whose length is knowable:
sourceSize()(lib/src/seekable.ts) reports a length for'buffer'and'file-browser'directly, and probes
'remote'withHEADfalling back to a one-byteContent-Rangerequest.'stream'and'chunker'reportundefined.undefined, never an exception: a size probe must not be able tofail an otherwise valid encrypt.
OpenTDF.createZTDFpasses the result through asknownSourceSize;writeStreamrejects viaestimateManifestBytes()beforegetReader()is ever called. A test counts source pulls andasserts zero.
The end-of-stream assert stays authoritative. The up-front number is an estimate —
manifestdoesn't yet carry the assertions or the root signature, neither of which exists until the payload
is done — and the error message says "an estimated N KiB" to match.
4.
ByteAccumulatorreplaces theBlobconcatenation. The spec asked for incremental roothashing; that isn't available. WebCrypto exposes no incremental HMAC, and
aggregateHashis alsothe message assertions bind to, so the concatenation genuinely has to exist as one buffer. What
was avoidable is its cost: the old code held one
Uint8Arrayper segment and joined them througha
Blob. At 3.3M segments that's 3.3M live typed arrays — several hundred MB of object overhead ontop of ~105 MB of actual digest — plus a full extra copy.
ByteAccumulatorwrites straight into oneexactly-sized buffer, on both the write and read paths. Output is byte-identical.
5. Mock server.
/filenow answersHEADand emitsContent-Rangeon 206, withAccess-Control-Expose-Headers. Without this the remote size probe was untestable anywhere butNode.
Not done — deliberately deferred
Recorded in the spec under item 1 rather than quietly dropped:
segmentInfosand the eagerchunksarray (tdf.ts:~1482).segmentHashListis gone, butthese two remain and are why item 4 alone will not deliver bounded memory — the scheduler paces
fetching, not this allocation. At 3.28M segments (16 MiB) they're manageable, which is what the
256 MiB cap buys; at 52.4M (1 MiB) each is independently multi-GB. Fixing them is invasive enough
to want its own PR.
JSON.parsetransient heap. The 300–400 MB figure behind the 256 MiB choice isreasoned, not measured. If a future cap raise pushes past ~512 MiB, a streaming parser has to be
revisited.
segmentSizeneeds no manifest-schema coordination withgo-sdk/java-sdk. fix(sdk): zip64/APPNOTE conformance in the TDF3 zip reader and writer (DSPX-4591) #1017's notes suggest
segmentSizeis already optional there, but no realround trip has been run.
Testing
cd lib && npm test— 449 mocha passing / 6 pending / 0 failing, 449 karma SUCCESS, 253web-test-runner passing, coverage gate green (
scale-limits.tsandbyte-accumulator.tsboth at100% lines).
npm run lintclean.New coverage:
unit/manifest-budget.spec.ts— read and write ceilings are one constant; the up-front rejectionfires with zero source pulls; the end-of-stream backstop still catches the unknown-size case
and does not claim to be an estimate;
estimateManifestBytesis exactly 1 byte over a realmanifest (the separating comma counted for the last entry, which has none).
unit/byte-accumulator.spec.ts— equivalence against a plain concatenation across sizes andhints, no reallocation given an exact hint, growth past an under-estimate, a run larger than
capacity, empty pushes, copy-not-alias, and no exposure of an over-allocated tail.
unit/scale-limits.spec.ts—chooseSegmentSizekeeps 1 MiB for ordinary files and reaches16 MiB at 50 TiB under both algorithms; ladder membership; monotonicity in source size; the 80%
budget invariant; staying above the AES-GCM invocation floor; largest-rung-not-throw when nothing
fits.
unit/seekable.spec.ts—sourceSizeacross all five source types, the HEAD→range fallback, andboth "report unknown rather than throw" paths.
unit/zip.spec.ts— the existing oversized-manifest test used 128 MiB, which is under the newcap; it was silently falling through to
JSON.parseand passing on the wrong error. Fixed, andgiven a sibling that proves a configured ceiling is enforced.
What isn't testable here: the 16 MiB auto-selection can't be exercised end-to-end without a
multi-TiB source. It's covered by unit tests on
chooseSegmentSizeplus the existing encrypt suiteconfirming small files still land on 1 MiB.
Risk
Touches crypto-adjacent code.
ByteAccumulatorproduces byte-identical input togetSignature, sothe root signature and assertion bindings are unchanged — the existing round-trip suite is the check
on that. The cap raise is backward compatible in one direction only: a reader on this version opens
anything an older writer produced, but a container written under the new 256 MiB cap is not readable
by an older SDK. At the sizes anyone is writing today that is unreachable, since the segment size
only climbs past ~3.6 TiB.