Skip to content

Built-in engines W27: archive speed round 5 (first-run optimisation, BCJ/BCJ2, Fast level) - #1713

Merged
KnifeLemon merged 2 commits into
builtin-enginesfrom
p4/w27-perf
Nov 20, 2026
Merged

KnifeLemon merged 2 commits into
builtin-enginesfrom
p4/w27-perf

Conversation

@KnifeLemon

Copy link
Copy Markdown
Owner

Wave 27 of the built-in engines: archive speed round 5, measured on an idle PC against 7-Zip 26.05 (median of 5 fresh-process runs, ReadyToRun).

Changes

  • First-run speed: hot archive loops start fully optimised (AggressiveOptimization on the LZMA/LZMA2 decoder loops, BCJ/BCJ2, bzip2 sort and block encoder; the Fast encoder's EncodeChunk, with AggressiveInlining on its helpers).
    • Before, these ran through ReadyToRun and instrumented tiers for most of a job: the first run was 0.3–0.6 s slower than later runs.
    • Applying it to the whole encoder, or turning TieredPGO off app-wide, made warm runs 2–7 % slower, so neither is done.
  • BCJ2: Read rewritten to scan while copying and write targets straight to the output (filter alone 1.5× faster). New ByteInput.ReadUInt32BigEndian.
  • BCJ: E8/E9 search uses vectorised IndexOfAny (filter alone 1.3× faster).
  • Fast level: hash chain depth 8 → 6. Output changes: about 0.25 % larger, about 1 % larger than 7-Zip -mx1.
  • Normal-level output and speed unchanged; bzip2, BCJ and BCJ2 output byte-identical.

Results (7-Zip time ÷ Filee time)

Item Round 4 Round 5
7z Fast pack 0.78× 1.03×
xz Fast pack 0.74× 1.01×
bz2 pack 0.69× 0.95×
tar.bz2 pack 0.76× 1.03×
7z LZMA unpack 0.93× 0.99×
7z LZMA2 unpack (2 blocks) 0.94× 0.94×
xz unpack 0.96× 0.94×
BCJ unpack 0.72× 0.77×
BCJ2 unpack 0.76× 0.83×

Round 4 ratios here are re-measured on the same idle PC. What's left: single-thread LZMA decoding of x86 data is 20–26 % behind 7-Zip's assembly decoder, which is most of the BCJ/BCJ2 gap.

Tests

  • Archive tests (LZMA, xz, 7z, bzip2, BCJ/BCJ2, zip and others) in Release: 851 passed, 18 skipped, 0 failed. New BCJ2_streams_decode_alike_in_reads_of_any_size.
  • Fast and bz2 output pass 7z t; xz and bz2 round-trip byte-identical.
  • dotnet format --verify-no-changes --severity warn clean on Filee.Engines and Filee.Engines.Tests.

🤖 Generated with Claude Code

KnifeLemon and others added 2 commits November 20, 2026 09:27
…ster BCJ2 and BCJ

Most of what was left between the built-in archive engines and 7-Zip was the start of each job: the hot loops ran
their ReadyToRun code and then the instrumented tier (on every thread at once when packing) for the first half
second or more, and a bzip2 job of a few dozen blocks never left the ReadyToRun sort at all.

- AggressiveOptimization on the long-running loops, so they are compiled fully optimised on first use: the LZMA
  decoder's runs, the BCJ and BCJ2 filters, the bzip2 sort, block encoder and run-length step, and the fast level's
  chunk encoder with its per-symbol code inlined (LzmaEncoder.Encode, EncodeOne and small helpers marked
  AggressiveInlining, as the tiered compiler ended up doing) plus the chain search and the emitters it calls. The
  price-based parse (normal, maximum) keeps the tiered compiler: its speed is unchanged.
- BCJ2: the main stream is copied and scanned in one loop with its state in locals, targets go straight into the
  output and are read as one big-endian load (the filter alone about 1.5x as fast).
- BCJ: the E8/E9 scan is a vectorised IndexOfAny (the filter alone about 1.3x as fast).
- Fast level: six hash-chain candidates instead of eight, about 4 % less work for 0.25 % larger output.

Fresh process, ReadyToRun, median of 5 (round 4 -> now, 7-Zip 26.05): 7z Fast 2.80 -> 2.11 s (7-Zip 2.17),
xz Fast 1.45 -> 1.07 s (1.08), bz2 1.85 -> 1.34 s (1.27), tar.bz2 3.15 -> 2.32 s (2.40), 7z LZMA 3.00 -> 2.83 s
(2.80), BCJ 1.40 -> 1.31 s (1.01), BCJ2 0.97 -> 0.89 s (0.74). Later runs in one process are as fast as before;
normal-level packing is unchanged. A test encodes BCJ2 streams in code and decodes them in reads of every size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ments

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@KnifeLemon KnifeLemon added the built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet label Nov 20, 2026
@KnifeLemon
KnifeLemon merged commit 61783ba into builtin-engines Nov 20, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

built-in engines Filee: built-in engines (Project 4), on the builtin-engines branch, not released yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant