Repository navigation
Built-in engines W27: archive speed round 5 (first-run optimisation, BCJ/BCJ2, Fast level) - #1713
Merged
Merged
Conversation
…ster BCJ2 and BCJ Most of what was left between the built-in archive engines and 7-Zip was the start of each job: the hot loops ran their ReadyToRun code and then the instrumented tier (on every thread at once when packing) for the first half second or more, and a bzip2 job of a few dozen blocks never left the ReadyToRun sort at all. - AggressiveOptimization on the long-running loops, so they are compiled fully optimised on first use: the LZMA decoder's runs, the BCJ and BCJ2 filters, the bzip2 sort, block encoder and run-length step, and the fast level's chunk encoder with its per-symbol code inlined (LzmaEncoder.Encode, EncodeOne and small helpers marked AggressiveInlining, as the tiered compiler ended up doing) plus the chain search and the emitters it calls. The price-based parse (normal, maximum) keeps the tiered compiler: its speed is unchanged. - BCJ2: the main stream is copied and scanned in one loop with its state in locals, targets go straight into the output and are read as one big-endian load (the filter alone about 1.5x as fast). - BCJ: the E8/E9 scan is a vectorised IndexOfAny (the filter alone about 1.3x as fast). - Fast level: six hash-chain candidates instead of eight, about 4 % less work for 0.25 % larger output. Fresh process, ReadyToRun, median of 5 (round 4 -> now, 7-Zip 26.05): 7z Fast 2.80 -> 2.11 s (7-Zip 2.17), xz Fast 1.45 -> 1.07 s (1.08), bz2 1.85 -> 1.34 s (1.27), tar.bz2 3.15 -> 2.32 s (2.40), 7z LZMA 3.00 -> 2.83 s (2.80), BCJ 1.40 -> 1.31 s (1.01), BCJ2 0.97 -> 0.89 s (0.74). Later runs in one process are as fast as before; normal-level packing is unchanged. A test encodes BCJ2 streams in code and decodes them in reads of every size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ments Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wave 27 of the built-in engines: archive speed round 5, measured on an idle PC against 7-Zip 26.05 (median of 5 fresh-process runs, ReadyToRun).
Changes
AggressiveOptimizationon the LZMA/LZMA2 decoder loops, BCJ/BCJ2, bzip2 sort and block encoder; the Fast encoder'sEncodeChunk, withAggressiveInliningon its helpers).Readrewritten to scan while copying and write targets straight to the output (filter alone 1.5× faster). NewByteInput.ReadUInt32BigEndian.IndexOfAny(filter alone 1.3× faster).-mx1.Results (7-Zip time ÷ Filee time)
Round 4 ratios here are re-measured on the same idle PC. What's left: single-thread LZMA decoding of x86 data is 20–26 % behind 7-Zip's assembly decoder, which is most of the BCJ/BCJ2 gap.
Tests
BCJ2_streams_decode_alike_in_reads_of_any_size.7z t; xz and bz2 round-trip byte-identical.dotnet format --verify-no-changes --severity warnclean on Filee.Engines and Filee.Engines.Tests.🤖 Generated with Claude Code