Repository navigation
Rust steps 1: monochrome recompression, Compare pixels and snap geometry in Rust - #1014
Merged
Merged
Conversation
The per-image work of the lossless monochrome mode (Flate inflate, packing pure black/white eight-bit gray/RGB to one bit, Flate and CCITT G4 encoding, choosing the smallest complete stream) now lives in the new crate open-pdf-monochrome. The Tauri command monochroom_beeld takes the raw stream bytes as a raw body plus the derived parameters in a header and returns an ipc::Response. PDF parsing, eligibility rules, the document loop and the replacement dictionaries stay in JS. - G4 output is byte-identical to js/pdf/ccitt-g4.js (golden vectors in the crate tests, 40 full-page encodings in the benchmark script). - Flate uses zlib-rs (already in Cargo.lock, Zlib licence): different bytes, same decoded samples; inflate is as strict as DecompressionStream. - The size rule compares complete stream objects: JS passes both candidate dictionary sizes at /Length 0. - JS falls back to its own encoder when not in Tauri, on Android, when the command is missing or fails (one warning), or with OPDS_RUST_MONOCHROME=0. - scripts/bench-rust/09-monochrome.mjs checks parity and times JS vs Rust: a 300 dpi A4 page goes from 53 ms (JS, no yields) to ~17 ms (Rust plus an IPC model), a 20-page document from 900 ms to 645 ms. Notes in docs/rust-stappen.md.
) New worker request `compare`: the worker renders both pages in tiles at the compare resolution, estimates the paper colour, builds the ink masks, runs the automatic alignment and the mask detection, and answers with the alignment, statistics and the clusters (binary, through the shared memory), so no bitmap crosses IPC. Masks are cached per worker (128 MB, cleared on trim). The algorithms are line-by-line ports of ink-mask.js, mask-detect.js, auto-align.js, fft.js and alignment.js, computing like V8 (Math.hypot and Math.round rebuilt, fdlibm-style trig from the libm crate). Parity fixtures generated from the JS modules (scripts/bench-rust/compare-golden.mjs) check 13 detections, 5 alignments, 4 ink masks and the background estimate bit for bit. Column FFTs use a contiguous copy and independent steps run on a few threads, both without changing the result. The Ready line now lists `caps` so the app only sends `compare` to workers that know it.
…fallback (#465) compare_pixel_pair routes a page pair to a pool worker that advertises the `compare` capability and returns the JSON header plus the packed clusters as raw bytes. computePairChanges() tries it first on desktop (rust-compare.js) and keeps the JS pipeline for the browser build, Android, in-memory sides, files whose length on disk differs from the app's copy, older workers and any failure (one console warning). OPDS_RUST_COMPARE=0 forces the JS pipeline.
…rison (#465) scripts/bench-rust/09-compare-pixels.mjs profiles the JS pipeline per stage on generated A3 and A0 drawing pairs with known edits, times the worker request end to end including stdin/stdout and the shared-memory read, and checks parity between pdf.js and PDFium rendering (--check). Results and the tolerance are recorded in docs/rust-stappen.md.
# Conflicts: # docs/rust-stappen.md
Snap points, edges and the spatial index for PDF linework now come from the PDFium worker on desktop instead of a walk over the PDF.js operator list in JavaScript. - pdfium-worker: new snap_geometry request. Walks the path objects of the already parsed page (form XObjects recursed with their matrices), flattens curves in four steps, applies the viewport matrix sent by the app, dedupes on the same 0.1 px grid and builds the same 48 px cell index. Compact binary result (OSG1) through the SHM region, fetched in chunks when it exceeds 64 MB. The Ready line lists caps so the app never sends the request to an older sidecar. - app: extract_snap_geometry command and WorkerPool::snap_geometry with the same page affinity as rendering; OPDS_RUST_SNAP=0 forces the JavaScript route. - js: pdf-snap-extractor uses the worker when the PDFium render path is available and the document has a file path; otherwise, or on failure, the PDF.js route as before (one warning, then JS for the session when the worker lacks the capability). Web and mobile builds are unchanged. - Parity and benchmark script under scripts/bench-rust: equal within 0.01 pt on lines, polylines, curves, rects, nested forms with /Matrix, CTM scaling, rotated pages, UserUnit; clip paths and annotation appearances are intentionally not snap geometry in the worker route. 200k segments: about 8 s in JavaScript versus about 0.4 s end to end through the worker, a third of the WebView heap. - docs/rust-stappen.md describes the step, the differences and the measurements.
# Conflicts: # docs/rust-stappen.md # open-pdf-studio/scripts/unit-testlijst.txt # open-pdf-studio/src-tauri/src/worker_pool/state.rs # pdfium-worker/src/main.rs # pdfium-worker/src/protocol.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First steps of moving CPU-heavy JavaScript work into Rust inside the existing Tauri app (not the Hayro kernel or the native rewrite). Each step keeps the JavaScript path as the fallback (web build, Android, older worker, errors) and has a parity test and a benchmark; numbers are in
docs/rust-stappen.md. No version bump and no release.Refs #902, Refs #465
What moved
open-pdf-monochrome, Tauri commandmonochroom_beeldOPDS_RUST_MONOCHROME=0compare; only clusters cross IPCOPDS_RUST_COMPARE=0snap_geometry, binary OSG1 through SHMn-only pathsOPDS_RUST_SNAP=0The worker announces new requests in its Ready line (
caps: ["compare", "snap_geometry"]); an older sidecar never receives them, so the app stays on the JavaScript path until the worker is rebuilt.Verification
cargo test -p pdfium-worker -p open-pdf-monochrome: 68 passcargo check -p open-pdf-studio -p pdfium-worker -p open-pdf-monochrome --lib --bins --tests(MSVC): passnpx vite build,npm run build:hosted: exit 0node scripts/draai-unit-tests.mjs: 8057 tests, 0 failnpm run test:quality, Python library tests, README image check: passNot verified
OPDS_RUST_*=0flag. IPC cost in WebView2 is modelled, not measured.🤖 Generated with Claude Code