Repository navigation
Rust steps 2: search text from the PDFium worker, faster annotation loading - #1018
Open
DutchSailor wants to merge 12 commits into
Open
DutchSailor wants to merge 12 commits into
DutchSailor wants to merge 12 commits into
Conversation
The mobile apps run in Tauri but have no PDFium worker. The worker path only checked for Tauri, so every page pair on Android made a compare_pixel_pair call that failed with "worker pool not ready" before the JS pipeline ran. Pass isMobile() to createRustCompare so the path is off there, like the PDFium render path (heeftRustRenderer).
A render or region answer with ok:false (for example PDF_OUTPUT_LIMIT on a large sheet at a high zoom) means the exchange went fine and the worker is healthy. The pool still treated it as a crash: it killed and respawned the worker, retried the render cold on a second worker (which failed the same way and was restarted too) and after three such answers in 30 s marked the worker DeadPermanent. Requests queued on that worker, such as snap geometry, failed with PDF_WORKER_UNAVAILABLE and fell back to JS. Type these answers as WorkerReported: no restart and no retry on another worker. Broken exchanges (EOF, timeout, garbage) still restart the worker.
pdf.js draws form field appearances with intent 'display', so a changed field value is a change in the JS pipeline. PDFium draws fields only in the form pass (FPDF_FFLDraw), which the matrix render used for compare tiles skips. On desktop a changed value in a title block was therefore not reported, while the overlay showed the new value and the web version reported it. On a page with widget annotations a tile is now rendered as the whole page at scale with a shifted origin, with annotations and form data. Pages without fields keep the matrix route.
zoekExtraKleuren scanned every key of the page's extras map, splitting each key, for every annotation whose /Rect was not an exact key - and it is called several times per annotation. More than a third of the app's own annotations miss the exact key (the saver writes 160.64000000000001, pdf.js reads 160.64), so converting a heavy markup set was quadratic per page: 37-65 s for 50,600 models on a 100-sheet set, now 2-4 s. The lookup now uses a per-map index (8 pt buckets on x0) and remembers each queried rectangle. Same rule as before: smallest distance below 8, ties go to the first key in map order. A test compares it with the full scan; deterministic model snapshots of the 100-sheet set are byte-identical before and after.
Adds a generated 100-sheet set (57,500 annotation dictionaries in 22 app types plus foreign ones, written with the real saver), a per-stage profile of the JS loader, a deterministic model snapshot for parity checks, and a lopdf measurement prototype for raw record extraction. Result, documented in docs/rust-stappen.md: moving the raw extraction to Rust is not clearly faster end to end. The shared pdf-lib load (15 s on this set) stays for seven other readers at open, PDFium cannot list dictionary keys and returns f32 numbers, lopdf 0.34 stores reals as f32, and the model depends on both pdf.js and pdf-lib number parsing and on pdf-lib's dictionary serialisation for source fingerprints. JS stays the default; no Rust route is wired in.
…es search The worker reads the glyphs of a page through the PDFium text page and the object tree (content order, XObject and marked-content boundaries) and builds the text items the way pdf.js getTextContent does: the same space, line-end and item-boundary rules, normalizeUnicode and bidi, plus the baseline span of every character. Space-only and duplicated text objects that PDFium skips are restored; pages where PDFium reads text differently (right-to-left script, vertical writing, glyphs without Unicode, small Type3 text) are flagged so the app reads them with pdf.js. The answer is binary (OPT1) through the SHM region, for a range of pages within a time budget so tile renders on the same worker are not held up.
Search (find bar, document and folder search, MCP) gets the text items of a page from the PDFium worker through the new extract_search_text command when the document has a file on disk that matches the bytes the app holds. The first page comes alone for a fast first result; then two workers read the document in chunks ahead of the search. Matching, unsaved OCR words, text edits and annotation text stay in JavaScript, and highlights use the real character spans the worker sends. pdf.js stays the fallback: web build, Android/iOS, flagged pages, a file that changed on disk, an older worker and any error (one warning). OPDS_RUST_SEARCH=0 or window.__opdsRustZoeken = false forces pdf.js.
scripts/bench-rust/10-zoektekst.mjs runs the app's search on pdf.js text and on worker text: generated PDFs with Latin, diacritics, ligatures, hyphens, right-to-left and CJK script, rotated text, forms, CropBox and Type3 fonts, then every PDF in a folder, then a timing run on a generated 500-page text document and heavy drawing sets (time to first result, total, heap). docs/rust-stappen.md gets the results and the choices.
# Conflicts: # docs/rust-stappen.md
The measurement crate is not part of the release and has no version of its own; with a version field the version-field check counted it as one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rust steps 2: search text from the PDFium worker, and a faster annotation loader that stays in JavaScript. Stacked on #1016 while that is open; it will be retargeted to
mainonce #1016 is merged. No version bump and no release. Details and numbers indocs/rust-stappen.md.Search text in the PDFium worker
page_textrebuilds the text items exactly as pdf.js 6.4getTextContentdoes (spaces, line ends, item boundaries,normalizeUnicode, bidi), with each character's span on the baseline; PDFium-specific artefacts are corrected (self-generated spaces and line breaks, line-end hyphen U+0002, skipped space-only and duplicated text objects, rotated char boxes, BDC/EMC item boundaries). BinaryOPT1through SHM, with a 150 ms budget per call so tile renders are not held up.extract_search_text; JSjs/search/rust-zoektekst.jsreads the current page first, then the document in chunks over 2 worker lanes, and is wired into the find bar,performSearchand the document search service (open documents, folder search, MCP). Highlights use the real character spans.OPDS_RUST_SEARCH=0forces pdf.js; web, Android/iOS and older workers use pdf.js.Annotation loading: investigated, stays in JavaScript
zoekExtraKleuren; it is now indexed with the same choice rule. Load time on that set 68–91 s → 24–31 s; reproducible model snapshots are byte-identical before and after.scripts/bench-rust/and is not part of the app.Verification
cargo test -p pdfium-worker -p open-pdf-monochrome: 89 pass;cargo check -p open-pdf-studio --lib --bins --tests(MSVC): passnpx vite build,npm run build:hosted: exit 0node scripts/draai-unit-tests.mjs: 8071 tests, 0 fail (after the version-field fix in the last commit)npm run test:quality: passNot verified
OPDS_RUST_SEARCH=0fallback, and opening a heavy markup set after the loader change.🤖 Generated with Claude Code