Skip to content

Commit 745e8dc

Browse files
Harden uv lockfile patch preservation and verify every uv release family (#239)
* Harden uv lockfile patch preservation after #238 Fail-closed revert for unwired Python entries: a ledger entry with no wiring (the shape `repair` re-synthesizes when state.json is lost) used to route into the uv / python-lock revert, iterate zero records, report success, and let the caller delete the vendored wheel and drop the entry while uv.lock, the pylock, the script, or requirements.txt still resolved through it. `revert_pypi_opts` now refuses while any Python project file references the uuid dir, mirroring the npm-family guard. Inventory: a PEP 723 script lock or PEP 751 lock supplements the project's poetry.lock / requirements.txt pins instead of hiding them; uv.lock keeps its exclusive precedence. Discovery: `python_lock_paths` follows symlinks and skips one unreadable entry instead of dropping every lock. Line endings: toml_edit re-emits every newline as LF, so CRLF pylock, script-lock and pyproject rewrites (and document-restore reverts) now restore the input's CRLF convention. Ledger recovery compares the purl name in PEP 503 form like `lookup`. Hosted pyproject edits render uv's own layout — header-less `[tool]` / `[tool.uv]` parents and a `[tool.uv.sources]` table after `[project]` — instead of dotted keys above `[project]`. The PEP 723 script block keeps fully dotted keys: only those vanish when empty after a parse round trip, which the out-of-order multi-package document revert relies on. `redirect_uv_missing_sha256` fires once per dep, not once per lock file; `repair` prefers uv.lock over an exported pylock when stamping flavor. Backtest harness: retry the cold-offline root-build failure with network for any uv version (not just 0.2.37), and pin first, latest, and every observed behaviour-boundary release of each 0.x family. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Follow each uv `[[distribution]]` lock shape when redirecting The experimental `[[distribution]]` grammar (uv 0.1.x–0.2.34) went through three shapes, and the rewriter keyed everything on the table name: - string sources + sub-table artifacts (`[distribution.sdist]`, `[[distribution.wheel]]`) through 0.2.5; - string sources + inline artifacts (`sdist = {…}`, `wheels = [...]`) from 0.2.6 through 0.2.17; - inline-table sources (`source = { registry = … }`) from 0.2.18. Emitting the string source into a 0.2.18–0.2.34 lock made uv reject it ("data did not match any variant of untagged enum SourceWire") and silently ignore the lock: an ordinary `uv sync` still installed the patch through the pyproject source, but `--frozen` / `--locked` failed. Emitting `[[distribution.wheel]]` into a 0.2.6–0.2.17 lock left the binary with a direct URL and no wheel, which it tried to build as a source archive ("Unsupported archive type: …whl"). Decide the source shape from the entry's own `source` syntax and the artifact shape from its own artifact keys (falling back to any sibling entry), independently. Vendored native wiring stays refused for the whole `[[distribution]]` era: every shape records absolute file paths for local artifacts. Reword the refusal to say so (it claimed "uv 0.1", but 0.2.34 refuses too). Add scripts/probe-uv-boundaries.py — the bisection tool behind every boundary pinned in scripts/backtest-uv.py (lock grammar, source and artifact shape, lock revision, `uv export`, `uv lock --script`, PEP 751 compilation, bare local wheel paths in requirements) — and pin the 0.2.5/0.2.6 and 0.2.17/0.2.18 pairs in the matrix. Document the exact boundaries. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Vendor into uv 0.2.35/0.2.36 locks that predate [package.metadata] The first two `[[package]]`-grammar releases wrote no root `[package.metadata]`; the metadata step refused them with `pypi_uv_lock_package_missing` although there was nothing to repoint. Skip the requires-dist rewrite when the root unit has no metadata table at all (a lock that HAS metadata but lacks the entry is stale and still refuses), so vendored native wiring covers every `[[package]]` release. Verified with real `uv sync --frozen` / `--locked` / ordinary installs on 0.2.35, 0.2.36 and 0.2.37. Pin 0.2.36/0.2.37 as the metadata boundary in the matrix and document the exact grammar shapes and coverage. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Record the 41-release uv compatibility matrix for source 310b904 Regenerate results.json, binaries.json, and the results section of docs/testing/uv-compatibility.md from a full run of scripts/backtest-uv.py against the rebuilt CLI: 41 pinned uv releases (first and latest of every 0.x family plus every observed behaviour boundary), 583 installed-byte comparisons and 240 lock-preservation checks, zero mismatches and zero lock changes. Every non-zero exit is a documented uv boundary or the cold-offline root-build case retried with network. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * utils/fs: export read_regular_to_string and is_symlink helpers Shared FIFO-safe reader (the shape every ecosystem module re-declared privately) and an lstat-based symlink probe, exported so the CLI crate's raw read_to_string sites and the Python lock writers can use them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * uv vendor: classify [tool.uv] dev-dependencies as direct; repoint every metadata entry classify_dependency scanned project.dependencies, optional-dependencies and [dependency-groups] but not the legacy `[tool.uv] dev-dependencies` array, which uv still honours (0.12.15 warns but records it exactly like `dependency-groups.dev`, in the root unit's [package.metadata.requires-dev]). Such a target classified Transitive: wire_uv took the override branch, [package.metadata.requires-dev] kept `specifier = "==…"`, and after a "successful" vendored scan every uv >= 0.2.37 `uv sync --locked` / `uv lock --check` exited non-zero while a plain `uv sync` rewrote the lock (ledger drift) and on 0.2.37/0.4.30 reinstalled the PRISTINE wheel. rewrite_root_metadata_entries also repointed only the FIRST matching element per array (a `break` in the requires-dist loop and per group in requires-dev) although its doc promised ALL: a package declared in both dependencies and an optional-dependencies extra has two requires-dist entries (the extra's with `marker = "extra == '…'"`) and uv repoints both, so the stale second entry kept --locked red. Fix: extend `declared` with the [tool.uv] dev-dependencies string members (pep508_name); repoint every matching requires-dist element as its own edit/wiring record (revert already replays N records per package), and rebuild each requires-dev group line once with every matching element rewritten so a group naming the package twice never yields overlapping spans. Both new fixtures are byte-exact uv 0.12.15 output. wire_uv now also returns a Vec<VendorWarning> of wiring-time advisories, threaded into the vendor outcome's warnings by pypi.rs (the channel the next commit's override-branch advisory uses). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * uv vendor: advise that override-dependencies wiring needs uv >= 0.5.6 The module doc's "claim 8" said [tool.uv.sources] applies to override-dependencies. Against real binaries that holds only from uv 0.5.6: on 0.2.35–0.5.3 a Transitive target wired through the override branch is silently reinstalled from the registry by a plain `uv sync` (the lock still names the vendored path, so `--frozen` installs it, which is why the matrix looked green). No lock-shape marker separates 0.5.3 from 0.5.6, so an offline refusal is impossible. Fix: the Transitive branch of wire_uv pushes an advisory VendorWarning with the stable code `pypi_uv_override_requires_uv_0_5_6` ("pin uv or make the package a direct dependency") through the outcome's warnings, and the module doc states the >= 0.5.6 boundary. The Direct branch stays silent. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * uv: repoint [manifest] constraints / build-constraints for the redirected package When the target is named in `[tool.uv] constraint-dependencies` (or build-constraint-dependencies), uv.lock carries `[manifest] constraints = [{ name = "urllib3", specifier = "==1.26.18" }]` (resp. build-constraints). Once the package has a source, uv >= 0.5.6 re-serializes that element as `{ name, path|url }` with the specifier dropped (uv 0.12.15 ground truth), so after vendoring or a hosted redirect `uv lock --check` / `uv sync --locked` exited 2 and a plain sync churned the lock. Neither backend touched those arrays: the vendored surgery only knew requires-dist / requires-dev / overrides, and the hosted rewrite_manifest returned early whenever the (script-lock-only) `requirements` key was absent. Vendored: rewrite_manifest_constraints repoints every matching element of both arrays, one whole-line `<key> = […]` fragment per key recorded as the new wiring kind `uv_lock_manifest_constraints` (unknown kinds are warned+skipped on revert, so the arm is added to the replace_fragment group). The records are pushed last so the reverse-order revert restores them before searching for a requires-dist element that may be byte-identical. Hosted: rewrite_manifest repoints matching `constraints` / `build-constraints` elements before the `requirements` early return, and still invents no `overrides` array for a direct dependency. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * uv vendor: keep a CRLF pyproject.toml / uv.lock pair CRLF through wire and revert wire_uv wrote `doc.to_string()` for pyproject.toml, and toml_edit re-emits every newline as LF: a pure-CRLF pyproject (git autocrlf on Windows) came back all-LF — whole-file churn — and revert_uv, which splices fragments rather than rewriting the file, could never restore it. The lock side had the same class in three places: rewrite_target_package_unit rebuilt the target unit with `join("\n")` and appended the reconstructed [package.metadata] block with "\n\n" (mixed endings inside a CRLF lock); add_manifest_override created `[manifest]\n…\n\n`, `overrides = […]\n` and the multi-line `,\n]` tail with LF; and revert_uv removed those fragments with hardcoded "{new}\n\n" / "{new}\n", so a CRLF lock's created overrides were left in place with a drift warning. Fix: render pyproject through preserve_line_endings, record the uv_override Rewritten old/new array fragments in the file's convention (a multi-line array renders LF and replace_fragment would miss), detect the lock's terminator once (newline_of) and build every spliced, appended and removed lock fragment with it; the target unit's span is trimmed of the trailing `\r` find_unit_span includes so the rebuilt unit splices back in front of the same `\r\n`. The regression test CRLF-converts the Direct, Transitive (created override + created [manifest]), Rewritten-override (multi-line arrays in both files) and requires-dev fixtures, asserts the wired pair has no bare LF and equals the LF wiring modulo terminators, and reverts byte-exactly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * uv: say what a [[distribution]] lock actually cannot do instead of "records absolute paths" Both refusals (load_uv_project and the vendored ArtifactSource::Path arm of rewrite_python_lock) claimed uv < 0.2.35 "records absolute file paths". Against the real 0.1.45–0.2.34 binaries that is not the limitation: <= 0.2.6 cannot parse a relative path source at all; 0.2.17–0.2.34 install one under --frozen / plain sync, but `--locked` rejects any non-canonical spelling, plain sync (0.2.34) and every `uv lock` absolutize it, resolution is CWD-relative, and 0.2.17/0.2.18 never verify the wheel hash. A user reading the old text would look for an absolute path to fix rather than upgrade or switch to a requirements.txt install. Both messages now read: uv `[[distribution]]` lockfiles (uv < 0.2.35, experimental `uv lock`) cannot carry a portable local wheel: `--locked` rejects relative paths and `uv lock`/`uv sync` rewrite them to absolute ones; upgrade to uv >=0.2.35 for native vendoring, or use a requirements.txt installation. The vendored refusal gains a pinning test; the existing hosted assertion (`contains("0.2.35")`) still holds. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * uv vendor: gate the metadata no-op on the sub-tables too, not the bare header The 0.2.35/0.2.36 early return in rewrite_root_metadata_entries checked `unit_text.contains("[package.metadata]")`. The requires-dev sub-table header `[package.metadata.requires-dev]` does NOT contain that substring (`.` where the bare header has `]`), so a root unit carrying only the sub-table would have been treated as "no metadata": the requires-dev entry kept its specifier, `uv lock --check` / `uv sync --locked` went red, and nothing refused. Every uv release since 0.2.37 happens to write the empty bare header first, which is why this never fired. The no-op now requires the absence of BOTH the bare header and any `[package.metadata.` sub-table; the later scan already works from the sub-table header alone, so such a unit is repointed rather than skipped. Pinned by a test whose root unit has only [package.metadata.requires-dev]. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * uv vendor: refuse a symlinked pyproject.toml / uv.lock before any write atomic_write_bytes_preserving_mode stages the new content next to the path and renames over it. uv itself writes THROUGH a symlinked uv.lock / pyproject.toml, but our rename REPLACES the link with a regular file: the target keeps the pre-vendor bytes, git shows a typechange, and a revert (which also renames) can never put the link back. wire_uv now refuses fail-closed before either write, and revert_uv keeps the artifact and fails, both with `pypi_uv_symlink_unsupported` naming the offending file (lstat via crate::utils::fs::is_symlink — the link itself, not its target). The orchestrator sweeps the freshly built wheel on a wiring refusal, so no `.socket/vendor` residue is left. The module's private FIFO-safe reader is replaced by the shared crate::utils::fs::read_regular_to_string. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * vendor/pypi: fail-closed listing, include probing and orphan reclaim for the unwired-revert guard The unwired-revert guard added after #238 protects a `repair`-reconstructed pypi ledger entry (no wiring records) from being deleted while a Python project file still resolves through the vendored wheel. Four gaps remained: - `keep_artifact` (`rollback/remove --preserve-state`) was still guarded, so a preserve-state revert of an unwired entry refused with `vendor_wiring_unknown_revert_blocked` although it would have deleted nothing. npm skips the guard there because the refusal exists only to protect the deletion; the pypi guard now does the same. - The lock enumeration failed OPEN: a `read_dir` error on the project root was read as "no Python locks here", `python_lock_paths` silently dropped any lock whose (followed) metadata failed, and the static probe list lacked uv.lock and pylock.toml. On a 0311 (execute-only) root, a listing-denied ACL or a transient EMFILE, `vendor --revert` deleted the referenced wheel and dropped the entry while uv.lock still resolved through it. The guard now lists the root itself (lstat only, so a symlinked lock with an unreadable target is still probed and fails closed on the read), and an unlistable root refuses with its own clause. - Once the guard cleared, the entry was still dispatched by flavor, which made a true orphan unreclaimable forever: flavor `uv` with uv.lock gone failed "cannot read uv.lock", and flavor `None` (what `repair` stamps for requirements/poetry/pdm/pipenv reconstructions) was unknown to the dispatch. An empty-wiring entry that passes the guard now skips the dispatch (it has nothing to replay) and proceeds to the uuid-dir removal. An UNKNOWN non-None flavor still fails closed: a newer backend's files may reference the artifact from a place this guard does not probe. - The requirements planner writes vendored pins into `-r` includes, but the guard probed only the root requirements.txt. `requirements_include_names` (new, exported from the core crate for the CLI's repair/orphan sweep) walks the same include tree as the planner — FIFO-safe, `Err` when a reached include exists but cannot be read — and the guard probes every in-root include, failing closed on an unreadable tree. Verified against real uv 0.9.30 and 0.5.31 through the built CLI: the refusal on a 0311 root keeps the wheel and `uv sync --frozen --offline` still installs from it; after `uv remove` the orphan is reclaimed and the ledger entry dropped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * repair/vendor: FIFO-safe reference scan that also sees `-r` include pins `repair`'s ledger reconstruction and `vendor --revert`'s orphan sweep share `scan_vendor_references`, which read every wiring-file candidate with a plain `tokio::fs::read_to_string`. A FIFO under one of those names — most easily the paired `<script>.py` derived from a `*.py.lock`, which the lock lister never stats — blocked forever in open(2) waiting for a writer, so `repair` never returned. The same raw read sat in `detect_reference_flavor` (the Python lock loop and the npm lock closure). All three now go through `read_regular_to_string`, keeping the existing skip-on-error tolerance. The scan also consulted the root requirements.txt alone, while the requirements planner writes a vendored pin where the original pin was — possibly inside a `-r`/`--requirement` include. After a state.json loss `repair` could not re-adopt such a wheel, and the orphan sweep deleted it as `vendor_orphan_removed` while `pip install -r` still resolved through it. The scan now appends the planner's in-root include tree via the new core `requirements_include_names` (an unreadable tree degrades to the root file, matching the per-file tolerance), so repair and the sweep see include-hosted pins. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * python_lock: fix uv version boundaries in comments; pin ≤0.2.5 sdist and sibling-shape rewrites The `[[distribution]]` shape comments disagreed with the PR's own probe matrix and docs: the inline-table-source era was labelled "0.2.20 through 0.2.34" (it begins at 0.2.18, the string→inline-table flip is 0.2.17→0.2.18) and the string-source/inline-artifact era "0.2.14–0.2.17" (it begins at 0.2.6, the sub-table→inline artifact flip is 0.2.5→0.2.6). The test doc for the latter also attributed bare `[[distribution.dependencies]]` edges to the whole era, but 0.2.6–0.2.15 still write source-qualified edges; only 0.2.16–0.2.17 write bare ones. Anyone deciding from the comments which real uv binary to reproduce a report with would pick the wrong one. Fix the four sites to the verified boundaries and pin two PR-introduced behaviours that had no test of their own: - `legacy_sdist_redirect_emits_sub_table`: a ≤0.2.5 lock (string source, `[distribution.sdist]` + `[[distribution.wheel]]` tables) redirected to a .tar.gz artifact gets a `[distribution.sdist]` sub-table with url/hash, loses the stale `[[distribution.wheel]]`, follows the source-qualified edge, and is byte-stable on rerun. - `artifactless_target_follows_sibling_shape`: an entry with no artifact keys takes its grammar from a sibling — `[[distribution.wheel]]` siblings yield a sub-table, inline `wheels = […]` siblings yield an inline array — and both are byte-stable on rerun. Both were GREEN on HEAD; they guard the shape selection against regression. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * lock_inventory: document that uv.lock exclusivity is keyed on parse success; pin poetry/script union and corrupt-uv.lock fall-through The comment above the pypi fallback chain said "uv.lock stays the EXCLUSIVE project inventory", which reads as presence-keyed. The code keys exclusivity on `python_lock_inventory` SUCCEEDING: an unparseable uv.lock contributes nothing and falls through to poetry.lock / requirements.txt exactly like a package-less poetry.lock does. That is the right behaviour — presence-keying would hide every requirements pin behind a corrupt lock and diverge from hosted, which skips the same file with `redirect_uv_lock_unsupported` and still reads the other pins — but a reader taking the comment at face value would "fix" it the wrong way. Reword the comment to say PARSEABLE and spell out why, and pin the two untested legs of the chain in `python_lock_union_tests`: - `poetry_lock_unions_with_script_lock_and_hides_requirements`: with no uv.lock, poetry.lock is the tool lock (hides requirements.txt) while a `tool.py.lock` still unions with it. - `unparseable_uv_lock_falls_through_to_requirements`: garbage-TOML uv.lock + requirements.txt yields the requirements pin. Both were GREEN on HEAD. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * python_script: scope the SourcesLayout revert claim to direct deps; pin where toml_edit places [tool.uv.sources] The SourcesLayout doc claimed an "out-of-order multi-package document revert converges" for the dotted script layout. That holds for DIRECT deps, each of which owns one `sources.<name>` key; TRANSITIVE pairs share a single `override-dependencies` array and `restore_value`'s array branch flags drift as soon as the entries stop lining up, so their revert is order-dependent whatever the layout. It also promised "a real `[tool.uv.sources]` table AFTER `[project]`", but toml_edit places a new table right after the last pre-existing table it visits in depth-first key order: after a trailing `[build-system]`, right after an existing `[tool.uv]`, and — with `[tool.ruff]` as the only tool table ahead of `[project]` — inside the tool block BEFORE `[project]`. Both statements would mislead whoever next touches revert or placement. Scope the doc to direct deps, describe the placement rule as observed, and pin the renderings with idempotency (a second rewrite returns None) after every assertion, including the two pre-existing rendering tests: - trailing `[build-system]` after `[project]`, LF and CRLF; - `[tool]` with `uv = { … }` as an inline table: the source lands as a dotted `sources.alpha` key inside the inline table, parses, settles; - `[tool.ruff]` before `[project]` (direct + transitive) and `[tool.uv]` followed by `[tool.ruff]`; - upgrade path from an older CLI's root-body dotted `tool.uv.sources.other = {…}`: the new source extends the dotted table instead of adding a conflicting header; - PEP 723 block with a user-authored `# [tool.uv.sources]` / `# [tool.uv]` header: our insertion joins the table, the extracted block parses, and every line is re-commented. All GREEN on HEAD; no rendering bug surfaced. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * pypi_lock: read symlinked locks like their targets, refuse to write through them Since Python lock discovery started following symlinks (#239), `read_file`'s lstat-based `symlink_metadata().is_file()` pre-check rejected every linked pylock.toml / *.py.lock with `pypi_lock_read_failed: … is not a regular file`. `contains_target` and `load_python_locks` bubble the FIRST unreadable sibling, so a stray non-referencing `pylock.dev.toml` link beside a REGULAR pylock.toml blocked vendoring of the whole project — on main the link was simply invisible. Reading through a link is safe (`open_regular_file`'s fstat already accepts a link to a regular file and still rejects FIFOs/directories), so `read_file` now delegates to the shared `read_regular_to_string`. Writing through one is not: every writer here stages next to the path and renames over it, which replaces the link with a detached regular copy — the target goes stale (uv itself writes THROUGH the link) and a later revert restores bytes but never the link (git: 120000→100644 typechange). The hosted replay side already refuses linked files fail-closed; `wire_python_locks` and `revert_python_locks` now check `is_symlink` on every file they are about to write (the lock AND the paired PEP 723 script) BEFORE the first write and refuse with `pypi_lock_symlink_unsupported` naming the file. A refused revert fails outright so the artifact and ledger entry stay for a retry. Tests (RED-first for the four symlink cases): stray sibling link no longer blocks the regular lock; dangling link is ignored by discovery; symlinked lock and symlinked script are refused before any write (link intact, target byte-identical, no .socket/); the revert twin fails and keeps the vendored bytes. Also pins `restore_document`'s CRLF re-application (LF ledger + CRLF live, the mirror, and the two-package out-of-order CRLF path that does not short-circuit) and documents the known order-dependence of reverting two TRANSITIVE script deps sharing one override-dependencies array. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * hosted: refuse symlinked rewrite targets before the ledger; FIFO-safe candidate reads The hosted write loop renames a staged copy over every `rewrite.files` key. Since discovery follows links, a symlinked uv.lock / pylock*.toml / *.py.lock / pyproject.toml (and, all along, a symlinked package-lock.json) was silently converted into a detached regular copy: the shared target stayed unpatched (uv writes THROUGH the link), and `--revert` could restore bytes but never the link. Reproduced against uv 0.12.15. The revert side (replay.rs) already refuses linked files; the write side now matches: before `save_redirect_state` and before any file write, every planned file is checked with the new `utils::fs::first_symlink`, and a hit fails the WHOLE rewrite (hosted rewrites are transactional) with exit 1, stable code `redirect_symlinked_file_unsupported` naming the file, `errorCode` in the JSON envelope, no ledger and no writes. Applies to every ecosystem and to dry runs, so a dry run predicts the refusal rather than a rewrite that will never happen. Separately, six candidate reads (REDIRECT_CANDIDATE_FILES, the Python locks and their paired scripts, the rush common/subspace locks, and the sync pnpm-workspace.yaml trust probe) used a plain `read_to_string`: a FIFO planted under any of those names wedged `scan --mode hosted` / `get --mode hosted` forever in open(2) waiting for a writer (reproduced). They now go through `read_regular_to_string` (non-blocking open + fstat), plus a new blocking twin `read_regular_to_string_sync` for the sync probe that keeps its NotFound/PermissionDenied/InvalidData classification. A non-regular file reads as unreadable and is skipped like a missing one. Tests (RED-first): `hosted_refuses_symlinked_lock_before_ledger_write` (real 0.12.15 uv.lock linked into a shared dir, wheel served by wiremock: exit 1, link intact, pyproject untouched, no .socket/), the package-lock.json twin with the human-output check, a regular-file control that redirects, and `hosted_scan_returns_with_fifo_candidate` (mkfifo pyproject.toml beside a real uv.lock, 90 s deadline with FIFO release on timeout — wedged before the fix, returns in ~1 s after). fs.rs unit tests cover the sync reader's error kinds and FIFO fail-fast and `first_symlink`'s ordering/dangling semantics. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * backtest-uv: variant lane, stdout export, and a doc-table renderer The real-uv matrix's only native-project fixture is `dependencies = ["urllib3==1.26.18"]`, which never reaches the `[tool.uv] dev-dependencies` / PEP 735 requires-dev paths, a duplicate requires-dist entry, a `[manifest]` constraints entry, or the transitive override branch — exactly the shapes where `uv sync --locked` was found to exit 2 after a "successful" scan. Add `variant_matrix`, run on every `[[package]]`-grammar binary: five pyproject shapes (tool-uv-dev, dependency-groups, extras-duplicate, constraints, transitive) locked fresh, scanned hosted and vendored, then installed into a fresh venv with --frozen, --locked (where listed) and a plain `uv sync`, recording exit code, lockUnchanged and the installed response.py hash. A plain sync that rewrites the lock is recorded, not raised: on uv < 0.5.6 the transitive fixture is expected to reinstall the registry wheel. Fixtures the binary cannot lock (no [dependency-groups] before 0.4.27, no [manifest] constraints on older uv) record formatSupported: false. The transitive cut-off lives in `[tool.uv] exclude-newer` rather than on the `uv lock` command line: a CLI flag is stored under `[options]` and its absence on sync makes uv re-resolve ("removal of global exclude newer"), failing --locked for a reason unrelated to the patch. Against the pre-fix CLI on 0.12.15 and 0.2.37 the lane is red where the review said it would be: vendored tool-uv-dev and extras-duplicate and both modes of constraints fail --locked on 0.12.15; on 0.2.37 tool-uv-dev and transitive fail --locked and plain sync reinstalls the pristine wheel. The export lane passed `uv export --output-file`, which only exists from 0.4.7, so the pinned export-boundary release 0.4.1 recorded four exit-2 rows and the doc showed "— / —" for a release whose export installs the patch. Read the export from stdout and write the file only on a successful exit (an empty file would pass for a lock in format_matrix); 0.4.1 now records Pass / Pass for the requirements export. write_summary carries the variant observations and gives each version row a `variants` verdict, plus `pythonVersion`. `--render-doc-table <results.json>` prints the generated section of docs/testing/uv-compatibility.md (run header, per-release table, project-variant table) so the doc is regenerated after a run instead of hand-edited; rendering the committed results.json reproduces the current 41 rows byte-for-byte. probe-uv-boundaries.py separates "the `uv lock` subcommand exists" (0.1.42) from "it writes a lock" (0.1.45; 0.1.42–0.1.44 panic "not yet implemented"), reports the panic line, and checks whether `uv export` accepts `--output-file` (0.4.7). Verified on real 0.1.41/0.1.44/0.1.45 and 0.4.6/0.4.7 binaries. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: correct the uv boundaries and describe the #239 hardening docs/testing/uv-compatibility.md carried three inaccuracies the real binaries contradict: `[[distribution]]` locks start at 0.1.45, not "0.1.x" (the `uv lock` subcommand appears in 0.1.42, panics through 0.1.44 and first writes a lock at 0.1.45); the root-build note blamed "uv 0.2.x and 0.3.0" when the cold --offline failure spans 0.2.35–0.3.5, 0.3.3 adds `--no-install-project` (which the harness passes), and 0.4.0 stops building a build-system-less root; and the vendored refusal for `[[distribution]]` locks claimed uv "records absolute file paths" when the real reasons are that a relative path source is unparseable through 0.2.6 and, on 0.2.17–0.2.34, rejected by --locked and absolutized by `uv lock` / plain `uv sync` with CWD-relative resolution and unverified hashes. Fix all three, add the `uv export --output-file` (0.4.7) boundary, define "lock-preservation check" precisely (every install attempt against a patched lock, failed installs included, with the --locked rows as the ones that measure preservation), and stop calling every non-zero exit a documented uv boundary: the 0.4.1 export rows were a harness gap. Document the hardening landing in #239: [tool.uv] dev-dependencies as a direct dependency with every duplicate requires-dist/requires-dev entry repointed, [manifest] constraints repointed, the pypi_uv_override_requires_uv_0_5_6 advisory (uv applies sources to overrides only from 0.5.6), symlinked project files refused before any write in both modes, vendored requirements' hash-enforcement boundaries (--require-hashes from 0.1.32, default from 0.5.x), and the CRLF-preserving vendored uv backend. Describe the new project-variant lane and `--render-doc-table`, and fence the results section with GENERATED markers so the merger regenerates it after the full rerun; the recorded numbers are unchanged and the rerun replaces the variant-table placeholder. docs/ecosystems.md and CHANGELOG.md: hosted uv coverage is requirements from 0.0.5 and native uv.lock from 0.1.45, not "every release since 0.1"; symlinked locks are discovered and refused before any write; the CRLF bullet covers the vendored backend; new bullets for the --locked fixes, the constraints repoint, the override advisory, the reworded [[distribution]] refusal, the unwired-revert guard fixes and the FIFO-safe CLI reads. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * uv: count [tool.uv] dev-dependencies as direct for hosted sources; bracket the uv lock boundary Hosted twin of the vendored classifier fix: rewrite_project_metadata decided "direct" from project.dependencies / optional-dependencies / dependency-groups only, so a package declared in the legacy [tool.uv] dev-dependencies array got an override-dependencies entry plus a trailing [manifest] overrides section it never needed — and uv < 0.5.6 does not apply [tool.uv.sources] to overrides at all. uv records that array under [package.metadata.requires-dev], which complete_python_lock_metadata already repoints, so the source alone carries the redirect. Pin uv 0.1.44 in the backtest matrix so the `uv lock` boundary (first lock written by 0.1.45; the subcommand panics through 0.1.44) is bracketed like every other one, and refresh the stale CRLF premise in the toml_surgery removal-helper test now that the uv wire renders in the file's convention. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * uv vendor: advise that the [manifest] constraints repoint needs uv >= 0.5.6 The 42-release matrix's new constraints variant showed the boundary the override branch already had: uv 0.2.37–0.5.3 serialize [manifest] constraints as { name, specifier } regardless of sources, so the repointed entry (the shape uv >= 0.5.6 writes and requires) makes `uv sync --locked` fail there, while --frozen and a plain sync still install the patch (the plain sync rewrites the entry back). No lock marker separates 0.5.3 from 0.5.6, so emit the advisory pypi_uv_constraints_require_uv_0_5_6 next to the repoint instead of refusing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Record the 42-release uv compatibility matrix for source 17d0dcb Regenerate results.json, binaries.json and the generated results section of docs/testing/uv-compatibility.md from a full scripts/backtest-uv.py run against the rebuilt CLI at 17d0dcb: 42 pinned uv releases (0.1.44 added to bracket the `uv lock` boundary), 1403 installed-byte comparisons and 1086 lock-preservation checks including the new project-variant lane ([tool.uv] dev-dependencies, [dependency-groups], extras duplicate, constraint-dependencies, transitive) on every [[package]] binary. Every mismatch and lock change is a documented uv boundary below 0.5.6: the 18 mismatches are the transitive variant's plain `uv sync` on 0.2.35–0.5.0 (sources not applied to override-dependencies), the 32 lock changes / non-zero --locked rows are those plus the constraints variant on 0.2.37–0.5.0 ({ name, specifier } serialization regardless of sources); the patch still installed in every constraints row. From 0.5.16 every shape passes in both modes with the lock byte-unchanged. The 0.4.1 export lane now records real installs (stdout export). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * utils/fs: accept Windows' PermissionDenied for a directory in the sync-reader test CreateFileW refuses to open a directory with ERROR_ACCESS_DENIED before the fstat regular-file check can classify it as InvalidInput; the callers skip either error kind, so the test now accepts both off Unix. Windows CI leg on 18d4046 was the only failure. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
1 parent ade011e commit 745e8dc

22 files changed

Lines changed: 24772 additions & 1201 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,75 @@ into the new version's section — see docs/releasing.md.
5151

5252
### Added
5353

54+
- **Python patches survive uv lockfiles in both hosted and vendored modes.**
55+
`scan --mode hosted|vendored` now rewrites native `uv.lock` together with
56+
the paired `pyproject.toml` source and metadata, PEP 723 script locks
57+
(`*.py.lock` plus the script's inline metadata), PEP 751 `pylock*.toml`,
58+
and uv-compiled hashed `requirements.txt`, so `uv sync --frozen|--locked`,
59+
`uv run --script`, and `uv pip sync --require-hashes` install the patched
60+
wheel instead of the registry artifact. Verified against real uv binaries
61+
from every 0.x release family (0.0 through 0.12) — first and latest release
62+
of each plus every observed behaviour boundary — see
63+
`docs/testing/uv-compatibility.md`: hosted mode covers requirements from
64+
uv 0.0.5 and native `uv.lock` from 0.1.45 — the first release whose
65+
`uv lock` writes one — through all three `[[distribution]]` lock shapes
66+
and `[[package]]`; vendored native covers every `[[package]]` release
67+
(uv ≥ 0.2.35), vendored requirements cover uv ≥ 0.1.24 (hash-enforced by
68+
`uv pip sync --require-hashes` from 0.1.32 and by default from 0.5.x).
69+
Follow-up hardening:
70+
`vendor --revert` refuses to delete a vendored Python wheel a lock still
71+
references when the ledger entry has no wiring to replay (the shape
72+
`repair` rebuilds), a script or PEP 751 lock supplements rather than hides
73+
`poetry.lock`/`requirements.txt` pins, symlinked locks are discovered, and
74+
refused before any write (never rewritten in place — uv writes through the
75+
link, an atomic rename would replace it; hosted
76+
`redirect_symlinked_file_unsupported`, vendored
77+
`pypi_uv_symlink_unsupported` / `pypi_lock_symlink_unsupported`), CRLF
78+
locks keep their line endings in both the hosted rewriter and the vendored
79+
uv backend (`pyproject.toml`, the rewritten `[[package]]` unit and the
80+
appended `[manifest]` / `[package.metadata]` fragments, plus their revert),
81+
and the hosted `[tool.uv.sources]` edit renders as a header after
82+
`[project]` the way uv writes it. (#238, #239)
83+
- **uv `--locked` survives the project shapes the plain fixture never
84+
reached.** A package listed only in `[tool.uv] dev-dependencies` is now
85+
classified as a direct dependency (it was wired as a transitive override
86+
and its `requires-dev` entry left stale), and every duplicate
87+
`requires-dist` / `requires-dev` entry for the package — extras, markers —
88+
is repointed rather than only the first, so `uv sync --locked` and
89+
`uv lock --check` accept the patched lock instead of exiting 2 and a plain
90+
`uv sync` no longer rewrites it. `[tool.uv] constraint-dependencies` /
91+
`build-constraint-dependencies` naming the package have their `[manifest]`
92+
`constraints` / `build-constraints` entries repointed too (uv ≥ 0.5.6
93+
serializes them with the package's source; 0.2.37–0.5.3 reject the
94+
repointed entry under `--locked`, so the repoint emits the advisory
95+
`pypi_uv_constraints_require_uv_0_5_6`). The transitive
96+
(override-dependencies) branch emits the advisory
97+
`pypi_uv_override_requires_uv_0_5_6`: uv applies `[tool.uv.sources]` to
98+
overrides only from 0.5.6, so on 0.2.35–0.5.3 `--frozen` installs the
99+
patch but a plain `uv sync` reinstalls the registry wheel. The
100+
`[[distribution]]`-grammar vendoring refusal now names the real reasons
101+
(relative path sources unparseable through 0.2.6, rejected by `--locked`
102+
and absolutized by `uv lock` / `uv sync` on 0.2.17–0.2.34) instead of
103+
"records absolute paths". `scripts/backtest-uv.py` gains a project-variant
104+
lane covering these shapes on every `[[package]]` binary and a
105+
`--render-doc-table` mode that prints the doc's results tables from
106+
`results.json`; its export lane reads `uv export` from stdout because
107+
`--output-file` only exists from 0.4.7. (#239)
108+
- **Unwired Python vendor entries revert safely.** `vendor --revert` /
109+
`rollback` on a ledger entry without wiring to replay (the shape `repair`
110+
reconstructs) skips the lock-reference guard under `--preserve-state`
111+
(nothing is deleted, so nothing needs protecting), refuses fail-closed when
112+
the project root cannot be listed or a lock's symlink target cannot be
113+
read (instead of treating "could not enumerate locks" as "no lock
114+
references it" and deleting the wheel), probes `-r` / `--requirement`
115+
includes of `requirements.txt` alongside the root file — the orphan sweep
116+
and `repair` see include-hosted pins too — and reclaims a genuinely
117+
orphaned artifact directory whose lock is gone or whose ledger flavor is
118+
unknown instead of failing forever. Hosted `scan` / `get`, `repair`, and
119+
ledger-less `rollback` read candidate lockfiles through the FIFO-safe
120+
reader, so a named pipe at a lockfile name no longer wedges the command in
121+
`open(2)`. (#239)
122+
54123
- **Path targeting on `scan` and `rollback`.** `scan [PATHS]...` scopes
55124
discovery to packages with an installed copy under a matching glob
56125
(ancestor rule: `scan packages/foo` covers the subtree; `*` never crosses

‎crates/socket-patch-cli/src/commands/repair_vendor.rs‎

Lines changed: 133 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,9 @@
1515
//! from the lockfile path itself (the contract's uuid-in-path rule), the
1616
//! record from the manifest (or the patch API, yielding a detached entry),
1717
//! and a fresh ledger entry is re-synthesized so sweep/GC/revert know the
18-
//! artifact again — stamped with the npm lockfile FLAVOR the reference was
19-
//! found in, so a later `vendor --revert` routes to the backend whose
20-
//! unwired-revert guard probes the right lockfile. WIRING reconstruction is
18+
//! artifact again — stamped with the lockfile FLAVOR the reference was
19+
//! found in (npm family and pypi), so a later `vendor --revert` routes to the
20+
//! backend whose unwired-revert guard probes the right lockfile. WIRING reconstruction is
2121
//! per-ecosystem: gem recognizes
2222
//! its own Gemfile/lock wiring and rebuilds full revert-capable records
2323
//! ([`socket_patch_core::vendor::gem::reconstruct_gem_wiring`]); the other
@@ -55,6 +55,7 @@ use socket_patch_core::api::client::{get_api_client_with_overrides, ApiClient};
5555
use socket_patch_core::crawlers::CrawlerOptions;
5656
use socket_patch_core::manifest::schema::{PatchManifest, PatchRecord};
5757
use socket_patch_core::patch::copy_tree::remove_tree;
58+
use socket_patch_core::utils::fs::read_regular_to_string;
5859
use socket_patch_core::utils::purl::{
5960
normalize_purl, percent_decode_purl_component, strip_purl_qualifiers,
6061
};
@@ -98,6 +99,8 @@ struct Candidate {
9899

99100
/// Files the vendor backends rewire — the search space for
100101
/// `.socket/vendor/<eco>/<uuid>/<leaf>` references when the ledger is gone.
102+
/// The Python locks the root LISTS (`pylock*.toml`, `*.py.lock` + script)
103+
/// and the requirements `-r` include tree are appended at scan time.
101104
const WIRING_FILES: &[&str] = &[
102105
"package-lock.json",
103106
"npm-shrinkwrap.json",
@@ -143,10 +146,23 @@ pub(crate) async fn scan_vendor_references(project_root: &Path) -> Vec<(String,
143146
files.push(path);
144147
}
145148
}
149+
// The requirements planner writes a vendored pin where the original pin
150+
// was — possibly inside a `-r` include — so the root requirements.txt
151+
// alone would miss it (and the orphan sweep, which reuses this scan,
152+
// would delete the include-referenced wheel). An unreadable include
153+
// tree degrades to the root file, matching the per-file tolerance
154+
// below.
155+
if let Ok(includes) =
156+
socket_patch_core::vendor::requirements_include_names(project_root).await
157+
{
158+
files.extend(includes);
159+
}
146160
files.sort();
147161
files.dedup();
148162
for file in files {
149-
let Ok(text) = tokio::fs::read_to_string(project_root.join(file)).await else {
163+
// FIFO-safe: a pipe under a wiring-file name must be skipped, not
164+
// waited on forever in open(2).
165+
let Ok(text) = read_regular_to_string(&project_root.join(file)).await else {
150166
continue;
151167
};
152168
let mut rest = text.as_str();
@@ -221,8 +237,14 @@ fn synth_entry(eco: &str, uuid: &str, artifact_path: &str, base_purl: &str) -> V
221237
async fn detect_reference_flavor(project_root: &Path, eco: &str, uuid: &str) -> Option<String> {
222238
if eco == "pypi" {
223239
let needle = format!(".socket/vendor/pypi/{uuid}/");
224-
for file in socket_patch_core::utils::python_lock::python_lock_paths(project_root).ok()? {
225-
if tokio::fs::read_to_string(project_root.join(&file))
240+
let mut files =
241+
socket_patch_core::utils::python_lock::python_lock_paths(project_root).ok()?;
242+
// uv.lock outranks the standalone locks (the vendor backend's own
243+
// precedence): a pylock EXPORTED from the wired project lock must not
244+
// relabel the entry `python-lock`. Alphabetical order would.
245+
files.sort_by_key(|file| file != "uv.lock");
246+
for file in files {
247+
if read_regular_to_string(&project_root.join(&file))
226248
.await
227249
.ok()
228250
.is_some_and(|text| text.contains(&needle))
@@ -244,7 +266,7 @@ async fn detect_reference_flavor(project_root: &Path, eco: &str, uuid: &str) ->
244266
}
245267
let needle = format!(".socket/vendor/npm/{uuid}/");
246268
let read = |name: &'static str| async move {
247-
tokio::fs::read_to_string(project_root.join(name))
269+
read_regular_to_string(&project_root.join(name))
248270
.await
249271
.ok()
250272
};
@@ -1446,6 +1468,96 @@ fn npm_coords(base_purl: &str) -> Option<(String, String)> {
14461468
mod tests {
14471469
use super::*;
14481470

1471+
/// A FIFO under a wiring-file name (here the paired `<script>.py` of a
1472+
/// `*.py.lock`, which the lister cannot filter because it derives the
1473+
/// script name without stat'ing it) used to wedge `repair` forever: a
1474+
/// plain `read_to_string` blocks in open(2) waiting for a writer. Every
1475+
/// reference read must go through the FIFO-safe reader and skip it.
1476+
#[cfg(unix)]
1477+
#[tokio::test]
1478+
async fn repair_returns_with_fifo_script() {
1479+
use std::time::Duration;
1480+
let tmp = tempfile::tempdir().unwrap();
1481+
let root = tmp.path();
1482+
let uuid = "11111111-1111-4111-8111-111111111111";
1483+
let path = format!(".socket/vendor/pypi/{uuid}/requests-2.28.1-py3-none-any.whl");
1484+
tokio::fs::write(
1485+
root.join("tool.py.lock"),
1486+
format!("archive = {{ path = '{path}' }}"),
1487+
)
1488+
.await
1489+
.unwrap();
1490+
let fifos = ["tool.py", "bun.lock"];
1491+
for name in fifos {
1492+
let c = std::ffi::CString::new(root.join(name).to_str().unwrap()).unwrap();
1493+
assert_eq!(unsafe { libc::mkfifo(c.as_ptr(), 0o644) }, 0, "mkfifo {name}");
1494+
}
1495+
// Release valve: if a read DID wedge in open(2), connecting a
1496+
// writer lets the blocking thread finish so the runtime can shut
1497+
// down and the test fails on the timeout instead of hanging.
1498+
let release = || {
1499+
use std::os::unix::fs::OpenOptionsExt as _;
1500+
for name in fifos {
1501+
let _ = std::fs::OpenOptions::new()
1502+
.write(true)
1503+
.custom_flags(libc::O_NONBLOCK)
1504+
.open(root.join(name));
1505+
}
1506+
};
1507+
1508+
let scanned =
1509+
tokio::time::timeout(Duration::from_secs(5), scan_vendor_references(root)).await;
1510+
release();
1511+
let refs = scanned.expect("scan_vendor_references must not wedge on a FIFO script");
1512+
assert_eq!(
1513+
refs,
1514+
vec![("pypi".to_string(), uuid.to_string(), path.clone())],
1515+
"the lock reference is still recovered around the FIFO"
1516+
);
1517+
1518+
let flavor = tokio::time::timeout(
1519+
Duration::from_secs(5),
1520+
detect_reference_flavor(root, "npm", uuid),
1521+
)
1522+
.await;
1523+
release();
1524+
assert_eq!(
1525+
flavor.expect("detect_reference_flavor must not wedge on a FIFO lock"),
1526+
None
1527+
);
1528+
}
1529+
1530+
/// The requirements planner writes vendored pins into `-r` includes,
1531+
/// so a reference may live ONLY in an include. The scan used to read
1532+
/// the root requirements.txt alone, leaving such a wheel unrecoverable
1533+
/// by `repair` and deletable by the orphan sweep.
1534+
#[tokio::test]
1535+
async fn scan_recovers_include_hosted_requirements_reference() {
1536+
let tmp = tempfile::tempdir().unwrap();
1537+
let root = tmp.path();
1538+
let uuid = "11111111-1111-4111-8111-111111111111";
1539+
let path = format!(".socket/vendor/pypi/{uuid}/six-1.16.0-py2.py3-none-any.whl");
1540+
tokio::fs::write(root.join("requirements.txt"), "-r requirements/base.txt\n")
1541+
.await
1542+
.unwrap();
1543+
tokio::fs::create_dir(root.join("requirements")).await.unwrap();
1544+
tokio::fs::write(
1545+
root.join("requirements/base.txt"),
1546+
format!(
1547+
"./{path} --hash=sha256:{} # socket-patch vendor: six==1.16.0\n",
1548+
"0".repeat(64)
1549+
),
1550+
)
1551+
.await
1552+
.unwrap();
1553+
let refs = scan_vendor_references(root).await;
1554+
assert_eq!(
1555+
refs,
1556+
vec![("pypi".to_string(), uuid.to_string(), path)],
1557+
"{refs:?}"
1558+
);
1559+
}
1560+
14491561
#[tokio::test]
14501562
async fn scan_recovers_script_and_pep751_vendor_references() {
14511563
let tmp = tempfile::tempdir().unwrap();
@@ -1462,14 +1574,27 @@ mod tests {
14621574
let references = scan_vendor_references(tmp.path()).await;
14631575
assert_eq!(
14641576
references,
1465-
vec![("pypi".to_string(), uuid.to_string(), path)]
1577+
vec![("pypi".to_string(), uuid.to_string(), path.clone())]
14661578
);
14671579
assert_eq!(
14681580
detect_reference_flavor(tmp.path(), "pypi", uuid)
14691581
.await
14701582
.as_deref(),
14711583
Some("python-lock")
14721584
);
1585+
// The project lock outranks a pylock exported from it.
1586+
tokio::fs::write(
1587+
tmp.path().join("uv.lock"),
1588+
format!("version = 1\n\n[[package]]\nname = \"requests\"\nversion = \"2.28.1\"\nsource = {{ path = \"{path}\" }}\n"),
1589+
)
1590+
.await
1591+
.unwrap();
1592+
assert_eq!(
1593+
detect_reference_flavor(tmp.path(), "pypi", uuid)
1594+
.await
1595+
.as_deref(),
1596+
Some("uv")
1597+
);
14731598
}
14741599

14751600
/// pnpm writes vendored paths in THREE spellings — override values,

0 commit comments

Comments
 (0)