Skip to content

Commit ff6aaae

Browse files
Support hosted and legacy Poetry patches (#241)
* Support Poetry patch lock generations Wire hosted and vendored wheels across native Poetry lock formats. Preserve independent rollback and verify installer compatibility. Assisted-by: Codex:gpt-6-astra * Keep Poetry rollback source and hashes together Preserve the lock when a legacy patch fragment has drifted, so rollback does not restore upstream hashes while retaining a patched source. Assisted-by: Codex:gpt-6-astra * Keep Poetry fixtures portable across checkouts Normalize checked-out fixtures before explicitly exercising LF and CRLF, so Windows auto-conversion cannot double carriage returns. Assisted-by: Codex:gpt-6-astra * fix(poetry): harden the shared lock rewriter and align hosted refusals with uv - Never panic on a user-editable lock: a `[metadata.files]` / `[metadata.hashes]` value that is not a table (array of tables, scalar, `[[metadata]]`) used to hit toml_edit's IndexMut and abort `scan --mode hosted|vendored` with exit 101 and no JSON; every table access is now guarded and refuses with a warning. - Accept any `lock-version = "2.<n>"` in the shared rewriter (the vendored loader already accepted newer minors with an advisory), so the same lock no longer succeeds LF-vendored, hard-fails CRLF-vendored and silently no-ops hosted. - Supersede an earlier Socket hosted URL for the same wheel in place (same origin + filename, fragment ignored) instead of refusing it as a foreign source, so a grant-token rotation or republished patch does not strand the pin — the bun / requirements / cargo rewriters make the same call. Foreign sources and vendored file sources are still refused. - Hosted rewriter: gate the missing-SHA-256 warning once per dep (not per lock), warn `redirect_poetry_entry_not_found` when a lock has no entry at the granted version (uv parity), and emit `redirect_poetry_stale_install_risk` when the lock was written by Poetry < 1.4 — measured on real 0.12.17–2.4.3: those releases never replace an already-installed same-version package after the redirect, 1.4+ do. - Write the SHA-256 lowercase (Poetry compares hexdigest strings) and anchor the legacy integrity fragment at its line break so a suffix-named sibling entry with the same value cannot make the splice ambiguous. - Unit tests for the shared rewriter (it had none) and hosted integration tests for the new warnings, the 2.x minor, and token rotation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(poetry): key the vendored pre-1.4 advisory on the lock's writer `pypi_poetry_integrity_unverified` fired for every lock-version 2.0 project, telling Poetry 1.4–1.8 users their installer verifies nothing. Lock 2.0 is written by 1.3 through 1.8; releases from 1.4 stamp `@generated by Poetry X.Y.Z`, so a stamped 2.0 lock is no longer flagged. The advisory now also names the second pre-1.4 behaviour measured on real releases: an already-installed same-version package is not replaced after the lock is rewired. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(poetry): add the per-release live matrix harness and its results scripts/backtest-poetry.py mirrors backtest-uv.py: it bootstraps every Poetry release (0.12.17 … 2.4.3) with uv, generates a native lock for a one-dependency project with a public free-tier patch, and drives the real CLI through hosted, vendored, agent and out-of-tree-venv agent mode plus `setup`, checking applied counts, idempotent re-scans, lock-driven installs into an emptied virtualenv, fresh-clone installs, `poetry check --lock`, `vex`, tampered-hash rejection, warm-virtualenv reinstall, Poetry's own relock, and `rollback` restoring every byte. 108 cases pass on the PR head; the results file feeds the docs table. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(poetry): document measured installer boundaries, warning codes and the matrix - CHANGELOG [Unreleased] entry for hosted + legacy Poetry support (the uv precedent had one; this PR did not). - poetry-compatibility.md rewritten around the measured matrix: per-generation rewrite shapes, the Poetry < 1.4 warm-virtualenv and hash-verification boundaries, which Poetry commands drop the patch source (with content-hash unchanged, so `poetry check --lock` cannot tell), the 0.12/1.0 cwd-relative file-source caveat, agent-mode discovery of Poetry's out-of-tree virtualenv, the lock-only vendored limitation, the harness recipe and the generated results table. - CLI_CONTRACT: poetry.lock joins the hosted candidate files; the vendored rows cover every lock generation; the new `redirect_poetry_*` and `pypi_poetry_integrity_unverified` codes are listed. - hosted-production-e2e.md no longer says Poetry locks are not rewritten by hosted mode; README no longer says hosted mode has no CLI revert (`rollback` unwinds it since the scan↔rollback duality change). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(pypi): discover Poetry's out-of-tree virtualenv in agent mode Poetry keeps a project's virtualenv OUTSIDE the project by default (`{cache-dir}/virtualenvs/<name>-<hash>-py<X.Y>`), so after a plain `poetry install` the crawler found no VIRTUAL_ENV / .venv / venv and fell through to the global interpreter's site-packages: `scan --mode agent` patched nothing for the project's dependencies (or the wrong interpreter) and reported success, and a bare `rollback` pruned the manifest while the venv stayed patched — measured on real Poetry 1.1.15, 1.8.5 and 2.4.3. The crawler now reproduces Poetry's own placement without running Poetry: `virtualenvs.create` / `in-project` / `path` and `cache-dir` from the `POETRY_*` environment, the project's `poetry.toml` and the user `config.toml` (Poetry's precedence), the platform default cache dir, the `{cache-dir}` / `{project-dir}` / `~` placeholders, and `EnvManager.generate_env_name` (lowercased sanitized name, 42-char cap, first 8 chars of url-safe base64 sha256 of normcase(realpath(cwd))) — unchanged from Poetry 1.0 through 2.x and pinned by known-answer vectors. Every `-py<X.Y>` sibling is scanned. A project-local venv still wins; a project that opted into in-project venvs or disabled creation is left to the existing paths. The matrix harness's out-of-tree leg now applies and rolls back BARE when the crawler sees the venv (falling back to `poetry run` otherwise) and records which path it took. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(pypi): vendor Poetry projects from a lock-only checkout `scan --mode vendored` on a fresh clone (pyproject + poetry.lock, nothing installed) was skipped with `vendor_fetch_unverifiable` + `package_not_installed` although the lock records the wheel's sha256 — the poetry.lock inventory was discovery-only (`LockIntegrity::None`), so the fetch gate refused before the vendor engine ever ran, while the identical uv.lock scenario vendored fine. That broke the CI story for Poetry: the machine that vendors had to have the package installed. The inventory now carries the pure-Python (`-none-any.whl`) wheel's sha256 from `files` (lock 2.x) or `[metadata.files]` (lock 1.0/1.1), lowercased, and the pypi fetcher resolves a hash-only entry through PyPI's JSON API (`urls[].digests.sha256`, `SOCKET_PYPI_JSON_API` overrides the endpoint) and verifies the download against the same digest, exactly like uv's lock-only path. Poetry 0.12's bare `[metadata.hashes]` names no wheel, and platform-only wheels offer no platform-independent choice, so those stay discovery-only. Measured on the fixed CLI: lock-only vendoring now applies on Poetry 1.0 (populated lock), 1.2, 1.8 and 2.4 fixtures (`vendor_fetched_missing` + `vendor_prebuilt_downloaded`), and the resulting checkout installs the patched wheel with every release. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(poetry): describe automatic virtualenv discovery and lock-only vendoring CHANGELOG Fixed bullets for the two pre-existing gaps the Poetry matrix surfaced (out-of-tree venv discovery, lock-only vendoring), the contract's pypi discovery-root sentence and `vendor_fetched_missing` row, and the compatibility doc's mode notes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(poetry): verify lock-1.0 hosted wheels on Poetry >= 1.2 and name the pip window - A lock-version 1.0 hosted rewrite only carried the patched hash in `[metadata.files]` and the URL fragment. Poetry >= 1.2 (which still reads 1.0 locks) verifies url sources against the package's own `files` and ignores `metadata.files` hashes, so it installed the hosted wheel with no hash check at all (measured on 1.2.2: every hash tampered, install exit 0). Write the package `files` entry for 1.0 too — Poetry 1.0.10 ignores the extra key (measured), 1.2.2 then rejects a tampered hash. - Poetry 1.0 installs url sources through pip, reading the hash from the URL fragment that Poetry suffixes with `#egg=<name>`. Sweeping pip 20.3–25.0 with real Poetry 1.0.10: the `#sha256=<hex>&` spelling installs and verifies on pip <= 22.2 and >= 23.1; pip 22.3 and 23.0 (what Python 3.8's ensurepip seeds) take the rest of the fragment as part of the digest and refuse (fail-closed); without the `&` every pip >= 22.3 refuses. The spelling stays; the lock-1.0 hosted advisory now names the pip window and the compatibility doc records the sweep. - Pin each generation's hosted shape in `poetry_hosted` (1.0 fragment + reference + dual files, 1.1 dual files, 2.x metadata untouched) and drop duplicated lines from a vendored test. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(vex): attest same-run hosted redirects whose ledger purl carries a qualifier `scan --mode hosted --vex` exempts the purls the run just confirmed from installed-tree verification and attests them from the redirect ledger (`assume_applied`). The confirmed purls come from the grant reference unqualified (`pkg:pypi/urllib3@1.26.18`) while the ledger records the API's artifact-qualified purl (`…?artifact_id=py2-py3-none-any-whl`), so for pypi redirects nothing matched: a lock-only Poetry (or uv) checkout redirected the lock and then exited 1 with `no_applicable_patches`. Match on the qualifier-stripped purl on both sides. Adds the first CLI-level Poetry hosted test: a lock-only project is redirected, attested in the same run, re-scanned idempotently, and rolled back byte for byte against a mocked API. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(vendor): let only fragment drift hold the atomic lock revert `revert_lock_fragment_splice_atomic` held the write whenever ANY warning was raised, so a record this build cannot replay (a newer ledger's unknown kind, or a foreign file skipped by the allowlist) blocked restoring the fragments it does understand — the lock stayed fully wired while the revert reported success. Track drift separately: a drifted fragment still holds the whole write (a half-restored legacy lock is uninstallable), forward-compat skips only warn. Also documents that `poetry lock --no-update` on Poetry 1.1/1.2 reshapes a hosted entry and drops the package-level files Poetry >= 1.2 verifies against. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(poetry): regenerate the matrix results on the fixed CLI 108/108 cases pass on the fix-branch head. The rerun records the fixes' effects: lock-only vendoring applies wherever the lock names a pure-Python wheel hash (1.0/1.1 populated, 1.2+), the out-of-tree agent leg applies and rolls back BARE on every 1.x/2.x release (the crawler finds Poetry's venv), and the pre-1.4 hosted advisory fires for 1.0–1.3 locks only. The table renderer reports the lock-only column per shape. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(poetry): keep hosted rollback invertible across Poetry's own relocks Two defects with one root cause — the recorded `[[package]]` fragment ended at the unit's last line while the rewrite APPENDS `[package.source]` after it, so the pristine fragment was a prefix of every rewritten (or relocked) unit: - Replay's "already converged" check (`content.contains(original) && !new.contains(original)`) fired when a relock (Poetry 1.1/1.2 `poetry lock --no-update`) or a hand edit dropped the inserted `files` line but kept the Socket source block: rollback reported success, deleted the ledger and restored `[metadata.files]` to upstream hashes while the lock still redirected — an inconsistent lock with no record left (lock 1.1 and 2.x). For lock 1.0 the guard's other arm meant a hand-restored pristine lock could never converge and rollback refused with a spurious drift. - A re-scan after such a relock appended edits recorded against the RELOCKED text; replay inverted the newer links back to the relocked state and then found neither `new` nor `original` of the older ones, so `rollback` and `remove` refused forever — and the refusal's remedy ("re-run scan") was what lengthened the chain (reproduced with real Poetry 1.1.15 and 1.2.2). The fragment now extends through the next top-level header (`[[package]]`, `[metadata]`, …), so a unit that grew a source block never contains the pristine fragment: the drifted shape is refused, the hand-restored lock converges. The hosted ledger merge REBASES `redirect_poetry_lock_package` edits whose (path, kind, key) already exists when the pre-run file no longer carried their `new` fragments — keeping the oldest `original`, adopting the fresh `new` — instead of appending, so one invertible link survives any number of relock/re-scan rounds. Idempotent re-scans and token-rotation supersedes still append as before. Tests: fragment boundary property per generation and for adjacent units; dropped-files-line-with-source-kept is refused not converged (1.1, 2.x); lock-1.0 hand-restored lock converges; CLI relock → re-scan → rollback chain lands on the pristine lock with a two-edit ledger. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(poetry): exercise relock → re-scan → rollback in the live matrix When Poetry's own relock keeps the patch source but re-lays the unit (1.1/1.2), the harness now re-scans instead of restoring the CLI's lock and lets the final rollback prove the rebased ledger still lands on pristine bytes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(poetry): final matrix results on the fixed CLI; isolate HOME for setup's Poetry 108/108 cases on the fix-branch head (0.12.17 … 2.4.3 × hosted / vendored / agent / out-of-tree agent / setup × direct / populated / crlf / pep621), including the live relock → re-scan → rollback chain on Poetry 1.1 and 1.2. The `setup` leg now hands the CLI-spawned `poetry lock` the case-isolated HOME as well (Poetry <= 1.1's shared HTTP-cache lock wedged one parallel run). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Fix Poetry CI line endings and stale hosted attestations Preserve LF/CRLF in the relock simulator and exercise both newline forms. Probe installed Python package hashes after hosted redirects, warn on stale bytes, and exclude them from same-run VEX even when another interpreter is patched. Cover re-scans, ledger fallback, missing files, dry runs, variants, and custom prefixes. Assisted-by: Codex:gpt-6-astra * Retry Windows delete-pending race in apply lock test Treat ERROR_ACCESS_DENIED from an open racing the deliberate lock-file deletion as a benign test interleaving on Windows. Retry within the existing bound and still require a clean mutually exclusive acquisition. Production lock acquisition and error handling are unchanged. Assisted-by: Codex:gpt-6-astra --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
1 parent b19c968 commit ff6aaae

54 files changed

Lines changed: 16625 additions & 130 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,38 @@ into the new version's section — see docs/releasing.md.
5151

5252
### Added
5353

54+
- **Poetry projects take hosted patches, and vendored patches now cover every
55+
`poetry.lock` generation.** `scan --mode hosted` rewrites `poetry.lock` to a
56+
`[package.source] type = "url"` pointing at the Socket-hosted, SHA-256-pinned
57+
wheel (Poetry 1.0 through 2.x; Poetry 0.12 ignores URL sources and is refused
58+
with `redirect_poetry_lock_unsupported`), and `scan --mode vendored` accepts
59+
the legacy `[metadata.hashes]` (0.12) and `[metadata.files]` (lock 1.0/1.1)
60+
layouts next to the 2.x `files` arrays, CRLF locks included. The rewrite keeps
61+
every other byte of the lock — dependency metadata, groups, markers, extras
62+
and the pyproject `content-hash` — and records independent rollback fragments
63+
per patch, so `rollback` restores the recorded originals in any order.
64+
Verified end-to-end against real Poetry 0.12.17, 1.0.10, 1.1.15, 1.2.2,
65+
1.3.2, 1.4.2, 1.5.1, 1.6.1, 1.7.1, 1.8.5, 2.0.1, 2.1.4, 2.2.1, 2.3.4 and
66+
2.4.3 in hosted, vendored and agent mode — see
67+
`docs/testing/poetry-compatibility.md` and `scripts/backtest-poetry.py`.
68+
Poetry releases before 1.4 neither verify local wheel hashes nor replace an
69+
already-installed package at the same version; both modes surface that as an
70+
advisory (`pypi_poetry_integrity_unverified`, `redirect_poetry_stale_install_risk`)
71+
keyed on the lock's writer, and the hosted rewriter warns
72+
`redirect_poetry_entry_not_found` when a lock has no entry for a granted
73+
patch (uv parity). A rotated grant token or republished patch supersedes the
74+
earlier hosted URL in place instead of being refused as a foreign source,
75+
a future `lock-version = "2.<n>"` is rewritten like 2.1 on every path
76+
(the vendored loader already accepted it), and a malformed
77+
`[metadata.files]` / `[metadata.hashes]` value is refused instead of
78+
panicking the scan. Rollback stays invertible across Poetry's own relocks:
79+
the recorded package fragment carries its boundary header, so a unit that
80+
Poetry 1.1/1.2 re-laid (source kept, inserted `files` line dropped) is
81+
refused rather than mistaken for an already-reverted lock, a lock-1.0
82+
redirect restored by hand converges instead of refusing, and a re-scan
83+
after such a relock REBASES the ledger's edits (pristine → current)
84+
instead of appending a chain whose older links match nothing — which made
85+
`rollback` and `remove` refuse forever. (#241)
5486
- **Python patches survive uv lockfiles in both hosted and vendored modes.**
5587
`scan --mode hosted|vendored` now rewrites native `uv.lock` together with
5688
the paired `pyproject.toml` source and metadata, PEP 723 script locks
@@ -168,6 +200,32 @@ into the new version's section — see docs/releasing.md.
168200

169201
### Fixed
170202

203+
- Hosted Python redirects now warn when installed files still contain upstream
204+
or modified bytes and omit those packages from same-run VEX. The read-only
205+
probe covers Poetry virtualenvs, repeats on re-scans, and uses persisted patch
206+
records if fetching fresh records fails.
207+
208+
- **Agent mode finds Poetry's out-of-tree virtualenv.** Poetry keeps a
209+
project's virtualenv under `{cache-dir}/virtualenvs/<name>-<hash>-py<X.Y>`
210+
by default, so after a plain `poetry install` the crawler saw no
211+
`VIRTUAL_ENV` / `.venv` / `venv` and fell through to the global interpreter:
212+
`scan --mode agent` patched nothing for the project's dependencies (or the
213+
wrong interpreter) while reporting success, and a bare `rollback` pruned the
214+
manifest while the venv stayed patched. The crawler now reproduces Poetry's
215+
own placement — `virtualenvs.create` / `in-project` / `path` and `cache-dir`
216+
from `POETRY_*`, the project's `poetry.toml` and the user `config.toml`, the
217+
platform default cache dir, and Poetry's env-name hash — without running
218+
Poetry, and scans every `-py<X.Y>` sibling. `poetry run socket-patch …` and
219+
`VIRTUAL_ENV` keep working as before.
220+
- **`scan --mode vendored` works from a lock-only Poetry checkout.** The
221+
`poetry.lock` inventory was discovery-only, so a fresh clone with nothing
222+
installed was skipped with `vendor_fetch_unverifiable` even though the lock
223+
records the wheel's sha256 (uv's lock vendored fine in the same scenario).
224+
The inventory now carries the pure-Python wheel's sha256 from `files` (lock
225+
2.x) or `[metadata.files]` (lock 1.0/1.1), and the pypi fetcher resolves a
226+
hash-only entry through PyPI's JSON API by that digest (verified again after
227+
download; `SOCKET_PYPI_JSON_API` overrides the endpoint). Poetry 0.12's bare
228+
`[metadata.hashes]` names no wheel and still needs an installed copy.
171229
- **`remove` no longer drops the manifest entry of a drift-kept vendored
172230
purl.** When the vendored revert keeps the artifact (`kept_artifact` —
173231
the lockfile drifted), the manifest entry is now kept too

‎README.md‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -233,7 +233,7 @@ The same patched bytes can reach your build three different ways. The modes diff
233233
|------|----------------------|--------------------------|-----------|
234234
| **agent** — `scan --mode agent` (or [`apply`](#apply)) | `.socket/` manifest + blobs, committed; the CLI re-applies after each install | The `socket-patch` CLI must run (install hook via [`setup`](#setup), or an `apply` step in CI) | Small repo footprint (per-file blobs, not whole packages); no lockfile edits; the only mode that needs CI / install-hook changes |
235235
| **vendored** — `scan --mode vendored` (or [`vendor`](#vendor)) | Patched packages committed under `.socket/vendor/`; the lockfile is rewired to consume them | **None** — the package manager installs the committed bytes | Fully airgapped and hermetic, at the cost of repo size |
236-
| **hosted** — `scan --mode hosted` | No patched bytes in your repo: the lockfile is rewritten so **only** the patched dependencies resolve to Socket-hosted, integrity-pinned packages on `patch.socket.dev`; the edits + patch records are ledgered in `.socket/vendor/redirect-state.json` (commit it — [`vex`](#vex) reads it, and it records the pre-redirect originals a future revert feature will need; hosted has no CLI revert yet, see [Undo things](#undo-things)) | Installs must be able to reach `patch.socket.dev` (no CLI, no install hook) | Smallest possible diff (lockfile + ledger); not for airgapped installs |
236+
| **hosted** — `scan --mode hosted` | No patched bytes in your repo: the lockfile is rewritten so **only** the patched dependencies resolve to Socket-hosted, integrity-pinned packages on `patch.socket.dev`; the edits + patch records are ledgered in `.socket/vendor/redirect-state.json` (commit it — [`vex`](#vex) reads it, and [`rollback`](#rollback) replays its recorded pre-redirect originals to unwind the redirect, see [Undo things](#undo-things)) | Installs must be able to reach `patch.socket.dev` (no CLI, no install hook) | Smallest possible diff (lockfile + ledger); not for airgapped installs |
237237
238238
Every mode pins the patched bytes: in agent mode the CLI verifies every file on each
239239
apply; vendored and hosted modes lean on your package manager's own lockfile integrity
@@ -356,11 +356,12 @@ and repair; pick by what you want back:
356356
357357
And `setup --remove` reverts the install hooks that `setup` added.
358358
359-
> Hosted mode has no CLI revert yet: `scan --mode hosted` makes plain lockfile /
360-
> registry-config edits, so undo them with your version control (e.g.
361-
> `git checkout -- <lockfile>`) and delete the `.socket/vendor/redirect-state.json`
362-
> ledger — once you've reverted by hand, its recorded original fragments are stale, and
363-
> a leftover ledger would still let [`vex`](#vex) attest the removed redirects.
359+
> Hosted mode is unwound by [`rollback`](#rollback), which replays the original
360+
> lockfile / registry-config fragments recorded in `.socket/vendor/redirect-state.json`
361+
> and drops the redirect records. If you revert a hosted edit by hand instead (e.g.
362+
> `git checkout -- <lockfile>`), also delete that ledger — its recorded originals are
363+
> then stale, and a leftover ledger would still let [`vex`](#vex) attest the removed
364+
> redirects.
364365
365366
## Command reference
366367

0 commit comments

Comments
 (0)