Support hosted Pipenv patches and safe vendoring (#242)

* Support hosted Pipenv patches safely

Preserve Pipenv categories and source identity when applying patches.
Handle legacy hosted references and vendored extras, reject unsupported
installers, and restore lock entries safely during rollback.

Assisted-by: Codex:gpt-6-astra

* Clarify Pipenv installer requirements

Explain the version requirement when Pipenv is unavailable, and point
local-wheel verification guidance at the existing VEX command.

Assisted-by: Codex:gpt-6-astra

Author: mikolalysenko

.github/workflows/pipenv-compatibility.yml

@@ -0,0 +1,100 @@
+name: Pipenv compatibility
+
+# Drives the real CLI and real Pipenv releases through hosted, vendored,
+# agent and out-of-tree agent mode on Linux and macOS (the `test` job already
+# runs the in-process hosted Pipenv CLI tests on all three platforms,
+# Windows included). The pre-2018 releases need Docker and stay in the local
+# matrix (`docs/testing/pipenv-compatibility.md`). Needs network (PyPI + the
+# public patch service); no Socket token.
+
+on:
+  pull_request:
+    paths:
+      - 'crates/socket-patch-core/src/patch/redirect/pipenv.rs'
+      - 'crates/socket-patch-core/src/vendor/pypi_pipenv.rs'
+      - 'crates/socket-patch-core/src/vendor/pypi.rs'
+      - 'crates/socket-patch-core/src/vendor/lock_inventory.rs'
+      - 'crates/socket-patch-core/src/crawlers/python_crawler.rs'
+      - 'crates/socket-patch-core/src/utils/pipenv.rs'
+      - 'crates/socket-patch-cli/src/commands/scan/hosted.rs'
+      - 'crates/socket-patch-cli/src/commands/vendor.rs'
+      - 'crates/socket-patch-cli/src/commands/rollback.rs'
+      - 'crates/socket-patch-cli/src/commands/vex.rs'
+      - 'crates/socket-patch-core/src/patch/redirect/replay.rs'
+      - 'scripts/backtest-pipenv.py'
+      - '.github/workflows/pipenv-compatibility.yml'
+  push:
+    branches: [main]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  matrix:
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          # Every 2018+ major, plus the shapes that exercise the relock and
+          # line-ending paths on the newest and the last Python-3.8 release.
+          - os: ubuntu-latest
+            versions: 2018.11.26 2020.11.15 2021.11.23 2022.12.19 2023.12.1 2024.4.1 2025.1.3 2026.8.0
+            shapes: direct
+          - os: ubuntu-latest
+            versions: 2022.12.19 2026.8.0
+            shapes: crlf marker-excluded extras category
+          - os: macos-latest
+            versions: 2018.11.26 2022.12.19 2023.12.1 2026.8.0
+            shapes: direct
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+      - name: Install Rust
+        run: rustup show
+      - name: Cache cargo
+        uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
+        with:
+          key: pipenv-compat
+          save-if: ${{ github.ref == 'refs/heads/main' }}
+      - name: Install uv
+        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6.8.0
+      - name: Install the Pythons the Pipenv releases run on
+        run: uv python install 3.8 3.12
+      - name: Build the CLI
+        run: |
+          cargo build --locked -p socket-patch-cli
+          mkdir -p "$RUNNER_TEMP/bin"
+          cp target/debug/socket-patch "$RUNNER_TEMP/bin/socket-patch"
+      - name: Run the Pipenv matrix
+        env:
+          SOCKET_NO_CONFIG: '1'
+          SOCKET_NO_UPDATE_CHECK: '1'
+          # The runners carry whatever patch release uv ships; the harness
+          # only needs one 3.8 and one 3.12.
+          BACKTEST_PY38: '3.8'
+          BACKTEST_PY312: '3.12'
+          PIPENV_VERSIONS: ${{ matrix.versions }}
+          PIPENV_SHAPES: ${{ matrix.shapes }}
+        run: |
+          # shellcheck disable=SC2086
+          python3 scripts/backtest-pipenv.py \
+            --socket-patch "$RUNNER_TEMP/bin/socket-patch" \
+            --socket-patch-revision "$GITHUB_SHA" \
+            --output "$RUNNER_TEMP/pipenv-compat" \
+            --versions $PIPENV_VERSIONS \
+            --shapes $PIPENV_SHAPES \
+            --modes hosted vendored agent agent-oot \
+            --jobs 4
+      - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        if: always()
+        with:
+          name: pipenv-compat-${{ matrix.os }}-${{ strategy.job-index }}
+          path: |
+            ${{ runner.temp }}/pipenv-compat/summary.json
+            ${{ runner.temp }}/pipenv-compat/summary.md
+          if-no-files-found: warn
+          retention-days: 7

CHANGELOG.md

@@ -51,6 +51,43 @@ into the new version's section — see docs/releasing.md.
 
 ### Added
 
+- **Pipenv projects can use hosted patches, and vendored patches keep every
+  category.** `scan --mode hosted` rewrites every `Pipfile.lock` category
+  (`default`, `develop`, Pipenv 2022+ named categories) that pins the patched
+  release to the hosted wheel — `file` references for Pipenv 2018 and later,
+  `path` for 7–11 (probed once with `pipenv --version`; `SOCKET_PIPENV_MAJOR`
+  pins it), pipfile-spec < 6 refused — preserving markers, extras, unrelated
+  entries, the Pipfile and its content hash, with per-entry rollback
+  (`redirect_pipenv_entry`). Vendored mode keeps custom categories and extras,
+  uses `path` for wheels with extras (Pipenv 2022's file-URL bug) and refuses
+  installers older than 2018 (`pypi_pipenv_installer_unsupported`). A stale
+  Pipfile.lock only vetoes the sibling Python rewriters on a real pin/source
+  conflict (`redirect_pipenv_refused`); anything else is
+  `redirect_pipenv_skipped`. Measured across the last stable release of all
+  18 published Pipenv majors — see `docs/testing/pipenv-compatibility.md` and
+  `scripts/backtest-pipenv.py`.
+- **`Pipfile.lock` is inventoried.** Lock-only Pipenv checkouts (a fresh
+  clone with nothing installed) now discover their pins in every mode —
+  hosted redirects them, vendored fetches the pristine wheel by one of the
+  lock's recorded digests (`LockIntegrity::Sha256AnyOf`, resolved through
+  PyPI's JSON API and verified against the same digest) and agent/scan list
+  them as lockfile-only packages. Previously they discovered nothing and
+  exited 0. Socket's own references stay discoverable, so a re-scan of an
+  already-redirected or already-vendored lock-only checkout re-confirms it;
+  a lock that resolves only from private indexes is never looked up on
+  pypi.org.
+- **Pipenv's out-of-tree virtualenv is discovered.** Agent mode (bare `scan`,
+  `rollback`, `vex`) now finds `$WORKON_HOME/<dir>-<hash>[-<python>]` (the
+  `.venv` file pointer, `PIPENV_CUSTOM_VENV_NAME` and `PIPENV_PIPFILE`
+  included) exactly as Pipenv 7 through 2026 place it, instead of falling
+  through to the global interpreter's site-packages.
+- **Pipenv stale-install guard.** Pipenv never reinstalls a release that is
+  already present, so a hosted or vendored rewrite over a warm venv leaves the
+  upstream bytes installed; `redirect_pypi_stale_install` /
+  `pypi_pipenv_stale_install` now say so, naming the site-packages dir and
+  the verified remedy (`pipenv run pip uninstall -y <pkg> && pipenv sync`, or
+  a clean `pipenv --rm && pipenv sync`), and the stale purl is excluded from
+  the same-run `--vex`.
 - **Poetry projects take hosted patches, and vendored patches now cover every
   `poetry.lock` generation.** `scan --mode hosted` rewrites `poetry.lock` to a
   `[package.source] type = "url"` pointing at the Socket-hosted, SHA-256-pinned
@@ -200,6 +237,23 @@ into the new version's section — see docs/releasing.md.
 
 ### Fixed
 
+- **Rollback after a Pipenv relock no longer refuses forever.** `pipenv lock`
+  (and `update`, and `install <other>` before 2024) regenerates a redirected
+  or vendored entry to registry shape on every Pipenv major; that is now the
+  desired end state — the hosted edit retires and the vendored record is
+  dropped (`vendor_lock_entry_relocked`) — instead of a permanent drift
+  refusal that held every pypi revert and kept the orphaned wheel dir. A
+  foreign `file`/`path` reference is still drift.
+- **Same-run `--vex` attests lock-only pypi redirects.** The confirmed purl is
+  unqualified while the ledger records the API's artifact-qualified purl;
+  both sides now match on the qualifier-stripped purl, so a lock-only Pipenv
+  (or uv) checkout no longer exits 1 `no_applicable_patches` after
+  redirecting its lock.
+- **The Pipenv installer probe runs only when a patch targets the lock**, warns
+  only when the lock was actually rewritten, resolves `pipenv` on absolute
+  `PATH` entries only (a relative entry would have executed a `pipenv` planted
+  in the scanned repository), finds `.bat`/`.cmd` shims on Windows, and takes
+  only the token after `version` (never a stray `Python 3.12` banner).
 - Hosted Python redirects now warn when installed files still contain upstream
   or modified bytes and omit those packages from same-run VEX. The read-only
   probe covers Poetry virtualenvs, repeats on re-scans, and uses persisted patch

README.md

@@ -254,6 +254,51 @@ Mode support varies by ecosystem — e.g. Go can't do hosted, Rush monorepos can
 vendored. See the full **[mode × ecosystem matrix](docs/ecosystems.md#mode--ecosystem-matrix)**
 for details and per-ecosystem caveats.
 
+### Pipenv compatibility
+
+Hosted mode rewrites every `Pipfile.lock` category that pins the patched
+release (`default`, `develop`, and Pipenv 2022+ named categories) and
+preserves the Pipfile, its content hash, markers, extras, and unrelated lock
+entries, so `pipenv install --deploy`, `pipenv sync` and `pipenv verify` keep
+passing. The reference shape follows the installing Pipenv: releases 7–11
+need `path` references, 2018 and later use `file` references, and lock
+formats before `pipfile-spec: 6` (Pipenv 0–6) are refused without changing
+the lock. Socket Patch probes `pipenv --version` once per run (only when a
+patch targets the lock); `SOCKET_PIPENV_MAJOR=<major>` pins the answer for
+machines without pipenv on PATH. Hosted references carry both the `#sha256=`
+URL fragment (verified by Pipenv 2023+) and a `hashes` entry (verified by
+2018–2022; Pipenv 11 verifies either), so a tampered lock fails to install
+on every supported release.
+
+Vendored mode requires Pipenv 2018 or later. Wheels with extras use `path`
+references to avoid Pipenv 2022's local-file URL parsing bug. Pipenv 2023+
+does not enforce hashes on local wheels; commit the wheel and run
+`socket-patch vex --product <purl>` (a Pipfile names no project, so pass the
+product purl explicitly).
+
+Fresh checkouts work in every mode: a clone with only `Pipfile` +
+`Pipfile.lock` is discovered from the lock (hosted redirects it, vendored
+fetches the pristine wheel by one of the lock's recorded digests), and agent
+mode finds Pipenv's default out-of-tree virtualenv under `WORKON_HOME`
+without `pipenv run`.
+
+Pipenv never reinstalls a release that is already present: `pipenv install`,
+`pipenv install --deploy` and `pipenv sync` all exit 0 and keep the installed
+bytes, on every Pipenv major. A hosted or vendored rewrite therefore
+protects fresh installs, and Socket Patch warns
+(`redirect_pypi_stale_install` / `pypi_pipenv_stale_install`) when a venv
+still holds the upstream release, with the verified remedy:
+`pipenv run pip uninstall -y <pkg> && pipenv sync` (or `pipenv --rm &&
+pipenv sync`). Do not use `pipenv uninstall <pkg>` for this — it rewrites the
+Pipfile and re-locks the patch away. `pipenv lock` / `pipenv update`
+regenerate the entry to its registry reference (a silent unpatch): re-run
+Socket Patch afterwards; `rollback` retires the stale record cleanly.
+
+`scripts/backtest-pipenv.py` drives the real CLI and the last stable release
+of every published Pipenv major through hosted, vendored, agent and
+out-of-tree agent mode, and `docs/testing/pipenv-compatibility.md` holds the
+measured boundaries and results.
+
 ## Common tasks
 
 ### Patch everything that can be patched