fix(node): gate GET /ipfs/{cid} tree objects so a withheld subtree's structure can't leak (#135)

.env.example

@@ -24,8 +24,11 @@ DATABASE_URL=postgresql://gitlawb:changeme@localhost:5432/gitlawb
 # ── Database pool & startup resilience ────────────────────────────────────
 # Maximum connections in the PostgreSQL pool. A cap, not a floor —
 # connections open lazily. Size against the DB server's max_connections,
-# remembering admin tooling opens its own pool.
-GITLAWB_DB_MAX_CONNECTIONS=20
+# remembering admin tooling opens its own pool. Each concurrent write pins one
+# connection for its whole duration (the connection-affine advisory lock), so the
+# node REJECTS at boot any value below GITLAWB_MAX_CONCURRENT_GIT_PUSHES + 8
+# headroom — keep this comfortably above that (default 48 for pushes 32).
+GITLAWB_DB_MAX_CONNECTIONS=48
 # Seconds a request waits for a pool connection before failing with 503.
 GITLAWB_DB_ACQUIRE_TIMEOUT_SECS=5
 # Upper bound on each startup connect+migrate attempt, in seconds. Keep it
@@ -109,15 +112,136 @@ GITLAWB_MAX_PACK_BYTES=2147483648
 
 # Max seconds a served git upload-pack / receive-pack (clone / push) may run
 # before it is aborted with a 504. Bounds a hung git that would otherwise pin a
-# worker and, on push, the repo write lock. Does NOT cover the info/refs
-# advertisement or the withheld-blob path, which remain unbounded. Default 600.
+# worker and, on push, the repo write lock. Also bounds both info/refs
+# advertisements, the withheld-blob pack build, and the push-side candidate
+# discovery (rev-list / cat-file), all reaped via process-group teardown (#174).
+# On the path-scoped upload-pack path the withheld-blob classification walk and
+# the pack serve share ONE deadline, so this value bounds their COMBINED duration
+# rather than giving each stage a full budget: a walk that consumes it leaves the
+# serve nothing and the clone gets a 504. Serving large path-scoped repos may
+# need a higher value here than when each stage was budgeted separately.
+# Must be 1..=3153600000 (100 years): the node derives deadlines from this value,
+# and a larger one cannot be represented. Default 600.
 GITLAWB_GIT_SERVICE_TIMEOUT_SECS=600
 
+# Max seconds the storage-ACQUISITION phase of a served git op may run before the
+# request is shed with a 503, separate from the git-run timeout above. A
+# concurrency permit is taken before this phase and GITLAWB_GIT_SERVICE_TIMEOUT_SECS
+# only starts once git spawns, so without this a stalled backend (a hung Tigris
+# HEAD/GET, or a hung pg advisory-lock iteration on push) pins the permit and drains
+# the pool until every later request 503s. On expiry the permit is released
+# (fail-closed). Kept separate because acquisition and git execution are distinct
+# cost centers. Must be positive; set very large to effectively disable. Default 30.
+GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS=30
+
+# Max concurrent git READ ops (upload-pack + the upload-pack info/refs
+# advertisement) served at once, a global pool separate from the push pool below.
+# The anon receive-pack info/refs advertisement has its OWN pool (see below), not
+# this one. Over-cap sheds a clean 503 + Retry-After. Anonymous reads draw from
+# here, so pair it with GITLAWB_MAX_CONCURRENT_READS_PER_CALLER (below) so one
+# caller cannot monopolize the pool. Default 128.
+GITLAWB_MAX_CONCURRENT_GIT_OPS=128
+
+# Max concurrent git-receive-pack (push) POST operations, in a pool separate from
+# the read pool (GITLAWB_MAX_CONCURRENT_GIT_OPS) so anonymous reads cannot shed an
+# authenticated push at admission. The anon receive-pack info/refs advertisement
+# runs in a SEPARATE pool of the same size (disjoint from this one), so an
+# advertisement flood cannot shed a push either. Over-cap sheds a 503 +
+# Retry-After. Default 32.
+GITLAWB_MAX_CONCURRENT_GIT_PUSHES=32
+
+# Max pushes PARKED at once waiting for the per-repo write lease of the SAME repo.
+# A parked push holds its already-buffered pack body in memory, so this bounds that
+# memory for a hot repo. Past the cap the newest push sheds a clean 503 +
+# Retry-After ("repo is busy with another push") instead of joining the queue; a
+# push to a DIFFERENT repo is never affected. The lease holder is not counted.
+# Raising GITLAWB_MAX_CONCURRENT_GIT_PUSHES does not raise this; set it explicitly.
+# Default 8.
+GITLAWB_REPO_LEASE_MAX_WAITERS=8
+
+# Max concurrent post-push pin loops (IPFS + Pinata pin_new_objects) across all
+# repos. Each loop holds a full per-push object-id list while pinning, so this
+# bounds that MB-scale memory even though the per-repo encrypt-task set already
+# caps the task COUNT. A loop DEFERS (waits) when the pool is full, never drops a
+# pin. Default 8.
+GITLAWB_MAX_CONCURRENT_PIN_TASKS=8
+
+# Max concurrent read ops (upload-pack + the upload-pack info/refs advertisement)
+# a single caller may hold, so one caller cannot monopolize the read pool. Keyed
+# on the resolved SOURCE IP, never the DID: a signature does not move a caller off
+# this cap. The source-IP key is only as granular as GITLAWB_TRUSTED_PROXY below:
+# left unset, a node behind an edge/NAT keys all callers on the edge IP and this
+# collapses to one global cap. Set GITLAWB_TRUSTED_PROXY for per-client keying; a
+# high-fanout caller (CI behind one NAT) then needs the operator to raise this.
+# Default 16.
+GITLAWB_MAX_CONCURRENT_READS_PER_CALLER=16
+
+# Two further per-source concurrency caps exist on the PUSH side but have NO
+# environment variable of their own: both are DERIVED as
+# GITLAWB_MAX_CONCURRENT_GIT_PUSHES / 8, with a floor of 1. They are not settable
+# independently, so raising the push pool raises both proportionally, and the
+# default push pool of 32 gives each a cap of 4.
+#   * The anon receive-pack info/refs advertisement cap: one source IP may hold at
+#     most this many slots in the dedicated advert pool, so saturating that pool
+#     takes ~8 distinct source IPs.
+#   * The authenticated receive-pack POST cap: one source IP may hold at most this
+#     many WRITE-pool slots. This one is load-bearing for push availability, since
+#     it is acquired before the global write permit: without it, one host minting
+#     disposable did:key identities could open enough slow pushes to monopolize the
+#     write pool and 503 every other source (owner enforcement defaults off, and
+#     the push rate limiter caps arrival rate, not in-flight concurrency).
+# Keyed on the resolved source IP, never the DID, so a DID farm does not defeat
+# them; keying granularity follows GITLAWB_TRUSTED_PROXY like the read cap above.
+
 # ── Push rate limiting (git-receive-pack flood brake) ─────────────────────
 # Max receive-pack requests (info/refs advertisement + push POST) per client
 # IP per hour. 0 disables. Default 600.
 GITLAWB_PUSH_RATE_LIMIT=600
 
+# ── /ipfs/{cid} visibility-walk admission (#174) ──────────────────────────
+# GET /ipfs/{cid} runs a per-repo full-history git walk in a blocking thread to
+# decide whether the caller may read a path-scoped blob. It is publicly reachable,
+# so it is bounded to keep a permissionless caller from fanning out unbounded
+# concurrent walks and exhausting blocking-pool threads + PIDs.
+# Max concurrent /ipfs walks across all callers (a pool of its own, disjoint from
+# the served-git pools). Over-cap sheds a 503. Default 32.
+GITLAWB_MAX_CONCURRENT_IPFS_WALKS=32
+# Max concurrent /ipfs walks a single SOURCE IP may hold (keyed like the git
+# per-caller caps via GITLAWB_TRUSTED_PROXY; reject-before-insert bounded map).
+# Default 4.
+GITLAWB_IPFS_WALK_PER_SOURCE=4
+# Max legacy (NULL-provenance) repos probed per single /ipfs request, bounding the
+# scan-fallback fan-out (git cat-file per candidate repo) for an anonymous caller. A
+# truncated scan sheds a retryable 503, never a false 404. Default 256.
+GITLAWB_IPFS_MAX_LEGACY_PROBES=256
+# Max EXPENSIVE path-scope visibility walks per single /ipfs request (only a
+# blob in a path-scoped repo costs a full-history walk). Over-cap repos are
+# skipped without a verdict and the scan continues; if the object is then found
+# nowhere the request sheds a retryable 503 instead of a false 404. Default 64.
+GITLAWB_IPFS_MAX_REPOS_WALKED=64
+# Ceiling on repos one /ipfs request may VISIT past the visibility gate. Each
+# visit costs a repo acquire — on a Tigris cache miss a full archive download,
+# so this is also the worst-case object-store fetch count per request — plus a
+# cat-file probe. On exhaustion the scan stops and sheds a retryable 503.
+# Default 1024.
+GITLAWB_IPFS_MAX_REPO_VISITS=1024
+# Absolute wall-clock budget for one admitted /ipfs request's acquire+walk
+# lifetime (all stages of the whole scan). Per-stage clamps bound the acquire
+# and walk stages to the remaining budget, and no stage starts once it is
+# exhausted (the scan then sheds a retryable 503). The object-type probe and
+# content-read cat-file subprocesses are budget-checked before starting AND each
+# run under their own deadline (the lesser of GITLAWB_GIT_SERVICE_TIMEOUT_SECS and
+# the remaining budget), reaped via process-group teardown, so a hung cat-file
+# cannot hold the request's walk slot past it. One hang path is still unbounded:
+# the probe's object-store readability check is a plain filesystem sweep with
+# nothing to reap, so a wedged filesystem can hold the slot past the deadline.
+# Must be 1..=3153600000 (100 years): the node derives an Instant deadline from
+# this value, and a larger one cannot be represented. Default 600.
+GITLAWB_IPFS_REQUEST_BUDGET_SECS=600
+# Max /ipfs/{cid} requests per client IP per hour (route flood brake, distinct
+# from the concurrency caps above). 0 disables. Default 600.
+GITLAWB_IPFS_RATE_LIMIT=600
+
 # ── Creation rate limiting (repo/agent/issue/PR flood brake) ──────────────
 # Max creation requests (POST /api/v1/repos, /api/register, fork, issues,
 # pulls) per client IP per hour, in addition to the per-DID limit. The per-DID

README.md

@@ -4,7 +4,7 @@
 
 Gitlawb Node is the open-source node software behind the Gitlawb network. It lets anyone run a self-hosted node, publish repositories under a DID, sign writes with Ed25519 HTTP signatures, replicate git activity across peers, and move toward a resilient app-delivery network where code and build assets can be served closer to users.
 
-Gitlawb is not trying to be only “another git host.” The long-term direction is:
+Gitlawb is not trying to be only "another git host." The long-term direction is:
 
 ```txt
 Decentralized GitHub
@@ -34,12 +34,12 @@ This is a Rust workspace with four crates:
 
 Most git hosting today depends on a small number of centralized platforms. Gitlawb Nodes are designed for a different model:
 
-- **Own your identity** — every user, agent, and node is an Ed25519 keypair represented as `did:key:z6Mk...`.
-- **Signed writes by default** — write requests use RFC 9421 HTTP Signatures instead of passwords.
-- **Git-native transport** — repositories are still real git repositories served over smart HTTP.
-- **Agent-native workflows** — the `gl` CLI and MCP server expose repo, issue, task, PR, and UCAN flows to AI agents.
-- **Peer-aware delivery** — nodes can announce, discover, gossip, and sync with each other.
-- **App CDN direction** — the network can evolve from decentralized code storage into code + asset + app delivery.
+- **Own your identity**: every user, agent, and node is an Ed25519 keypair represented as `did:key:z6Mk...`.
+- **Signed writes by default**: write requests use RFC 9421 HTTP Signatures instead of passwords.
+- **Git-native transport**: repositories are still real git repositories served over smart HTTP.
+- **Agent-native workflows**: the `gl` CLI and MCP server expose repo, issue, task, PR, and UCAN flows to AI agents.
+- **Peer-aware delivery**: nodes can announce, discover, gossip, and sync with each other.
+- **App CDN direction**: the network can evolve from decentralized code storage into code + asset + app delivery.
 
 ---
 
@@ -189,14 +189,14 @@ For public-network use, make sure `GITLAWB_NODE` points to the node you want. Th
 
 Public nodes (e.g. `node.gitlawb.com`) require two things on writes:
 
-1. **RFC 9421 HTTP Signatures** — every write is signed by your identity key. `gl`
+1. **RFC 9421 HTTP Signatures**: every write is signed by your identity key. `gl`
    and the `git-remote-gitlawb` helper do this automatically. An old/unsigned CLI
    fails with `401 not_an_agent`; `gl` will tell you to upgrade and register.
 2. **An iCaptcha proof** on the spam-gated writes (**repo create, fork, register**).
    `gl` solves this for you: on the node's `403 icaptcha_proof_required` it reads the
    `x-icaptcha-url` / `x-icaptcha-level` hints, requests a challenge, solves it
    locally (arithmetic / algebra / sequence), and **retries the same signed request**
-   with the `x-icaptcha-proof` header — no manual steps, no env vars.
+   with the `x-icaptcha-proof` header. No manual steps, no env vars.
 
 ```bash
 gl identity new                                   # create did:key identity
@@ -215,14 +215,14 @@ Notes:
 - **Proofs are short-lived (~5 min TTL) and single-use.** If one expires between
   solving and use, the client transparently solves a fresh one and retries.
 - **What needs what:** create / fork / register are signed **and** iCaptcha-gated;
-  `git push` is **signed-only** (owner signature is the gate — no per-push challenge);
+  `git push` is **signed-only** (owner signature is the gate, no per-push challenge);
   reads (clone / fetch / `repo info`) need no proof. A non-existent repo returns a
   clear `404`, never a placeholder.
 - **API-key iCaptcha deployments:** set `GITLAWB_ICAPTCHA_URL` to your iCaptcha
   origin and `GITLAWB_ICAPTCHA_API_KEY` to its key. The client only talks to an
   `https` origin whose host is allowlisted (that URL or the public default), and
-  sends the bearer token **only** to your configured origin — never to a URL a
-  node advertises — so a hostile node can't capture the key or redirect the solve.
+  sends the bearer token **only** to your configured origin, never to a URL a
+  node advertises, so a hostile node can't capture the key or redirect the solve.
 
 ---
 
@@ -343,13 +343,24 @@ Important node settings:
 | `GITLAWB_REQUIRE_SIGNED_PEER_WRITES` | Require signed peer announce/sync writes. |
 | `GITLAWB_AUTO_SYNC` | Enable automatic sync from known peers. |
 | `GITLAWB_MAX_PACK_BYTES` | Max git pack body size for smart-HTTP routes. |
-| `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack/receive-pack may run before it is aborted (504). Default 600. Does not bound `info/refs` or the withheld-blob path. |
+| `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack, receive-pack, or `info/refs` advertisement may run before it is aborted (504). Default 600. Also bounds the withheld-blob classification walk (on both the upload-pack serve and receive-pack replication paths) and the push-side pin-candidate discovery (`rev-list` / `cat-file`), each reaped via process-group teardown at the deadline. On the path-scoped upload-pack path the classification walk and the pack serve share ONE deadline, so this value bounds their combined duration rather than granting each stage a full budget: a walk that consumes it leaves the serve nothing and the clone gets a 504. Serving large path-scoped repos may therefore need a higher value than they did when each stage was budgeted separately. Accepted range is 1 to 3153600000 (100 years), since the node derives deadlines from this value and a larger one cannot be represented. |
+| `GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS` | Max seconds the storage-acquisition phase (Tigris HEAD/GET, push advisory-lock) of a served git op may run before the request is shed with a 503, separate from the git-run timeout. The concurrency permit is released on expiry so a stalled backend cannot pin the pool. Default 30. |
+| `GITLAWB_REPO_LEASE_MAX_WAITERS` | Max pushes parked at once waiting for the same repo's write lease. Each waiter pins its buffered pack body, so this bounds that memory for a hot repo; past the cap the newest push sheds a 503 + Retry-After instead of queueing. Pushes to other repos are unaffected, and the lease holder is not counted. Default 8. |
+| `GITLAWB_MAX_CONCURRENT_IPFS_WALKS` | Max concurrent `GET /ipfs/{cid}` visibility walks across all callers (own pool, disjoint from the served-git pools); over-cap sheds 503. Default 32. |
+| `GITLAWB_IPFS_WALK_PER_SOURCE` | Max concurrent `/ipfs` walks a single source IP may hold. Default 4. |
+| `GITLAWB_IPFS_MAX_LEGACY_PROBES` | Max legacy (NULL-provenance) repos probed per `/ipfs/{cid}` request, bounding the scan-fallback fan-out. A truncated scan returns a retryable 503, not a false 404. Default 256. |
+| `GITLAWB_IPFS_MAX_REPOS_WALKED` | Max expensive path-scope visibility walks per `/ipfs/{cid}` request; over-cap repos are skipped and the scan continues, shedding a retryable 503 (not a false 404) if the object is then found nowhere. Raised to `MAX_PIN_SOURCES + 1` if set below it, so a provenanced request is never truncated before its full source set is tried. Default 64. |
+| `GITLAWB_IPFS_MAX_REPO_VISITS` | Ceiling on repos one `/ipfs/{cid}` request may visit (acquire + probe) past the visibility gate. Also the worst-case per-request Tigris fetch count. On exhaustion the scan stops with a retryable 503. Default 1024. |
+| `GITLAWB_IPFS_REQUEST_BUDGET_SECS` | Absolute wall-clock budget for one admitted `/ipfs/{cid}` request's acquire+walk lifetime. Per-stage clamps bound the acquire and walk stages to the remaining budget, and no stage starts once it is exhausted; the scan then stops with a retryable 503. The object-type probe and content-read `cat-file` subprocesses are budget-checked before starting and each also run under their own deadline (the lesser of `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` and the remaining budget), reaped via process-group teardown, so a hung `cat-file` cannot hold the request's walk slot past it. One hang path is still unbounded: the probe's object-store readability check is a plain filesystem sweep with nothing to reap, so a wedged filesystem can hold the slot past the deadline. Default 600. Accepted range is 1 to 3153600000 (100 years), since the node derives a deadline from this value and a larger one cannot be represented. |
+| `GITLAWB_IPFS_RATE_LIMIT` | Max `/ipfs/{cid}` requests per client IP per hour (route flood brake). 0 disables. Default 600. |
 | `GITLAWB_TIGRIS_BUCKET` | Optional S3/Tigris shared repo storage bucket. |
 | `GITLAWB_PINATA_JWT` | Optional Pinata/IPFS warm-storage pinning. |
 | `GITLAWB_IRYS_URL` | Optional Irys/Arweave permanent anchoring. |
 
 Production note: change the default Postgres password before exposing a node publicly.
 
+Legacy-pin window: releases before the CID-resolver work stored the provider CID (Kubo dag-pb / Pinata) as a pinned object's resolver key. The `/ipfs/{cid}` resolver now recomputes the raw-content CID from the object bytes and refuses to serve a key that does not match, so `GET /api/v1/ipfs/pins` can still advertise an unrepaired legacy CID that 404s. Such a row is repaired opportunistically the next time a push carries the object again (its key is rewritten to the raw CID, the old value kept in `legacy_provider_cid`), but git negotiation omits objects the node already has, so most legacy rows never re-enter a push delta. A deferred one-shot startup sweep, not this opportunistic path, is what fully retires the advertise-then-404 window. Rows whose object bytes are gone stay withheld.
+
 ---
 
 ## Optional node staking