bazel l1 [fork 3/5]: client-side forwarding + marker-based degradation - #29
Conversation
b61bf50 to
702cd1c
Compare
d066700 to
1a46c90
Compare
|
The idempotency guard in withOutgoingStoragePrefix/withOutgoingS3Backend checks key presence in the outgoing metadata, not equality — if an outer layer ever attached a different prefix or endpoint, the inner call would silently defer to it rather than flag the conflict. Today the only re-entry is Contains(RAW/AC) delegating to Get on the identical context, so this can't fire, but the presence check encodes "someone already set this, trust them" where the intent is "don't attach twice". Worth either asserting equality (log + prefer the request-scoped value) or a comment noting the assumption that all layering paths share one context. |
|
isTransportFailure counts codes.Canceled into transport_miss_total, but a Canceled here is usually the caller's context dying (Bazel gave up, request torn down) — not L1 trouble. Mixing it in means the transport-miss series inflates with client-induced cancellations, which muddies any "L1 unhealthy" alert on it. Two options: distinguish caller-cancel from backend failure by checking ctx.Err() == context.Canceled before classifying, and either skip the meter entirely (degrading to a miss is still right, the caller is gone anyway) or count it under a separate label/reason (e.g. transport_miss_total{reason="client_canceled"}), leaving the alertable series clean. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 1a46c90. Configure here.
|
@piob-io thanks for the review
|
piob-io
left a comment
There was a problem hiding this comment.
FA's forwarded-CAS→created metrics mapping keys on successful forwards carrying an empty reason (if key.Reason == "" before tagging l1_forwarded). Both forwarded emit sites here do pass "", but nothing pins that: TestObserveUploadCarriesEntryKind asserts Method/Kind/Status/Bytes and never Reason. Adding if outcome.Reason != "" { t.Errorf(...) } to that test — with a comment naming the fa consumer (build_cache_metrics.go's forwarded→created mapping) — turns the cross-repo assumption into a tripwire on the side that can actually break it.
…etered misses The upstream client now forwards the request-scoped storage prefix and (endpoint, bucket) selection as outgoing gRPC metadata on reads and asynchronous uploads. Read-path fail-open contract: transport failures, backend auth rejections, and rejections carrying the trust-interceptor ErrorInfo marker degrade to cache misses on distinct alertable counters (unary and streaming paths, including an eager first-Recv on streams); unmarked errors keep failing strictly. Also: StreamReadCloser.Read no longer returns n=-1 on mid-stream errors, upload timeouts and queue metrics, an operation-observer seam that reports uploads as "forwarded" with the entry Kind carried on OperationOutcome, StopUploaders for embedders, and CheckCapabilities now accepts a caller context. Co-authored-by: Cursor <cursoragent@cursor.com>
Review (fork 3/5, Piotr + bugbot), four fixes to the same seam: - Caller-canceled requests (ctx dead + Canceled) still degrade to misses but meter on client_canceled_miss_total, keeping transport_miss_total a clean L1-health signal. - isTransportFailure no longer blanket-classifies non-status errors; only context/net errors qualify, and fetchBlobDigest preserves the application status code (ResourceExhausted fails strictly again). - Outgoing-metadata idempotency guards compare values, not key presence: a conflicting outer value is logged and overridden by the request-scoped one instead of silently deferred to. - The unknown-size CAS Contains path classifies backend errors on the same meters as every other read path. Co-authored-by: Cursor <cursoragent@cursor.com>
Piotr's fork 3/5 approval note: fa's forwarded-CAS-to-created metrics mapping lives downstream of the convention that successful forwards carry no reason. fa now overwrites the reason unconditionally with l1_forwarded, so a fork-attached reason would be silently swallowed rather than break attribution — this assertion makes anyone who adds one reconcile with that mapping deliberately instead of by surprise. Co-authored-by: Cursor <cursoragent@cursor.com>
38b0818 to
4f5ebb6
Compare
f389172 to
0b2a6aa
Compare

Piece 3 of 5 peeled off
shreyas/bazel-l1-goal-state(design record: FastActions/fa#4779). Base: piece 2 (#28). One commit (840d6fb).Summary
grpcproxy(what the embedded FA host runs): forwards the tenant's storage prefix, auth secret, and pinned S3(endpoint, bucket)selector as outgoing gRPC metadata on the synchronous path, and captures them per queued item so asynchronous write-throughs carry the same identity after the request context is gone.cache.blacksmith.shErrorInfo marker (piece 2'sSTORAGE_PREFIX_REJECTED/ piece 4'sS3_BACKEND_SELECTOR_REJECTED) degrades to a metered local miss — never a build failure; unmarked errors keep their existing semantics.Dark until an FA host is configured with L1 mode + the secret; merging is harmless.
Evidence
Staging E2E 2026-07-29→30: negative-trust and unknown-selector runs degraded client-side to metered misses (exit 0) while the node counted and journaled the rejections.
Made with Cursor
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled. (Staging)Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.