Skip to content

bazel l1 [fork 3/5]: client-side forwarding + marker-based degradation - #29

Open
shreyas-blacksmith wants to merge 3 commits into
shreyas/bazel-l1-piece-2-trustfrom
shreyas/bazel-l1-piece-3-forwarding
Open

bazel l1 [fork 3/5]: client-side forwarding + marker-based degradation#29
shreyas-blacksmith wants to merge 3 commits into
shreyas/bazel-l1-piece-2-trustfrom
shreyas/bazel-l1-piece-3-forwarding

Conversation

@shreyas-blacksmith

@shreyas-blacksmith shreyas-blacksmith commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Piece 3 of 5 peeled off shreyas/bazel-l1-goal-state (design record: FastActions/fa#4779). Base: piece 2 (#28). One commit (840d6fb).

Summary

  • The client half of the trust + routing contracts, in grpcproxy (what the embedded FA host runs): forwards the tenant's storage prefix, auth secret, and pinned S3 (endpoint, bucket) selector as outgoing gRPC metadata on the synchronous path, and captures them per queued item so asynchronous write-throughs carry the same identity after the request context is gone.
  • Typed rejection handling: an L1 response carrying the cache.blacksmith.sh ErrorInfo marker (piece 2's STORAGE_PREFIX_REJECTED / piece 4's S3_BACKEND_SELECTOR_REJECTED) degrades to a metered local miss — never a build failure; unmarked errors keep their existing semantics.

Dark until an FA host is configured with L1 mode + the secret; merging is harmless.

Evidence

Staging E2E 2026-07-29→30: negative-trust and unknown-selector runs degraded client-side to metered misses (exit 0) while the node counted and journaled the rejections.

Made with Cursor


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled. (Staging)


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

@shreyas-blacksmith
shreyas-blacksmith force-pushed the shreyas/bazel-l1-piece-2-trust branch from b61bf50 to 702cd1c Compare July 30, 2026 21:22
@shreyas-blacksmith
shreyas-blacksmith force-pushed the shreyas/bazel-l1-piece-3-forwarding branch 2 times, most recently from d066700 to 1a46c90 Compare July 31, 2026 13:55
@piob-io

piob-io commented Jul 31, 2026

Copy link
Copy Markdown

The idempotency guard in withOutgoingStoragePrefix/withOutgoingS3Backend checks key presence in the outgoing metadata, not equality — if an outer layer ever attached a different prefix or endpoint, the inner call would silently defer to it rather than flag the conflict. Today the only re-entry is Contains(RAW/AC) delegating to Get on the identical context, so this can't fire, but the presence check encodes "someone already set this, trust them" where the intent is "don't attach twice". Worth either asserting equality (log + prefer the request-scoped value) or a comment noting the assumption that all layering paths share one context.

@piob-io

piob-io commented Jul 31, 2026

Copy link
Copy Markdown

isTransportFailure counts codes.Canceled into transport_miss_total, but a Canceled here is usually the caller's context dying (Bazel gave up, request torn down) — not L1 trouble. Mixing it in means the transport-miss series inflates with client-induced cancellations, which muddies any "L1 unhealthy" alert on it. Two options: distinguish caller-cancel from backend failure by checking ctx.Err() == context.Canceled before classifying, and either skip the meter entirely (degrading to a miss is still right, the caller is gone anyway) or count it under a separate label/reason (e.g. transport_miss_total{reason="client_canceled"}), leaving the alertable series clean.

@shreyas-blacksmith
shreyas-blacksmith marked this pull request as ready for review July 31, 2026 14:54

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 1a46c90. Configure here.

Comment thread cache/grpcproxy/grpcproxy.go
Comment thread cache/grpcproxy/grpcproxy.go Outdated
Comment thread cache/grpcproxy/grpcproxy.go
@shreyas-blacksmith

Copy link
Copy Markdown
Contributor Author

@piob-io thanks for the review

  • idempotency guard: agreed, now idempotent by value. Same value no-ops, a conflict is logged and the request-scoped value wins.
  • Canceled: caller-dead cancellations still degrade to misses but meter on a separate client_canceled_miss_total, so transport_miss_total stays a clean L1-health signal.

@piob-io piob-io left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FA's forwarded-CAS→created metrics mapping keys on successful forwards carrying an empty reason (if key.Reason == "" before tagging l1_forwarded). Both forwarded emit sites here do pass "", but nothing pins that: TestObserveUploadCarriesEntryKind asserts Method/Kind/Status/Bytes and never Reason. Adding if outcome.Reason != "" { t.Errorf(...) } to that test — with a comment naming the fa consumer (build_cache_metrics.go's forwarded→created mapping) — turns the cross-repo assumption into a tripwire on the side that can actually break it.

shreyas-blacksmith and others added 3 commits July 31, 2026 19:58
…etered misses

The upstream client now forwards the request-scoped storage prefix and
(endpoint, bucket) selection as outgoing gRPC metadata on reads and
asynchronous uploads. Read-path fail-open contract: transport failures,
backend auth rejections, and rejections carrying the trust-interceptor
ErrorInfo marker degrade to cache misses on distinct alertable counters
(unary and streaming paths, including an eager first-Recv on streams);
unmarked errors keep failing strictly.

Also: StreamReadCloser.Read no longer returns n=-1 on mid-stream errors,
upload timeouts and queue metrics, an operation-observer seam that
reports uploads as "forwarded" with the entry Kind carried on
OperationOutcome, StopUploaders for embedders, and CheckCapabilities
now accepts a caller context.

Co-authored-by: Cursor <cursoragent@cursor.com>
Review (fork 3/5, Piotr + bugbot), four fixes to the same seam:
- Caller-canceled requests (ctx dead + Canceled) still degrade to misses
  but meter on client_canceled_miss_total, keeping transport_miss_total a
  clean L1-health signal.
- isTransportFailure no longer blanket-classifies non-status errors;
  only context/net errors qualify, and fetchBlobDigest preserves the
  application status code (ResourceExhausted fails strictly again).
- Outgoing-metadata idempotency guards compare values, not key presence:
  a conflicting outer value is logged and overridden by the
  request-scoped one instead of silently deferred to.
- The unknown-size CAS Contains path classifies backend errors on the
  same meters as every other read path.

Co-authored-by: Cursor <cursoragent@cursor.com>
Piotr's fork 3/5 approval note: fa's forwarded-CAS-to-created metrics
mapping lives downstream of the convention that successful forwards
carry no reason. fa now overwrites the reason unconditionally with
l1_forwarded, so a fork-attached reason would be silently swallowed
rather than break attribution — this assertion makes anyone who adds
one reconcile with that mapping deliberately instead of by surprise.

Co-authored-by: Cursor <cursoragent@cursor.com>
@shreyas-blacksmith
shreyas-blacksmith force-pushed the shreyas/bazel-l1-piece-2-trust branch from 38b0818 to 4f5ebb6 Compare August 1, 2026 00:27
@shreyas-blacksmith
shreyas-blacksmith force-pushed the shreyas/bazel-l1-piece-3-forwarding branch from f389172 to 0b2a6aa Compare August 1, 2026 00:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants