Make OTLP a lossless wire in both directions - #259
Merged
Conversation
`TaskLogBuffer` was cloud-local while cloud was its only consumer. The OTLP logs exporter needs the identical bounds and the identical retention rules — a cache hit's bytes belong to the run that executed them, and a failed tail is never evicted to keep a success — and a second implementation of those rules is how two sinks come to disagree about which task's output survived. So it moves beside the telemetry contract that decides its vocabulary (`TaskStatus`, `CacheSource`) and onto the façade, on the same demonstrated need that widened it for `splitTaskId` and `escapeMarkdownCell`. Cloud keeps one import path via a re-export shim, so no call site moves; its suite for the buffer moves to core with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019x8vExRpvPDkvwcDZ7wKDE
The exporter was dropping fields a consumer needs and cannot re-derive: the workspace identity that keys a multi-workspace store, the default branch that decides whether a run is trunk or branch work, the output fingerprint that is the entire cross-machine hermeticity signal, the wallclock offsets, and every run-level tally. A backend reading the trace could describe what ran but not which workspace it belonged to, and a `--verify=fingerprint` divergence was invisible. All of them now ride the spans, and the keys live in one `VX_ATTR` map so a rename is a compile error rather than a field that silently stops arriving. Nanosecond offsets ride as OTLP int64 strings — the first value past the float-safe range is a real cache dedup key, not a rounding question. Run start/end additionally ride as millisecond attributes, because the span's own times are nanos and that precision loss lands a row in the wrong storage partition. The fingerprint's per-file map is the one attribute allowed not to survive: it is large and a limiting collector will truncate it. Detection keys on the fixed-width tree digest, so that costs a diff its detail, never its verdict. Tests pin the contract's field set, so a new telemetry field fails until someone decides how it maps. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019x8vExRpvPDkvwcDZ7wKDE
Traces described what ran; the output itself still had nowhere to go but a vx-specific endpoint, which is the half a developer actually opens when a build goes red. It now rides the standard Logs signal, one record per executed task, linked to that task's span by traceId/spanId so a viewer opens the log from the span rather than by correlating ids by hand. Per task, not per chunk: a build writes its output in thousands of tiny pieces, so a record each would multiply the payload to deliver the same bytes and make every receiver reassemble them before it could show anything. Capture goes through core's shared buffer, so the bounds and the retention rules are the ones the cloud sink already applies — a cache hit ships nothing, because those bytes belong to the run that executed them, and a failure's tail is never dropped to keep a success. On by default once an endpoint is configured, matching the cloud sink; `OTEL_LOGS_EXPORTER=none` is honoured so a pipeline already configured that way does not start receiving build logs because it upgraded vx. With the signal off the sink stops asking for `task.log` at all, so core never projects a chunk and the run pays nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019x8vExRpvPDkvwcDZ7wKDE
The exporter now describes a run completely, so the standard wire is
enough to rebuild one. This makes vx Cloud read it: POST an OTLP trace or
logs payload to /v1/otlp/v1/{traces,logs} and it decodes into the same
records the native endpoints take and goes through the same ingest — one
store, one set of read queries, no second schema. The existing dedup
covers a retried OTLP batch for free, which matters because OTLP
exporters retry and carry no dedup semantics of their own.
Two things OTLP cannot say for itself, handled explicitly. It has no run
boundary, so the vx.run root span is required: it is the span that ends
last, and a payload without one is refused rather than stored half-formed.
And a collector may drop spans, so the run tallies are recomputed from
the ones that arrived, via the same function the native path uses — a
partial batch yields a header consistent with the rows it stored instead
of a count that outruns them.
The decoder repeats the exporter's attribute keys on purpose: cloud takes
no runtime dependency on the exporter, and a wire only one package can
write is not much of a wire — the point is that anyone can build their
own analytics off these payloads. The copy is safe because it is
differentially guarded: the tests drive the real exporter and assert the
decode reproduces the record that went in, so a renamed key fails there
rather than costing a field in production.
/v1/ingest stays the default and is not replaced. Its per-run
transaction, its incremental task rows and the catalog push have no OTLP
analogue, and a config snapshot is not an observation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019x8vExRpvPDkvwcDZ7wKDE
The job burned all 15 minutes inside `playwright install --with-deps` and was cancelled before a single test ran: apt crawled, with `fonts-freefont-ttf` (5.6 MB) alone taking 7m49s from the Azure Ubuntu mirror. Nothing about the run was vx's — the core job on the same commit passed in 2m44s. The install chain here is long by construction (apt postgres, bun install, npm playwright, the `--with-deps` apt fonts, the chromium download, the SPA build) and 15 left no room for a slow mirror. Raising the budget is the right move for the reason it is usually the wrong one: this fails by running LONG, which more time actually fixes, and 25 is still far short of letting a genuinely wedged suite sit unnoticed against a ~2 minute test run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019x8vExRpvPDkvwcDZ7wKDE
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The OTel exporter now describes a run completely, and vx Cloud accepts
OTLP as an ingest path — so the standard wire carries everything, feeds
the dashboard, and is enough for anyone else to build their own analytics
on.
The design call
OTLP became the export plus a receiver, not core's internal contract.
Making it the contract would hardcode a vendor spec into core — the
coupling the provider-neutral directive removed — and the two audiences
differ: a tracing backend wants spans, cloud wants a lossless, idempotent,
atomic per-run record. One shape serving both means the lossy one wins.
So
TelemetryRecordstays core's contract,@vzn/vx-otelstays the onlyplace OTel is named, and cloud grew a receiver.
What the exporter was dropping
Measured by diffing what cloud's ingest writes against what the mapper
emitted:
workspaceId/workspaceName— the multi-workspace routing key. Abackend could describe a run but not say which workspace it belonged to.
defaultBranch— the trunk-vs-branch trust axis.outputFp— the entire cross-machine hermeticity signal. A--verify=fingerprintdivergence was invisible over OTel.rerun-failed's ownexitCode(a receiver knew the re-run failed but not with what).
a developer actually opens when a build goes red.
Encoding decisions that are not aesthetic
9007199254740993isthe first integer a JS number cannot hold, and a receiver derives a task's
dedup key from it.
carries times, because the span's are unix-nanos (epoch-ms × 1e6 is past
MAX_SAFE_INTEGER) and those two values are a storage partition key.Losing precision there is a row in the wrong partition, not a rounding error.
filename and these are the actionable half of a verdict, so a split path
names a file that does not exist.
vx.requested_tasksdeliberately stayscomma-joined — read by humans far more often than parsed — and that is
written down as a stated limit, not a guarantee.
largest attribute and a limiting collector truncates it first. Detection
keys on the fixed-width tree digest, so that costs a diff its detail,
never its verdict.
Logs
One record per task, not per chunk: a build writes output in thousands
of tiny pieces, so a record each would multiply the payload to deliver the
same bytes and force every receiver to reassemble them before showing
anything. Records link to their task span by trace and span id.
On by default once an endpoint is configured (the cloud sink's convention),
with the standard
OTEL_LOGS_EXPORTER=nonehonoured so a pipeline alreadyconfigured that way does not start receiving build logs because it upgraded
vx. With the signal off the sink stops declaring
task.loginwants, socore never projects a chunk and the run pays nothing.
TaskLogBuffermoved into core rather than being reimplemented: theotel sink needs the identical bounds and retention rules, and a second copy
of those rules is how two sinks come to disagree about which task's output
survived. Cloud keeps one import path via a re-export shim, so no call site
moved.
The receiver
POST /v1/otlp/v1/{traces,logs}decodes into the same records the nativeendpoints take and calls the same
Analytics.ingest*— one store, one setof read queries, no second schema. The existing
(started_at, run_id, project, task)idempotency covers a retried OTLPbatch for free, which matters because OTLP exporters retry and carry no
dedup semantics of their own.
Two things OTLP cannot say for itself:
vx.runroot span is required — it is the spanthat ends last — so a payload without one is a 400 rather than a
half-stored run.
arrived, via the same function the native path uses, so a partial batch
yields a header consistent with the rows it stored instead of a count that
outruns them.
The decoder repeats the exporter's attribute keys on purpose: cloud takes no
runtime dependency on the exporter, and a wire only one package can write is
not much of a wire. The copy is guarded differentially — the tests drive
the real
OtelSinkand assert the decode reproduces the record that went in.Deliberately not done
/v1/ingestis not replaced. Its per-run transaction, its incremental taskrows (what makes run detail fill in live) and the catalog push have no OTLP
analogue, and a config snapshot is not an observation. Both wires are
documented, with the OTLP caveats stated rather than papered over.
Verification
Mutation-checked rather than assumed: renaming
vx.task.hashin theencoder fails exactly 2 of the receiver's 12 tests (restore → 12/0);
dropping
defaultBranchfrom the encoder fails exactly 1 of vx-otel's 34(restore → 34/0). A completeness pin on the
RunContextRecordfield setmeans a new telemetry field fails until someone decides how it maps.
One existing test was repinned honestly: the sink used to assert
wantsnever contains
task.log— true when it had nowhere to send them; it nowasserts the refusal is conditional, with the logs-off arm still pinned so
the zero-cost property cannot regress.
No
CACHE_VERSION/ schema / migration / wire bump.TELEMETRY_SCHEMA_VERSIONstays 2 — no contract field was added; theexporter was simply not carrying fields that already existed.
Gates from the root: fmt/lint 0, core 2631 / 0 (23 skip = sandbox),
cloud 1280 / 0 across 58 suites (38 skip = the browser suites, which
need playwright + a built
ui/distthis container lacks), vx-otel43 / 0, docs site 168 pages with a zero-broken-link crawl over both
edited pages.
🤖 Generated with Claude Code
https://claude.ai/code/session_019x8vExRpvPDkvwcDZ7wKDE
Generated by Claude Code