Skip to content

Make OTLP a lossless wire in both directions - #259

Merged
Exelord merged 5 commits into
mainfrom
claude/otel-vx-core-iiqah3
Aug 19, 2026
Merged

Make OTLP a lossless wire in both directions#259
Exelord merged 5 commits into
mainfrom
claude/otel-vx-core-iiqah3

Conversation

@Exelord

@Exelord Exelord commented Aug 19, 2026

Copy link
Copy Markdown
Member

The OTel exporter now describes a run completely, and vx Cloud accepts
OTLP as an ingest path — so the standard wire carries everything, feeds
the dashboard, and is enough for anyone else to build their own analytics
on.

The design call

OTLP became the export plus a receiver, not core's internal contract.
Making it the contract would hardcode a vendor spec into core — the
coupling the provider-neutral directive removed — and the two audiences
differ: a tracing backend wants spans, cloud wants a lossless, idempotent,
atomic per-run record. One shape serving both means the lossy one wins.
So TelemetryRecord stays core's contract, @vzn/vx-otel stays the only
place OTel is named, and cloud grew a receiver.

What the exporter was dropping

Measured by diffing what cloud's ingest writes against what the mapper
emitted:

  • workspaceId / workspaceName — the multi-workspace routing key. A
    backend could describe a run but not say which workspace it belonged to.
  • defaultBranch — the trunk-vs-branch trust axis.
  • outputFp — the entire cross-machine hermeticity signal. A
    --verify=fingerprint divergence was invisible over OTel.
  • Wallclock offsets, every run tally, and rerun-failed's own exitCode
    (a receiver knew the re-run failed but not with what).
  • Task logs, which had nowhere to go but a vx-specific endpoint — the half
    a developer actually opens when a build goes red.

Encoding decisions that are not aesthetic

  • Nanosecond offsets ride as OTLP int64 strings. 9007199254740993 is
    the first integer a JS number cannot hold, and a receiver derives a task's
    dedup key from it.
  • Run start/end also ride as millisecond attributes even though the span
    carries times, because the span's are unix-nanos (epoch-ms × 1e6 is past
    MAX_SAFE_INTEGER) and those two values are a storage partition key.
    Losing precision there is a row in the wrong partition, not a rounding error.
  • Verify path lists became JSON arrays. A comma is a legal byte in a
    filename and these are the actionable half of a verdict, so a split path
    names a file that does not exist. vx.requested_tasks deliberately stays
    comma-joined — read by humans far more often than parsed — and that is
    written down as a stated limit, not a guarantee.
  • The fingerprint's per-file map may legally not survive. It is the
    largest attribute and a limiting collector truncates it first. Detection
    keys on the fixed-width tree digest, so that costs a diff its detail,
    never its verdict.

Logs

One record per task, not per chunk: a build writes output in thousands
of tiny pieces, so a record each would multiply the payload to deliver the
same bytes and force every receiver to reassemble them before showing
anything. Records link to their task span by trace and span id.

On by default once an endpoint is configured (the cloud sink's convention),
with the standard OTEL_LOGS_EXPORTER=none honoured so a pipeline already
configured that way does not start receiving build logs because it upgraded
vx. With the signal off the sink stops declaring task.log in wants, so
core never projects a chunk and the run pays nothing.

TaskLogBuffer moved into core rather than being reimplemented: the
otel sink needs the identical bounds and retention rules, and a second copy
of those rules is how two sinks come to disagree about which task's output
survived. Cloud keeps one import path via a re-export shim, so no call site
moved.

The receiver

POST /v1/otlp/v1/{traces,logs} decodes into the same records the native
endpoints take and calls the same Analytics.ingest* — one store, one set
of read queries, no second schema. The existing
(started_at, run_id, project, task) idempotency covers a retried OTLP
batch for free, which matters because OTLP exporters retry and carry no
dedup semantics of their own.

Two things OTLP cannot say for itself:

  • No run boundary. The vx.run root span is required — it is the span
    that ends last — so a payload without one is a 400 rather than a
    half-stored run.
  • Collectors may drop spans. Tallies are recomputed from the spans that
    arrived, via the same function the native path uses, so a partial batch
    yields a header consistent with the rows it stored instead of a count that
    outruns them.

The decoder repeats the exporter's attribute keys on purpose: cloud takes no
runtime dependency on the exporter, and a wire only one package can write is
not much of a wire. The copy is guarded differentially — the tests drive
the real OtelSink and assert the decode reproduces the record that went in.

Deliberately not done

/v1/ingest is not replaced. Its per-run transaction, its incremental task
rows (what makes run detail fill in live) and the catalog push have no OTLP
analogue, and a config snapshot is not an observation. Both wires are
documented, with the OTLP caveats stated rather than papered over.

Verification

Mutation-checked rather than assumed: renaming vx.task.hash in the
encoder fails exactly 2 of the receiver's 12 tests (restore → 12/0);
dropping defaultBranch from the encoder fails exactly 1 of vx-otel's 34
(restore → 34/0). A completeness pin on the RunContextRecord field set
means a new telemetry field fails until someone decides how it maps.

One existing test was repinned honestly: the sink used to assert wants
never contains task.log — true when it had nowhere to send them; it now
asserts the refusal is conditional, with the logs-off arm still pinned so
the zero-cost property cannot regress.

No CACHE_VERSION / schema / migration / wire bump.
TELEMETRY_SCHEMA_VERSION stays 2 — no contract field was added; the
exporter was simply not carrying fields that already existed.

Gates from the root: fmt/lint 0, core 2631 / 0 (23 skip = sandbox),
cloud 1280 / 0 across 58 suites (38 skip = the browser suites, which
need playwright + a built ui/dist this container lacks), vx-otel
43 / 0, docs site 168 pages with a zero-broken-link crawl over both
edited pages.

🤖 Generated with Claude Code

https://claude.ai/code/session_019x8vExRpvPDkvwcDZ7wKDE


Generated by Claude Code

claude added 5 commits August 19, 2026 18:22
`TaskLogBuffer` was cloud-local while cloud was its only consumer. The
OTLP logs exporter needs the identical bounds and the identical retention
rules — a cache hit's bytes belong to the run that executed them, and a
failed tail is never evicted to keep a success — and a second
implementation of those rules is how two sinks come to disagree about
which task's output survived.

So it moves beside the telemetry contract that decides its vocabulary
(`TaskStatus`, `CacheSource`) and onto the façade, on the same
demonstrated need that widened it for `splitTaskId` and
`escapeMarkdownCell`. Cloud keeps one import path via a re-export shim,
so no call site moves; its suite for the buffer moves to core with it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019x8vExRpvPDkvwcDZ7wKDE
The exporter was dropping fields a consumer needs and cannot re-derive:
the workspace identity that keys a multi-workspace store, the default
branch that decides whether a run is trunk or branch work, the output
fingerprint that is the entire cross-machine hermeticity signal, the
wallclock offsets, and every run-level tally. A backend reading the trace
could describe what ran but not which workspace it belonged to, and a
`--verify=fingerprint` divergence was invisible.

All of them now ride the spans, and the keys live in one `VX_ATTR` map so
a rename is a compile error rather than a field that silently stops
arriving. Nanosecond offsets ride as OTLP int64 strings — the first value
past the float-safe range is a real cache dedup key, not a rounding
question. Run start/end additionally ride as millisecond attributes,
because the span's own times are nanos and that precision loss lands a
row in the wrong storage partition.

The fingerprint's per-file map is the one attribute allowed not to
survive: it is large and a limiting collector will truncate it. Detection
keys on the fixed-width tree digest, so that costs a diff its detail,
never its verdict.

Tests pin the contract's field set, so a new telemetry field fails until
someone decides how it maps.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019x8vExRpvPDkvwcDZ7wKDE
Traces described what ran; the output itself still had nowhere to go but
a vx-specific endpoint, which is the half a developer actually opens when
a build goes red. It now rides the standard Logs signal, one record per
executed task, linked to that task's span by traceId/spanId so a viewer
opens the log from the span rather than by correlating ids by hand.

Per task, not per chunk: a build writes its output in thousands of tiny
pieces, so a record each would multiply the payload to deliver the same
bytes and make every receiver reassemble them before it could show
anything. Capture goes through core's shared buffer, so the bounds and
the retention rules are the ones the cloud sink already applies — a cache
hit ships nothing, because those bytes belong to the run that executed
them, and a failure's tail is never dropped to keep a success.

On by default once an endpoint is configured, matching the cloud sink;
`OTEL_LOGS_EXPORTER=none` is honoured so a pipeline already configured
that way does not start receiving build logs because it upgraded vx.
With the signal off the sink stops asking for `task.log` at all, so core
never projects a chunk and the run pays nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019x8vExRpvPDkvwcDZ7wKDE
The exporter now describes a run completely, so the standard wire is
enough to rebuild one. This makes vx Cloud read it: POST an OTLP trace or
logs payload to /v1/otlp/v1/{traces,logs} and it decodes into the same
records the native endpoints take and goes through the same ingest — one
store, one set of read queries, no second schema. The existing dedup
covers a retried OTLP batch for free, which matters because OTLP
exporters retry and carry no dedup semantics of their own.

Two things OTLP cannot say for itself, handled explicitly. It has no run
boundary, so the vx.run root span is required: it is the span that ends
last, and a payload without one is refused rather than stored half-formed.
And a collector may drop spans, so the run tallies are recomputed from
the ones that arrived, via the same function the native path uses — a
partial batch yields a header consistent with the rows it stored instead
of a count that outruns them.

The decoder repeats the exporter's attribute keys on purpose: cloud takes
no runtime dependency on the exporter, and a wire only one package can
write is not much of a wire — the point is that anyone can build their
own analytics off these payloads. The copy is safe because it is
differentially guarded: the tests drive the real exporter and assert the
decode reproduces the record that went in, so a renamed key fails there
rather than costing a field in production.

/v1/ingest stays the default and is not replaced. Its per-run
transaction, its incremental task rows and the catalog push have no OTLP
analogue, and a config snapshot is not an observation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019x8vExRpvPDkvwcDZ7wKDE
The job burned all 15 minutes inside `playwright install --with-deps`
and was cancelled before a single test ran: apt crawled, with
`fonts-freefont-ttf` (5.6 MB) alone taking 7m49s from the Azure Ubuntu
mirror. Nothing about the run was vx's — the core job on the same commit
passed in 2m44s.

The install chain here is long by construction (apt postgres, bun
install, npm playwright, the `--with-deps` apt fonts, the chromium
download, the SPA build) and 15 left no room for a slow mirror. Raising
the budget is the right move for the reason it is usually the wrong one:
this fails by running LONG, which more time actually fixes, and 25 is
still far short of letting a genuinely wedged suite sit unnoticed against
a ~2 minute test run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019x8vExRpvPDkvwcDZ7wKDE
@Exelord
Exelord merged commit 7d4f999 into main Aug 19, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants