[haggs]

I found two telemetry failures on large Compass runs: oversized OTLP batches were rejected, and some spans never appeared in LangSmith even after successful export acknowledgements. Missing signal spans made completed analysis look as though it had never run, and surviving spans could give an incomplete picture of usage.

Approach

I buffer completed spans on disk until shutdown, then export parents before their children. Each request contains at most 64 spans, and a parent must be acknowledged in an earlier request before its children are sent. Files are removed after acknowledgement; failed requests get one retry, and undelivered spans remain on disk.

I wrap embedding requests in OpenInference’s suppress_tracing() context. Embedding spans are never collected, buffered, or uploaded; the application still receives the original embedding results. Other agent, model, tool, and signal spans remain traced.

This is a workaround for the observed losses after acknowledgement; I have not established the receiving-side cause.

Alternatives tested

I tested alternatives on a 10,000-trace corpus with four signals and remote embeddings. Raising only the client limit to 128 MiB still lost 470 of 2,678 spans. At 512 MiB, the endpoint returned 11 HTTP 413 responses without any client size-limit drops, so I stopped that attempt. The experiment is recorded in closed PR #126.

Hiding vectors and setting OTEL_BSP_MAX_EXPORT_BATCH_SIZE=8, without disk buffering, avoided size failures: the largest request was 3.65 MiB and every export succeeded. LangSmith nevertheless missed 68 of 2,710 acknowledged spans, including three signal roots. The missing count stayed unchanged across ten backend checks.

Validation

Verified locally: I ran 807e0fa through the normal Compass CLI against all 10,000 traces in a large corpus, with every signal enabled and remote embeddings. Five signals completed; grader analysis skipped because the inputs contained no grader results. The run produced seven insights that passed schema validation. A local observer recorded every span ID, and I checked LangSmith storage after export: all 1,429 non-embedding spans ended and were stored with their original parent relationships. No embedding spans were collected. All 27 export requests succeeded, with no HTTP 413 responses; the largest request was 27.9 MiB before compression. Analysis took 113.9 minutes, followed by 76.9 seconds until CLI exit, including telemetry upload.