fix(o11y): record output request + HEC ACK-poll latency in ms (PIPE-1404) - #292
Open
Dylan-M wants to merge 1 commit into
Open
fix(o11y): record output request + HEC ACK-poll latency in ms (PIPE-1404)#292Dylan-M wants to merge 1 commit into
Dylan-M wants to merge 1 commit into
Conversation
This was referenced Aug 11, 2026
feat(o11y): per-instance metrics via injectable MeterProvider, drop package globals (PIPE-1066)
#285
Merged
Merged
Contributor
Author
2 tasks
Dylan-M
force-pushed
the
dylanmyers/pipe-1404-output-latency-ms-buckets
branch
from
August 14, 2026 19:22
b24bf34 to
792856e
Compare
Dylan-M
force-pushed
the
dylanmyers/pipe-1066-lifecycle-metrics
branch
from
August 14, 2026 19:22
be2b515 to
c52f955
Compare
Dylan-M
force-pushed
the
dylanmyers/pipe-1066-lifecycle-metrics
branch
from
August 26, 2026 15:43
c52f955 to
e49f88b
Compare
Dylan-M
force-pushed
the
dylanmyers/pipe-1404-output-latency-ms-buckets
branch
from
August 26, 2026 15:43
792856e to
91ba669
Compare
Dylan-M
force-pushed
the
dylanmyers/pipe-1404-output-latency-ms-buckets
branch
from
September 4, 2026 13:06
91ba669 to
fbc35ad
Compare
Dylan-M
force-pushed
the
dylanmyers/pipe-1066-lifecycle-metrics
branch
from
September 4, 2026 13:06
e49f88b to
7594cdb
Compare
eKuG
approved these changes
Sep 4, 2026
Dylan-M
force-pushed
the
dylanmyers/pipe-1066-lifecycle-metrics
branch
from
September 4, 2026 21:00
7594cdb to
a67296a
Compare
Dylan-M
force-pushed
the
dylanmyers/pipe-1404-output-latency-ms-buckets
branch
from
September 4, 2026 21:00
fbc35ad to
70e85fe
Compare
…404) Assisted-by: Claude Opus 4.8
Dylan-M
force-pushed
the
dylanmyers/pipe-1066-lifecycle-metrics
branch
from
September 4, 2026 22:54
a67296a to
ba081e7
Compare
Dylan-M
force-pushed
the
dylanmyers/pipe-1404-output-latency-ms-buckets
branch
from
September 4, 2026 22:54
70e85fe to
113a78a
Compare
graphite-app
Bot
changed the base branch from
dylanmyers/pipe-1066-lifecycle-metrics
to
graphite-base/292
September 10, 2026 13:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Proposed Change
blitz.output.request_latencyandblitz.output.hec.ack_poll_latencywere declared in seconds and recorded via.Seconds(). Both use OTel's default histogram buckets (0, 5, 10, 25, ... 10000), where the first boundary is 5. Output latencies almost always run well under 5 seconds, so every sample landed in the first bucket. The histograms carried no distribution signal.Both now record in milliseconds, where the default buckets fit. A new
output.DurationMillishelper converts a duration to fractional milliseconds. It stays float so a sub-millisecond sample is not truncated to zero. Truncating would also understate the histogram sum. The request-latency feeders (tcp, otlp_grpc, hec) and the HEC ACK-poll site record through it. Bothmetric.yamlunits change fromstoms.Stacked on #291.
How the reviewer can validate
Manual:
make buildnc -lk 127.0.0.1 5140 >/dev/null &curl -s localhost:9100/metrics | grep request_latencyblitz_blitz_output_request_latency_milliseconds_*(previously..._seconds_*). On a localhost sink, sub-5ms sends correctly sit in the first bucket. Any latency above 5ms now lands in a higher bucket, where in seconds it would have collapsed into the first.The HEC ACK-poll histogram (
blitz_blitz_output_hec_ack_poll_latency_milliseconds_*) validates the same way against a HEC endpoint with ack enabled.Automated:
go test ./output/....TestDurationMilliscovers the conversion including sub-millisecond preservation. Manual-reader tests assert themsunit on bothblitz.output.request_latencyandblitz.output.hec.ack_poll_latency.Checklist