Skip to content

fix(o11y): record output request + HEC ACK-poll latency in ms (PIPE-1404) - #292

Open
Dylan-M wants to merge 1 commit into
graphite-base/292from
dylanmyers/pipe-1404-output-latency-ms-buckets
Open

fix(o11y): record output request + HEC ACK-poll latency in ms (PIPE-1404)#292
Dylan-M wants to merge 1 commit into
graphite-base/292from
dylanmyers/pipe-1404-output-latency-ms-buckets

Conversation

@Dylan-M

@Dylan-M Dylan-M commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Proposed Change

blitz.output.request_latency and blitz.output.hec.ack_poll_latency were declared in seconds and recorded via .Seconds(). Both use OTel's default histogram buckets (0, 5, 10, 25, ... 10000), where the first boundary is 5. Output latencies almost always run well under 5 seconds, so every sample landed in the first bucket. The histograms carried no distribution signal.

Both now record in milliseconds, where the default buckets fit. A new output.DurationMillis helper converts a duration to fractional milliseconds. It stays float so a sub-millisecond sample is not truncated to zero. Truncating would also understate the histogram sum. The request-latency feeders (tcp, otlp_grpc, hec) and the HEC ACK-poll site record through it. Both metric.yaml units change from s to ms.

Stacked on #291.

How the reviewer can validate

Note: the manual validation procedure below is an AI-generated script. Review it before running.

Manual:

  1. make build
  2. Start a throwaway TCP sink: nc -lk 127.0.0.1 5140 >/dev/null &
  3. Run blitz to it with the self-metrics endpoint on:
    ./blitz --generator-type apache-common --generator-apache-common-rate 50ms \
      --output-type tcp --output-tcp-host 127.0.0.1 --output-tcp-port 5140 \
      --metrics-port 9100
    
  4. curl -s localhost:9100/metrics | grep request_latency
  5. Confirm the histogram renders in milliseconds: blitz_blitz_output_request_latency_milliseconds_* (previously ..._seconds_*). On a localhost sink, sub-5ms sends correctly sit in the first bucket. Any latency above 5ms now lands in a higher bucket, where in seconds it would have collapsed into the first.

The HEC ACK-poll histogram (blitz_blitz_output_hec_ack_poll_latency_milliseconds_*) validates the same way against a HEC endpoint with ack enabled.

Automated: go test ./output/.... TestDurationMillis covers the conversion including sub-millisecond preservation. Manual-reader tests assert the ms unit on both blitz.output.request_latency and blitz.output.hec.ack_poll_latency.

Checklist
  • Changes are tested
  • CI has passed

@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1404-output-latency-ms-buckets branch from b24bf34 to 792856e Compare August 14, 2026 19:22
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1066-lifecycle-metrics branch from be2b515 to c52f955 Compare August 14, 2026 19:22
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1066-lifecycle-metrics branch from c52f955 to e49f88b Compare August 26, 2026 15:43
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1404-output-latency-ms-buckets branch from 792856e to 91ba669 Compare August 26, 2026 15:43
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1404-output-latency-ms-buckets branch from 91ba669 to fbc35ad Compare September 4, 2026 13:06
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1066-lifecycle-metrics branch from e49f88b to 7594cdb Compare September 4, 2026 13:06
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1066-lifecycle-metrics branch from 7594cdb to a67296a Compare September 4, 2026 21:00
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1404-output-latency-ms-buckets branch from fbc35ad to 70e85fe Compare September 4, 2026 21:00
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1066-lifecycle-metrics branch from a67296a to ba081e7 Compare September 4, 2026 22:54
@Dylan-M
Dylan-M force-pushed the dylanmyers/pipe-1404-output-latency-ms-buckets branch from 70e85fe to 113a78a Compare September 4, 2026 22:54
@graphite-app
graphite-app Bot changed the base branch from dylanmyers/pipe-1066-lifecycle-metrics to graphite-base/292 September 10, 2026 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants