Skip to content

fix(server): bound Prometheus requests and handle response failures - #1709

Open
OskarEichler wants to merge 2 commits into
OutlineFoundation:masterfrom
OskarEichler:codex/prometheus-request-reliability
Open

OskarEichler wants to merge 2 commits into
OutlineFoundation:masterfrom
OskarEichler:codex/prometheus-request-reliability

Conversation

@OskarEichler

@OskarEichler OskarEichler commented Aug 27, 2026

Copy link
Copy Markdown

Why

Response parsing inside HTTP event callbacks can throw outside the returned promise, and stalled response bodies/readiness probes can retain resources.

Changes

  • Use an awaited fetch/JSON path so malformed/truncated bodies, non-success status codes, and Prometheus API errors reject normally.
  • Cover both headers and body with a 30-second abort deadline; clean up failed requests while retaining successful keep-alive reuse.
  • Drain readiness response bodies and destroy stalled probes after one second.
  • Use built-in promise-based directory creation/writes, removing the undeclared transitive mkdirp dependency from this module.

Verification

  • After splitting, all 65 existing manager-metrics, shared-metrics, access-key, and file-operation specs passed again on this isolated branch under Node 22.23.2.

  • Controlled source checks passed for malformed/truncated JSON, non-2xx responses, and request cleanup.

  • Targeted semantic TypeScript and formatting checks passed. The relevant existing server suites also passed in the combined-source verification.

  • Patch isolation and git diff --check passed. Recombining all focused patches reproduces the reviewed source changes exactly.

  • No test/spec files were added or modified; controlled probe drivers are kept outside the repository.

Compatibility and remaining validation

  • Requires Node with native AbortController, which is available in the existing Node 18 baseline; independent of the Node 22 PR.
  • No real Prometheus/container startup integration or production timing measurement was performed.

Full npm installation/build checks remain incomplete: npm 12 rejected existing lockfile Git dependencies (EALLOWGIT). Isolated registry-only tooling was used without disabling that safeguard.

Scope

This is one focused part of the reliability review, based directly on upstream master; it does not include the other review patches. No installed client, live VPN/DNS setting, or production service was changed by this patch.

Second review — 2026-08-27

Use one readiness polling loop across subprocess retries instead of recursively creating abandoned async loops. Recover spawn failures/exits with bounded exponential backoff (1–30 seconds, reset after a stable minute). Replace the readiness inactivity timeout with an absolute one-second deadline, including body draining.

Controlled supervisor checks passed for spawn failure, repeated crashes, the backoff cap/reset, and a single readiness poller. Real local HTTP checks passed for malformed JSON, HTTP errors, stalled headers/body, and readiness deadlines. All 65 selected existing server specs and targeted TypeScript checks passed on this isolated branch.

All touched files and nearby callers were reviewed again. Each focused patch was also checked in combination with the other seven patches for this repository. External probe drivers remain outside the repositories; no test/spec files were added or modified. Existing full-build and platform-validation limitations above still apply.

@OskarEichler
OskarEichler requested a review from fortuna as a code owner August 27, 2026 18:49
@greptile-apps

greptile-apps Bot commented Aug 27, 2026

Copy link
Copy Markdown

Greptile Summary

The PR bounds local Prometheus HTTP operations and makes query failures reject through the normal promise path.

  • Replaces callback-based Prometheus queries with awaited fetch and JSON handling under a 30-second abort deadline.
  • Drains readiness responses and destroys probes after one second of inactivity.
  • Replaces the local mkdirp usage and callback write with promise-based built-in filesystem operations.

Confidence Score: 5/5

The PR appears safe to merge with no concrete correctness or security regression identified.

The changed request paths consistently reject HTTP, parsing, API, and timeout failures, while readiness probes and filesystem operations retain the required startup behavior.

Important Files Changed

Filename Overview
src/shadowbox/infrastructure/prometheus_scraper.ts Adds bounded Prometheus query and readiness lifecycles, normalizes response failures into promise rejections, and modernizes configuration-file writes without an identified regression.

Sequence Diagram

sequenceDiagram
  participant S as Shadowbox
  participant P as Prometheus
  S->>P: Query with 30-second abort signal
  alt Successful HTTP and Prometheus response
    P-->>S: JSON result data
  else HTTP, JSON, API, or timeout failure
    P-->>S: Rejected query
  end
  S->>P: Readiness probe
  alt Headers received
    P-->>S: Status code
    S->>P: Drain response body
  else Socket idle for one second
    S->>P: Destroy request
  end
Loading

Reviews (1): Last reviewed commit: "fix(server): bound Prometheus requests a..." | Re-trigger Greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant