feat(generator): add per-process scraper to hostmetrics - #295
feat(generator): add per-process scraper to hostmetrics#295chelseawright7 wants to merge 1 commit into
Conversation
The existing `processes` scraper reports host-wide counts (system.processes.count) only, so nothing in blitz emitted per-process telemetry. The new `process` scraper emits process.memory.usage, process.memory.virtual, process.cpu.time, process.disk.io, process.threads and process.open_file_descriptors. Each record carries its own resource map with the process identity: pid, parent_pid, executable name and path, command, command_line, command_args, owner and — on Linux — cgroup. That resource-level identity is what makes process metrics high cardinality, and therefore what reduction and normalization pipelines act on; placing it in datapoint attributes instead would not exercise the same code paths downstream. The simulated process table varies by the configured OS (Linux daemons vs Windows services), and always contains daemons under 1 MiB resident so memory-threshold filters have something to drop.
eKuG
left a comment
There was a problem hiding this comment.
One cardinality concern worth a decision before merge; two smaller notes folded in.
|
|
||
| var records []output.MetricRecord | ||
| for _, tmpl := range templates { | ||
| pid := int64(r.Intn(30000) + 100) // #nosec G404 |
There was a problem hiding this comment.
PIDs are re-rolled on every Scrape, and pid sits on the resource. Each cycle therefore mints ~30 brand-new resource identities, so a long run produces unbounded distinct resources downstream (backends index per resource). The existing scrapers keep one stable host resource, so this is new in kind. Suggest assigning PIDs once per template (seed-derived or a map initialized on first Scrape) so identities stay stable across cycles.
Related, lower priority: process.cpu.time (line 125) and process.disk.io (line 138) are Sum-typed but emit fresh random values each scrape, so rate() over them goes negative. That matches the existing system.cpu.time behavior, so fine if intended for a load generator, just noting it now extends to per-process series.
Proposed Change
Split out of #294 per review — that PR is now journald data library only, and this one carries the scraper on its own.
The existing
processesscraper reports host-wide counts (system.processes.count) only, so nothing in Blitz emitted per-process telemetry. The newprocessscraper emits:process.memory.usageByprocess.memory.virtualByprocess.cpu.timestatesprocess.disk.iodirectionByprocess.threads{thread}process.open_file_descriptors{count}Each record carries its own resource map with the process identity —
process.pid,process.parent_pid,process.executable.name,process.executable.path,process.command,process.command_line,process.command_args,process.owner, and on Linuxprocess.cgroup. That resource-level identity is what makes process metrics high cardinality, and therefore what reduction and normalization processors act on; putting it in datapoint attributes instead would not exercise the same downstream code paths.The simulated process table varies by
generator.hostmetrics.os(Linux daemons vs Windows services), and always contains daemons under 1 MiB resident so memory-threshold filters have something to drop.Checklist
go build ./...,go test ./...,make lintandmake security(178 files, 0 issues) all pass.New unit tests in
generator/hostmetrics/process_test.gocover: resource-attribute placement, that the shared base resource map is never mutated by a scrape, that each template gets a distinct resource map, that a scrape yields processes both under and over 1 MiB resident, Windows-specific behavior (.exenames, nocgroup), and command-line assembly with and without args. Existing scraper-count assertions updated 8 → 9.Smoke-tested through the CLI:
blitz --generator-type hostmetrics --generator-hostmetrics-scrapers process \ --output-type stdout --generator-count 4