[Feature]: Add --output-format=json|logfmt flag for machine-readable output

[fuleinist]

Feature Description

Add a --output-format=json|logfmt|raw flag (matching the existing --input-format=auto|json|logfmt enum) so hl can re-emit parsed records as structured output instead of (or in addition to) the human-readable default. The default behavior stays unchanged; the flag is opt-in.

Concretely: after parsing + filtering + sorting, emit one record per line in the chosen machine-readable format. For --output-format=json this means NDJSON (one JSON object per line, matching how jq expects input). For --output-format=logfmt it means the standard key=value key="quoted value" shape.

Motivation and Use Case

hl already does all the work to parse structured logs — JSON, logfmt, even mixed/prefixed inputs. Today the parsed records are formatted only for the human eye, which means there's no way to:

• Convert formats mid-pipeline. hl input.logfmt --output-format=json | jq '.level == "error"' — re-emit logfmt as JSON, then filter with jq. Today: not possible without a separate converter (logfmt2json, etc.).
• Sort + filter + re-emit as JSON. hl -s -l error -P app.log --output-format=json | jq -r '.trace_id' — sort by timestamp, filter to errors only, then extract a field across all matching lines. Today: impossible without leaving hl, since hl's output is human-formatted text that jq can't parse back.
• Pipe into dashboards / log shippers that want JSON over stdout. hl -F app.log --output-format=json | vector --config ... — follow a log file and emit structured records to a log shipper. Today: hl emits ANSI-colored human text; the shipper has to reverse-parse it.

The maintainer's existing InputFormat enum (Auto | Json | Logfmt, in src/cli.rs) already covers the input-side taxonomy. This feature just mirrors that taxonomy on the output side, turning hl from "human reader only" into "structured-log preprocessor".

Example Usage

# Convert logfmt → NDJSON, filter to errors, extract trace_id
hl --input-format=logfmt --output-format=json -l error app.logfmt | jq -r '.trace_id'

# Convert JSON → logfmt (a frequently-requested but missing direction)
hl --input-format=json --output-format=logfmt -P app.json

# Sort JSON logs by timestamp, then pipe to a structured sink
hl -s --output-format=json --paging=never app.jsonl

# Follow a log file and emit NDJSON as lines arrive
hl -F --output-format=json --output-delimiter=newline app.log

Expected output of --output-format=json (NDJSON, one record per line):

{"timestamp":"2026-07-06T10:00:00Z","level":"INFO","message":"Request received","request_id":"12345","duration_ms":150}
{"timestamp":"2026-07-06T10:00:01Z","level":"ERROR","message":"DB timeout","request_id":"12346","duration_ms":3001}

Expected output of --output-format=logfmt:

timestamp="2026-07-06T10:00:00Z" level=INFO message="Request received" request_id=12345 duration_ms=150
timestamp="2026-07-06T10:00:01Z" level=ERROR message="DB timeout" request_id=12346 duration_ms=3001

Alternatives Considered

• --raw already exists — but --raw dumps the original source bytes; if input was a 2KB pretty-printed JSON object, --raw gives you 2KB of pretty JSON, not the parsed-and-rewritten-as-compact-JSON record. Different feature.
• Customizable output templates (issue Customizable Output Formatting and Custom Fields #611) — much larger scope: needs the internal model refactor the maintainer is mid-way through (see maintainer comment on Customizable Output Formatting and Custom Fields #611, 2025-11-25: "I don't expect it to land for at least a couple of months"). A structured-output flag is the inverse direction: instead of a user-defined human template, it's a fixed machine schema, and only requires a small re-emit pass at the existing output boundary.
• External converter (logfmt2json, jq, etc.) — adds dependencies and breaks hl's value proposition as a single-binary pipeline tool. Users would have to know their input format, pick a converter, and chain it manually.
• Just emit --raw and let jq parse it — only works if input was already JSON; doesn't help the logfmt-input case, and doesn't preserve hl's parsed/sorted/filtered state (e.g., timestamp reordering from -s is lost).

Additional Context

• The maintainer has shipped every input-format flag (--input-format=auto|json|logfmt) and the InputFormat enum in src/cli.rs. Mirroring this on output is symmetric and fits the existing clap ValueEnum pattern.
• This complements issue Customizable Output Formatting and Custom Fields #611 (template-based human output) rather than competing: Customizable Output Formatting and Custom Fields #611 is human-readable templating for display; this proposal is machine-readable emission for piping. A user could legitimately want both.
• Suggested implementation scope: a new OutputFormat enum (Pretty | Json | Logfmt, with Pretty as the default preserving current behavior) in src/cli.rs, a new src/output/json.rs and src/output/logfmt.rs formatter module paralleling src/output.rs, and a dispatch site in the render pipeline. ~150-300 lines, no internal model refactor required.
• Happy to send a PR if the maintainer agrees this is a useful direction. Filing from fuleinist, a recurring GitHub-issue contributor.

[hiSandog]

NDJSON after parsing, filtering, and sorting fits the pipeline use cases described here. How should lines that fail parsing under --input-format=auto be represented: omitted, emitted as structured errors, or preserved in a message field? Machine-readable modes should also guarantee that ANSI and progress output never appear on stdout.

[fuleinist]

Thanks for engaging — this is exactly the design fork I was hoping to surface. Here's my recommendation:

Recommended: structured error record in-band on stdout (NDJSON)

When --input-format=auto fails to parse a line, emit a discriminated record in the same NDJSON stream using a reserved prefix (e.g., __hl_ or _hl_) so downstream jq filters can drop or surface them cleanly:

{"timestamp":"2026-07-06T10:00:00Z","level":"INFO","message":"Request received"}
{"_hl_error":"input_format_auto_detect_failed","_hl_line":42,"_hl_raw":"2026-07-06 [garbled bytes not matching json or logfmt"}
{"timestamp":"2026-07-06T10:00:01Z","level":"ERROR","message":"DB timeout"}

Why in-band stdout (not stderr):

• Pipelines stay simple: hl --input-format=auto --output-format=json | jq -c 'select(._hl_error | not)' — no separate stream merging
• Matches the precedent from vector, fluent-bit, and journalctl -o json for structured log streams
• A user can opt into a "pure records only" view with a single jq filter, or surface errors with jq 'select(._hl_error)'

Why not silent omission:
Loses visibility for debugging the auto-detector itself; users hit a "missing record" bug with no diagnostic.

Why not stderr:
Splits the data across two file descriptors; downstream tooling has to merge them and worry about ordering; breaks the "stdout is the structured payload" contract.

Suggested record shape

Use a single _hl_error key (string) plus _hl_line (1-based source line) and _hl_raw (the original bytes). The leading underscore convention signals "this is metadata, not a parsed field" so downstream filters can confidently pattern-match on it.

One clarifying question

Do you want a --strict mode that fails the process (non-zero exit) when any line fails to parse? My take: no, because hl is positioned as a pipeline-friendly preprocessor — strict mode would push the user to wrap in || true or add a guard, which defeats the purpose. But if you think --strict is useful as an opt-in for CI/log-validation use cases, the natural place is a separate boolean flag, not a behavior change of --input-format=auto itself.

Happy to include the error-record shape + --strict (if you want it) in the eventual PR. Let me know which direction you prefer.