Real-corpus report: standard CompressedVideo MCAP cannot be canonicalized without AUD

[WilliamK112]

Context

Following the steer in the review of #148, I ran HFlow against a public corpus rather than taking another starter issue. I checked the existing discussions and open PRs first; this is distinct from #48's WebDataset/LeRobot ingestion boundary.

Environment:

• HFlow 0.2.0; the tested doctor/transform/storage sources are identical to current main at f1a4fa2
• macOS 26.1 arm64, Python 3.14.7
• no GPU

Corpus:

• nominal-io/xplane-mcap, a public recorded X-Plane telemetry/video MCAP
• file: xplane.mcap
• 170,784,196 bytes
• SHA-256: 2596d0c450fd3322c147ae396a2abcc6c76840943ce37203ab29919c29cc30d7 (matches the Hub LFS OID)

What is in the file

The MCAP is readable and indexed:

• 214 chunks
• 24,792 messages over 411.466666666 seconds
• video: 24,689 foxglove.CompressedVideo protobuf messages at a 16,666,667 ns median interval (about 60 Hz)
• video/keyframes: 103 messages
• no metadata records

The first decoded video payload reports format h264 and has Annex B NAL types [6, 7, 8, 5] (SEI, SPS, PPS, IDR). The next two have [1]. There is no type 9 access-unit delimiter.

What breaks

A full doctor pass finishes quickly and diagnoses the file accurately:

$ hflow doctor xplane.mcap
[error] missing-provenance
[warning] missing-episode-record
[error] video-not-aud-delimited: video message 0 ...
... video-not-aud-delimited: 24686 further occurrences suppressed
verdict: NOT CONFORMING

Wall time: 0.67 s.

The transform cannot turn this otherwise readable standard MCAP into a canonical episode:

ValueError: pass-through video topic 'video' violates the canonical convention:
no access-unit delimiter (AUD, NAL type 9) found ... (re-encode upstream --
v1 does not transcode CompressedVideo)

app.process() fails in 0.033 s on the first video message. This is expected from the implementation: CompressedImage is transcoded, while foxglove.CompressedVideo is validate-only pass-through.

So the real-corpus boundary here is one level deeper than #48: the container and schema get through the door, but a common valid H.264 representation cannot cross the canonicalization boundary.

What is fast

The indexed MCAP summary and full doctor scan are fast. This is not a scanning-performance problem; it is a transform-capability/contract problem.

What is awkward

The module-level transform contract says "any input MCAP in", and the quickstart invites "any standard MCAP". A user with a standard foxglove.CompressedVideo MCAP can reasonably expect canonicalization to bridge HFlow's stricter H.264 requirements, but the existing path only explains after ingestion starts that re-encoding must happen upstream.

The useful part of the failure is that it is immediate and precise. The awkward part is that HFlow provides no preparation path for this already-MCAP case.

Design question before code

I see two honest shapes:

1. Let the transform conditionally re-encode a non-conforming foxglove.CompressedVideo channel, preserving message timestamps and retaining zero-copy pass-through for streams that already conform.
2. Keep the core transform strict and add a standalone callable + CLI preparation tool for standard MCAP to canonical MCAP, with the quickstart/transform wording narrowed accordingly.

I have not started either implementation. Which boundary would you prefer? If the standalone-tool shape is preferred, I would start with this one corpus and prove that doctor passes after round-trip before generalizing.

[kstonekuan]

Thanks for running this.

Both your shapes assume a re-encode. I don't think you need one. An AUD is a NAL you insert into the Annex B stream, so the slice data is untouched: h264_metadata=aud=insert, in the ffmpeg we already pin. Lossless.

That changes the trade. The reason to keep the transform strict is that silently re-encoding changes pixels nobody asked to change. Inserting AUDs doesn't, so option 1 gets much easier to say yes to.

One thing to know: AUD is one of four canonical constraints, and your file happens to satisfy the other three. A repair path still has to decide what it does about the rest.

Try the lossless insert on this corpus and see whether doctor passes. If it does, the design question mostly answers itself. If you'd rather go straight at one of your two shapes, say so.

You're right about the docs. "Any input MCAP in" is not true today whichever way this lands. I'll fix that separately.

[kstonekuan]

Checked this rather than leaving you to find out.

I took a canonical episode, stripped the AUDs to imitate your file, and ran the stripped stream back through h264_metadata=aud=insert. All 15 messages came back byte-identical to the originals, and the message boundaries were recoverable from the AUD positions alone.

So the lossless path works and the concatenate-then-re-split shape does too. Caveat: that was 15 messages from our own encoder, not 24,689 from X-Plane.

One shortcut if it helps: you already know your message boundaries from MCAP, so you may not need ffmpeg at all. One message is one access unit in your file, so prepending an AUD NAL to each payload gets you there directly.

[WilliamK112]

The lossless path works on the full X-Plane corpus. I implemented it in #184.

On the exact 170,784,196-byte file from this discussion (matching SHA-256 2596d0c...30d7), the transform repaired all 24,689 video messages in 0.800 s. For every message, the output H.264 data was exactly six bytes longer and the complete original payload remained byte-for-byte identical as its suffix; topic and both MCAP times plus the protobuf timestamp/frame_id/format were unchanged. A full doctor pass then returned CONFORMING in 0.282 s (with only the existing missing episode/v1 metadata warning).

The PR keeps all other constraints strict, leaves already-AUD-delimited messages byte-for-byte unchanged, covers protobuf and ROS2 CDR envelopes, and bumps the transform behavior version because canonical output bytes change.

[kstonekuan]

#184 is merged, so this is resolved. Transform now inserts a missing AUD losslessly and refuses everything else as before.

One thing worth recording here rather than leaving in the PR: repairing a missing AUD quietly broke the multiple-access-unit check, because that check counted AUDs and a payload holding two frames read as one after repair. William fixed it by counting pictures from first_mb_in_slice instead, which does not depend on AUDs existing.

Thanks for running the corpus and measuring the round-trip rather than asserting it. That is what made this a small change instead of an argument.