Wave 8: UDP, QUIC, IKEv2, tunnels, the cleartext inventory, and the docs

[unprovable]

The last wave of the backlog. PR-40, PR-41, PR-42 and PR-43, plus a documentation pass, built by five agents concurrently and integrated here. 1149 tests -> 1508.

pcapscan/datagrams.py -- the UDP seam (new, integrator)
pcapscan/quic.py -- PR-40, QUIC Initial decryption (RFC 9001)
pcapscan/ikev2.py -- PR-41, IKEv2 SA proposals over UDP
pcapscan/tunnels.py -- PR-42, GRE/ERSPAN/VXLAN/GENEVE (#12, offline)
pcapscan/cleartext.py -- PR-43, the inventory of what is not protected
docs/ -- eight pages; README cut from 344 lines to 228

-------------------------------------------------------------------------- The blocker: this tool could not read a UDP packet -------------------------------------------------------------------------- decode_frame walked to TCP and returned None for everything else, with no counter and no mention in the stats. Measured over the capture corpus that was 30,530 of 159,929 packets -- 19% -- dropped in silence, including 11,722 QUIC packets on port 443. A monitor whose purpose is to report what cryptography is in use was blind to the transport that now carries a growing share of the web's TLS 1.3 handshakes, and the three UDP protocols this wave adds would each have had to invent their own plumbing to get at it.

So the seam came first, and it is the integrator's work rather than any one PR's:

• decode_datagram, sharing the link-layer, VLAN, IPv4-options and IPv6-extension-chain walk with decode_frame through a new _walk_network. One walk, not two. Every defect that walk has ever had -- the fixed 14-byte Ethernet header, the fixed 20-byte IPv4 header, the unwalked IPv6 chain -- produced plausible wrong offsets rather than an error, and a second copy would be a second place to reintroduce them. It also refuses non-initial IPv4 fragments, which TCP never needed and a near-MTU QUIC Initial very much does.
• pcapscan/datagrams.py, which keys UDP flows and discovers handlers rather than requiring registration: HANDLER_MODULES names the modules, each is imported if present and skipped if not. That is what let four people write four protocols at once without editing one shared list.
• Wired into SessionBuilder, so the CLI, iter_sessions and the upload sandbox all get UDP without further change.

-------------------------------------------------------------------------- PR-40 -- QUIC, and the post-quantum traffic nobody could see -------------------------------------------------------------------------- This is the headline and it is a large one.

RFC 9001 derives Initial packet protection from a published constant salt and the client's Destination Connection ID, which is in the clear in the header. No keys are needed and none are used: the secrecy of a QUIC handshake never rested on the Initial keys. The handler walks long headers, removes header protection, opens the packet with AES-128-GCM, reassembles CRYPTO frames by offset across coalesced packets, and hands the result to pcapscan.sessions.Session -- the same object the TCP path builds.

That reuse is the design. There is no second ClientHello parser and no second document shape, so JA4, the extension census, ECH, the key-share group, the resumption logic and the post-quantum classification all arrived with no change to cryptomon/analysis.py, the CBOM, the CSV export or the dashboard.

Over the corpus: 86 QUIC flows claimed, 468 Initials seen, 468 decrypted, 0 undecryptable, 86 ClientHellos and 60 ServerHellos recovered. 69 of the 86 hellos were still in flight across more than one packet, so a single-packet reader would have got a fragment of each.

clients offering a post-quantum hybrid 69 / 86 = 80.2% (TCP: 71.6%)
servers accepting one 35 / 60 = 58.3% (TCP: 20.4%)
ECH offered 74 / 86 = 86.0% (TCP: 36.5%)
ALPN h3 on all 86; no TCP hello offers it

Whole-corpus effect, measured before and after:

hybrid key exchanges 81 -> 116 (+43%)
quantum-safe fraction 14.4% -> 18.6%
TLS 1.3 sessions 499 -> 585 (+17%)

Sixty-nine post-quantum key exchanges were happening over QUIC and this tool could not see one of them. Thirty-five of them completed.

The honest bound on that: of the 11,722 QUIC packets, 10,984 are short-header 1-RTT and will never be read by any keyless tool. They are now counted, not read. The reachable population is 762 long-header packets and the 468 Initials inside them, and that is the whole handshake, which is the point.

Verification was not straightforward and is worth recording. tshark 3.4 on this machine cannot decrypt QUIC v1 -- it predates RFC 9000, carries draft-29's salt, and fails every Initial with a checktag error while calling version 1 "Unknown". Rather than take that as a disagreement, the agent decrypted one packet under three candidate salts to show which was right, then built a different oracle: it re-framed all 86 decrypted CRYPTO streams as ordinary TLS-over-TCP in a synthetic capture and let tshark's TLS dissector read them. 86/86 parsed as ClientHello, SNI agreed 86/86, and tshark reported key-share code points 0x6399 and 0x11EC exactly where we name X25519Kyber768Draft00 and X25519MLKEM768. RFC 9001 Appendix A's published test vectors are pinned in the test file as well.

-------------------------------------------------------------------------- PR-41 -- IKEv2
-------------------------------------------------------------------------- IKEv2 is the one protocol here that states its cryptography instead of implying it: RFC 7296 puts the encryption algorithm, the PRF, the integrity algorithm and the key exchange on the wire as four separately numbered transforms, in the clear, before anything is encrypted. So almost nothing is inferred.

Transform names are chosen to be exactly the strings cryptomon/analysis.py already knows, so no additions to the marker tables were needed. The RFC 9370 case is the good one: a KE=x25519 with ADDKE1=ml-kem-768 is emitted as the single name x25519+ml-kem-768, which normalises to x25519mlkem768, in which the existing table finds one post-quantum marker and one classical one and answers hybrid -- the true answer, and the one neither half gives alone.

Port 4500 carries IKE, ESP and NAT keepalives on the same port, distinguished by a four-byte non-ESP marker. Getting that wrong means parsing ciphertext as an IKE header and reporting a proposal made of noise; a marker-shaped datagram is therefore tried both ways. ESP is reported as present-and-opaque, which is a useful finding rather than a failure -- and opaque_flows is a new readiness field so that "we could not see it" is never reported as "there was none".

There is no IKEv2 in the corpus, verified independently rather than taken on trust: 0 packets on UDP 500 or 4500 in all 160,221. Everything quoted about its behaviour comes from eight generated fixtures. The one corpus-backed claim is the negative: 30,531 real datagrams offered to the detector, 0 claimed, 0 exceptions.

The CBOM now fills ikev2TransformTypes -- encr, prf, integ, ke -- which is a CycloneDX 1.6 field this project has had a schema for and nothing to put in. esn and auth are deliberately left empty: ESN is not carried, and IKEv2's authentication method is negotiated inside the encrypted IKE_AUTH, so a passive observer never sees it. Emitting either would invent a value the traffic did not contain.

-------------------------------------------------------------------------- PR-42 -- tunnels (#12, the offline half)
-------------------------------------------------------------------------- Every measurement in this repository comes from a capture taken on an endpoint. That is not how anyone monitors a corporate network: they mirror a port, and the traffic arrives wrapped in GRE inside IP. To this tool, every one of those frames was unreadable.

Unwrapping happens before framing, in its own module, not as recursion inside decode_frame. Peeling a tunnel does not yield a transport header -- it yields another whole frame needing the entire walk run over it again -- and putting that inside decode_frame would hand every caller, the live eBPF adapter included, a recursion it did not ask for with an attacker-chosen depth. The stage generalises: GRE (RFC 2784/2890, with the variable header its flag bits imply), ERSPAN Types I, II and III, VXLAN, GENEVE, IP-in-IP and 6in4, bounded to four levels.

ERSPAN Type I is the trap and is handled: it is distinguished from Type II only by the GRE sequence flag, so a parser assuming a header is always present eats eight bytes of the real Ethernet frame and then decodes plausible nonsense.

The proof is the one that matters: tls12_certificate.pcap wrapped frame-by-frame in ERSPAN Type II. Today's pipeline drops 19 of 19 frames and reports nothing. With the stage, the pipeline returns the identical session document -- == on the dict, same ciphersuite, same sha384.badssl.com, same timestamp to the microsecond.

The ERSPAN truncation bit is carried through to a counter and onto the session, because a switch saying "I cut this frame short" is labelled missing data, and both times this project has reported a wrong answer that looked right -- truncated ClientHellos parsed as whole, tshark's desegmenter giving up in silence -- the data was missing without saying so.

The live eBPF path is still blind to tunnels, and #12 should not be closed as though it covered both halves.

-------------------------------------------------------------------------- PR-43 -- what is not protected at all
-------------------------------------------------------------------------- Scope changed on evidence. The backlog's PR-43 leads with DTLS; there is not one DTLS record in the corpus, nor any WireGuard. What is there is 30,531 UDP datagrams, most of them entirely unencrypted. "This uses RSA-2048, which a quantum computer breaks" and "this uses nothing, which anyone on the segment breaks today" are both CBOM findings, and the second is actionable this afternoon.

Fourteen protocols identified by content: DNS (with DNSSEC state read properly, from the EDNS0 DO bit and the RRSIG/DS/DNSKEY records), mDNS, LLMNR, NetBIOS-NS and -DGM, DHCP, NTP, CLDAP, STUN/TURN, HSRP v1 and v2, SNMP v1/v2c/v3, syslog, SSDP and BigFix. 1,058 flows claimed over the corpus, zero false positives, and no flow on port 443 taken.

Of the non-QUIC UDP: 61.4% has no confidentiality and none of it is encrypted. The four-rung protection ladder earns its place on the HSRP result -- all 15,867 hellos carry an all-zero plaintext auth field and a keyed-MD5 authentication TLV, so the gateway redundancy is authenticated by a 1991 hash. That is obsolete, which is neither none nor authenticated only, and the distinction is the finding.

Also: zero DNSSEC and zero EDNS0 in 2,134 DNS messages -- not one client on this network even asks for signatures. 764 of those queries (35.8%) are for HTTPS resource records, which is where the ECH configuration lives: this corpus asks for ECH keys in the clear 764 times.

Nothing identifying is recorded. Not a query name, not an mDNS instance name, not a NetBIOS name, not a DHCP hostname. Service types and record types survive; names are counted via keyed BLAKE2 tokens under a per-process random key, because a stable digest of a hostname is a hostname when the name space is a corporate network. No community string, HSRP auth string or ICE username is ever written down -- only that one is present and in the clear.

-------------------------------------------------------------------------- Documentation
-------------------------------------------------------------------------- README.md restructured to route rather than explain (344 -> 228 lines), and docs/ added: install, configuration, offline analysis, the service, the live sensor, reading a report, architecture, and a 623-line troubleshooting page mined from this repository's own commit history and module docstrings. Every command with a prompt was run; anything needing Linux, root, bcc or a live interface is marked "not verified on this machine" rather than implied.

Thirteen inaccuracies in the old README are fixed, among them: port 990 documented as sftp when it is FTPS, "IPv6 support" still listed as a TODO four waves after it shipped, http://0.0.0.0:8000/docs offered as a browsable address, and an enp0s1 interface default that does not exist.

-------------------------------------------------------------------------- Defects found and fixed

• Every non-SSH record was counted as a TLS session. Summary.add fell through to _add_tls, so the 1,058 cleartext UDP flows arrived in the readiness report as resumed TLS sessions -- moving the quantum-safe fraction by inventing sessions that never negotiated anything. Found independently by two agents. Fixed with an explicit branch, plus _add_ikev2 for the protocol that does have a key exchange to classify.
• The upload sandbox was TCP-only. pcapscan/sandbox.py never got the UDP seam, so a capture uploaded through the browser was analysed by a strictly weaker parser than the same file on the command line -- silently, because a report of nothing looks like a capture with nothing in it.
• --no-certificates lost the data it promised to keep. --help said "keep the raw chain"; the flag passed lambda _body: [], so the chain was parsed to nothing and the DER discarded. Now a KEEP_DER sentinel.
• Four scripts were committed without their executable bit, so ./ubuntu-setup.sh was "permission denied" on a fresh clone -- and deploy/README.md's own quickstart begins sudo ./create-service.sh.
• BrokenPipeError reached stderr on | head and | mongoimport, both of which the README recommends: catching the write is not enough, because CPython flushes stdout again at shutdown where no except can reach it.
• The UDP flow counters did not sum. flows_unrecognised was only incremented at the 64-datagram cap, so a two-datagram exchange was counted nowhere and 1,071 of 1,164 flows fell in the gap. A stats line whose parts do not add up is worse than no stats line. Now flows_undetected closes it: 1058 + 86 + 8 + 18 + 2 = 1172.
• DatagramRouter.push was unguarded while _detect was guarded. One handler raising ended the capture for every other protocol sharing the loop.
• DES, DES40 and the IKEv2 ENCR_DES family had no strength entry, so they were reported with no symmetric strength at all. This also fixes TLS_RSA_EXPORT_WITH_DES40_CBC_SHA on the TLS side.
• The upload form read "up to 0 MB" below a 1 MiB cap, and said MB for MiB.
• And one of mine, in the seam: self.router = router or None, three lines below a comment warning that DatagramRouter defines __len__ so an empty one is falsy. The comment now names the incident.

-------------------------------------------------------------------------- Verified

pytest -q 1508 passed, 19 skipped (was 1149, 18)
pytest -m smoke -q 1370 passed (was 1032)
with a corpus MongoDB 1523 passed, 4 skipped
compileall clean
fuzz, 11 targets 249,193 cases, 0 failures
tshark oracle no drift

The whole real corpus, end to end: 2,404 sessions (1,346 TLS, 1,058 cleartext), 624 key exchanges performed, 116 hybrid, 508 classical, 61 post-quantum offers refused, quantum-safe fraction 18.6%.

-------------------------------------------------------------------------- Fixes from the code review of both waves
-------------------------------------------------------------------------- A high-effort review over main..HEAD found 15 defects; 13 were real and are fixed here. None was found by the test suite, which was green at 1508 throughout, and four existed only because two halves written separately met for the first time.

• A capture starting with the server's packet lost the QUIC ClientHello. pcapscan.datagrams calls whichever end sent first the client, so a capture beginning between the client's Initial and the server's reply inverts that and every later client Initial arrives with from_client=False. _candidates offered header.dcid only when from_client was already true, so the list came back empty and a readable Initial was counted undecryptable -- reproduced on a real corpus flow as initials_decrypted 1 -> 0. It also made direction_corrected dead code. The DCID is now offered again as a last resort; on a server Initial it is the client's SCID and will not verify, so the wrong guess costs one AEAD check. Two regression tests, both mutation-checked against the defect.
• An IPsec SA that encrypts nothing was reported as AES-256. RFC 4543's ENCR_NULL_AUTH_AES_GMAC is null encryption with a GMAC, and SYMMETRIC_BITS matches by substring in order, so AES_256 won. For a tool whose headline distinction is encrypted versus cleartext that was the worst single answer it could give. NULL_AUTH now goes first, and the CBOM calls the transform a mac rather than a block-cipher.
• The dashboard told every deployment its alerts carried no direction, in a red block directly above rows reading (server). charts.py looked for 'from '; integration had spelled it (server) to match pcapscan/export.py. The bucket now carries direction as its own key as well as in the name, so the "Sent by" column works and the two spellings cannot drift apart again.
• The timeline dropped the newest buckets silently. One document with a broken ts made an all-time chart render years of zeros from 1970 and omit today. The oldest go now, and the number dropped is in notes.
• DatagramRouter.finish() was unguarded while push and _detect were guarded -- and finish is where the expensive parsing of attacker bytes happens, so one raise ended the whole capture in a traceback.
• api.env.example shipped ROOT_PATH=/cryptomon live, and create-service.sh installs it whether or not nginx is in front. The default install 404'd /docs and every post-upload redirect.
• A DNSKEY answer with no signature was called authenticated -- only RRSIG authenticates an RRset -- and dnssec_signed froze at the first datagram, so a flow that saw a query then an RRSIG reported 0.
• The dashboard asserted "not one certificate is quantum-safe" over a truncated rollup, and drew a truncated stack against a full denominator. Both now account for what the row limit left out.
• MONGO_PORT= empty never unpublished the database, though two files said it would; compose's :- substitutes the default for an empty value too.

Verified after: 1515 passed / 19 skipped on Python 3.10, 3.11 and 3.12; smoke 1375; 237,415 fuzz cases clean; oracle unchanged; eBPF compile, verify and behaviour green in the pinned container. The corpus results are byte-identical -- 2,404 sessions, 624 key exchanges, 116 hybrid, 18.6% -- because every one of these was an edge this corpus does not exercise, which is exactly why reading found them and running did not.