feat(raw-reads-processing): validate against a warm readtools JVM

[corneliusroemer-agent]

Depends on loculus-project/readtools#1.

The Dockerfile currently pulls the readtools jar from a personal fork, corneliusroemer-agent/readtools v1.1.0-rc1, because the agent account cannot push to loculus-project/readtools to cut a release there. That makes this PR actually buildable and testable rather than pointing at a tag that does not exist, but it must not merge like this — cut the equivalent release on loculus-project/readtools and this becomes a one-line repoint. There is a TODO on the line.

That preview jar is built from the head of loculus-project/readtools#1 and was checked before upload: its CLI output is byte-identical to the released v1.0.0 jar across valid, content-error and structural cases.

Every raw-reads submission forks java -jar readtools.jar, and roughly half that call is classloading and JIT warm-up the process immediately throws away. The number that makes this worth doing: JVM boot to main is 0.036s, so this is not process-startup overhead that a lighter launcher would fix — it is the compiled parse loop being rebuilt from scratch for every submission.

This starts one long-lived readtools JVM alongside the deacon server, warms it, and POSTs each validation to it on loopback. On 100k-read pairs: ~1.3s per validation today, ~0.9s warm, ~0.3s once several run concurrently in that one JVM.

Be aware the ~0.3s figure does not land from this PR alone. process_all in the preprocessing pipeline still calls /process-files one entry at a time and waits for each, so today you get the warm-JVM half, roughly 1.3s to 0.9s. I've deliberately not touched prepro.py here — parallelising that loop is a separate change with its own risks (it needs a concurrency cap, and deacon's serial accept loop becomes the next bottleneck). What this PR does is make the service able to absorb that concurrency when it comes, and doing it inside one JVM is what makes it affordable at all: the resident deacon index leaves too little room under the 8Gi limit to fork several JVMs side by side.

Measured on a live preview, not just locally

Four 53 MB paired-FASTQ submissions (SRR21755542, 1,455,282 reads / 190 Mbp) through the preview for this branch, all NO_ISSUES, then approved.

step	before, same file	with the warm server
readtools validation	~3 s	0.90 / 0.96 / 1.01 / 1.13 s
whole /process-files	~4.6 s	~2.3 s
submit → PROCESSED	8.2 s	6.2 s first, then 3.1 s
deacon	0.57 s	0.19 – 0.30 s
pod memory, peak	4.89 – 5.05 GB	5.15 GB

That is about 3x on the readtools step in production — a bigger win than the ~1.5x measured on a 16-vCPU dev box, because the pod pays more for a cold JVM than a fat container does. Worth knowing for anyone who benchmarks this kind of change locally and assumes the local ratio carries over.

The pod's own startup log shows the JIT warm-up curve directly, which is the clearest evidence that the mechanism is what we think it is rather than noise:

23:00:46 Starting readtools validation server: java -Xmx1g -jar ... server --threads 4
23:00:48 warm-up 1/3: 1.06s
23:00:48 warm-up 2/3: 0.71s
23:00:49 warm-up 3/3: 0.55s
23:00:49 readtools validation server ready

Warm-up costs 3 seconds of pod startup, against a startupProbe that allows 600 s. Memory moving 4.89 → 5.15 GB is the resident JVM, and is why I have not touched the pod's 8 Gi sizing — the deacon index still dominates it.

Two things the live run taught us that the bench could not

The service used to log nothing at all about readtools. The only breadcrumb was a debug line inside the subprocess branch, so on the warm path the duration had to be reconstructed from the gap between the last "Successfully downloaded" line and deacon's first line. That is fixed here: both paths now log the elapsed time and which one ran. Found only by trying to measure the thing in production.

A Connection refused to loculus-raw-reads-processing:5000 is what an unscheduled pod looks like. The pod requests 8 Gi and sat Pending/unschedulable for ~15 minutes because no preview node had that much free, producing zero log lines. Submissions during that window failed with a clean Internal Error ... Connection refused, which reads like an application bug and is not one. Incidentally this is a decent negative control: the pipeline does surface a missing validator rather than silently passing the submission, which is the failure mode that would actually matter.

Full validation: measured, and off by default

readtools_full_validation validates every read instead of the first 100k per file. Measured on the same 53 MB submission (727,641 reads per mate, so 7.3x more data than the default cap):

	quick (default)	full
fork per call	~1.15 s	~8.6 s
warm server	0.86 s	7.40 s
peak memory, one validation	0.39 GB	1.33 GB

Two things worth knowing before anyone turns this on.

The warm server barely helps in full mode. The win it provides is a roughly fixed amount of reclaimed JIT compilation, so it is most of a ~1 s job but only ~14% of an ~8.6 s one. The ~3x this PR buys in production becomes about 1.15x with full validation on.

Memory does not scale with the file, it scales with the limit. ValidatorWrapper sizes the pairing Bloom filter as readCountLimit / 2, so full mode allocates for 50M reads even for a tiny submission, and copies it per mate. Four concurrent full validations against a 1 GB heap ran out of memory on three of the four. That is why the flag defaults to off, and why turning it on needs readtools_server_max_heap raised or readtools_server_threads lowered rather than just flipping the boolean.

That experiment also turned up two real bugs, both fixed here and in loculus-project/readtools#1: the OOM left the HTTP exchange unanswered so callers hung for their full timeout instead of erroring, and any non-zero exit was reported to the submitter as an invalid file - meaning our validator running out of memory would have been shown to someone as a problem with their perfectly good FASTQ.

What you should push on

A long-lived JVM is a genuinely new failure mode. Today a validation bug that leaks memory or wedges dies with the process and costs one submission; now it can affect every subsequent one. Three things buy that back, and none of them is free:

readtools_server_enabled: false restores the old fork-per-call path without a rebuild. I did not add an automatic fallback when the server is unreachable, on purpose — a silent fallback would hide a permanently broken server behind merely-slower validations, and I'd rather that page someone. Say the word if you'd prefer it fall back anyway.

The JVM runs under an explicit -Xmx with concurrency capped, so it cannot grow into deacon's memory. I did not raise the pod's 8Gi request: measured resident use of the warm JVM was ~440MB against a 1g heap, well inside the headroom the deacon index leaves, and this actually lowers peak use compared with several concurrent forked JVMs. Flagging it because it's the kind of thing worth disagreeing with.

Startup blocks until the JVM reports healthy, so no submission is ever served by a cold JVM. That makes pod startup slower by the warm-up, which the existing startupProbe already tolerates (600s allowance, warm-up takes seconds). /health now also fails if the JVM has exited, or if its own probe — a real one-read validation, not a liveness ping — fails.

The part I care most about being right

The submitter-facing error message must not depend on which path ran. Rather than defining a JSON verdict schema, the server returns the exact stdout, stderr and exit code the CLI would have printed, so _parse_validation_error is untouched and both paths are literally the same text. test_server_and_subprocess_agree runs each fixture through both and compares the resulting message string, so any drift fails the suite rather than silently reaching a submitter.

There's also a test that fires mixed valid and invalid files concurrently and checks no request gets another request's verdict, which is the failure I'd most fear from sharing a JVM.

🚀 Preview: Add preview label to enable

[claude]

This PR may be related to: #7187