feat(preprocessing): process a batch's raw read files concurrently

[corneliusroemer-agent]

Submitting raw reads is slow in batches, and the reason is that preprocessing talks to the raw reads processing service strictly one entry at a time. process_all loops over the batch and each iteration makes a blocking HTTP call that has to come back before the next entry starts. On preview-raw-reads-feature.pathoplexus.org a 100-entry batch ran at about 5.2 s per entry with no overlap at all, which is just the per-entry cost multiplied by the batch size. Nextclade alignment is already batched across the whole batch before this loop, so file processing is the one step that is still serialised per entry.

This sends the files of the whole batch to the service up front from a thread pool and then resolves each entry's result inside the existing per-entry loop, so entries still get processed and returned in the same order as before. Only the waiting is overlapped; metadata processing, nextclade handling and error handling are untouched. Both mates of a paired submission are also now downloaded from S3 at the same time inside the service, which was a sequential loop over the two files.

The thing worth knowing that is not visible in the diff: process_single_unaligned calls the file processing service too, and Config.finalize forces alignment_requirement to NONE whenever an organism has no segments or no nextclade dataset — which is plausible for an organism that only takes raw reads. Parallelising only the aligned branch would have been a no-op for exactly the case this is meant to speed up, so the prefetch sits above the branch and both arms use it.

What I'd like a decision on

The default concurrency of 4 is a guess on the conservative side, and it is the number I would most like a second opinion on. The load that actually reaches the single raw reads processing pod is this number times the number of preprocessing replicas across all organisms pointing at it. Each concurrent request forks its own java -jar readtools.jar on top of the roughly 4.9 GB that the deacon index occupies in an 8 Gi pod, so the ceiling is memory, not CPU. Happy to raise it, but I would want it load-tested against a preview first rather than reasoned about.

There is also a failure mode that this change makes more likely, and I deliberately did not fix it here. raw_reads_processing_service_timeout_seconds (default 600) is a socket timeout on each request. With concurrent requests, a request can sit queued inside the pod — in FastAPI's thread pool, and then behind deacon, which serves filter requests one at a time — and burn its timeout while waiting rather than working. If that timeout fires, the client closes the socket mid-request, and a client that disconnects part-way through is what drives the deacon server into a busy loop it never recovers from: every later filter hangs, the health probe does not notice, and the pod needs restarting. At a concurrency of 4 with a 600 s budget this is unlikely, but the clean fix is a semaphore inside raw-reads-processing so that excess requests wait in FastAPI with no socket at risk yet, instead of waiting inside a half-served deacon call. That felt like a separate change; I would do it before raising the concurrency much.

On the numbers

I did not measure this against a preview. Locally, driving the real client code against a stub service that sleeps 3 s per request (the readtools plus deacon cost measured on the preview), a batch of 20 goes from 60.1 s at concurrency 1 to 15.0 s at 4 and 6.1 s at 16. That only demonstrates that the requests really do overlap and that the setting bounds them — the stub has no server-side cost, so it scales perfectly and the real curve will flatten much earlier, bounded by deacon's serial handling and by how many JVMs fit in the pod. Please don't read those numbers as expected production speedups.

Both new test files use a threading.Barrier in the fake service so that they would time out rather than quietly pass if the calls were still sequential, which a wall-clock assertion would not catch reliably in CI.

🚀 Preview: https://rawreads-parallel-file-pr.loculus.org

[claude]

This PR may be related to: #7110 — the PR description discusses a failure mode where a client-side socket timeout during concurrent requests can drive the deacon server into a busy loop requiring a pod restart, which is exactly the scenario #7110 proposes alerting on.