Tolerate per-URL operational errors instead of aborting the run

[jbothma]

What

Make a run tolerate per-URL failures: gather and count them, summarise at the end, and complete regardless.

Why

A kolkhoz-manual job aborted mid-run when one URL's capture raised:

File ".../pravda/pravda.py", line 113, in _finalize_capture
    async with asyncio.timeout(HAR_PROCESSING_TIMEOUT_S):
TimeoutError

Pravda deliberately does not persist this kind of failure — its own code notes that inner HAR-processing / storage timeouts are operational errors that it raises, as opposed to HTTP/browser failures which it stores on the snapshot with error set. The raised error propagated out of asyncio.gather(*(snap(url) …)) in capture_urls and took down the entire run, discarding all the work that had already succeeded.

Change

• capture_urls now catches exceptions per URL, logs a warning, records an OperationalError(stage, url, error), and returns (captures, errors). A URL that fails operationally is simply absent from the captures mapping (Pravda-persisted errored snapshots still appear there, unchanged).
• The extraction loop is wrapped the same way, so one page's extraction failing (LLM/parse/IO) doesn't abort the run either.
• summarise_errors logs, at the end of the run, a total plus counts by stage and by exception type, then the per-URL detail.
• The pipeline now writes outputs and completes even when some URLs failed.

except Exception is deliberate breadth for "let the run finish"; CancelledError / KeyboardInterrupt are BaseException and still propagate.

Operational errors go to the log only, not the database — the database stays reserved for the capture failures Pravda persists on the snapshot.

Testing

• ruff clean.
• Behavioural test: with 4 URLs where 2 raise TimeoutError, capture_urls returns the 2 good captures and 2 recorded errors, and summarise_errors logs the by-stage/by-type/per-URL summary without raising.
• Not yet run end-to-end on the cluster.

🤖 Generated with Claude Code