fix: move usage persistence off the inference response path

[tsonglew]

Problem

infer() currently waits for usage inserts, the API key's last_used_at update, transaction commit and connection close before returning. On a deployment with slow persistent storage, model prediction took 23–104 ms while backend response times were about 2–3 seconds. An isolated WAL write/commit/close probe took 541–1695 ms on the data volume. Usage persistence can also turn an otherwise successful prediction into 503 USAGE_UNAVAILABLE.

Change

Move usage persistence to one lifespan-managed background writer per process. Requests enqueue a small event and return the original prediction, including its token usage. The writer reuses its SQLite connection, commits batches of up to 100 events, flushes partial batches after a one-second collection interval, and coalesces last-used updates without moving timestamps backwards.

The queue is bounded to 1,000 pending events plus one in-flight batch. Shutdown stops accepting events and attempts to drain for up to five seconds without blocking the ASGI event loop. API-key authentication, revocation and admin writes remain synchronous; there is no schema change.

Telemetry tradeoff

Persisted usage is now eventually consistent and best-effort, not a billing/quota ledger. Queue overflow drops new statistics with rate-limited warnings. SQLite failures roll back where possible and drop/log the affected batch; later batches reconnect. Failed batches are deliberately not retried because an uncertain commit could double-count them. A forced exit or shutdown timeout can lose pending events, and an in-flight SQLite call may outlive the shutdown wait. These semantics are documented in both READMEs.

Validation

• Python 3.12: python -m pytest backend/tests -q (55 tests).
• HTTP regression tests for both /v1/systemone and /internal/playground hold the background write behind an event and confirm responses complete before it is released, including simulated storage failure.
• Writer tests cover batching/connection reuse, timestamp coalescing, timed flush, concurrent overflow, bounded shutdown, transaction rollback, reconnection and restart.
• Existing authentication, revocation, validation and concurrent inference tests remain covered; statistics assertions now wait for eventual persistence.
• No GPU performance claim: model calls are mocked in the tests. The deployed end-to-end latency still needs remeasurement after rollout.