Skip to content

feat(lapis): distinguish a SILO timeout from SILO being unreachable - #1867

Open
corneliusroemer wants to merge 1 commit into
GenSpectrum:mainfrom
corneliusroemer-agent:silo-request-observability
Open

feat(lapis): distinguish a SILO timeout from SILO being unreachable#1867
corneliusroemer wants to merge 1 commit into
GenSpectrum:mainfrom
corneliusroemer-agent:silo-request-observability

Conversation

@corneliusroemer

Copy link
Copy Markdown
Contributor

A timeout was reported as "Could not connect to silo", which is wrong and misleads whoever reads it. HttpTimeoutException means LAPIS did connect, sent the request and gave up waiting - SILO is reachable and may be answering other callers perfectly well. Only ConnectException means not reachable.

Related to #1866 in that the misleading exception came up while figuring out what caused info to return silo connection error.

SiloTimeoutException maps to 503, the same status the other SILO-unavailable paths use.

Also edits the message for an uncaught silo exception: the problem isn't necessarily connection failure, could be anything so message shouldn't assume.

PR Checklist

  • All necessary documentation has been adapted.
  • All necessary changes are explained in the llms.txt.
  • The implemented feature is covered by an appropriate test.

Copilot AI lite review requested due to automatic review settings September 10, 2026 13:15
@vercel

vercel Bot commented Sep 10, 2026

Copy link
Copy Markdown

@corneliusroemer-agent is attempting to deploy a commit to the cov-spectrum Team on Vercel.

A member of the Team first needs to authorize it.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

User-facing error messages now include full SILO URIs (host/details) that are returned in API responses, which can leak internal service addresses.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR improves SILO error reporting in the LAPIS service by distinguishing request/connection timeouts from true “SILO not reachable” scenarios, and by mapping timeouts to HTTP 503 to align with other SILO-unavailable cases.

Changes:

  • Introduces SiloTimeoutException and throws it on HttpTimeoutException with a more specific timeout message.
  • Updates “could not connect” messaging to be more accurate and less assumption-based for non-connect exceptions.
  • Adds a MockServer-based test to verify slow /info responses trigger SiloTimeoutException.
File summaries
File Description
lapis/src/main/kotlin/org/genspectrum/lapis/silo/SiloClient.kt Adds timeout-specific exception and messaging; refines connect vs other failure messages.
lapis/src/main/kotlin/org/genspectrum/lapis/controller/ExceptionHandler.kt Maps SiloTimeoutException to HTTP 503 with warning-level logging.
lapis/src/test/kotlin/org/genspectrum/lapis/silo/SiloClientTest.kt Adds coverage ensuring slow /info triggers the new timeout exception.
Review details

Suppressed comments (2)

lapis/src/main/kotlin/org/genspectrum/lapis/silo/SiloClient.kt:257

  • This generic SILO error message also embeds the full URI (including host). Since Throwable messages are returned in error responses (including 500s), consider removing host details from the message to avoid exposing internal topology to clients.
            val message = "Error talking to silo at $uri: ${exception::class} ${exception.message}"

lapis/src/main/kotlin/org/genspectrum/lapis/silo/SiloClient.kt:295

  • The timeout message includes the full SILO URI (including host) and is returned to clients as the 503 body. Consider limiting it to the endpoint path to avoid leaking internal service addresses while still distinguishing timeout vs connect-timeout.
        val elapsedMillis = (System.nanoTime() - startedAtNanos) / 1_000_000
        return when (exception) {
            is HttpConnectTimeoutException -> "Timed out connecting to silo at $uri after ${elapsedMillis}ms"
            else -> "Timed out waiting for a response from silo at $uri after ${elapsedMillis}ms"
        }
  • Files reviewed: 3/3 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

}
} catch (connectException: ConnectException) {
val message = "Could not connect to silo: ${connectException::class} ${connectException.message}"
val message = "Could not connect to silo at $uri: ${connectException::class} ${connectException.message}"
A timeout was reported as "Could not connect to silo", which is wrong and
misleads whoever reads it. HttpTimeoutException means LAPIS did connect, sent
the request and gave up waiting - SILO is reachable and may be answering other
callers perfectly well. Only ConnectException means not reachable.

Seen in the wild: LAPIS evicted itself from its Kubernetes Service after its
/sample/info readiness probes failed, while SILO was answering /info in under a
millisecond throughout. The message pointed at networking; the cause was
callInfo()'s 100 ms budget. The new message therefore also names the target URI
and how long the call actually took, since neither is otherwise visible - a
/sample/info that fails in ~140 ms gives no hint that the bound was 100 ms
rather than the probe's 5 s.

SiloTimeoutException maps to 503, the same status the other SILO-unavailable
paths use.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants