You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf: SentenceTransformer model reloaded on every search request #178
In both server/app.py and server-https/app.py, the SentenceTransformer
model is instantiated inside milvus_search(), meaning it is loaded from disk
on every single query:
defmilvus_search(query: str, top_k: int=5):
encoder=SentenceTransformer(EMBEDDING_MODEL) # ← re-loaded every callquery_vec=encoder.encode(query).tolist()
Impact
Loading a transformer model takes 500ms–2s per call depending on hardware.
In an agentic RAG workflow where search_kubeflow_docs may be invoked
multiple times per conversation turn, this compounds into significant latency.
Proposed Fix
Move the encoder to a module-level singleton initialized once at startup:
Hi! I'd like to work on this issue. I can see the problem - the SentenceTransformer model is being reloaded on every query causing significant latency. I'll move the encoder initialization to module-level as suggested and submit a PR. Thanks!
Hey everyone! Just wanted to give an update that I have resolved this issue.
I’ve implemented a thread-safe, double-checked locking mechanism for lazy-loading SentenceTransformer in both server/app.py and server-https/app.py. This ensures that the embedding model is instantiated only once per process.
This eliminates the ~500ms+ loading penalty per milvus_search call and drastically reduces memory overhead across concurrent workers. I've opened PR #186 with the fix! Let me know what you think.
Sharing some findings from digging into this, since a few PRs are open against it and the ground has shifted underneath them.
The files moved.server/app.py and server-https/app.py were relocated to legacy/ in #207 (703c5f2). Neither path exists at the repo root on main any more, so the open PRs here (#156, #159, #180, #184, #186, #190) all target files that are no longer there and won't apply cleanly without a rebase onto the new locations.
The bug is half-fixed.#233 (a4c92f3) landed a lazy singleton, but only in one of the two files:
File
State on main
legacy/server/app.py
Fixed — has _ENCODER plus a _get_encoder() lazy singleton
legacy/server-https/app.py
Still reloads — SentenceTransformer(EMBEDDING_MODEL) inside milvus_search()
So the behaviour this issue describes is still real, but only in legacy/server-https/app.py.
pytest.ini excludes it by default (addopts = -m "not integration and not slow and not legacy")
pyproject.toml has exclude = ["legacy"] for ruff
CI only compilealls it — nothing runs or deploys it
The live retrieval path is now the MCP server (docs-agent-mcp/mcp-server/), which calls the TEI embeddings service over HTTP via embeddings_client.py rather than loading a model in-process, so it doesn't have this problem at all.
Given that, it might be worth a maintainer confirming whether legacy/server-https is still worth patching before anyone spends time rebasing — otherwise that's six PRs of effort going into code that is archived and unexecuted. Not my call to make, just flagging it so the information is in one place.
Activity
24f3005089 commentedon Mar 27, 2026
Hi! I'd like to work on this issue. I can see the problem - the SentenceTransformer model is being reloaded on every query causing significant latency. I'll move the encoder initialization to module-level as suggested and submit a PR. Thanks!
Ayush-kathil commentedon Apr 6, 2026
Hey everyone! Just wanted to give an update that I have resolved this issue.
I’ve implemented a thread-safe, double-checked locking mechanism for lazy-loading SentenceTransformer in both server/app.py and server-https/app.py. This ensures that the embedding model is instantiated only once per process.
This eliminates the ~500ms+ loading penalty per milvus_search call and drastically reduces memory overhead across concurrent workers. I've opened PR #186 with the fix! Let me know what you think.
abhishekKokadwar commentedon Sep 19, 2026
Sharing some findings from digging into this, since a few PRs are open against it and the ground has shifted underneath them.
The files moved.
server/app.pyandserver-https/app.pywere relocated tolegacy/in #207 (703c5f2). Neither path exists at the repo root onmainany more, so the open PRs here (#156, #159, #180, #184, #186, #190) all target files that are no longer there and won't apply cleanly without a rebase onto the new locations.The bug is half-fixed. #233 (
a4c92f3) landed a lazy singleton, but only in one of the two files:mainlegacy/server/app.py_ENCODERplus a_get_encoder()lazy singletonlegacy/server-https/app.pySentenceTransformer(EMBEDDING_MODEL)insidemilvus_search()So the behaviour this issue describes is still real, but only in
legacy/server-https/app.py.Worth checking whether that's worth fixing.
legacy/looks deliberately fenced off:pytest.iniexcludes it by default (addopts = -m "not integration and not slow and not legacy")pyproject.tomlhasexclude = ["legacy"]for ruffcompilealls it — nothing runs or deploys itThe live retrieval path is now the MCP server (
docs-agent-mcp/mcp-server/), which calls the TEI embeddings service over HTTP viaembeddings_client.pyrather than loading a model in-process, so it doesn't have this problem at all.Given that, it might be worth a maintainer confirming whether
legacy/server-httpsis still worth patching before anyone spends time rebasing — otherwise that's six PRs of effort going into code that is archived and unexecuted. Not my call to make, just flagging it so the information is in one place.chasecadet commentedon Nov 20, 2026
Thank you for reporting this!
We are closing pre-restructure issues targeting the legacy
server/,server-https/,manifests/,kagent-feast-mcp/, or in-processSentenceTransformerprototype code. In #207, #210, #233, #240, and #249,kubeflow/docs-agentmigrated to dedicated TEI (text-embeddings-inference), FastMCP (docs-agent-mcp/mcp-server/), shared modular KFP pipelines (docs-agent-mcp/pipelines/), and Helm charts (docs-agent-mcp/charts/).If you still observe an equivalent issue on the current
mainstack, please feel free to open a new issue using the current issue templates.