Skip to content

perf: SentenceTransformer model reloaded on every search request #178

Closed as not planned

Description

@Maxencejules

Bug Description

In both server/app.py and server-https/app.py, the SentenceTransformer
model is instantiated inside milvus_search(), meaning it is loaded from disk
on every single query:

def milvus_search(query: str, top_k: int = 5):
    encoder = SentenceTransformer(EMBEDDING_MODEL)  # ← re-loaded every call
    query_vec = encoder.encode(query).tolist()

Impact

Loading a transformer model takes 500ms–2s per call depending on hardware.
In an agentic RAG workflow where search_kubeflow_docs may be invoked
multiple times per conversation turn, this compounds into significant latency.

Proposed Fix

Move the encoder to a module-level singleton initialized once at startup:

# Module level — loaded once
encoder = SentenceTransformer(EMBEDDING_MODEL)

def milvus_search(query: str, top_k: int = 5):
    query_vec = encoder.encode(query).tolist()  # reuses loaded model
    ...

Files Affected

  • server/app.py
  • server-https/app.py

Activity

  1. 24f3005089 commented on Mar 27, 2026

    @24f3005089

    Hi! I'd like to work on this issue. I can see the problem - the SentenceTransformer model is being reloaded on every query causing significant latency. I'll move the encoder initialization to module-level as suggested and submit a PR. Thanks!

  2. Ayush-kathil commented on Apr 6, 2026

    @Ayush-kathil

    Hey everyone! Just wanted to give an update that I have resolved this issue.

    I’ve implemented a thread-safe, double-checked locking mechanism for lazy-loading SentenceTransformer in both server/app.py and server-https/app.py. This ensures that the embedding model is instantiated only once per process.

    This eliminates the ~500ms+ loading penalty per milvus_search call and drastically reduces memory overhead across concurrent workers. I've opened PR #186 with the fix! Let me know what you think.

  3. abhishekKokadwar commented on Sep 19, 2026

    @abhishekKokadwar
    Member

    Sharing some findings from digging into this, since a few PRs are open against it and the ground has shifted underneath them.

    The files moved. server/app.py and server-https/app.py were relocated to legacy/ in #207 (703c5f2). Neither path exists at the repo root on main any more, so the open PRs here (#156, #159, #180, #184, #186, #190) all target files that are no longer there and won't apply cleanly without a rebase onto the new locations.

    The bug is half-fixed. #233 (a4c92f3) landed a lazy singleton, but only in one of the two files:

    File State on main
    legacy/server/app.py Fixed — has _ENCODER plus a _get_encoder() lazy singleton
    legacy/server-https/app.py Still reloads — SentenceTransformer(EMBEDDING_MODEL) inside milvus_search()

    So the behaviour this issue describes is still real, but only in legacy/server-https/app.py.

    Worth checking whether that's worth fixing. legacy/ looks deliberately fenced off:

    • pytest.ini excludes it by default (addopts = -m "not integration and not slow and not legacy")
    • pyproject.toml has exclude = ["legacy"] for ruff
    • CI only compilealls it — nothing runs or deploys it

    The live retrieval path is now the MCP server (docs-agent-mcp/mcp-server/), which calls the TEI embeddings service over HTTP via embeddings_client.py rather than loading a model in-process, so it doesn't have this problem at all.

    Given that, it might be worth a maintainer confirming whether legacy/server-https is still worth patching before anyone spends time rebasing — otherwise that's six PRs of effort going into code that is archived and unexecuted. Not my call to make, just flagging it so the information is in one place.

  4. chasecadet commented on Nov 20, 2026

    @chasecadet
    Contributor

    Thank you for reporting this!

    We are closing pre-restructure issues targeting the legacy server/, server-https/, manifests/, kagent-feast-mcp/, or in-process SentenceTransformer prototype code. In #207, #210, #233, #240, and #249, kubeflow/docs-agent migrated to dedicated TEI (text-embeddings-inference), FastMCP (docs-agent-mcp/mcp-server/), shared modular KFP pipelines (docs-agent-mcp/pipelines/), and Helm charts (docs-agent-mcp/charts/).

    If you still observe an equivalent issue on the current main stack, please feel free to open a new issue using the current issue templates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions