Skip to content

feat: OpenAI-compatible LLM provider (#616) - #648

Merged
coaxk merged 19 commits into
mainfrom
feat/616-openai-compatible-provider
Nov 1, 2026
Merged

feat: OpenAI-compatible LLM provider (#616)#648
coaxk merged 19 commits into
mainfrom
feat/616-openai-compatible-provider

Conversation

@coaxk

@coaxk coaxk commented Nov 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

What this changes

Adds an OpenAI-compatible LLM provider next to Ollama. Settings → Integrations offers Provider: Ollama | OpenAI-compatible (base URL with /v1, optional API key, text model, vision setting), also settable through LLM_PROVIDER / OPENAI_* env vars. All three LLM features run on it: the title-language guess, Arena explanations and the vision check. One code path serves local servers (Ollama /v1, Unsloth Studio, llama-server, LM Studio, vLLM) and hosted APIs (OpenAI, OpenRouter). It works out per server and model which JSON and reasoning-off request formats are accepted and remembers them, and it never returns an all-reasoning reply as a silent empty answer. Test connection runs six named checks. The API key is masked on read, never logged, never in telemetry and never overwritten by its own mask. The title-language prompt now sends library name + path relative to the library root, for both providers.

Why

Closes #616.

Testing

  • PYTHONPATH=src pytest -q passes locally: 3435 passed, 9 skipped (after rebase onto 7c9e0b4)
  • Regression tests added throughout (client, config, routers, telemetry, settings UI). The 29 new settings UI tests were run against the old settings.jsx: 26 of 29 fail there. Full vitest 53 files / 366 tests; check:frontend rc 0 with bundles unchanged; ruff 0.15.15 clean on src/ tests/.
  • Smoke-tested live in an isolated container running this branch's source (fresh DB, no media mount, no docker socket, telemetry off), not on the dev stack:
    • Ollama /v1, gemma4:12b-it-qat: test connection 6/6; /api/enrichment/lang "Shingeki no Kyojin" → ja; /api/vision/check parsed a real Plex thumbnail; Arena explanation non-empty (explainer called on the live client with a synthetic result, since the container has no subgen).
    • OpenRouter google/gemini-2.5-flash (reasoning) and openai/gpt-4o-mini: the same four checks pass on both; neither needed a format fallback or a reasoning retry (no #616 ... retrying lines in the logs).
    • Not tested live: OpenAI direct, Unsloth Studio, llama-server. Implemented from their documentation and source; the CHANGELOG says so.
  • Prompt eval (scripts/eval_title_language.py --arm both, 60 titles scored against the arrs' originalLanguage, gemma4): old 33/60, new 33/60 (en 27/30, other 6/30). Acceptance was new ≥ old − 1.

Migration impact

N/A: no schema change. New settings persist through the existing config store.

Telemetry impact

  • No new top-level field. Two booleans inside the existing integrations object, openai_local and openai_hosted, set only when the OpenAI-compatible provider is active with a base URL and model. No URL, key or model name enters the payload.

Known limits

  • Pulling and unloading models stay Ollama-only (no OpenAI equivalent); those controls are hidden for the OpenAI-compatible provider.
  • The Integrations rail item is still labelled "Ollama" when the OpenAI-compatible provider is active (renaming it would change what Ollama users see; follow-up).

🤖 Generated with Claude Code

https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi

coaxk and others added 19 commits November 1, 2026 19:23
@coaxk @claude
docs(616): OpenAI-compatible provider design and server research
6d28f92 
Design approved by Judd 2026-10-31. Includes the live Ollama v1 probe
results and the Unsloth, llama-server, OpenAI and OpenRouter research.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk @claude
docs(616): implementation plan for the OpenAI-compatible provider
06802d0 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk @claude
feat(616): settings for the OpenAI-compatible provider
2327447 
Six persisted, env-guarded settings: llm_provider, openai_base_url,
openai_api_key, openai_model, openai_vision, openai_server_type. One set
of normalisers is shared by the env path and the persisted-override path:
whitespace stripped, keyword values lower-cased, model ids kept as typed,
and an unknown provider or server type falls back to the default with a
warning instead of failing config load.

Built by a DeepSeek flash worker (dspilot run 563207aa), reviewed here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk @claude
feat(616): OpenAI-compatible client core
7cf44ca 
Built by the DeepSeek flash worker (dspilot run dbae24ba) from brief L4;
reviewed and accepted. Three docstring/comment rationales reworded in
review where the builder had invented a reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk @claude
feat(616): parameter negotiation and reasoning retry
22d0a7b 
A 400 naming an optional parameter drops, swaps or steps it down and
retries, at most 3 times per call; the accepted shape is remembered per
base URL, model and profile. An empty reply cut off by the cap while the
model reasoned is retried once with a raised cap.

Built by the DeepSeek flash worker (run a45279e0, brief L7). Review added
three tests: raised cap never below the request, use_learned False
overwrites the stored shape, no bare-word parameter match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk @claude
feat(616): rate-limit handling and breaker for the LLM client
8f35f5f 
Built by DeepSeek run 7ead6df9 (brief L9), reviewed and re-verified here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk @claude
feat(616): vision and model listing for the OpenAI-compatible client
82745d4 
Built by the DeepSeek flash worker (dspilot run fa0222a5, brief L10),
reviewed and accepted. Review fixup: dropped an unused noqa directive.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk @claude
feat(616): a rejected API key opens the LLM breaker
24a0e63 
A 401 or 403 from /chat/completions now records a breaker failure, so a
revoked hosted key stops being retried every call and the client cools
down instead. 400 and 404 still count as success and models() stays
outside the breaker. Built by the DeepSeek flash worker (dspilot run
cbfbd23c, brief L12) and reviewed in the main session: full suite 3273
passed, 3 mutants caught.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk @claude
docs(616): record the 401 and 403 breaker decision in the plan
7e05205 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk @claude
feat(616): one active LLM client on app.state.llm
cc01047 
build_llm_client() picks Ollama or the OpenAI-compatible client from
llm_provider; every caller uses app.state.llm and catches LLMError.
GPU gate only for local providers, bulk enrichment stops on cooling
down, unload and pull are provider-aware, health and probes list
/models for the OpenAI-compatible client.

DeepSeek flash run f0a7b2f7 (L15), reviewed and accepted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk @claude
feat(616): credentials and Test connection for the OpenAI-compatible …
5b8725e 
…provider

Adds the openai and llm credential integrations, skips masked secrets on
save for every integration, and a step-by-step Test connection
(reachable, key, model, JSON, thinking, vision) on a fresh client.

Built by dspilot run 4d51def3 (brief L16).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk @claude
feat(616): language prompt sends library name and relative path; eval…
8ad0f68 
… script

The title-language prompt now names the library a row is filed in and its
path inside that library instead of the raw canonical path. A reply that is
neither a JSON object nor a bare language code raises NotJSONReply and is
stored as an error row; before, a prose reply was parsed by its first word,
so "It is Japanese." was stored as Italian. OllamaClient.generate accepts
schema_name so both clients take the same arguments.

scripts/eval_title_language.py compares the old and new prompts against the
arrs originalLanguage without writing to the enrichment store.

Built by the DeepSeek flash worker (dspilot run eb2c0297, brief L18).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk @claude
docs(616): plan Task 8b, clear guesses the old fallback stored
f56ad77 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk @claude
fix(616): clear language guesses the old free-text fallback stored
361ca55 
Before #616 a prose or truncated JSON reply was stored as a guessed
language code and cached forever. EnrichmentStore.requeue_unparseable
re-parses every cached row at startup and clears the ones the current
parser rejects, so they are asked again. Fail-soft at boot.

Built by DeepSeek flash (dspilot run e5bee333, brief L19).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk @claude
docs(616): Task 9 reports the provider as integrations booleans
6d444b5 
The telemetry ingest drops unknown top-level keys, so llm_provider would
never be stored. Judd chose openai_local and openai_hosted booleans in
the integrations object, which the ingest already stores and counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk @claude
feat(616): telemetry reports the LLM provider kind
db7b964 
openai_local and openai_hosted booleans inside the integrations object,
set only when the OpenAI-compatible provider is active with a base URL
and a model. The ingest drops unknown top-level keys, so no new
top-level field. No URL, key or model name enters the payload.

Built by DeepSeek flash (dspilot run 3dcc7729, brief L20).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk @claude
feat(616): settings UI for the OpenAI-compatible provider
9e7226f 
Provider select on the LLM panel, the OpenAI-compatible credential
editor (base URL, key, model, vision, server type), the hosted-profile
disclosure line, per-step Test connection rows, and the Ollama-only
vision card and unload row hidden for the OpenAI-compatible provider.

Built by deepseek-flash, dspilot run 5bcbaa41 (brief L22).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk @claude
test(616): positive control on the SystemPanel unload-hidden test
16ec306 
The absence check passed on a panel that never rendered. It now
asserts the panel itself rendered first, and fails against the base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk @claude
docs(616): changelog, README provider section, eval usage
1fd011d 
CHANGELOG Unreleased: Added entry for the OpenAI-compatible provider naming
the servers tested live (Ollama /v1, OpenRouter gemini-2.5-flash and
gpt-4o-mini) and those implemented from documentation only (OpenAI direct,
Unsloth Studio, llama-server); Changed entry for the title-language prompt
with the gemma4 eval result (old 33/60, new 33/60).

README: provider section with a compose example, six env vars in the table.

eval_title_language.py: the usage line piped into subarr-next, whose code
predates 616 and lacks the modules the script imports; document the
throwaway-container route used for the live run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ShzWm3urD55AUk5yrSgXsi
@coaxk
coaxk merged commit 48381bc into main Nov 1, 2026
@coaxk
coaxk deleted the feat/616-openai-compatible-provider branch November 1, 2026 09:29
@coaxk coaxk mentioned this pull request Nov 1, 2026
Closed
github-advanced-security[bot]
github-advanced-security Bot found potential problems Nov 1, 2026
View reviewed changes
Comment thread src/subarr/routers/integrations.py
continue
if key in _SECRET_KEYS and _is_masked(raw):
skipped_masked.append(settings_attr)
log.info("creds: %s.%s masked value, kept the stored secret", svc, settings_attr)
@coaxk coaxk mentioned this pull request Nov 1, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@github-advanced-security github-advanced-security[bot] github-advanced-security[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature] Add option for openAI-compatible providers

2 participants

@coaxk @github-advanced-security