Skip to content

feat(llm): Atlas Cloud LLM provider in both SDKs - #227

Open
binyangzhu000-sudo wants to merge 1 commit into
PatterAI:mainfrom
binyangzhu000-sudo:feat/atlascloud-llm
Open

binyangzhu000-sudo wants to merge 1 commit into
PatterAI:mainfrom
binyangzhu000-sudo:feat/atlascloud-llm

Conversation

@binyangzhu000-sudo

Copy link
Copy Markdown

Summary

Adds AtlasCloudLLM to both SDKs — Atlas Cloud is an OpenAI-compatible aggregator gateway that fronts 100+ open and frontier models (OpenAI, DeepSeek, Qwen, GLM, Kimi, MiniMax) behind one key and one endpoint.

The motivation is the one a voice agent actually hits: you rarely need a model forever. You tune for latency, a cheaper open-weights model lands, or one vendor rate-limits you at 6pm. Behind an aggregator that is a model= string; without one it is a new provider, a new key and a new pricing table.

Why a named provider when CustomLLM (#165) already covers any OpenAI-compatible endpoint — the same reason GroqLLM, CerebrasLLM and LiteLLMLLM (#183) exist: a named provider carries the base URL, the env var, the model catalog, real pricing so the dashboard shows cost instead of $0, a telemetry vendor tag, and a patter init row. CustomLLM gives you the transport; it can't give you any of those.

The provider is OpenAILLMProvider with an Atlas base URL, so streaming, tool calling and every OpenAI sampling knob are inherited unchanged. On the TS side it reuses the existing parseOpenAISseStream exported from providers/groq-llm.ts rather than duplicating the parser.

Changes

Provider (both SDKs)

  • libraries/python/getpatter/providers/atlascloud_llm.py — AtlasCloudLLMProvider + AtlasCloudModel (StrEnum, 8 ids), provider_key = "atlascloud", reads ATLASCLOUD_API_KEY, re-applies the parent's User-Agent to the swapped-in client, tags cost attribution atlascloud (without the override, spend is attributed to the parent's openai).
  • libraries/typescript/src/providers/atlascloud-llm.ts — the mirror, same defaults, camelCase options.
  • Namespace adapters getpatter.llm.atlascloud / getpatter/llm/atlascloud exporting LLM, plus the flat AtlasCloudLLM alias on both package barrels and the lazy providers/__init__.py registry entry.

Default model — chosen on measurement, not vibes. openai/gpt-4.1-mini. Against a live transfer_call tool-calling turn it answered in 14 completion tokens with zero reasoning tokens; the reasoning-tuned models in the catalog (kimi-k2.5, minimax-m2.5, qwen3.5-35b-a3b) spent 28–64 reasoning tokens first. On a phone call those are dead air, so they're reachable via model= rather than being the default.

Pricing — all eight AtlasCloudModel ids, read from the gateway's own catalog (GET /v1/models → pricing.prompt / pricing.completion / pricing.input_cache_read, USD per token) rather than a docs snapshot. Two details worth flagging:

  • cache_read is populated even where it equals input (two models get no cache discount at this gateway). Omitting the key would have billed cached tokens at $0, since the provider subtracts cached_tokens from prompt_tokens before billing — exactly the silent-under-billing class the existing Groq/Cerebras pricing tests were written for.
  • Both pricing tests now iterate the enum instead of a hardcoded list, so adding a model without a rate fails CI.

Wiring

  • atlascloud pip extra; ATLASCLOUD_API_KEY in both .env.example files.
  • Atlas Cloud row in patter init's pipeline LLM table, both runtimes — verified end to end: getpatter init --mode pipeline --llm atlascloud emits llm=AtlasCloudLLM(), getpatter[atlascloud]==0.7.1 in requirements.txt, and the env line; the TS wizard emits new AtlasCloudLLM().
  • atlascloud added to the telemetry STACK_VENDORS allowlist (both files, kept byte-for-byte in sync) and to _infer_llm_provider — without it the stack field collapses to other.

Tests

  • libraries/python/tests/unit/test_atlascloud_llm.py — 13 tests. No mocks: real construction, real client inspection.
  • libraries/typescript/tests/atlascloud-llm.mocked.test.ts — 12 tests, globalThis.fetch mocked at the paid boundary only; the provider, SSE parser, namespace adapter and pricing lookup are all real code.
  • Both cover: defaults and override, endpoint/base-URL, bearer + Patter UA, env-var resolution and the named error when it's missing, SSE text/tool-call parsing, the cached-token subtraction, and throw-don't-silently-end on a non-2xx.

Docs — providers/atlascloud.mdx in both SDK sections (model + live pricing table, prompt-cache semantics, why-an-aggregator-for-voice), AtlasCloudLLM entries in both llm.mdx, and docs.json nav.

Verification

Run against the real gateway, both SDKs, not just mocks:

Python TypeScript
Streaming turn ttft=2.55s → "Patter is connected." ttft=2064ms → "Patter is connected."
Tool call transfer_call {"department": "billing"} transfer_call {"department":"billing"}
Usage chunk in=13 out=2 inputTokens=15 outputTokens=6 cacheRead=0
Cost attribution patter.llm.provider = atlascloud (captured live) calculateLlmCost → $0.020000 / 10k+10k

Suites:

  • Python pytest tests/ — 3030 passed, 33 skipped, 2 xfailed
  • Python pytest tests/security/ -m security — 90 passed, 2 xfailed
  • TypeScript vitest run — 2629 passed, 9 skipped (167 files)
  • tsc --noEmit clean; npm run build clean
  • bash scripts/pr-validate.sh — all TypeScript checks PASS. The Python jobs report skipped (missing: pip/pytest) on my machine because the script looks for pip/pytest on PATH and this checkout uses a uv venv; I ran both Python commands the script would have run, directly against that venv — results above.

Pre-merge checklist

  • Local validation is green: bash scripts/pr-validate.sh — see the note above about the Python jobs being run directly.
  • Both SDKs updated — same API shape and defaults, snake_case ↔ camelCase.
  • CHANGELOG.md updated — entry under ## Unreleased → ### Added.
  • Tests added; only the paid/external boundary is mocked (TS fetch; the Python tests mock nothing).
  • No secrets, credentials, or real phone numbers / PII in the diff — placeholders only (apikey-...).
  • No external license headers or "ported from " provenance comments.

Breaking change?

No. Purely additive: a new optional provider, a new optional extra, and new optional rows in existing tables. No existing default changes and no new credential is required unless you opt into this provider.


Disclosure: I work at Atlas Cloud. Happy to adjust the default model, trim the docs page, or drop any of the wiring (init/telemetry) if you'd rather keep the surface smaller.

Adds `AtlasCloudLLM` — an OpenAI-compatible aggregator gateway
(https://api.atlascloud.ai/v1) that fronts 100+ open and frontier models
behind one key, so moving between models is a `model` string rather than a
new provider, credential and pricing table.

The provider is `OpenAILLMProvider` with an Atlas base URL, so streaming,
tool calling and every OpenAI sampling knob are inherited unchanged. The
TypeScript side reuses the existing `parseOpenAISseStream` from
`providers/groq-llm.ts` rather than duplicating the parser.

Default model is `openai/gpt-4.1-mini`: in a live tool-calling turn it
answered in 14 completion tokens with zero reasoning tokens, where the
reasoning-tuned models in the catalog spent 28-64 first — on a phone call
those are dead air.

Pricing for all eight `AtlasCloudModel` ids is read from the gateway's own
catalog (GET /v1/models -> pricing.prompt / pricing.completion /
pricing.input_cache_read) rather than a docs snapshot. `cache_read` is
populated even where it equals `input`, because omitting the key bills
cached tokens at $0. Both pricing tests iterate the enum, so adding a model
without a rate fails CI instead of under-billing silently.

Also wired: the `atlascloud` pip extra, `ATLASCLOUD_API_KEY` in both
.env.example files, an Atlas Cloud row in `patter init`'s pipeline LLM table
in both runtimes, and `atlascloud` in the telemetry vendor allowlist so the
stack field reports the real vendor instead of collapsing to "other".

Docs: per-provider page in both SDK sections plus an `AtlasCloudLLM` entry
in each llm.mdx.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant