Repository navigation
feat(llm): Atlas Cloud LLM provider in both SDKs - #227
Open
binyangzhu000-sudo wants to merge 1 commit into
Open
binyangzhu000-sudo wants to merge 1 commit into
binyangzhu000-sudo wants to merge 1 commit into
Conversation
Adds `AtlasCloudLLM` — an OpenAI-compatible aggregator gateway (https://api.atlascloud.ai/v1) that fronts 100+ open and frontier models behind one key, so moving between models is a `model` string rather than a new provider, credential and pricing table. The provider is `OpenAILLMProvider` with an Atlas base URL, so streaming, tool calling and every OpenAI sampling knob are inherited unchanged. The TypeScript side reuses the existing `parseOpenAISseStream` from `providers/groq-llm.ts` rather than duplicating the parser. Default model is `openai/gpt-4.1-mini`: in a live tool-calling turn it answered in 14 completion tokens with zero reasoning tokens, where the reasoning-tuned models in the catalog spent 28-64 first — on a phone call those are dead air. Pricing for all eight `AtlasCloudModel` ids is read from the gateway's own catalog (GET /v1/models -> pricing.prompt / pricing.completion / pricing.input_cache_read) rather than a docs snapshot. `cache_read` is populated even where it equals `input`, because omitting the key bills cached tokens at $0. Both pricing tests iterate the enum, so adding a model without a rate fails CI instead of under-billing silently. Also wired: the `atlascloud` pip extra, `ATLASCLOUD_API_KEY` in both .env.example files, an Atlas Cloud row in `patter init`'s pipeline LLM table in both runtimes, and `atlascloud` in the telemetry vendor allowlist so the stack field reports the real vendor instead of collapsing to "other". Docs: per-provider page in both SDK sections plus an `AtlasCloudLLM` entry in each llm.mdx.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
AtlasCloudLLMto both SDKs — Atlas Cloud is an OpenAI-compatible aggregator gateway that fronts 100+ open and frontier models (OpenAI, DeepSeek, Qwen, GLM, Kimi, MiniMax) behind one key and one endpoint.The motivation is the one a voice agent actually hits: you rarely need a model forever. You tune for latency, a cheaper open-weights model lands, or one vendor rate-limits you at 6pm. Behind an aggregator that is a
model=string; without one it is a new provider, a new key and a new pricing table.Why a named provider when
CustomLLM(#165) already covers any OpenAI-compatible endpoint — the same reasonGroqLLM,CerebrasLLMandLiteLLMLLM(#183) exist: a named provider carries the base URL, the env var, the model catalog, real pricing so the dashboard shows cost instead of$0, a telemetry vendor tag, and apatter initrow.CustomLLMgives you the transport; it can't give you any of those.The provider is
OpenAILLMProviderwith an Atlas base URL, so streaming, tool calling and every OpenAI sampling knob are inherited unchanged. On the TS side it reuses the existingparseOpenAISseStreamexported fromproviders/groq-llm.tsrather than duplicating the parser.Changes
Provider (both SDKs)
libraries/python/getpatter/providers/atlascloud_llm.py—AtlasCloudLLMProvider+AtlasCloudModel(StrEnum, 8 ids),provider_key = "atlascloud", readsATLASCLOUD_API_KEY, re-applies the parent's User-Agent to the swapped-in client, tags cost attributionatlascloud(without the override, spend is attributed to the parent'sopenai).libraries/typescript/src/providers/atlascloud-llm.ts— the mirror, same defaults,camelCaseoptions.getpatter.llm.atlascloud/getpatter/llm/atlascloudexportingLLM, plus the flatAtlasCloudLLMalias on both package barrels and the lazyproviders/__init__.pyregistry entry.Default model — chosen on measurement, not vibes.
openai/gpt-4.1-mini. Against a livetransfer_calltool-calling turn it answered in 14 completion tokens with zero reasoning tokens; the reasoning-tuned models in the catalog (kimi-k2.5,minimax-m2.5,qwen3.5-35b-a3b) spent 28–64 reasoning tokens first. On a phone call those are dead air, so they're reachable viamodel=rather than being the default.Pricing — all eight
AtlasCloudModelids, read from the gateway's own catalog (GET /v1/models→pricing.prompt/pricing.completion/pricing.input_cache_read, USD per token) rather than a docs snapshot. Two details worth flagging:cache_readis populated even where it equalsinput(two models get no cache discount at this gateway). Omitting the key would have billed cached tokens at$0, since the provider subtractscached_tokensfromprompt_tokensbefore billing — exactly the silent-under-billing class the existing Groq/Cerebras pricing tests were written for.Wiring
atlascloudpip extra;ATLASCLOUD_API_KEYin both.env.examplefiles.patter init's pipeline LLM table, both runtimes — verified end to end:getpatter init --mode pipeline --llm atlascloudemitsllm=AtlasCloudLLM(),getpatter[atlascloud]==0.7.1inrequirements.txt, and the env line; the TS wizard emitsnew AtlasCloudLLM().atlascloudadded to the telemetrySTACK_VENDORSallowlist (both files, kept byte-for-byte in sync) and to_infer_llm_provider— without it the stack field collapses toother.Tests
libraries/python/tests/unit/test_atlascloud_llm.py— 13 tests. No mocks: real construction, real client inspection.libraries/typescript/tests/atlascloud-llm.mocked.test.ts— 12 tests,globalThis.fetchmocked at the paid boundary only; the provider, SSE parser, namespace adapter and pricing lookup are all real code.Docs —
providers/atlascloud.mdxin both SDK sections (model + live pricing table, prompt-cache semantics, why-an-aggregator-for-voice),AtlasCloudLLMentries in bothllm.mdx, anddocs.jsonnav.Verification
Run against the real gateway, both SDKs, not just mocks:
ttft=2.55s→"Patter is connected."ttft=2064ms→"Patter is connected."transfer_call {"department": "billing"}transfer_call {"department":"billing"}in=13 out=2inputTokens=15 outputTokens=6 cacheRead=0patter.llm.provider = atlascloud(captured live)calculateLlmCost→$0.020000/ 10k+10kSuites:
pytest tests/— 3030 passed, 33 skipped, 2 xfailedpytest tests/security/ -m security— 90 passed, 2 xfailedvitest run— 2629 passed, 9 skipped (167 files)tsc --noEmitclean;npm run buildcleanbash scripts/pr-validate.sh— all TypeScript checks PASS. The Python jobs reportskipped (missing: pip/pytest)on my machine because the script looks forpip/pytestonPATHand this checkout uses auvvenv; I ran both Python commands the script would have run, directly against that venv — results above.Pre-merge checklist
bash scripts/pr-validate.sh— see the note above about the Python jobs being run directly.snake_case↔camelCase.CHANGELOG.mdupdated — entry under## Unreleased→### Added.fetch; the Python tests mock nothing).apikey-...).Breaking change?
No. Purely additive: a new optional provider, a new optional extra, and new optional rows in existing tables. No existing default changes and no new credential is required unless you opt into this provider.
Disclosure: I work at Atlas Cloud. Happy to adjust the default model, trim the docs page, or drop any of the wiring (init/telemetry) if you'd rather keep the surface smaller.