fix(models): resolve and merge raw ids in models report - #1053
Conversation
codeburn models asked each provider for a label and bucketed by raw id, so gpt-5.6-* and Fireworks path-form ids stayed raw and two ids that share a display name became two rows. Keep provider-first labels. Fall back to the global short-name table on a local miss, then merge by provider + display name.
Extra High on getagentseal#1053: three raw ids sharing a provider + display name cleared then refilled savingsBaselineModel because empty meant both "none seen" and "conflict". Track distinct baselines in a Set and emit one only when a single non-empty value remains.
|
Reviewed in depth on a rebased branch with real-data runs. The display-resolution half is right and verified ($-identical on a 31-row real corpus, Blocker — the merge key is a display string. Also:
Suggest splitting: land |
Maintainer review on getagentseal#1053: display-name keys collide for distinct SKUs (GPT-5 / GPT-5.3 Codex / Kimi K2 Thinking / Opus 4.6). Key on the alias-resolved canonical id, keep the first raw id, and partial-sum Codex credits when a merge mixes rated and unrated buckets.
ozymandiashh
left a comment
There was a problem hiding this comment.
The original canonical-ID merge work looks sound, but this head cannot be merged safely against current main. Its conflicted report paths still compute billable output as output + reasoning; current main deliberately uses billableOutputTokens because Codex output already includes reasoning. Taking the branch side would reintroduce double counting (for example 500k output + 500k reasoning becomes 1M instead of 500k). Please rebase, retain the current billableOutputTokens pricing semantics in both audit and models reports, and rerun the report/model suites on the resolved tree.
Resolves conflicts from main's getagentseal#1078 (billableOutputTokens for the output bucket) and getagentseal#1084 (models-report test exemplar swap) against this branch's canonical-id row merging. - src/audit-report.ts: import-line collision only. Union both sides' imports (billableOutputTokens + fallbackRawModelDisplayName/ getShortModelName); both are used elsewhere in the file and neither side's logic needed further changes. - src/models-report.ts: import-line collision resolved the same way. The row-construction conflict was structural, not a data conflict: main's side pushed one row per raw bucket (a simpler variant this branch's canonical-id folding already made obsolete downstream -- rowsByKey/foldedCategoryCost/foldKey are used unconditionally past this point). Kept this branch's rowsByKey merge-by-canonical-id structure, which already computes credits from bucket.outputTokens (already billable-output-summed at the accumulation stage, main's change, untouched by this conflict) using the identical formula main used; folded main's explanatory comment about billable output into the kept credits block.
Resolves conflicts against main after getagentseal#1078/getagentseal#1084/getagentseal#1088/getagentseal#1090/getagentseal#1092/ getagentseal#1053/getagentseal#1056 landed since this branch's last upstream merge. - CHANGELOG.md: kept both entries (this PR's getagentseal#968/getagentseal#1050 note plus main's getagentseal#1079/getagentseal#1088, getagentseal#1082, getagentseal#1075 notes that had moved into the same "### Fixed" slot). - Everything else (src/models.ts, main.ts, usage-aggregator.ts, daily-cache.ts, tests/*) merged cleanly with no conflict markers; git's recursive merge combined getagentseal#1050's flat-rate classifier changes with getagentseal#1056's codex-auto-review -> gpt-5.5 alias without overlap. Verified the codex-auto-review / getagentseal#1056 interaction post-merge: isBuiltInFlatRateModel no longer matches codex-auto-review (dropped per getagentseal#1050), while MODEL_ALIASES still aliases it to gpt-5.5 (getagentseal#1056), so it prices at GPT-5.5 rates rather than $0. Covered by the existing tests/models.test.ts "Codex activity ids (getagentseal#1047)" describe block and the "does not treat a priced sibling as expected-free" case. getFlatRateModelsConfigHash's output is folded into getDailyCacheConfigHash's template literal unconditionally (flatRateModels=<hash>), so the flat-rate section always participates in the daily-cache invalidation hash even when empty -- no DAILY_CACHE_VERSION bump needed for this change.
Problem
codeburn modelsshows raw ids (gpt-5.6-sol,accounts/fireworks/models/kimi-k2p6) thatcodeburn reportalready resolves, and splits one model into two rows when two raw ids are the same SKU.Closes #967.
Root cause
modelsasked each provider for a label, then bucketed by raw id. Providers with a local table miss fell back to the raw id. A later merge keyed on the display string, so live label collisions (GPT-5 / GPT-5.3 Codex / Kimi K2 Thinking / Opus 4.6) would sum distinct SKUs into one row.Change
cursor-agentandvercel-gatewayconsult that table after their local transform (suffix / vendor peel).models --format jsonis not a lex-smallest rewrite).creditsIncomplete.User impact
codeburn modelsmatchesreportfor gpt-5.6-* and Fireworks path-form ids. Path-formglm-5p2merges with the bare slug. Distinct SKUs that only share a label stay two rows.Preservation
No new LiteLLM rates. No desktop / By Model rewrite (
aggregateModelTotalsalready merges). No Keychain / Buzz / #868 / #916 / #998 work. Audit still buckets by raw id (it recomputes rates per id). No second Extra High.Testing
resolveCanonicalModelIdunit cases (230 passing in the focused files; 3 pre-existing DeepSeek snapshot fails)gpt-5.6-sol→GPT-5.6 Sol (est.)), path-form (accounts/fireworks/models/kimi-k2p6→Kimi K2.6), canonical merge of fireworksglm-5p2+glm-5p2, display-only collisions stay split, first-seen raw id, mixed credit partial-sum