Skip to content

fix(codex): stop double-billing reasoning output, price cache writes at the explicit rate only - #1078

Merged
iamtoruk merged 1 commit into
mainfrom
fix/codex-pricing-1075
Aug 21, 2026
Merged

fix(codex): stop double-billing reasoning output, price cache writes at the explicit rate only#1078
iamtoruk merged 1 commit into
mainfrom
fix/codex-pricing-1075

Conversation

@iamtoruk

Copy link
Copy Markdown
Member

Fixes claims 2 and 1 from @chr-evensen's report, in the form the verification comment on #1075 landed on. Claim 3 (long-context tiers) is deferred to #1076 at the real 272k threshold; the missing gpt-5.6-codex snapshot rows are #1077.

Fix A — reasoning was billed twice

OpenAI bills reasoning tokens as part of output_tokens. On a 1,396-rollout corpus, all 134,316 token_count events carrying a total satisfy input + output == total; zero satisfy the exclusive reading. CodeBurn added reasoning_output_tokens on top in four places:

site what it did
src/providers/codex.ts (fresh parse) priced output + reasoning
src/parser.ts cachedCallToApiCall (cache rehydration) same, with an existing carve-out for claude only
src/models-report.ts, src/audit-report.ts displayed output + reasoning

The two cost sites are twins — codex is not on the reported-cost pass-through allowlist, so a warm run re-prices from the stored token buckets while a cold run uses the parser's number. If only one had been fixed, a user's total would change between a cold and a warm run. Both now call one helper:

const REASONING_INCLUDED_IN_OUTPUT = new Set(['claude', 'codex'])
export function billableOutputTokens(provider, outputTokens, reasoningTokens) {  }

so the two sites cannot drift, and each carries a comment pointing at the other. claude is in the set to preserve the carve-out parser.ts already had; the claude parser always emits reasoningTokens: 0, so no claude number moves. The raw reasoningTokens field is untouched and still reported on its own — only the double-count is gone.

Fix B — cache_write_input_tokens, guarded

The field (codex PR #33454) was never read and cacheCreationInputTokens was hardcoded to 0. It is now read on both decode paths, clamped to max(0, min(cache_write, input − cached)), carved out of the uncached-input bucket, and passed as cacheCreationInputTokens.

The guard. buildCosts in src/models.ts fabricates cacheWriteCostPerToken = 1.25 × input when the pricing source omits one. That default is right for Anthropic-style pricing, where a cache write genuinely costs more. OpenAI charges nothing extra for a cache write before gpt-5.6, and the snapshot reflects that:

model cache-write rate in the snapshot
gpt-5.6-terra, -sol, -luna, gpt-5.6 explicit (1.25× input)
gpt-5.5, gpt-5.4, gpt-5.3-codex, gpt-5 null → fabricated

Routing tokens through the fabricated rate would have invented a surcharge OpenAI never billed on every pre-5.6 model. So ModelCosts now carries cacheWriteCostIsExplicit (set once, in buildCosts, from whether the raw tuple's cache-write slot was non-null), and the codex path only moves tokens into the cache-write bucket when it is true. Otherwise they stay in plain input and the price is unchanged to the cent.

This is deliberately a decision about which bucket codex tokens land in. buildCosts' defaults are not changed, and no other provider's behaviour moves.

Corpus A/B

Read-only, CODEBURN_PRICING_SNAPSHOT_ONLY=1, scratch cache dirs, models -p lifetime --provider all --format json on origin/main vs this branch:

codex   cost  $4713.12 -> $4547.09   -$166.03  (-3.52%)
codex   out   22,644,840 -> 16,819,446 tokens  (-25.73% displayed;
                                                the old number was 34.63% too high)
codex   cacheWriteTokens  0 -> 0     (claim 1 contributes $0 on this corpus today)

Every other provider is byte-identical row for row, with one exception: the two claude rows move, and they move base-vs-base too (I was using Claude Code while the runs were in flight — claude-opus-5 calls 32243 -> 32244 between two consecutive origin/main runs). No non-claude, non-codex row differs at all.

The upgrade-path corpus reaches the same conclusion independently, and asserts the stronger half: codex tokens 476,815 -> 476,815 and calls 123 -> 123 identical, cost 1.451527 -> 1.207871.

Cache versions

A cost change invalidates persisted output, so three layers move:

  • CODEX_CACHE_VERSION 10 → 11codex-results.json stores each call's costUSD and token buckets verbatim.
  • PROVIDER_PARSE_VERSIONS.codex gains -codex-pricing-v1 — the session cache serves unchanged files without invoking the provider parser. Fix A self-heals through it (cost is re-derived on read), but Fix B's bucket move does not: cached entries store the buckets, not the raw event. Without this bump the cache-write carve-out would never reach an existing user.
  • DAILY_CACHE_VERSION / MIN_SUPPORTED_VERSION 20 → 23. Not 21: v21 is claimed by the feat(copilot): read per-request input/cache from session-store.db #946 landing branch and v22 by PR fix(models): price Codex activity ids via the official underlying model #1056, so reusing either would let two incompatible schemas share a filename. feat/core-extraction sits at 26 and reconciles at its final merge by keeping the max. scripts/upgrade-path/run.mjs tracks the bump (daily-cache.v23.json), which the upgrade-path CI asserts.

compare.mjs gains a COST_CHANGED_BY_DESIGN list holding codex: it keeps the full exact treatment for the call count and every token field, and only lifts the 0.5% cost tolerance, reporting the delta instead. Drop codex from that list once a published CLI carries this fix.

Verification

  • npm test: 2970 passed, 5 skipped, 0 failed (219 files).
  • npx tsc --noEmit: clean. npm run build: clean.
  • npm run verify:upgrade: PASSED, including daily-cache.v23.json re-derived.
  • New tests are revert-proof — each fails with only its own site reverted:
    • codex.ts cost site → prices a fresh codex parse from output_tokens alone
    • parser.ts cost site → codex-pricing-1075-rehydrate.test.ts (drives the full pipeline cold then warm)
    • the guard → THE GUARD: leaves cache writes in the input bucket… (gpt-5.5 with a nonzero cache_write; cost identical to before the fix)
    • the clamp, both display sums, CODEX_CACHE_VERSION, and DAILY_CACHE_VERSION/MIN_SUPPORTED_VERSION each have their own failing-on-revert test.
  • Two existing assertions were updated rather than worked around: tests/audit-report.test.ts and tests/models-report.test.ts each asserted the old additive behaviour for a codex/claude call. The additive branch is still covered — the models-report case moved to hermes, and the new suite asserts codex 1000 vs hermes 1400 side by side.

Known adjacent, deliberately not in this PR

Codex throughput double-counts reasoning the same way (src/providers/codex.ts activeGeneratedTokens/taskGeneratedTokens, and src/codex-throughput.ts's generatedTokens), so tok/s reads high. That is a display metric with its own test surface and no bearing on cost; it deserves its own PR rather than being smuggled into a pricing fix.

Closes #1075

…at the explicit rate only

Reasoning tokens are a subset of output_tokens for OpenAI models, not an
extra bucket: on a 1,396-rollout corpus all 134,316 token_count events
carrying a total satisfy input + output == total. CodeBurn added
reasoning_output_tokens on top when pricing a codex call, in the
cache-rehydration re-price, and in the models/audit display sums. That
overstated codex cost by $166.03 (3.5%) and displayed output tokens by
34.6% on that corpus. Both cost sites and the display sums now go through
one shared billableOutputTokens() so a cold parse and a warm read cannot
drift apart.

cache_write_input_tokens (codex PR #33454) was never read and
cacheCreationInputTokens was hardcoded to 0. It is now carved out of the
uncached-input bucket and clamped to it, but routed to the cache-write
bucket ONLY when the pricing source publishes an explicit cache-write rate.
buildCosts fabricates 1.25x input when a source omits one, which is correct
for Anthropic and would have invented a surcharge OpenAI never charged on
gpt-5.5 / 5.4 / 5.3-codex / gpt-5. ModelCosts now carries
cacheWriteCostIsExplicit so that distinction survives getModelCosts.

A cost change invalidates persisted output: codex-results.json v10 -> v11
(stores costUSD verbatim), the codex parse version moves (the token-bucket
change does not self-heal on read), and the daily cache goes 20 -> 23 (21 is
claimed by the #946 landing branch and 22 by PR #1056). The upgrade-path
corpus asserts codex tokens and calls exactly and reports the repricing.

Closes #1075
@iamtoruk
iamtoruk merged commit 91ddf58 into main Aug 21, 2026
15 checks passed
iamtoruk added a commit that referenced this pull request Aug 21, 2026
fix(pricing): harden the #1078 follow-ups — cache-flag survival, credits trap, harness cost bound
pull Bot pushed a commit to TheTechOddBug/codeburn that referenced this pull request Aug 21, 2026
…reasoning param

The pricing cache written to disk had no schema version, so a cache written
by a pre-getagentseal#1078 binary lacked cacheWriteCostIsExplicit on every entry. Reading
it back resolved the missing key to undefined (falsy), silently reintroducing
the surcharge-fabrication bug getagentseal#1078 killed for up to CACHE_TTL_MS after an
upgrade. loadCachedPricing now rejects any cache whose version doesn't match
the current schema instead of reading it verbatim.

codexCredits() still accepted an optional reasoningTokens param that added it
to output - the exact double-count getagentseal#1078 removed from every real caller. The
only caller never passed it; deleted it so it can't be reintroduced by
accident.

parser.ts's activeGeneratedTokens fallback went through billableOutputTokens
in getagentseal#1078, but codex is the only caller of activeDurationMs/activeGeneratedTokens
and always sets both together, so the fallback branch is unreachable for it.
Reverted to reduce diff noise.
pull Bot pushed a commit to TheTechOddBug/codeburn that referenced this pull request Aug 21, 2026
… stale comments

The COST_CHANGED_BY_DESIGN carve-out in compare.mjs left codex cost entirely
unasserted after getagentseal#1075/getagentseal#1078. It now requires the upgraded cost to be strictly
lower than baseline and within 25% of it, and the row verdict says "repriced"
instead of the misleading "identical (cost N% drift)".

grok.ts's comment on the reasoning/output split still claimed provider-side
splitting was the repo's only mechanism; it now also names
billableOutputTokens/REASONING_INCLUDED_IN_OUTPUT (models.ts), which is the
other half since getagentseal#1078. usage-aggregator.ts's "folds reasoning into output"
comment was true pre-getagentseal#1078 but is backwards for codex now (reasoning is
already inside output, not added to it).

Test exemplars for the "reasoning is additive" case used hermes, whose
upstream is OpenAI-shaped and may not stay a safe example; swapped to gemini,
which documents "thoughts" as genuinely separate output.

CHANGELOG's getagentseal#1075 entry gets one line noting days whose codex transcripts
have aged out keep their pre-fix totals via the daily-cache never-lose guard,
matching the disclosure already given for getagentseal#1040.
iamtoruk added a commit that referenced this pull request Aug 21, 2026
fix(codex): port the #1078 pricing fixes (reasoning double-count + cache-write bucketing) to feat/core-extraction
avs-io pushed a commit to avs-io/codeburn that referenced this pull request Aug 21, 2026
The same reasoning-inclusion bug getagentseal#1075/getagentseal#1078 fixed for cost also affected
the throughput display: activeGeneratedTokens/taskGeneratedTokens in the
Codex parser and generatedTokens in the live codex-tps reader summed
outputTokens + reasoningTokens, but reasoning is already inside output.
All three sites now route through the billableOutputTokens('codex', ...)
helper getagentseal#1078 introduced, so the throughput numerator can never drift from
the billed one.

Cost and every token count are unchanged (verified byte-identical on a
real 51,753-call corpus); Tok/s drops 20-42% depending on how
reasoning-heavy the model is.

activeGeneratedTokens/activeDurationMs/toolWaitMs are stored verbatim in
both the Codex result cache and the session cache rather than re-derived
on read, so neither self-heals: CODEX_CACHE_VERSION moves 11 -> 13 (12 is
claimed by feat/core-extraction's own port of this feature) and
PROVIDER_PARSE_VERSIONS.codex gains a codex-tps-v1 suffix, forcing Codex
sessions to re-parse once.

Closes getagentseal#1079
iamtoruk added a commit to avs-io/codeburn that referenced this pull request Aug 22, 2026
Resolves conflicts from main's getagentseal#1078 (billableOutputTokens for the
output bucket) and getagentseal#1084 (models-report test exemplar swap) against
this branch's canonical-id row merging.

- src/audit-report.ts: import-line collision only. Union both sides'
  imports (billableOutputTokens + fallbackRawModelDisplayName/
  getShortModelName); both are used elsewhere in the file and neither
  side's logic needed further changes.
- src/models-report.ts: import-line collision resolved the same way.
  The row-construction conflict was structural, not a data conflict:
  main's side pushed one row per raw bucket (a simpler variant this
  branch's canonical-id folding already made obsolete downstream --
  rowsByKey/foldedCategoryCost/foldKey are used unconditionally past
  this point). Kept this branch's rowsByKey merge-by-canonical-id
  structure, which already computes credits from bucket.outputTokens
  (already billable-output-summed at the accumulation stage, main's
  change, untouched by this conflict) using the identical formula
  main used; folded main's explanatory comment about billable output
  into the kept credits block.
iamtoruk added a commit to avs-io/codeburn that referenced this pull request Aug 22, 2026
Resolves conflicts against main after getagentseal#1078/getagentseal#1084/getagentseal#1088/getagentseal#1090/getagentseal#1092/
getagentseal#1053/getagentseal#1056 landed since this branch's last upstream merge.

- CHANGELOG.md: kept both entries (this PR's getagentseal#968/getagentseal#1050 note plus
  main's getagentseal#1079/getagentseal#1088, getagentseal#1082, getagentseal#1075 notes that had moved into the same
  "### Fixed" slot).
- Everything else (src/models.ts, main.ts, usage-aggregator.ts,
  daily-cache.ts, tests/*) merged cleanly with no conflict markers;
  git's recursive merge combined getagentseal#1050's flat-rate classifier changes
  with getagentseal#1056's codex-auto-review -> gpt-5.5 alias without overlap.

Verified the codex-auto-review / getagentseal#1056 interaction post-merge:
isBuiltInFlatRateModel no longer matches codex-auto-review (dropped
per getagentseal#1050), while MODEL_ALIASES still aliases it to gpt-5.5 (getagentseal#1056),
so it prices at GPT-5.5 rates rather than $0. Covered by the existing
tests/models.test.ts "Codex activity ids (getagentseal#1047)" describe block and
the "does not treat a priced sibling as expected-free" case.

getFlatRateModelsConfigHash's output is folded into
getDailyCacheConfigHash's template literal unconditionally
(flatRateModels=<hash>), so the flat-rate section always participates
in the daily-cache invalidation hash even when empty -- no
DAILY_CACHE_VERSION bump needed for this change.
iamtoruk added a commit to kelchm/codeburn that referenced this pull request Aug 22, 2026
Resolves the conflicts getagentseal#946 accumulated while it was in validation. Eight
files conflicted; the session-store accounting is unchanged.

src/daily-cache.ts — version collision. This PR minted 25 when main was at
24; getagentseal#1056 (`codex-auto-review` pricing) then spent 25 on main. The bump moves
to 26/MIN 26 and daily-cache.v26.json, with main's full comment ladder kept as
the foundation and this PR's paragraph rewritten to name 26 and record the
collision. PENDING_REDERIVE_PROVIDERS and the B1 migration semantics from
b6481c1 carry over intact, retargeted at 26.

src/models.ts, src/parser.ts, src/audit-report.ts, src/models-report.ts —
getagentseal#1075/getagentseal#1078 replaced the per-site "reasoning is already inside output" tests
with billableOutputTokens() and REASONING_INCLUDED_IN_OUTPUT. This PR had
added copilot to that case at three sites independently. Union: all three
sites take main's helper call verbatim, and copilot joins claude and codex in
the set — same accounting this PR shipped, now through main's single source of
truth. It also reaches parser.ts activeGeneratedTokens (a fourth site, from
getagentseal#1079), which is the same correction: a copilot supplementary call carries
reasoning with output 0, so counting it as generated repeats the per-turn
output. The audit legend already said so on this side.

src/providers/copilot.ts — comment-only. getagentseal#1054's lastEventTimestamp-first
shutdown fallback was derived from this branch, so the code was already
identical on both sides: the `shutdownTimestamp` expression and the
`copilot:<sid>:shutdown:<model>:<n>` key are byte-for-byte main's. Both
rationales are kept (leg-collapse on date, and residual anchoring).

src/session-cache.ts — PROVIDER_PARSE_VERSIONS.copilot takes this PR's
`-session-store-v3` suffix; main's getagentseal#1051 note about why a fingerprint change
is expensive is kept above it. Codex keeps main's getagentseal#1092 suffix chain untouched.

src/main.ts — getagentseal#1067 deleted the unreachable live dailyMap fallback that this
PR had taught behavioral weight. Main's deletion wins; the now-unused
isBehavioralTurn import goes with it.

tests/parser.test.ts — import union.

Also: scripts/upgrade-path/run.mjs NEW_DAILY_CACHE -> daily-cache.v26.json,
CODEBURN_COPILOT_SESSION_STORE_DB added to the getagentseal#1064 env-isolation CLEARED
list, and the CHANGELOG entry's stale "v21" corrected to v26.

Verified: tsc clean; 3132 tests pass across 223 files; test:locks 26/26;
verify:upgrade PASSED, re-deriving daily-cache.v26.json and holding durable
copilot history across the bump. getagentseal#1054's regression ("keeps three stampless
shutdown legs as :n keys with lastEventTimestamp") passes on the merged tree.
Real-corpus A/B against origin/main over 2026-07-01..2026-08-22: codex, grok,
kimicode and opencode byte-identical in export, audit and models; claude drifts
only monotonically with run order (a live session writing transcripts, confirmed
by interleaving four runs). This machine has no copilot data, so the copilot
recovery semantics rest on the suites and the upgrade-path corpus.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Codex: price cache_write_input_tokens, avoid reasoning-output double count, and apply long-context tiers

1 participant