Repository navigation
V1.8.7 - #1370
Merged
Merged
V1.8.7#1370
Conversation
- Remove cacheReadTokens truncation to promptTokens (#1343) - Skip /count_tokens endpoint in tokenRequestFinishHandler (#1344) - Add deducted guard to prevent duplicate deduction (#1345) Also add SC11 integration tests and update R&D workflow skill. Signed-off-by: zhangmiao <zhangmiao@yf-networks.com>
… output_cost_per_video billing - Add ModeResponses and /v1/responses, /v1/video/generations path detection - Add ImageInputTokens/VideoCount to TokenUsage and QuotaUsage - Add input_cost_per_image_token and output_cost_per_video price keys - Extend calcCostUnits with responses/video_generation branches - Support image input token billing in calcChatCost and calcImageGenerationCost - Skip /count_tokens endpoint and add deducted idempotency flag - Add ai_image_input_tokens/ai_video_count access log fields - Add SC05 integration tests TC11-TC14 and update design docs Signed-off-by: Kimi Code <kimi-code@example.com> Signed-off-by: zhangmiao <zhangmiao@yf-networks.com>
…che hit (#1343) The previous fix removed the cacheReadTokens > promptTokens truncation, but the problem persisted: Anthropic input_tokens only counts fresh (cache-missing) tokens and excludes cache read/write, so normalInput = promptTokens - cacheReadTokens clamped to 0 and fresh tokens were billed nothing. A 100% cache hit (input_tokens = 0) was even misclassified as guess and dropped entirely. - Normalize PromptTokens to total input (input_tokens + cache read + cache write) in all three Claude fallback parsers, matching OpenAI prompt_tokens semantics; UsedQuota derived from the normalized value - calcChatCost: normalInput = max(promptTokens - cacheRead - cacheWrite, 0) so cache creation tokens are billed once at cache_creation price - Update unit tests and SC11 TC01 integration test (812300 -> 956940), add parse-layer normalization tests - Update rmb_quota.md, claude_protocol_support.md and the modification design doc OpenAI/DeepSeek behavior is unchanged (cacheWrite is always 0).
…er layer Organize protocol knowledge (auth injection, version headers, usage field extraction, error normalization) by model_protocol instead of scattering it across bfe_basic/reverseproxy/mod_ai_token_auth/mod_body_process. - add bfe/bfe_model_protocol: ProtocolAdapter interface, compile-time registry (Get/Supports/ValidateProtocols), detect.go, openai and anthropic adapters, utils leaf package with UsageFields/ProtocolError - reverseproxy: delegate auth/version headers to adapter, add ErrorNormalizer seam before fallback status-code whitelist (default no-op), clusterSupportsAuthStyle -> modelprotocol.Supports - mod_ai_token_auth: SetApiKey/UpdateCtxByUsage delegate to adapter; accumulate semantics unchanged - mod_body_process: SSEEvent/RawEvent.GetQuotaUsage converge the three duplicated usage field chains via extractUsageFields helper - bfe_basic: GetApiKey/DetectAuthStyle delegate to detect.go - validate ModelProtocols against known protocols at startup/reload Zero behavior change; SC06 integration tests (TC-01~TC-08) all pass. Docs: sys_design/model_protocol_adapter.md, modifications/2026-09-03-model-protocol-adapter/design-changes.md
…mal input Five integration cases (SC03 TC08/TC09, SC05 TC08, SC09 TC03/TC04) still expected the pre-#1343 formula normal_input = prompt - cacheRead, which double-bills cacheWrite tokens (charged both in normal input and as cache_creation). Align test expectations and design docs with the current calcChatCost formula: normal_input = prompt - cacheRead - cacheWrite, matching rmb_quota.md. No production code changes. Full integration suite SC01-SC11 passes.
…t abort (#1352) Issue #1352: with EstimateToken=true, a client sending TCP RST right after message_start was billed by full-request estimation (prompt estimate seeded at auth time + content estimate), even though BFE had set ErrClientWrite and never received the final usage. Root causes: - tokenRequestFinishHandler only checked res.StatusCode == 200, which is meaningless for SSE (headers flushed before the stream); it never looked at req.ErrCode. - calcCostUnits bills from the token fields directly, so the prompt estimate seeded by tokenFoundProductHandler flowed into the deduction regardless of the UsedQuota guard. Changes: - bfe_basic: add responseCompleted/finalUsageSeen state to AiBasicInfo (shared by mod_body_process and mod_ai_token_auth) - mod_ai_token_auth: abort guard (ErrClientWrite/ErrClientClose/ ErrClientReset without final usage -> no deduction); reset estimate-seeded token fields when estimation is not billable; EstimateToken estimation only applies to completed responses; tokenReadResponseHandler marks non-streaming responses completed/final-usage-seen - mod_body_process: QuotaUsage gains IsFinalUsage/IsTermination flags; message_stop/[DONE]/final usage chunk detection; final usage from Anthropic message_delta is merged with prompt tokens parsed from message_start (previously dropped by the UsedQuota <= 0 gate) - bfe_model_protocol: parse the real streaming shape message_start. message.usage (issue #1352 item 3) Billing rule after fix: client abort without final usage -> free; abort after final usage -> bill actual usage; normal completion unchanged. Tests: unit tests in mod_ai_token_auth/mod_body_process/anthropic; new integration scenario SC12 (client abort billing, 3 cases) with SSE-capable mock backend and raw-TCP RST client. Full suite (go build/test/vet) passes. Docs: sys_design/rmb_quota.md 6.4/7.4/7.5, claude_protocol_support.md, ai_access_log_fields.md, model_protocol_adapter.md, configuration bfe.conf.md (EstimateToken semantics narrowed), modifications/2026-09-04-client-abort-billing-fix/design-changes.md, tests/integration/测试设计文档/scenario-SC12-客户端中断计费 Related-to: #1343 Signed-off-by: zhangmiao <zhangmiao@yf-networks.com>
fix(mod_ai_token_auth): skip full-request estimation billing on client abort (#1352)
…g Redis balance as exhausted - token_rule_load: reject only Quota < 0 so a quota plan with Quota=0 loads - QuotaPlan.HasBalance: treat a missing Redis balance key as zero balance and return QuotaExhausted instead of a 500 - redis_client: add IsKeyNotFound helper - add unit tests and SC03 integration cases TC-13/TC-14 - update token_rule.data docs (en_us/zh_cn) and rmb_quota.md Relates to #1348 Signed-off-by: zhangmiao <zhangmiao@yf-networks.com>
fix(mod_ai_token_auth): allow zero total_token quota and treat missin…
…with path TLSConfReload only relocated server_cert_conf/tls_rule_conf into the versioned dir given by ?path=, while client CA/CRL were still read from the activated dir. A client CA that exists only in the version dir made reload fail with ClientCALoad, so conf-agent never switched the tls_conf symlink and hot reload deadlocked permanently: rainway-ai-gateway/conf-agent#19 - relocate ClientCABaseDir/ClientCRLBaseDir into the version dir when they live under <confRoot>/tls_conf; custom absolute paths untouched - tlsConfLoad takes the two base dirs as parameters; the startup call site passes the parsed config values (semantics unchanged) - add unit tests and SC13 integration tests for the path redirection Docs: sys_design/tls_conf_reload_path.md, modifications/2026-09-05-tls-conf-reload-path-fix/ Signed-off-by: mileszhang2016 <zhangmiao@yf-networks.com>
fix(bfe_server): relocate client CA/CRL base dirs on tls_conf reload with path
The raw consumer API key was still persisted in pb_access3.log via three fields that the 2026-08-19 field-701 rework did not cover: - field 49 authorization: the raw Authorization header - field 801 ai_route_rule_hits[].rule_owner: apikey-type rule owner is the raw key itself - field 841 ai_auth_hit_quota_plans[]: plan id is composed of the raw key Add a unified credential masking gate (credential_mask.go), invoked as the last step of requestLogGen. It recursively scans the whole RequestLog via protobuf reflection: authenticated requests get the raw key replaced with the internal key_id; unauthenticated requests (including brute-forged keys) get affected fields emptied. Field 49 is no longer logged at all. Future fields are covered by the generic scan. Regression: SC05 integration tests assert (for every TC) a serialized byte-level scan against the raw key and a nil authorization; unit tests cover authenticated/unauthenticated/no-AiInfo paths. Fixes #1357 Signed-off-by: zhangmiao <zhangmiao@yf-networks.com>
fix(mod_access_pb3): mask raw API Key in access log (#1357)
…d SC14 integration tests P0/P1 (data plane): - inject llm-d.ai/inference-pool metadata (cluster name) into ext-proc RequestHeaders so EPP demux can route to the right cell - ordered primary/backup EPPAddr consumption with gRPC health checking, failover/failback hysteresis (EPPCheck) and error-driven per-request retry - idempotent EppClient.Close/CloseRespBody (double close panicked when the response body was closed in copyResponse and again by defer) - gRPC client productionization: per-address persistent connections, TLS verification (EPPTLS), connect/call timeouts (EPPTimeout), cluster-level sliding-window circuit breaker (EPPBreaker) - fix response body relay: bounded buffer, whole-stream abort on overflow instead of silently dropping chunks, EndOfStream always delivered - observability: labeled prometheus metrics at /monitor/epp_metrics (epp_calls_total, failover/failback, local fallback, breaker transitions, active addr index) plus flat counters on the status page Config (cluster_conf.data GslbBasic): - new optional fields EPPCheck/EPPTimeout/EPPTLS/EPPBreaker with defaults and load-time validation; docs updated in zh_cn and en_us Integration tests (tests/integration, scenario SC14): - in-process mock EPP (ext-proc v3 + health, TLS) recording pool metadata, request/response headers and bodies; real llm-d-inference-sim backends - 7 test cases: decision forwarding, SSE body relay, health-check failover, error-driven retry, unknown-pool/local fallback, EPP-down fallback, circuit breaker open - config builder support for static (real-process) backends and EPP clusters Docs: sys_design/epp_integration.md (system design), SC14 test design docs, modifications/2026-09-06-epp-ai-gateway-integration design changes.
…ses (#1359) Fixes #1359. With mod_header loaded and a global response rule hit, BFE panicked on internally-constructed responses (e.g. mod_ai_route 404 "AI route not found") whose request.HttpResponse was nil, tearing down the connection (client saw EOF). - 3.1 core fix: CreateSpecifiedContentResp now assigns resp to request.HttpResponse - 3.2 centralized fallback: after both response_got labels in ServeHTTP and ServeHTTPForAI, backfill basicReq.HttpResponse from res when still nil, covering any other internal-response construction paths - unit test: TestCreateSpecifiedContentRespAttachesHttpResponse - SC01 regression: load mod_header with a global RSP_HEADER_SET rule, add TC11 asserting the internal 404 is returned normally (status, body) with the header rule applied; test design docs updated Signed-off-by: zhangmiao <zhangmiao@yf-networks.com>
joinPath relocated reload target files by strings.Split(suffix, "/"), so on Windows an absolute backslash config path (C:\...\server_cert_conf.data) had no "/" and the whole path was appended to the version dir, producing an invalid path and breaking TLSConfReload (and ServerDataConfReload/ GslbDataConfReload) with a "path" param on Windows. Use filepath.Base, which accepts both separators on Windows and keeps the same last-segment semantics. Fixes the two pre-existing Windows failures: TestTLSConfReload_ClientCARelocatedWithPath and TestTLSConfReload_CustomAbsoluteCABaseDirNotRelocated. Signed-off-by: zhangmiao <zhangmiao@yf-networks.com>
EPP and 2 fixes
… billing item Model catalog prices may need 10-12 decimal places (e.g. 7.6234102728e-08 yuan/token). The previous load-time fixed-point conversion truncated such prices to 1e-8 yuan integers, undercharging by ~8% per token. - cluster_conf: drop pricesInt/tierPricesInt pre-conversion, the *_int key constants and the priceKeyToIntKey map; replace GetPriceInt(tier, key) int64 with GetPrice(tier, key) float64. Prices (including TierPrices) stay float64 and load-time validation now only checks non-negativity. - cluster_conf: remove the custom PriceMap/TierPriceMap MarshalJSON that forced decimal notation; scientific notation (e.g. 1.5e-6) is now accepted in cluster_conf.data. - mod_ai_token_auth: calcChatCost/calcImageGenerationCost/ calcVideoGenerationCost convert each billing item via the new quota.CalcCostUnits (round(usage * (price * 1e8))) and sum integers, so sub-item split/fallback semantics are unchanged and Redis Lua still only sees integers. - go.mod: bump github.com/bfenetworks/go-lib to v0.0.4 (adds quota.CalcCostUnits, rounds RmbToFixedPoint). - tests: update GetPriceInt cases to GetPrice; add high-precision unit tests; add SC03 TestTC15 and SC09 TestTC07 integration tests covering scientific-notation prices end to end, including tier fallback. - docs: update rmb_quota.md and add modifications design doc. For prices with at most 8 decimal places the deduction amounts are identical to the previous convert-at-load behavior. Signed-off-by: zhangmiao <zhangmiao@yf-networks.com>
feat(mod_ai_token_auth): keep model prices as float64 and convert per billing item
…reaming responses Issue #1364: when a Bearer (openai-style) request is answered with an Anthropic body (type:"message", input_tokens/output_tokens), the final usage was not recognized and the request-finish guard zeroed the billing fields, leaving only cache_read to be deducted (chunked framing) or nothing at all (Content-Length framing). - guard in tokenRequestFinishHandler now clears all 10 billing fields (prompt/completion/total/quota/cache/audio/image/video) instead of UsedQuota alone - UpdateCtxByUsage falls back to the composed cross-protocol parse chain (ParseUsageFieldsCrossProtocol) when the single auth-style adapter yields all-zero prompt/completion - isFinalUsage accepts a top-level type:"message" marker; RawEvent marks IsFinalUsage/IsTermination for a complete non-SSE JSON body - SC03 integration tests: TestTC16 (chunked) / TestTC17 (Content-Length), mock backend gains NoContentLength switch Signed-off-by: zhangmiao <zhangmiao@yf-networks.com>
The llm-d EPP framework routes request parsers by the :path pseudo-header; without it the request falls back to the legacy /v1/completions default parser and chat-completions payloads fail to parse, so EPP errors and BFE silently falls back to local WRR (200 but no EPP scheduling). Verified with integration-test SC28: without this change 6/7 cases fail, with it 7/7 pass. Signed-off-by: zhangmiao <zhangmiao@yf-networks.com>
fix(mod_ai_token_auth): bill complete usage for cross-protocol non-streaming responses
Add hardcoded length-tier price keys input/output_cost_per_token_above_
{200k,256k,272k,512k}_tokens: ModelTableCheck parses them into a sorted
lengthTier table and calcChatCost picks the highest tier exceeded by total
input tokens (tier-first/fallback semantics aligned with GetPrice; models
without tier keys keep bitwise-identical billing).
Add cache_creation_input_token_cost_1h: the 1h-TTL portion of cache writes
(usage.cache_creation.ephemeral_1h_input_tokens, fallback
cache_creation_input_tokens_1h) is parsed in the protocol adapter layer,
carried through QuotaUsage/TokenUsage as CacheWriteTokens1h, billed at the
1h price with the remainder at the base 5m price.
Access log gains ai_cache_write_1h_tokens (field 788) via bfe-access-pb
v0.3.6. Includes unit tests, integration scenarios SC15 (billing) and
SC05 TC15 (log field), plus design docs.
Signed-off-by: zhangmiao <zhangmiao@yf-networks.com>
feat(billing): support length-tier pricing and 1h-TTL cache write cost
- add gemini adapter (x-goog-api-key auth, usageMetadata extraction, stream terminal/final-usage judgment) and registry/detect wiring - sink hardcoded stream termination/final-usage judgment from mod_body_process into protocol adapters (StreamEvent abstraction); openai/anthropic behavior unchanged - extend cross-protocol usage fallback chain to three stages (openai -> anthropic -> gemini) - shouldTriggerFallback now resolves ErrorNormalizer by AuthStyle - add SC16 integration scenario and design docs Signed-off-by: zhangmiao <zhangmiao@yf-networks.com>
…docs Signed-off-by: zhangmiao <zhangmiao@yf-networks.com>
feat(model_protocol): support gemini protocol
Signed-off-by: zhangmiao <zhangmiao@yf-networks.com>
chore(release): prepare v1.8.7 release
Signed-off-by: zhangmiao <zhangmiao@yf-networks.com>
ci(license): ignore SC16 testdata_routeonly fixture files
… in openai adapter The gemini refactor (fac4e09) sank the hard-coded final-usage check into per-protocol adapters, but the openai adapter (the fallback for Bearer requests and unknown styles) kept only 'message'/'', dropping 'message_delta'. A cross-protocol stream (Bearer request answered with an Anthropic body) then lost the final usage: the completion tokens in message_delta were never merged, finalUsageSeen stayed false, and the request was undercharged (complete stream) or billed for free (client abort after final usage). SC12 TC02/TC03 integration tests caught this. Restore the legacy combined rule in the openai adapter, mirroring IsStreamTerminal which already keeps message_stop. Genuine OpenAI streams are unaffected: their chunks carry type chat.completion.chunk or no type, never message_delta. Signed-off-by: zhangmiao <zhangmiao@yf-networks.com>
… restoration Signed-off-by: zhangmiao <zhangmiao@yf-networks.com>
fix one problem
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.