Skip to content

feat(ai): make the AI decision provider selectable and add Drex - #346

Open
chichengmessi wants to merge 2 commits into
OpenByteInc:mainfrom
chichengmessi:drex/decision-filter
Open

chichengmessi wants to merge 2 commits into
OpenByteInc:mainfrom
chichengmessi:drex/decision-filter

Conversation

@chichengmessi

@chichengmessi chichengmessi commented Nov 3, 2026 •

Copy link
Copy Markdown

Summary

Adds AI_DECISION_PROVIDER so the pre-trade AI decision filter can call Nace Drex instead of Jev. Drex serves the same POST /v1/systemone contract, so it goes through the existing questions, answer validation, confidence threshold, decision rules, billing, LLM fallback and fail-open. With the setting unset or jev, nothing changes.

Pointing the JEV_* settings at Drex already works, but the decision timeline then labels Drex decisions as jev.

To switch: in System Settings → AI / LLM, set AI Decision Provider to "Nace Drex" and paste the Nace API key (or set AI_DECISION_PROVIDER=drex and NACE_API_KEY). No JEV_* setting changes.

Related issue

No issue yet. This touches the entry filter, which CONTRIBUTING treats as execution-critical; I'm happy to open a scoped issue or discussion first if you prefer.

Changes

  • ai_decision_filter.py: AI_DECISION_PROVIDER=drex reads NACE_API_KEY and NACE_DEFAULT_DECISION_MODEL (default drex-max-finance) with the base URL fixed to https://console.nace.ai/v1; timeout and threshold reuse JEV_TIMEOUT_SECONDS / JEV_MIN_CONFIDENCE. Decisions are stored with provider drex, reasons drex_entry_approved / drex_entry_rejected:*, and fallback reasons drex:.... Jev requests, reasons and messages are unchanged; the shared settings-refresh debug log no longer says Jev. Drex requests send X-Nace-Client and User-Agent as quantdinger/<APP_VERSION> so Nace can attribute traffic per integration; Jev requests get neither. _jev_config / _evaluate_jev are renamed to _system_one_config / _evaluate_system_one. JEV_ENABLED applies to Jev only: choosing Drex turns Drex on.
  • settings.py: AI_DECISION_PROVIDER, NACE_API_KEY and NACE_DEFAULT_DECISION_MODEL in System Settings → AI / LLM.
  • env.example, .env.example, README.md, README_CN.md: the three keys and one paragraph.
  • test_ai_decision_filter.py: Drex settings, a Drex rejection audited as drex (with its attribution headers asserted), a Drex error falling back to the LLM, and no Nace headers on Jev requests.

Frontend follow-up: several decision-filter strings say JEV whatever the provider. For example, a Drex fallback shows "JEV confidence was below the configured threshold", and the drex_entry_* reasons have no translations, so they show as raw codes. The settings page places only JEV_* keys in the AI Decision Filter card, so the three new fields appear under the common AI settings. Using record.provider in those strings and adding the three keys to that card would cover it.

Test plan

  • pytest tests/test_ai_decision_filter.py tests/test_strategy_v2_order_gateway.py tests/test_settings_secret_masking.py tests/test_sync_env.py tests/test_bootstrap_env.py tests/test_professional_report_settings_coverage.py tests/test_ai_agent_contract_alignment.py tests/test_strategy_runtime_settings.py (73 passed)
  • ruff check, scripts/check_docs.py, git diff --check
  • Default path compared with main: identical Jev request (URL, headers, body) and stored result for a valid, a low-confidence and a malformed answer, with and without NACE_API_KEY set
  • Live, Flask app with Postgres, StrategyV2OrderGateway.submit with the filter on (no pending-order worker, no exchange): Drex rejected a BTCUSDT long against the trend (drex_entry_rejected:signal_conflict, 370 ms) and approved the short (357 ms); an invalid key fell back to the LLM (drex:401 ...); no keys gave ai_not_configured as before; provider unset used Jev as before
  • Live Drex call through AIDecisionFilter.evaluate sent X-Nace-Client / User-Agent: quantdinger/<APP_VERSION> and returned a normal decision
  • Live, provider drex, settings-file refresh forced to fail: no log line, result or stored decision mentions Jev, for a Drex decision or a Drex 401 that fell back to the LLM
  • GitHub CI

API documentation (if routes/schemas changed)

  • No routes or OpenAPI schemas changed (export_openapi.py output identical)

Drex vs Jev

Nace's Financial Decisions Benchmark (48 held-out finance benchmarks, 3,361 decisions):

Benchmark Drex Max Finance Jev 1.14
Finance Index 77.3 63.6
Accuracy % 85.7 76.7
Credit & risk 48.3 30.0
Markets & reasoning 41.5 28.1

Benchmark chart and details: Drex for Finance announcement.

On this filter: 100 entries on 10 market states, each in a strong trend on 1h, 4h and 1d. Per state there are 4 clean entries and 6 entries that each break one of the filter's own blocking criteria. Each model was called through AIDecisionFilter at the default 0.55 threshold; jev-latest answered as jev-1.14.0.

This filter (n=100) Drex Max Finance Jev 1.14
Decided at the 0.55 default 82 2
Violations rejected (of 60) 42 2
Clean entries passed (of 40) 40 0
Wrong decisions 0 of 82 0 of 2
Correct, with the LLM fallback (gpt-4o-mini) 98 68
Identical answers on a repeat request 100 81
Cost per 1,000 decisions $0.066 $0.118
  • Jev's risk_check or execution_quality confidence is under 0.55 on 98 of 100 entries. The filter hands those to the LLM fallback, or allows the order when no LLM is configured.
  • Jev's median latency is lower (0.17 s vs 0.37 s); both are far below the LLM fallback (3.8 s).
  • Cost: Drex Max Finance is $0.03 per 1M input tokens, with output free, at 2,206 input tokens per decision. Jev is priced at the $0.042 per 1M input rate implied by its own usage.cost, at 2,798 tokens per decision.
  • Full results, all slices: gist.

Questions about Drex: Nace Discord.

AI_DECISION_PROVIDER picks the System One model the pre-trade filter calls
first: jev (default, unchanged) or drex. Drex is called with NACE_API_KEY
and NACE_DEFAULT_DECISION_MODEL (default drex-max-finance) through the same
questions, validation, confidence threshold, decision converger and
fallbacks. The audit trail records provider "drex" and its reasons as
drex_entry_*; Jev requests, reasons and error text are unchanged.

Signed-off-by: Cheng Chi (Bill) <cheng@nace.ai>
Drex requests now send X-Nace-Client and User-Agent as
quantdinger/<APP_VERSION>, so Nace can attribute traffic per
integration. Jev requests are unchanged.

Signed-off-by: Cheng Chi (Bill) <cheng@nace.ai>
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant