_{What it does · Results · Install · Use · How it works · Scope · License}

---

What it does

Effective challenge has needed a number for fifteen years. This is the number.

SR 11-7 and its successor SR 26-2 both require "effective challenge" of a financial model: critical analysis by objective, informed parties who can identify limitations and assumptions. Neither defines it as a measurable quantity. Banks are now inserting LLMs into that second-line role as validation copilots, with no way to state, in numbers, how much of a violation one of those models can be trusted to catch.

Challenge Integrity prices it: the Maximum Undetectable Materiality (MUM), the largest seeded model-validation violation, in basis points, that a blind AI validator fails to flag. A synthetic corpus of validation artifacts carries known, seeded violations at three materiality tiers plus clean controls. Validators score the corpus blind, with no access to the answer key. score.py turns their verdicts into detection rates by tier, a false-positive rate, and a KILL/GO read against pre-registered falsification conditions.

Results

A falsification pilot: 13 synthetic model-validation artifacts (10 seeded violations across 3 materiality tiers, 3 clean controls), scored by 3 blind validators spanning capability tiers (Haiku, Sonnet, Opus). Both kill conditions cleared, and the pilot found a sharper result than the starting hypothesis.

Materiality tier	Pooled detection
Tier 1 — LOW (~1-5bp)	66.7% (6/9)
Tier 2 — MED (~10-30bp)	91.7% (11/12)
Tier 3 — HIGH (~100+bp)	77.8% (7/9)

Detection does not scale monotonically with materiality: the high tier sits below the mid tier. A 140bp, capital-relevant lookahead leak was missed by two of three validators. The naive "small violations are the blind spot" hypothesis is falsified. What actually separates caught from missed is verification depth: violations a competent reader catches by understanding the setup (target leakage, cherry-picked backtests, mislabeled limit breaches) are caught by all three validators regardless of size. Violations that require recomputing a statistic or tracing a data pipeline slip through, independent of how much money is at stake. MUM >= 140bp in this pilot. False-positive rate on clean controls: 0/9. Full readout in RESULTS.md.

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#eef1ff','primaryBorderColor':'#1B3DFF','lineColor':'#8a8a8a','fontFamily':'ui-monospace, monospace'}}}%%
flowchart LR
    A[Synthetic corpus<br/>13 artifacts] --> B[Seeded violations<br/>10 violations, 3 tiers,<br/>3 clean controls]
    B --> C[Blind validators<br/>Haiku · Sonnet · Opus]
    C --> D[score.py<br/>detection rate, FP rate,<br/>KILL/GO read]

Loading

Install

git clone https://github.com/sarthakguptaquant/challenge-integrity.git
cd challenge-integrity

No dependencies beyond the Python 3 standard library. No API keys, no network calls.

Use / Reproduce

# Score the pilot verdicts already in verdicts/ against answer-key.json
python3 score.py

# Score a different answer key or verdicts directory
python3 score.py --answer-key answer-key.json --verdicts-dir verdicts

# See all options
python3 score.py --help

To score a new validator: prompt it as a second-line model-risk reviewer against every file in corpus/, save its verdicts as a JSON file into verdicts/ shaped as:

{
  "validator_id": "your-validator-name",
  "verdicts": [
    {"id": "artifact-01", "flagged": true, "issue": "one-line description"},
    {"id": "artifact-02", "flagged": false, "issue": ""}
  ]
}

Then re-run python3 score.py.

How it works

The corpus design

10 seeded violations across 3 materiality tiers, plus 3 clean controls that structurally mirror three of the violation artifacts (same document type), so the false-positive rate is tested like-for-like rather than on an easy giveaway. Five violation archetypes, two of each, deliberately spread across tiers rather than tier-sorted, so materiality and detectability are not correlated by construction:

• VaR model with a hidden lookahead / forward-fill data leak
• PD or CECL model with target leakage (a feature that encodes the outcome)
• ES/VaR backtest run on a cherry-picked (survivorship- or window-selected) sample
• a fabricated or mis-transcribed regression statistic in an analyst memo
• a limit / risk-appetite breach mislabeled as within-appetite

answer-key.json holds ground truth (which artifacts carry a violation, its archetype, materiality tier, synthetic bp figure, and where to look) and is never referenced from inside any artifact — artifacts are blind on their own.

The KILL/GO falsification test

Three pre-registered kill conditions decide whether the pilot is worth building out:

1. Saturation (no blind spot) — killed if detection is ~100% across the board. Cleared: real misses exist, including a 140bp capital-relevant one.
2. Zero-information (no detection / no discrimination) — killed if validators cannot discriminate violations from clean controls. Cleared: 6/10 violations caught by all three validators, 0/9 false positives on clean controls.
3. Prior-art occupancy — killed if an existing materiality-priced AI-validator evaluation already occupies this framing. Requires a targeted literature search before treating a GO as final; not scored by score.py.

score.py scores conditions 1 and 2 automatically and prints a KILL/GO read.

Why verification depth, not materiality, drives detection

Violations caught by all three validators regardless of size: target leakage, cherry-picked/survivorship backtests, mislabeled limit breaches — conceptual defects a competent reader catches by understanding the setup. Violations that slipped through required recomputing a statistic or tracing a data pipeline: a 140bp VaR lookahead leak (missed by 2/3), a fabricated R-squared (missed by 2/3), a forward-fill lookahead (missed by 1/3), an inconsistent F-statistic (missed by 1/3). Materiality (bp) is close to orthogonal to detection in this pilot. Full artifact-by-artifact breakdown in RESULTS.md.

Scope

This is a falsification pilot, not a benchmark. Read the numbers with that frame:

• Every figure is synthetic. Every basis-point number, statistic, and dollar amount in corpus/ and answer-key.json is invented for this test. No real portfolio, model, or organization data appears anywhere in this repository.
• Small sample. 3 validators, 13 artifacts, 3-4 items per tier cell. A single flip moves a cell result by roughly 33 percentage points; the 11-point tier spread sits inside that noise band.
• Model-family correlation. All three validators, and the corpus author, are Claude-family. Detection and blind spots could be correlated by shared training. The next rigor upgrade is genuinely independent non-Claude validators (a GPT and a Gemini) plus human-verified violations.
• No rules-based validator yet. The current three comparison points are LLM-only; a deterministic rules-based fourth validator is not wired in.
• MUM is a floor, not a ceiling. MUM >= 140bp states the largest violation missed in this pilot, not a bound on what a larger corpus would surface.

None of these caveats are load-bearing for the pilot's go/no-go read (both kill conditions cleared cleanly), but they bound how much weight the specific percentages can carry until a larger, cross-model, human-verified build replaces them. Full list in RESULTS.md.

License

MIT. See LICENSE. Please cite if you build on it (see CITATION.cff).

---

_{sarthakguptaquant · AI x quantitative finance · sarthakgpt.com}