AI Guard: redaction tests for the full provider-bound context (multi-turn history) - #7595
Draft
avara1986 wants to merge 2 commits into
Draft
AI Guard: redaction tests for the full provider-bound context (multi-turn history)#7595avara1986 wants to merge 2 commits into
avara1986 wants to merge 2 commits into
Conversation
…message APPSEC-68867 Backend RFC v1.1 and RFC v1.2 make the scope distinction normative: attack analysis targets the latest logical message, but SDS scans every model-visible string in the messages array of the current /evaluate call, so redaction_replacements may target the system prompt and historical user, assistant and tool messages as well as the latest one. Only REDACT_MIXED redacted a non-last message alongside the latest, so a tracer that redacted only the latest message passed 22 of the 24 existing scenarios while still shipping the whole history to the provider on turn 2. Adds Test_RedactionMultiTurnContext and nine generator scenarios: the RFC multi-turn example (historical SSN plus a new email, with an already redacted assistant message that must survive byte for byte), history-only redaction with a benign latest message, one replacement per role in a single call, historical tool call/result and content part, non-contiguous replacements in an eight-message conversation, a value restated across turns, and request-local paths replayed over three calls that grow then reorder the conversation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
|
🎉 All green!🧪 All tests passed 🔗 Commit SHA: c0b0b65 | Docs | View more details | Give us feedback! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Jira: APPSEC-68867
RFCs: RFC AI Guard Sensitive Data Redaction (v1.2) and RFC AI Guard Sensitive Data Redaction. Backend (v1.1)
Follow-up to #7385, which introduced the redaction corpus.
Both RFCs were revised to make one point normative:
This matters because redaction is copy-on-write: the tracer sends a redacted copy to the provider and leaves the caller's list alone, so an earlier message still holds its original value on the next turn.
The gap
Of the 24 redaction scenarios on
main, onlyREDACT_MIXEDredacted a non-last message alongside the latest one, andREDACT_SYSTEM_PROMPTredacted a two-message system prompt. A tracer that redacted only the latest message passed 22 of 24 and would still ship the entire conversation history to the provider on turn 2 — the exact failure the RFC revision was written to prevent.What this PR adds
Test_RedactionMultiTurnContext(9 tests) and 9 scenarios ingen_redaction_cassettes.py:REDACT_TURN_1+REDACT_HISTORY_AND_LATESTmessages[1]plus a new email atmessages[3], with the already-redacted assistant message atmessages[2]surviving byte for byte (no double redaction)REDACT_HISTORY_ONLYREDACT_EVERY_ROLE_IN_HISTORYREDACT_HISTORICAL_TOOL_CALLREDACT_HISTORY_CONTENT_PARTmessages[1].content[0].textin a historical multimodal messageREDACT_DEEP_HISTORYREDACT_SAME_VALUE_ACROSS_TURNSREDACT_REORDERED_CONTEXTtest_redact_paths_are_request_local, which replays turn 1 → turn 2 → reordered turn 2 across three calls. The reorder swaps the strings behindmessages[1]/messages[3], so a tracer reusing the previous response's paths writes the SSN replacement over the email message and failsPlus
test_redacted_history_in_sdk_response, asserting the list handed back to the caller — and therefore sent to the provider — has no history left. That is the guarantee redaction exists for.Keeping fixtures in sync
Unchanged from #7385: scenarios live in
SCENARIOSinutils/scripts/gen_redaction_cassettes.py, which writes both the cassettes andtests/ai_guard/redaction_scenarios.json, and cross-checks every declaredexpect_redacted/expect_removedagainst its reference implementation of the RFC algorithm../format.shruns it, so the fixtures in the repo are always the fixtures the script produces.Test results
./run.sh AI_GUARD tests/ai_guard/test_ai_guard_sdk.py::Test_RedactionMultiTurnContext→ 9 passed againstpython@4.14.0-rc1. Python'sredact_messagesis index- and cardinality-agnostic, so it passes unchanged — the new tests document a contract it already satisfies rather than asking for new work.Test_RedactionInSDKResponsegap).TEST_THE_TEST→ 437 passed../format.sh→ mypy, ruff, yamlfmt, yamllint, parser checks and the redaction-fixture drift check all clean.Notes for reviewers
Test_RedactionMultiTurnContextis activated for python (flask-poc: v4.14.0-dev) and declared at the existing redaction ref for nodejs (express4: *ref_6_13_0);missing_featurefor cpp_httpd, cpp_nginx, dotnet, golang, java, php, ruby.Test_Redactionon purpose: full-context scope is its own normative concern, so a tracer that implements last-message redaction first can be declared separately.docs/understand/scenarios/ai_guard.mdgains a subsection explaining the scope distinction and why the class exists.🤖 Generated with Claude Code