Conformance tests - #592
Conversation
238e64f to
dfbf395
Compare
There was a problem hiding this comment.
Pull request overview
This PR introduces a live-API conformance test harness for the VRChat OpenAPI specification. Because VRChat publishes no official spec, the repo reverse-engineers it from real traffic; this PR adds the machinery to drive ~500 Arazzo workflows against the live API, capture the responses as HAR, and compare them against the bundled description to surface undocumented fields (drift) and untested routes (coverage). It also adds CLAUDE.md codifying the evidence-first documentation discipline and updates dependencies/scripts to support the new flow.
Changes:
- Adds
test/run.sh(session-cache-aware suite runner with skip handling) andtest/.env.example(credential template), plus atest/test:coverage/test:driftscript set inpackage.json. - Adds a
vrchat/close-schemasRedocly decorator (redocly/plugin.ts) and an unpublishedtestbundle (redocly.yaml) that setsunevaluatedProperties: falseon object schemas so drift can detect undocumented response fields. - Updates
.gitignoreto exclude captures/credentials and refreshes dependencies (upgrades Redocly, drops unused packages, addshttpexec/totp-generator/typescript).
Reviewed changes
Copilot reviewed 6 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
test/run.sh |
New suite runner: boots httpexec, manages session-cache login, applies default skips, writes the whole-suite capture. |
test/.env.example |
Credential template; two comments still use stale conformance naming (flagged). |
redocly/plugin.ts |
Adds close-schemas decorator closing object schemas via unevaluatedProperties, with an APIConfig opt-out. |
redocly.yaml |
Adds the unpublished test bundle that applies close-schemas for drift checking. |
package.json |
Reworks lint/test scripts, upgrades Redocly, adds test deps, removes unused ones. |
CLAUDE.md |
Documents the evidence-first workflow, tooling traps, and commands. |
.gitignore |
Ignores test/.out, .env* (except the example), and all *.har captures. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| # Credentials for the conformance suite. Copy to `conformance/.env` and fill in. | ||
| # `.env` is gitignored; this example is not. | ||
|
|
||
| # Login, used by `pnpm conformance:login` to mint a session. That step also |
This reverts commit d7998f8.
a9f54ad to
17dd7dd
Compare
4357bcb to
48e486b
Compare
No description provided.