Skip to content

Conformance tests - #592

Draft
ariesclark wants to merge 17 commits into
mainfrom
conformance-tests
Draft

Conformance tests#592
ariesclark wants to merge 17 commits into
mainfrom
conformance-tests

Conversation

@ariesclark

Copy link
Copy Markdown
Member

No description provided.

@ariesclark
ariesclark changed the base branch from main to spec-updates August 3, 2026 23:23
@ariesclark
ariesclark marked this pull request as draft August 3, 2026 23:31
@ariesclark
ariesclark requested a balanced review from Copilot August 8, 2026 23:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a live-API conformance test harness for the VRChat OpenAPI specification. Because VRChat publishes no official spec, the repo reverse-engineers it from real traffic; this PR adds the machinery to drive ~500 Arazzo workflows against the live API, capture the responses as HAR, and compare them against the bundled description to surface undocumented fields (drift) and untested routes (coverage). It also adds CLAUDE.md codifying the evidence-first documentation discipline and updates dependencies/scripts to support the new flow.

Changes:

  • Adds test/run.sh (session-cache-aware suite runner with skip handling) and test/.env.example (credential template), plus a test/test:coverage/test:drift script set in package.json.
  • Adds a vrchat/close-schemas Redocly decorator (redocly/plugin.ts) and an unpublished test bundle (redocly.yaml) that sets unevaluatedProperties: false on object schemas so drift can detect undocumented response fields.
  • Updates .gitignore to exclude captures/credentials and refreshes dependencies (upgrades Redocly, drops unused packages, adds httpexec/totp-generator/typescript).

Reviewed changes

Copilot reviewed 6 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
test/run.sh New suite runner: boots httpexec, manages session-cache login, applies default skips, writes the whole-suite capture.
test/.env.example Credential template; two comments still use stale conformance naming (flagged).
redocly/plugin.ts Adds close-schemas decorator closing object schemas via unevaluatedProperties, with an APIConfig opt-out.
redocly.yaml Adds the unpublished test bundle that applies close-schemas for drift checking.
package.json Reworks lint/test scripts, upgrades Redocly, adds test deps, removes unused ones.
CLAUDE.md Documents the evidence-first workflow, tooling traps, and commands.
.gitignore Ignores test/.out, .env* (except the example), and all *.har captures.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread test/.env.example Outdated
Comment thread test/.env.example Outdated
# Credentials for the conformance suite. Copy to `conformance/.env` and fill in.
# `.env` is gitignored; this example is not.

# Login, used by `pnpm conformance:login` to mint a session. That step also
Base automatically changed from spec-updates to main August 9, 2026 02:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants