Skip to content

verification-integration.md

Latest commit

 

History

History
50 lines (33 loc) · 3.33 KB

verification-integration.md

File metadata and controls

50 lines (33 loc) · 3.33 KB
Raw

Verification integration

Verification is pluggable behind a single interface:

interface VerificationAdapter {
  name: string;
  verify(input: VerificationInput): Promise<VerificationResult>;
}

VerificationInput is the skill package directory plus the normalized requirement and spec. VerificationResult carries static check results, per-case results, and four metrics: passRate, schemaValidRate, unsupportedClaimRate, toolErrorRate. Every adapter's result feeds the same quality gate.

local-demo (default)

Offline and deterministic. Demo-level by design — it does not execute the skill against a live model. What it actually does:

  1. Static checks: required files exist; skill.manifest.json, tools.json, eval-cases.json, and verification-rules.json parse and satisfy their schemas; every requirement tool is defined; write tools are confirmation-gated; SKILL.md contains the required sections and statuses; no forbidden claims appear.
  2. Case evaluation: for each eval case, it evaluates the case's capability checks (see skill-package-format.md) against the package content. A case passes when the package actually declares the behavior the case requires — e.g. a partial_success case fails if the skill dropped partial failure handling.
  3. Metrics: computed from case results; failed tool_failure cases count as tool errors, failed no_unsupported_claims checks count as unsupported claims.
  4. Replay data: the pipeline persists a replay artifact (input + expectation + observed result) for every failed case.

local-demo-flaky

The same verifier with deterministic failure injection (hash-picked cases are force-failed). Use it to demonstrate the failed-gate path: repair cannot fix injected failures, the gate stays red, and no package is produced.

npx tsx src/cli/forge.ts run --verifier local-demo-flaky

verification-template (optional)

Bridges to a local checkout of agent-skill-verification-template, a generic eval / observability / quality gate framework. This adapter is optional and partial — the default demo never requires it.

What it does today:

  1. Locates the repo (default ../agent-skill-verification-template, override with --verification-template-path). Missing repo → clear error, pipeline stage fails safely.
  2. Reads the reports/latest/summary.json produced by that framework's own eval run. Missing report → clear error telling you to run npm run eval there first.
  3. Maps its metrics (passRate, schemaValidRate, unsupportedClaimRate, toolErrorRate) and per-case results into the forge result shape, so the same quality gate applies.

What it does not do yet (roadmap): translate forge-generated eval-cases.json into the template's testcase format and execute them there. Until that lands, the bridge gates on the external framework's own report rather than on the freshly generated skill.

npx tsx src/cli/forge.ts run --verifier verification-template \
  --verification-template-path ../agent-skill-verification-template

Writing your own adapter

Implement VerificationAdapter, return a VerificationResult (validated with zod at the stage boundary), and register it in createVerificationAdapter in src/cli/forge.ts. Any framework that can produce the four gate metrics can drive the release decision.