Skip to content

feat(admin): automatically retest and recover failed account models - #5169

Open
apple-ouyang wants to merge 1 commit into
Wei-Shaw:mainfrom
apple-ouyang:codex/model-error-auto-recovery
Open

feat(admin): automatically retest and recover failed account models#5169
apple-ouyang wants to merge 1 commit into
Wei-Shaw:mainfrom
apple-ouyang:codex/model-error-auto-recovery

Conversation

@apple-ouyang

@apple-ouyang apple-ouyang commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add an error_recovery scheduled-test mode with a simple minute interval (5 minutes by default) and an optional advanced 5-field cron schedule
  • discover system-recorded account and model errors, test affected models independently, and stop retrying each model as soon as it recovers
  • let operators multi-select probe models; error recovery defaults to all available models and stores the full selection canonically as model_ids: []
  • enable auto recovery by default while preserving an explicit probe-only opt-out, and add click/keyboard-accessible help describing the recovery boundary
  • clear only the successful model's unchanged JSON state and immediately synchronize the scheduler snapshot

Safety boundaries

  • only structured errors recorded by the system are eligible
  • AICredits, manually disabled or unschedulable accounts, expired accounts, elapsed historical windows, and bare status=error states are not retried or recovered
  • model-level errors always test their actual model_id; a non-empty model_ids list acts only as a failed-model allowlist
  • account-level structured errors are probed once with the first selected model
  • model recovery uses a PostgreSQL JSONB compare-and-delete so a concurrent new error or sibling-model update cannot be cleared accidentally
  • account-level recovery is also conditional on the observed account version
  • each account can have at most one error-recovery plan, and a process-local guard prevents overlapping runs of the same plan

Database and compatibility

  • migration 192_add_scheduled_test_error_recovery.sql adds the trigger and retry schedule fields and backfills scheduled_test_results.model_id
  • migration 193_add_scheduled_test_error_model_ids.sql adds model_ids TEXT[] NOT NULL DEFAULT '{}'
  • repository writes normalize nil/full selections to an empty PostgreSQL array, never SQL NULL
  • existing scheduled plans retain their current behavior through the default scheduled trigger mode
  • no dependencies or additional worker systems are introduced; the existing per-minute scheduled-test runner is reused

Validation

  • go test ./internal/service ./internal/repository ./internal/handler/admin -count=1 (5,361 tests passed)
  • go test -tags=unit ./internal/service ./internal/repository ./internal/handler/admin -count=1 (8,187 tests passed)
  • PostgreSQL integration tests for non-empty and canonical empty model_ids create/update round-trips (2 tests passed)
  • focused PostgreSQL integration coverage for conditional per-model recovery, account recovery, sibling-model success, and same-reset rewrites
  • ScheduledTestsPanel.spec.ts (8 tests passed) and ModelMultiSelect.spec.ts (1 test passed)
  • focused runner/service tests for disabled auto recovery (7 tests passed)
  • pnpm run typecheck, pnpm run lint:check, and pnpm run build
  • independent code review: PASS, no blocking findings

The feature was deployed to a local Sub2API instance as sub2api:0.1.169-model-error-recovery-models-optout-ac75612b by replacing only the app container. PostgreSQL and Redis were not recreated. Health, migrations 192/193, desktop UI, default-all model selection, subset selection, help text, and create/read/delete of an AIINPUT recovery plan all passed.

Live AIINPUT evidence also showed the intended state machine: failed models were retried independently, successful models were cleared, later upstream regressions created new active errors and resumed retrying, and failed probes did not clear those new errors. Implementation validation actions were limited to AIINPUT accounts; no validation plan or test request was sent to other providers.

The approved behavior contract and detailed evidence are included in docs/superpowers/specs/2026-08-01-model-error-auto-recovery.md and PROGRESS.md.

Relation to #4821

#4821 adds a one-click way to create ordinary periodic scheduled tests across accounts. This PR adds a distinct failure-only trigger mode and model-granular recovery semantics. The two features are complementary rather than competing implementations.

Closes #5184

@apple-ouyang
apple-ouyang marked this pull request as ready for review August 1, 2026 13:18
@apple-ouyang
apple-ouyang force-pushed the codex/model-error-auto-recovery branch from c94da3e to ac75612 Compare August 2, 2026 08:24
@apple-ouyang
apple-ouyang force-pushed the codex/model-error-auto-recovery branch from ac75612 to 288e8aa Compare August 3, 2026 09:31
Retry only structured system-detected account and model errors, then recover each model independently after a successful probe.

Support minute or advanced cron schedules, selected failed-model probes, durable running results, and live progress in the admin panel.
@apple-ouyang
apple-ouyang force-pushed the codex/model-error-auto-recovery branch from 288e8aa to 075a97a Compare August 3, 2026 11:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(admin): automatically retry and recover failed account models

1 participant