feat(admin): automatically retest and recover failed account models - #5169
Open
apple-ouyang wants to merge 1 commit into
Open
feat(admin): automatically retest and recover failed account models#5169apple-ouyang wants to merge 1 commit into
apple-ouyang wants to merge 1 commit into
Conversation
apple-ouyang
marked this pull request as ready for review
August 1, 2026 13:18
apple-ouyang
force-pushed
the
codex/model-error-auto-recovery
branch
from
August 2, 2026 08:24
c94da3e to
ac75612
Compare
apple-ouyang
force-pushed
the
codex/model-error-auto-recovery
branch
from
August 3, 2026 09:31
ac75612 to
288e8aa
Compare
Retry only structured system-detected account and model errors, then recover each model independently after a successful probe. Support minute or advanced cron schedules, selected failed-model probes, durable running results, and live progress in the admin panel.
apple-ouyang
force-pushed
the
codex/model-error-auto-recovery
branch
from
August 3, 2026 11:07
288e8aa to
075a97a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
error_recoveryscheduled-test mode with a simple minute interval (5 minutes by default) and an optional advanced 5-field cron schedulemodel_ids: []Safety boundaries
AICredits, manually disabled or unschedulable accounts, expired accounts, elapsed historical windows, and barestatus=errorstates are not retried or recoveredmodel_id; a non-emptymodel_idslist acts only as a failed-model allowlistDatabase and compatibility
192_add_scheduled_test_error_recovery.sqladds the trigger and retry schedule fields and backfillsscheduled_test_results.model_id193_add_scheduled_test_error_model_ids.sqladdsmodel_ids TEXT[] NOT NULL DEFAULT '{}'NULLscheduledtrigger modeValidation
go test ./internal/service ./internal/repository ./internal/handler/admin -count=1(5,361 tests passed)go test -tags=unit ./internal/service ./internal/repository ./internal/handler/admin -count=1(8,187 tests passed)model_idscreate/update round-trips (2 tests passed)ScheduledTestsPanel.spec.ts(8 tests passed) andModelMultiSelect.spec.ts(1 test passed)pnpm run typecheck,pnpm run lint:check, andpnpm run buildThe feature was deployed to a local Sub2API instance as
sub2api:0.1.169-model-error-recovery-models-optout-ac75612bby replacing only the app container. PostgreSQL and Redis were not recreated. Health, migrations 192/193, desktop UI, default-all model selection, subset selection, help text, and create/read/delete of an AIINPUT recovery plan all passed.Live AIINPUT evidence also showed the intended state machine: failed models were retried independently, successful models were cleared, later upstream regressions created new active errors and resumed retrying, and failed probes did not clear those new errors. Implementation validation actions were limited to AIINPUT accounts; no validation plan or test request was sent to other providers.
The approved behavior contract and detailed evidence are included in
docs/superpowers/specs/2026-08-01-model-error-auto-recovery.mdandPROGRESS.md.Relation to #4821
#4821 adds a one-click way to create ordinary periodic scheduled tests across accounts. This PR adds a distinct failure-only trigger mode and model-granular recovery semantics. The two features are complementary rather than competing implementations.
Closes #5184