Skip to content

feat(workflow_engine): Add API for evaluation artifacts in EAP - #126358

Draft
saponifi3d wants to merge 7 commits into
masterfrom
jcallender/wfe-evals/evals-api
Draft

saponifi3d wants to merge 7 commits into
masterfrom
jcallender/wfe-evals/evals-api

Conversation

@saponifi3d

Copy link
Copy Markdown
Contributor

Description

This PR creates an API to access the evaluation-artifacts in EAP; /organization/:slug/evaluation-artifacts/.

The API can be filtered in a number of ways;

  • Identities: detector_id, workflow_id, group_id, event_id, project_id,
    detector_type
  • Evaluation Attributes: outcome, error, evaluation_type, evaluation_phase

This will allow us to lookup specific detectors, search for any problems based on the detector type, and discover any potential problems hiding in workflow_engine processing.

This API will be used for:

  • Seer Code Mode -- allowing users to ask seer why their monitor created an issue or why their alert was/wasn't fired
  • Metric Issue Graphs -- we can search for specific monitors and look at the data that was passed into the evaluations, allowing us to power the graphs with exactly the data that was evaluated
  • Internal Admin -- we'll also want to create an internal admin page. This page should be accessible by support or the team, to allow for fast investigations and supporting users

@saponifi3d
saponifi3d added this pull request to stack #126093 October 2, 2026 18:37
@github-actions github-actions Bot added Scope: Backend Automatically applied to PRs that change backend components Scope: Frontend Automatically applied to PRs that change frontend components labels Oct 2, 2026
request.GET, default_stats_period=timedelta(days=EAP_RETENTION_DAYS)
)
except InvalidParams as error:
raise ParseError(detail=f"Invalid date range: {error}")
@saponifi3d
saponifi3d force-pushed the jcallender/wfe-evals/evals-api branch from c3b4d58 to 22376cd Compare October 9, 2026 19:53
@sentry

sentry Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Sentry Snapshot Testing

Name Added Removed Changed Renamed Unchanged Skipped Status
sentry-frontend
sentry-frontend
0 0 0 0 1044 0 ✅ Unchanged

⚙️ sentry-frontend Snapshot Settings

Base automatically changed from jcallender/wfe-evals/to-eap-3 to master October 21, 2026 23:16
@getsantry

getsantry Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!


"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

@getsantry getsantry Bot added the Stale label Nov 12, 2026
@saponifi3d
saponifi3d force-pushed the jcallender/wfe-evals/evals-api branch from 22376cd to d009117 Compare November 16, 2026 23:18
@getsantry getsantry Bot removed the Stale label Nov 17, 2026
saponifi3d and others added 7 commits November 20, 2026 12:01
- Annotate every EvaluationArtifactResponse field with ApiDescription and type condition input as JsonValue, as the response coverage check on master now requires.
- Enable the evaluation artifacts flag in the agent-token public GET matrix so the endpoint row returns 200.
@saponifi3d
saponifi3d force-pushed the jcallender/wfe-evals/evals-api branch from d009117 to ceb019f Compare November 21, 2026 00:40

This branch was successfully deployed

1 active (outdated) deployment
Preview — 878a9441 Deployed Oct 2, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components Scope: Frontend Automatically applied to PRs that change frontend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants