refactor(sre): configure the SRE agent at install; aep-api serves its handoff - #859
Merged
tharindulak merged 7 commits intoOct 4, 2026
Merged
Conversation
The agents' discovery MCP hand-writes MCP's Streamable HTTP in its single-response form: JSON-RPC types, initialize/ping/tools/list dispatch, notification 202s and the tool-result writers. Move that protocol layer into platform/mcprpc so a second surface can serve its own tools over it without an MCP SDK; mcpdiscovery keeps its tools and behaves exactly as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… handoff The OpenChoreo SRE agent runs one Deployment per plane with one model and one static MCP header, so it can only ever serve one org. AE still kept per-org machinery around it: an SRE model connection table, a per-org handoff token in org_secrets, a reconciler pushing both into the agent's Secret, and aep-mcp-server forwarding the agent's bearer to two REST operations through a public-edge gate. Replace all of it with install-time values and an in-process MCP surface. aectl sre install: - probes --llm-api-key-file/--llm-model/--llm-base-url (https only, no redirects) and writes the model straight into sre-agent-aep; a re-run without them keeps the model, and with no model the agent waits at 0 replicas, as before; - generates the handoff key once (32 random bytes, reused unless --rotate-handoff-token) into both sre-agent-aep and aep-api's sre-handoff Secret, and passes its hash to the platform release so a rotation rolls aep-api; - removes the aep-api-sre-push Role/RoleBinding and the sre-model-seed Secret earlier versions installed. aep-api: - serves search_related_issues and create_issue at POST /internal/v1/sre-handoff/mcp (sourcecontrol/issues, over platform/mcprpc), calling the issue service in process with the handoff's incident context bound, behind a constant-time check of the install-time key (SRE_HANDOFF_ORG / SRE_HANDOFF_TOKEN); - drops the REST handoff gate, the sreagent reconciler, the kubeobs client, the SRE model connection service and the SREAgent connection capability (contract regenerated, console fixtures updated); - migration phase23 drops org_sre_model_connections and the SRE org_secrets rows; auto-RCA now follows SREHandoff.Enabled. Removed: services/aep-mcp-server, its image (CI, release, skaffold, dev Makefile, aectl --mcp-server-image) and chart templates; the platform chart routes exactly the handoff path to aep-api instead. The handoff skill moves next to the extension it is mounted with. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0040 records the decision and supersedes ADR-0038's two SRE model connection amendments. services/aep-api/design/sre-handoff.md replaces sre-model-connection.md; the runbook, security notes, install design note, glossary and READMEs describe the install-time model and handoff key and aep-api's handoff surface. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Review skippedToo many files! This PR contains 129 files, which is 29 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. ⚙️ Run configuration
⛔ Files ignored due to path filters (3)
📒 Files selected for processing (129)
You can disable this status message by setting the
Comment |
server_gen.go embeds the contract gzip-compressed, and compress/flate's output differs between Go releases: the copy generated with Go 1.27.1 did not match what CI's Go 1.26.0 (go.work) produces, so gen-api-check failed. Regenerated with GOTOOLCHAIN=go1.26.0; only the embedded spec's bytes change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tharindulak
marked this pull request as ready for review
October 2, 2026 18:40
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
Shall we update the SRE agent to send the org namespace with each mcp tool call? Right now aep-api always uses SRE_HANDOFF_ORG, so incidents from other orgs go to the wrong org or fail. With the org namespace being sent in each tool call, we wouldn't need |
Conflicts: - migrate: main shipped phase23_agent_guardrail_applications, so the SRE drop migration is renumbered phase24_drop_sre_model_connections and runs after it (design note and ADR-0040 follow). - mcpdiscovery: main's new list_guardrail_policies tool moves onto the shared mcprpc helpers. - gen: server_gen.go regenerated from the merged contract with Go 1.26.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rver One SRE agent serves every org on its observability plane, but the handoff filed every incident under the one org `--org` named: another org's incident failed, or, on a project-name clash, landed in the wrong org's repo with that org's RCA in it (review of wso2#859). Both tools now take a required `namespace`, the alert's OpenChoreo namespace, and `SRE_HANDOFF_ORG` / `aectl sre install --org` are gone. That value reaches aep-api through a model that reads pod logs, so it is a claim: before either tool reads or writes, aep-api asks the observer (POST /api/v1alpha1/alerts/query, clients/observability.AlertQuerier, with its own service token) whether an alert fired in the last hour for that namespace and project, and, for create_issue, that component (as given or with the project prefix). No alert is a 403 tool error, an observer that cannot answer a 503; either way nothing is read or filed. The handoff key now authenticates the agent, not an org, and aep-api refuses to start the handoff without OBSERVER_URL and its service credential. The skill and CONTEXT.md tell the agent to copy the namespace from the alert; ADR-0040, the design note, runbook and security notes record the check and its bound. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tharindulak
added a commit
to wso2-tharindu-SF/sample-6
that referenced
this pull request
Oct 3, 2026
…ervice1 (#5) (#8)" (#9) This reverts commit cff7021, restoring service1 to v1 so the divide-by-zero on an empty catalog can be triggered again to test the SRE handoff (wso2/labs-agentic-engineer#859). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
kaje94
approved these changes
Oct 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Configures the OpenChoreo SRE agent entirely at install and serves its handoff from aep-api, as proposed in #846's review. The agent runs one Deployment per observability plane, with one model and one static MCP header. The per-org machinery around it (an SRE model connection table, a per-org token, a reconciler and a separate MCP server) served exactly one org anyway.
Following the review on this PR, one installation now serves every org on its plane. Each handoff call names its org, and aep-api verifies that claim against OpenChoreo's own alert record. There is no
--org.Decision recorded in ADR-0040; design note:
services/aep-api/design/sre-handoff.md.What changes
aectl sre installprobes--llm-api-key-file/--llm-model/--llm-base-url(https only, no redirects) against<base URL>/models, then writes the model straight into the agent's Secretsre-agent-aep. A re-run without the flags keeps the model; without any model the agent waits at 0 replicas, as before.--rotate-handoff-token) intosre-agent-aepand aep-api'ssre-handoffSecret. It passes the key's hash to the platform release, so a rotation rolls aep-api. The key authenticates the agent, not an org.POST /internal/v1/sre-handoff/mcpservessearch_related_issuesandcreate_issuein process, over the JSON-RPC plumbing now shared with the discovery MCP (platform/mcprpc, no MCP SDK). The tool descriptions and wire shapes are unchanged fromaep-mcp-server.namespace, which the agent copies from the alert.POST /api/v1alpha1/alerts/query, with aep-api's own service token) whether an alert fired in the last hour for that namespace and project, and, for a create, that component. It accepts the component as given or with the project prefix.403tool error; an observer that cannot answer is a503. Either way nothing is read or filed.OBSERVER_URLand the service credential; aep-api and the chart refuse to enable it without them.SRE_HANDOFF_ORGandaectl sre install --org;sreagentreconciler, thekubeobsclient, the SRE model connection service and theSREAgentconnection capability (contract regenerated, console fixtures updated);aep-mcp-server, with its image (images/release workflows, skaffold,make dev-update,aectl platform update --mcp-server-image) and chart templates. The chart'saep-api-sre-handoffHTTPRoute matches exactly the handoff path.phase24_drop_sre_model_connectionsdropsorg_sre_model_connectionsand the SREorg_secretsrows. It runs aftermain'sphase23_agent_guardrail_applications;phase22stays because databases have run it.sre-model-seedSecret.CONTEXT.mdtell the agent to copy the namespace from the alert. The design note, runbook, security notes, install note and glossary are updated.What the org check does and does not prevent
wc-…namespaces are not org handles, so the handoff resolves to no org there. It is not enabled on WSO2 Cloud today.Testing
go vet, the full suite,make deadcode-check, golangci-lint (0 issues) andmake gen-api-checkpass with Go 1.26.0. Thephase22→phase23→phase24migration tests pass against Postgres (testcontainers).actionStatuses, the 409 conflict, plan-issue annotation and the tool list.go vet, all tests and golangci-lint pass. New tests cover model resolution, the probe (accept, reject without echoing the key, no redirects), key reuse and rotation, the agent Secret and scaling, and the removal of--org.observer.baseURL.make dev-envfrom this branch (WITH_AGENT_MANAGER=0, seeded with agpt-5.4key), using a sample app whose service1 divides by zero on an empty catalog:openai:gpt-5.4with theaeextension, and aep-api loggedSRE handoff enabledwith noSRE_HANDOFF_ORG. From the agent's pod, the endpoint answered200with both tools requiringnamespace, and401with a wrong key.GET /average-score(only those two requests) returned a500/ by zero, and the alert fired.search_related_issuesandcreate_issuewithnamespace=default. aep-api's observer check accepted both, so aep-api's service token can read alerts. The issue was filed in that org's repo.GET /average-scoreon the empty catalog then returned200 {"average":0}, with no error log and no new alert.Known, not caused by this PR
make deadcode-ts-checkreportspackages/web-search/src/aep-web.tsandmcp.ts, as onmain.promote-task-from-issuehas no caller in this repo, and had none before; its comments now say so. Removing the operation is left for a follow-up.aep-mcp-server.bao server -dev, so a k3d node restart wipes it (and the CoreDNShost.k3d.internalentry). This hit testing twice on an overloaded VM; persistent OpenBao storage for dev is a separate fix.🤖 Generated with Claude Code