[FEATURE] Run the live UI Playwright suite in CI against the built UI image

[Charlesthebird]

Problem

CI's ui-tests job proves the UI is self-consistent, not that it works. It runs yarn typecheck, yarn lint, yarn test, and yarn test:pw — and test:pw is the mock suite, served by MSW with no backend at all. A backend contract change cannot fail it.

#2653 is what that looks like in practice: #2638 stopped populating three fields the Substrate page renders, every check stayed green, and it was found by hand.

A live suite already exists and CI never runs it:

• ui/playwright/live/ holds pages.spec.ts, agent-lifecycle.spec.ts, and substrate.spec.ts, run by yarn test:pw:live.
• No workflow references test:pw:live or UI_LOOP_LIVE.

Live mode also does not test the shipped artifact. LIVE_COMMAND in ui/playwright.config.ts is yarn dev, so the suite hits a Vite dev server proxying to the cluster — not the nginx image an operator runs.

Proposal

Run the live suite in CI against the real built UI image.

• Serve the image from ui/Dockerfile, not yarn dev, so the run exercises what ships: nginx addressing and the runtime window.environmentVariables rewrite, which no dev-server run touches.
• Point the suite at the port-forwarded svc/kagent-ui rather than a Vite port. playwright.config.ts is already written for this — the comment above LIVE_COMMAND names a probed port-forward as the case that replaces that line.
• Reuse what exists rather than building a second path: scripts/setup-cluster/setup-cluster.sh already builds ui/Dockerfile, sets it on deploy/kagent-ui, and port-forwards the service on 8080. The test-e2e job already stands up a Kind cluster with helm-install, and build (ui) already builds the image.

Keep chat on the mock backend

Chat stays where it is. Its journeys need deterministic streaming deltas, tool call and result ordering, cancellation, and a failed turn with retry — none of which a real model gives you reliably, and all of which the mock suite already asserts.

The live directory has no chat spec today, and it should not gain one. Live coverage is for the surfaces where a controller response is the thing under test.

Notes

• ui/playwright/DEFERRED.md already anticipates this: the cleanup.spec.ts entry is deferred "while the suite runs on the mock backend" and says to revisit "if the suite gains a live-backend mode".
• Worth adding once it runs: assertions on the Actor Templates columns, which is the gap [BUG] Harness column is always empty in the Substrate page's Actor Templates table #2653 fell through.
• The mock suite keeps its job. It is fast, runs both engines, and covers error paths a real cluster will not produce on demand. This adds a lane rather than replacing one.

---

🤖 written by Claude