test(#8037): save logs and a failure summary for every failed behaviour scenario

docs/guides/dev/behaviour-drivers.md

@@ -71,8 +71,9 @@ Use `forge.Client` for operations it already exposes; add REST helpers inside th
 ## Adding a CI driver
 
 1. Implement `ci.Driver` — `WaitForWorkflow`, `FindCompletedWorkflowRun`, `AssertNoWorkflow`, `GetRunLogs`, `DownloadArtifacts`, `DownloadNamedArtifactFromRun`, `DownloadNamedArtifactAfter`, `WaitForHarnessAgent`, `WaitForHarnessAgentRound`, `WaitForFailedHarnessAgent`, `AssertNoHarnessAgentArtifact`, `CountHarnessDispatches`.
-2. Map forge `WorkflowRun` types to portable polling logic; reuse patterns from the GitHub Actions driver in `pkg/behaviourtest/drivers/ci/githubactions/`.
-3. Register in `behaviourtest.RunSuite` for the matching `BEHAVIOUR_CI` value.
+2. Also implement the optional `ci.RunLister` interface (`ListRecentRuns`). A failed scenario's After hook uses it to save the logs of runs that no step resolved before the leased repository is deleted. Without it, only the run a step resolved has its logs collected.
+3. Map forge `WorkflowRun` types to portable polling logic; reuse patterns from the GitHub Actions driver in `pkg/behaviourtest/drivers/ci/githubactions/`.
+4. Register in `behaviourtest.RunSuite` for the matching `BEHAVIOUR_CI` value.
 
 ## Adding an install driver
 

docs/guides/dev/behaviour-testing.md

@@ -269,7 +269,7 @@ The `Given the enrolled test repository` step allocates a repo via `Driver.Alloc
 3. Creates the repo (the forge's `auto_init` provides the initial commit) and runs `fullsend github setup` (after resolving the inference WIF provider when configured).
 4. Caches the successful ensure for the duration of this lease so duplicate `EnsureRepo` calls skip redundant work.
 
-The After hook runs `CleanupScenario` (issues, PRs, ephemeral forks, hosting repos) and then `Driver.DeallocateRepo`, which deletes the leased base — after in-scenario debug collection has written workflow logs and agent artifacts under `BEHAVIOUR_ARTIFACT_DIR` — and returns the name to the pool. The next lessee of that name recreates the repo from scratch. Mint enrollment of the numbered *names* can remain pre-provisioned; the GitHub repos themselves are ephemeral around a lease. `Driver.Finalize` tears down suite-scoped resources (e.g. preview mint) and reclaims outstanding leases with an error.
+The After hook runs `CleanupScenario` (issues, PRs, ephemeral forks, hosting repos) and then `Driver.DeallocateRepo`, which deletes the leased base — after in-scenario debug collection has written workflow logs and agent artifacts under `BEHAVIOUR_ARTIFACT_DIR` — and returns the name to the pool. For a failed scenario the After hook first runs `steps.CollectFailureLogs`. It saves the logs of the repository's recent workflow runs that no step already saved, which covers a wait that timed out or a step that failed before it resolved a run. It always writes `debug-scenario-<name>-<suffix>/failure-summary.txt` with the scenario error and each run's outcome. A run whose logs cannot be fetched gets a `workflow-logs-unavailable.txt` stating why. The summary states when the run listing may be truncated or the per-scenario fetch cap was reached. A scenario that passes but whose repo deallocation fails also gets a failure summary. Everything is redacted and written `0600`, like the in-scenario logs; the redaction also masks the runner's `World.Token` and the values of credential-named environment variables (for example `*_TOKEN`, `*_PAT`, `*_KEY`), including in directory names. The next lessee of that name recreates the repo from scratch. Mint enrollment of the numbered *names* can remain pre-provisioned; the GitHub repos themselves are ephemeral around a lease. `Driver.Finalize` tears down suite-scoped resources (e.g. preview mint) and reclaims outstanding leases with an error.
 
 Concurrent callers for the same repo are serialized via `singleflight.Group` — only one goroutine runs the create+install flow while others wait. This removes the requirement for numbered `test-repo-NN` repos to be pre-provisioned in the pool org.
 
@@ -606,4 +606,6 @@ suiteRunner := godog.TestSuite{
 
 **`scm.Driver.ListPullRequestCommits` addition (breaking change):** The `scm.Driver` interface now includes `ListPullRequestCommits(ctx, owner, repo, number) ([]string, error)`, returning the commit SHAs on a pull request, oldest first. The dummy-playback suite uses the first entry as the code stage's published commit: review and fix only append commits to the branch, so unlike the head branch's live tip, that commit cannot be displaced by a later stage that races ahead. The GitHub and GitLab reference implementations pass through to the new `forge.Client.ListPullRequestCommits` method (GitHub caps results at 250 commits). This widens the required method set, so external `scm.Driver` implementations must add this method when upgrading past this release or they will no longer satisfy the interface.
 
+**`ci.RunLister` optional interface:** `ListRecentRuns(ctx, owner, repo string, limit int) ([]forge.WorkflowRun, error)` lists a repository's most recent workflow runs across all workflows. Failure log collection uses it to find runs that no step resolved. It is a separate interface, not a `ci.Driver` method, so external `ci.Driver` implementations keep compiling. Without it, failed scenarios still get a failure summary but only the run a step resolved has its logs collected. The GitHub Actions and GitLab CI reference drivers implement it.
+
 Bump the pinned version when behaviour step vocabulary or `pkg/behaviourtest` APIs change.

internal/forge/github/github.go

@@ -3985,16 +3985,23 @@ func (c *LiveClient) ListRepositoryArtifacts(ctx context.Context, owner, repo st
 	return artifacts, nil
 }
 
+// maxJobLogBytes caps how much of each job's log GetWorkflowRunLogs reads.
+const maxJobLogBytes = 1 << 20 // 1 MiB per job
+
 // GetWorkflowRunLogs downloads the logs for a workflow run.
 // It fetches the job list for the run and concatenates each job's log output.
+// When the output is not complete — more jobs than the single page of up to
+// 100 fetched, or a job log over maxJobLogBytes — a "[job list truncated:" or
+// "[log truncated:" note is embedded in the returned text.
 func (c *LiveClient) GetWorkflowRunLogs(ctx context.Context, owner, repo string, runID int) (string, error) {
 	// List jobs for this run.
-	resp, err := c.get(ctx, fmt.Sprintf("/repos/%s/%s/actions/runs/%d/jobs", owner, repo, runID))
+	resp, err := c.get(ctx, fmt.Sprintf("/repos/%s/%s/actions/runs/%d/jobs?per_page=100", owner, repo, runID))
 	if err != nil {
 		return "", fmt.Errorf("list jobs for run %d: %w", runID, err)
 	}
 	var jobsResult struct {
-		Jobs []struct {
+		TotalCount int `json:"total_count"`
+		Jobs       []struct {
 			ID         int    `json:"id"`
 			Name       string `json:"name"`
 			Status     string `json:"status"`
@@ -4039,14 +4046,23 @@ func (c *LiveClient) GetWorkflowRunLogs(ctx context.Context, owner, repo string,
 			fmt.Fprintf(&buf, "[logs unavailable: HTTP %d]\n\n", jobResp.StatusCode)
 			continue
 		}
-		logData, readErr := io.ReadAll(io.LimitReader(jobResp.Body, 1<<20)) // 1 MB per job
+		logData, readErr := io.ReadAll(io.LimitReader(jobResp.Body, maxJobLogBytes+1))
 		jobResp.Body.Close()
 		if readErr != nil {
 			fmt.Fprintf(&buf, "[failed to read logs: %v]\n\n", readErr)
 			continue
 		}
+		if len(logData) > maxJobLogBytes {
+			fmt.Fprintf(&buf, "%s\n[log truncated: job %d exceeds %d bytes]\n", string(logData[:maxJobLogBytes]), job.ID, maxJobLogBytes)
+			continue
+		}
 		fmt.Fprintf(&buf, "%s\n", string(logData))
 	}
+	// The jobs listing is a single page; say so when jobs were left out so
+	// callers do not mistake the snapshot for a complete one.
+	if jobsResult.TotalCount > len(jobsResult.Jobs) {
+		fmt.Fprintf(&buf, "[job list truncated: %d of %d jobs included]\n", len(jobsResult.Jobs), jobsResult.TotalCount)
+	}
 	return buf.String(), nil
 }
 

internal/forge/github/github_test.go

@@ -4152,6 +4152,64 @@ func TestListWorkflowRunJobs(t *testing.T) {
 	assert.Equal(t, "dispatch / Harness run (triage)", jobs[1].Name)
 }
 
+// TestGetWorkflowRunLogs_TruncationIsMarked verifies that an oversized job
+// log and a jobs listing that omits jobs are each reported in the returned
+// text, so callers can tell the snapshot is incomplete.
+func TestGetWorkflowRunLogs_TruncationIsMarked(t *testing.T) {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.URL.Path {
+		case "/repos/org/repo/actions/runs/42/jobs":
+			assert.Equal(t, "100", r.URL.Query().Get("per_page"))
+			json.NewEncoder(w).Encode(map[string]any{
+				"total_count": 3,
+				"jobs": []map[string]any{
+					{"id": 1, "name": "big", "status": "completed", "conclusion": "success"},
+					{"id": 2, "name": "small", "status": "completed", "conclusion": "success"},
+				},
+			})
+		case "/repos/org/repo/actions/jobs/1/logs":
+			_, _ = w.Write([]byte(strings.Repeat("x", maxJobLogBytes+5)))
+		case "/repos/org/repo/actions/jobs/2/logs":
+			_, _ = w.Write([]byte("small log"))
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	defer srv.Close()
+
+	client := newTestClient(t, srv)
+	logs, err := client.GetWorkflowRunLogs(context.Background(), "org", "repo", 42)
+	require.NoError(t, err)
+	assert.Equal(t, 1, strings.Count(logs, "[log truncated:"), "only the oversized log is marked")
+	assert.Contains(t, logs, "[log truncated: job 1 exceeds 1048576 bytes]")
+	assert.Contains(t, logs, "small log")
+	assert.Contains(t, logs, "[job list truncated: 2 of 3 jobs included]")
+}
+
+// TestGetWorkflowRunLogs_CompleteHasNoTruncationMarkers verifies that a
+// complete snapshot carries no truncation note.
+func TestGetWorkflowRunLogs_CompleteHasNoTruncationMarkers(t *testing.T) {
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.URL.Path {
+		case "/repos/org/repo/actions/runs/42/jobs":
+			json.NewEncoder(w).Encode(map[string]any{
+				"total_count": 1,
+				"jobs":        []map[string]any{{"id": 1, "name": "build", "status": "completed", "conclusion": "success"}},
+			})
+		case "/repos/org/repo/actions/jobs/1/logs":
+			_, _ = w.Write([]byte(strings.Repeat("x", maxJobLogBytes)))
+		default:
+			http.NotFound(w, r)
+		}
+	}))
+	defer srv.Close()
+
+	client := newTestClient(t, srv)
+	logs, err := client.GetWorkflowRunLogs(context.Background(), "org", "repo", 42)
+	require.NoError(t, err)
+	assert.NotContains(t, logs, "truncated")
+}
+
 // TestListWorkflowRunJobs_EscapesPathComponents is a regression test
 // (#7996 review): owner and repo were interpolated into the request URL
 // without escaping, so a delimiter-containing value (e.g. "#") could alter

internal/forge/gitlab/ci.go

@@ -883,14 +883,21 @@ func (c *LiveClient) GetWorkflowRunLogs(ctx context.Context, owner, repo string,
 		if remaining > maxTraceSize {
 			remaining = maxTraceSize
 		}
-		data, readErr := io.ReadAll(io.LimitReader(resp.Body, remaining))
+		data, readErr := io.ReadAll(io.LimitReader(resp.Body, remaining+1))
 		resp.Body.Close()
 		if readErr != nil {
 			fmt.Fprintf(&b, "=== Job %d (%s): error reading trace: %v ===\n", j.ID, j.Name, readErr)
 			continue
 		}
+		truncated := int64(len(data)) > remaining
+		if truncated {
+			data = data[:remaining]
+		}
 		totalRead += int64(len(data))
 		fmt.Fprintf(&b, "=== Job %d (%s) ===\n%s\n", j.ID, j.Name, string(data))
+		if truncated {
+			fmt.Fprintf(&b, "=== Job %d (%s): trace truncated at %d bytes ===\n", j.ID, j.Name, remaining)
+		}
 	}
 	return b.String(), nil
 }

internal/forge/gitlab/methods_test.go

@@ -3405,6 +3405,28 @@ func TestGetWorkflowRunLogs_TraceError(t *testing.T) {
 	assert.Contains(t, logs, "test output")
 }
 
+// TestGetWorkflowRunLogs_TruncatedTraceIsMarked verifies that a trace over
+// the per-job limit is cut and says so, so callers can tell the snapshot is
+// incomplete.
+func TestGetWorkflowRunLogs_TruncatedTraceIsMarked(t *testing.T) {
+	client, mux := setupTest(t)
+	mux.HandleFunc("/api/v4/projects/o%2Fr/pipelines/30/jobs", func(w http.ResponseWriter, _ *http.Request) {
+		w.Header().Set("Content-Type", "application/json")
+		fmt.Fprint(w, `[{"id":1,"name":"build","status":"success"},{"id":2,"name":"test","status":"success"}]`)
+	})
+	mux.HandleFunc("/api/v4/projects/o%2Fr/jobs/1/trace", func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte(strings.Repeat("x", 10<<20+5)))
+	})
+	mux.HandleFunc("/api/v4/projects/o%2Fr/jobs/2/trace", func(w http.ResponseWriter, _ *http.Request) {
+		_, _ = w.Write([]byte("small trace"))
+	})
+	logs, err := client.GetWorkflowRunLogs(context.Background(), "o", "r", 30)
+	require.NoError(t, err)
+	assert.Equal(t, 1, strings.Count(logs, "trace truncated at"), "only the oversized trace is marked")
+	assert.Contains(t, logs, "Job 1 (build): trace truncated at 10485760 bytes")
+	assert.Contains(t, logs, "small trace")
+}
+
 func TestGetWorkflowRunLogs_ListJobsError(t *testing.T) {
 	client, mux := setupTest(t)
 	mux.HandleFunc("/api/v4/projects/o%2Fr/pipelines/30/jobs", func(w http.ResponseWriter, _ *http.Request) {

pkg/behaviourtest/drivers/ci/driver.go

@@ -55,3 +55,16 @@ type Driver interface {
 	AssertNoHarnessAgentArtifact(ctx context.Context, owner, repo, agent string, after time.Time) error
 	CountHarnessDispatches(ctx context.Context, owner, repo, agent string, after time.Time) (int, error)
 }
+
+// RunLister is an optional extension of Driver for listing a
+// repository's most recent workflow runs across all workflows, newest
+// first, returning at most limit runs. The suite uses it to collect the
+// logs of every run in a failed scenario's repository before the lease
+// ends — including runs no step got hold of (a wait that timed out, a
+// failure before any run was resolved). It is a separate interface so
+// that external Driver implementations keep compiling; a Driver that
+// does not implement it still gets an explanatory failure summary in
+// place of the missing logs.
+type RunLister interface {
+	ListRecentRuns(ctx context.Context, owner, repo string, limit int) ([]forge.WorkflowRun, error)
+}

pkg/behaviourtest/drivers/ci/githubactions/githubactions.go

@@ -333,6 +333,11 @@ func (d *Driver) GetRunLogs(ctx context.Context, owner, repo string, runID int)
 	return d.Client.GetWorkflowRunLogs(ctx, owner, repo, runID)
 }
 
+// ListRecentRuns implements ci.RunLister.
+func (d *Driver) ListRecentRuns(ctx context.Context, owner, repo string, limit int) ([]forge.WorkflowRun, error) {
+	return d.Client.ListRecentWorkflowRuns(ctx, owner, repo, limit)
+}
+
 func (d *Driver) DownloadNamedArtifactFromRun(ctx context.Context, owner, repo string, runID int, artifactName string, destDir string) error {
 	artifacts, err := d.Client.ListWorkflowRunArtifacts(ctx, owner, repo, runID)
 	if err != nil {

pkg/behaviourtest/drivers/ci/githubactions/githubactions_test.go

@@ -18,6 +18,7 @@ import (
 	"github.com/stretchr/testify/require"
 
 	"github.com/fullsend-ai/fullsend/internal/forge"
+	"github.com/fullsend-ai/fullsend/pkg/behaviourtest/drivers/ci"
 )
 
 // instantAfter returns a timer function that fires immediately, allowing
@@ -3709,3 +3710,24 @@ func TestWaitForHarnessAgent_ArtifactFirstFailureReturnsRun(t *testing.T) {
 	require.NotNil(t, run, "the failed run is returned with the error so its logs can be saved")
 	assert.Equal(t, 88, run.ID)
 }
+
+func TestListRecentRuns_ImplementsRunLister(t *testing.T) {
+	t.Parallel()
+
+	client := forge.NewFakeClient()
+	client.RecentWorkflowRuns = map[string][]forge.WorkflowRun{
+		"org/repo": {{ID: 3}, {ID: 2}, {ID: 1}},
+	}
+	lister, ok := New(client, "tok").(ci.RunLister)
+	require.True(t, ok, "driver must implement ci.RunLister for failure log collection")
+
+	runs, err := lister.ListRecentRuns(context.Background(), "org", "repo", 2)
+	require.NoError(t, err)
+	require.Len(t, runs, 2)
+	assert.Equal(t, 3, runs[0].ID)
+	assert.Equal(t, 2, runs[1].ID)
+
+	client.Errors["ListRecentWorkflowRuns"] = fmt.Errorf("boom")
+	_, err = lister.ListRecentRuns(context.Background(), "org", "repo", 2)
+	require.ErrorContains(t, err, "boom")
+}

pkg/behaviourtest/drivers/ci/gitlabci/gitlabci.go

@@ -275,6 +275,11 @@ func (d *Driver) GetRunLogs(ctx context.Context, owner, repo string, runID int)
 	return d.Client.GetWorkflowRunLogs(ctx, owner, repo, runID)
 }
 
+// ListRecentRuns implements ci.RunLister.
+func (d *Driver) ListRecentRuns(ctx context.Context, owner, repo string, limit int) ([]forge.WorkflowRun, error) {
+	return d.Client.ListRecentWorkflowRuns(ctx, owner, repo, limit)
+}
+
 // DownloadArtifacts downloads all artifacts from a pipeline's jobs.
 func (d *Driver) DownloadArtifacts(ctx context.Context, owner, repo string, runID int, destDir string) error {
 	artifacts, err := d.Client.ListWorkflowRunArtifacts(ctx, owner, repo, runID)

pkg/behaviourtest/drivers/ci/gitlabci/gitlabci_test.go

@@ -15,6 +15,7 @@ import (
 	"github.com/stretchr/testify/require"
 
 	"github.com/fullsend-ai/fullsend/internal/forge"
+	"github.com/fullsend-ai/fullsend/pkg/behaviourtest/drivers/ci"
 )
 
 // instantAfter returns a timer function that fires immediately, allowing
@@ -1179,3 +1180,24 @@ func TestWaitForHarnessAgent_ArtifactFirstFailureReturnsRun(t *testing.T) {
 	require.NotNil(t, run, "the failed run is returned with the error so its logs can be saved")
 	assert.Equal(t, 88, run.ID)
 }
+
+func TestListRecentRuns_ImplementsRunLister(t *testing.T) {
+	t.Parallel()
+
+	client := forge.NewFakeClient()
+	client.RecentWorkflowRuns = map[string][]forge.WorkflowRun{
+		"org/repo": {{ID: 3}, {ID: 2}, {ID: 1}},
+	}
+	lister, ok := New(client, "tok").(ci.RunLister)
+	require.True(t, ok, "driver must implement ci.RunLister for failure log collection")
+
+	runs, err := lister.ListRecentRuns(context.Background(), "org", "repo", 2)
+	require.NoError(t, err)
+	require.Len(t, runs, 2)
+	assert.Equal(t, 3, runs[0].ID)
+	assert.Equal(t, 2, runs[1].ID)
+
+	client.Errors["ListRecentWorkflowRuns"] = fmt.Errorf("boom")
+	_, err = lister.ListRecentRuns(context.Background(), "org", "repo", 2)
+	require.ErrorContains(t, err, "boom")
+}