[update-status] enable stuck saga reporting

[karencfv]

After the work marking sagas with non-transient errors as abandoned #10602 has been merged, we can enable stuck saga reporting again.

This PR just reverts #10539

Closes: #10538

[karencfv]

Update:

@jgallagher found that the same sagas that were stuck in unwinding state were still there

root@[fd00:1122:3344:109::3]:32221/omicron> select id,current_sec,creator,time_created,now() - time_created as age,saga_state from saga where saga_state in ('running', 'unwinding') order by age;
                   id                  |             current_sec              |               creator                |         time_created          |            age            | saga_state
---------------------------------------+--------------------------------------+--------------------------------------+-------------------------------+---------------------------+-------------
  b9df69fa-73a4-4a2e-be4a-d0832b88d01e | 0b1cc325-3c30-400d-80e4-59918f68cd65 | 0b1cc325-3c30-400d-80e4-59918f68cd65 | 2026-05-01 18:16:52.270956+00 | 81 days 20:21:30.353103   | unwinding
  8dbd27f3-711d-46d2-9c27-050d18e15fdb | 0b1cc325-3c30-400d-80e4-59918f68cd65 | 0b1cc325-3c30-400d-80e4-59918f68cd65 | 2026-05-01 18:16:51.678499+00 | 81 days 20:21:30.94556    | unwinding
  3aba3dda-32d7-46dd-8cf7-4e0c1ed07018 | 0b1cc325-3c30-400d-80e4-59918f68cd65 | 0b1cc325-3c30-400d-80e4-59918f68cd65 | 2026-05-01 18:16:49.807687+00 | 81 days 20:21:32.816372   | unwinding
  8b9bc46d-1b91-485d-8333-8eb502a71f60 | e6810ed6-a7ee-4980-aaa2-cafb40e98299 | e6810ed6-a7ee-4980-aaa2-cafb40e98299 | 2026-05-01 15:22:36.639842+00 | 81 days 23:15:45.984217   | unwinding
  1d249d2a-156f-43d6-881c-d1ea03c1e356 | e6810ed6-a7ee-4980-aaa2-cafb40e98299 | e6810ed6-a7ee-4980-aaa2-cafb40e98299 | 2026-05-01 15:22:33.172165+00 | 81 days 23:15:49.451894   | unwinding
  34f5a000-259d-4abe-ac93-4b3c787a9ea8 | e6810ed6-a7ee-4980-aaa2-cafb40e98299 | e6810ed6-a7ee-4980-aaa2-cafb40e98299 | 2026-05-01 15:22:32.592305+00 | 81 days 23:15:50.031754   | unwinding
  52fd4cb7-71ed-42f4-b7d2-32a167a07b53 | e6810ed6-a7ee-4980-aaa2-cafb40e98299 | e6810ed6-a7ee-4980-aaa2-cafb40e98299 | 2026-05-01 15:22:32.39906+00  | 81 days 23:15:50.224999   | unwinding
  7edd6cd9-28dc-4c68-8788-0ab57683b58a | 96d03c44-3ca6-420c-948f-64a9429a2eaf | 96d03c44-3ca6-420c-948f-64a9429a2eaf | 2026-05-01 00:51:33.820687+00 | 82 days 13:46:48.803372   | unwinding
  abe40088-e88a-4cff-bf9b-bf16b77991c4 | e1e02556-5dca-4e33-b1b6-ef534ad69f14 | 65a11c18-7f59-41ac-b9e7-680627f996e7 | 2025-07-04 04:56:50.332554+00 | 383 days 09:41:32.291505  | unwinding
  ebc5ae43-1e67-42c7-b3da-1c52e8800ec4 | e1e02556-5dca-4e33-b1b6-ef534ad69f14 | 65a11c18-7f59-41ac-b9e7-680627f996e7 | 2023-10-09 14:39:15.803991+00 | 1016 days 23:59:06.820068 | unwinding
(10 rows)

I double checked that #10602 was indeed deployed in dogfood and it was. The easiest way to check was to see if the saga table had the new columns.

It did

root@[fd00:1122:3344:109::3]:32221/omicron> SHOW COLUMNS FROM saga
                                         -> ;
    column_name    |         data_type          | is_nullable | column_default | generation_expression |            indices             | is_hidden
-------------------+----------------------------+-------------+----------------+-----------------------+--------------------------------+------------
  id               | UUID                       |    false    | NULL           |                       | {lookup_saga_by_sec,saga_pkey} |   false
  creator          | UUID                       |    false    | NULL           |                       | {saga_pkey}                    |   false
  time_created     | TIMESTAMPTZ                |    false    | NULL           |                       | {saga_pkey}                    |   false
  name             | STRING(128)                |    false    | NULL           |                       | {saga_pkey}                    |   false
  saga_dag         | JSONB                      |    false    | NULL           |                       | {saga_pkey}                    |   false
  saga_state       | public.saga_state          |    false    | NULL           |                       | {saga_pkey}                    |   false
  current_sec      | UUID                       |    true     | NULL           |                       | {lookup_saga_by_sec,saga_pkey} |   false
  adopt_generation | INT8                       |    false    | NULL           |                       | {saga_pkey}                    |   false
  adopt_time       | TIMESTAMPTZ                |    false    | NULL           |                       | {saga_pkey}                    |   false
  abandon_time     | TIMESTAMPTZ                |    true     | NULL           |                       | {saga_pkey}                    |   false
  abandon_reason   | public.saga_abandon_reason |    true     | NULL           |                       | {saga_pkey}                    |   false
  abandon_comment  | STRING                     |    true     | NULL           |                       | {saga_pkey}                    |   false
(12 rows)

My next step was to find any trace of said stuck-in-unwinding sagas in the logs in relation to saga recovery. I couldn't find any in any of the nexuses (nexi?) logs, which led me back to re-reading the saga recovery code to try to understand why we weren't hitting abandon_non_transient_failed_sagas() even though I know for certain there are non-transient failed sagas.

In the beginning of the background task, it retrieves which sagas we are going to attempt to recover:

    // Fetch the list of not-yet-finished sagas that are assigned to
    // this Nexus instance.
    let result = list_sagas_in_progress(
        &self.saga_recovery_opctx,
        datastore,
        self.sec_id,
    )
    .await;

and later the results of this list are processed for saga recovery

match result {
    Ok(db_sagas) => {
        let plan = nexus_saga_recovery::Plan::new(
            log,
            &self.rest_state,
            db_sagas,
        );
        self.recovery_check_done(log, &plan).await;
        let (execution, future) =
            self.recovery_execute(log, &plan, &recovery).await;
        self.rest_state.update_after_pass(&plan, &execution);
        let last_pass_success =
            nexus_saga_recovery::LastPassSuccess::new(
                &plan, &execution,
            );
        self.abandon_non_transient_failed_sagas(
            log,
            &execution.failed,
        )
        .await;
...
}

The clue here was in the comment when listing saga recovery candidates "sagas that are assigned to this Nexus instance". I double checked what this method did, and indeed, it is only fetching sagas from a specific SEC

/// List all in-progress sagas assigned to the given SEC
async fn list_sagas_in_progress(
    opctx: &OpContext,
    datastore: &DataStore,
    sec_id: db::SecId,
) -> Result<BTreeMap<SagaId, nexus_db_model::saga_types::Saga>, Error> {
    let log = &opctx.log;
    debug!(log, "listing candidate sagas for recovery");
    let result = datastore
        .saga_list_recovery_candidates_batched(&opctx, sec_id)
        .await
...

This means any abandoned sagas whose current SEC is non-existent any more will never show up in the saga recovery candidate list.

I'm not entirely sure what the most correct way to approach this is yet, but I'll think about this and it'd be great if we can chat about this in tomorrow's update watercooler