daemon cache routines catch up under load and amplify CPU starvation

[muqsitnawaz]

Impact

On an 18-core macOS workstation, agents-cli background work drove load average to 125.96 / 90.06 / 80.35 (about 7x CPU oversubscription). Interactive typing became visibly delayed because WindowServer was starved.

Installed agents-cli: 1.22.35.

Profile evidence

At the first sample:

load averages: 125.96 90.06 80.35
18 logical CPUs

agents __daemon-run       ~97-101% CPU, PPID 1
agents sessions focus     ~98-99% CPU
tmux server               ~100% CPU, PPID 1
WindowServer              ~43% CPU
MenubarHelper             ~0% CPU

The menu-bar LaunchAgent was healthy: one run, one live PID, and no exit. It was not the hot process. The daemon LaunchAgent was also a single live process rather than a launchd crash loop.

The daemon log showed the recurring cache routines firing continuously:

00:42 usage-refresh + fleet-cache-warm + session-cache-warm spawned
00:43 usage-refresh spawned
00:44 usage-refresh spawned
00:45 usage-refresh + fleet-cache-warm + session-cache-warm spawned
...
01:03 8 routines missed their fire
01:03 fleet-cache-warm + session-cache-warm + usage-refresh spawned, then caught up

After the first burst subsided, load was still 42.69 / 39.98 / 50.50 and the shared agents-cli tmux server still occupied one full core. The server held 26 sessions / 26 panes (10 dead panes) and only three attached clients, so the hot tmux process was not explained by a large active-client count.

Memory pressure compounded the CPU starvation: 4.46 GiB of 6 GiB swap was in use.

For comparison, a healthy 16-core Linux fleet worker sampled at load 0.17 / 0.35 / 0.51.

Mechanism in current source

apps/cli/src/lib/daemon-ticks.ts:1-20 says the daemon maintenance loops were converted into detached one-shot CLI processes executed by system routines. The three observed jobs are registered at daemon-ticks.ts:183-192; usage refresh is intended to run every 60 seconds at daemon-ticks.ts:123-140.

apps/cli/src/lib/daemon.ts:791-820 spawns each scheduled job through executeJobDetached. daemon.ts:903-940 runs backlog recovery and re-executes overdue routines unless catch-up is disabled. On an already starved interactive machine, missed ephemeral cache refreshes therefore create more work instead of being dropped.

The Factory extension has coalescing code for duplicate session subprocesses (apps/factory/src/core/cachedInFlight.ts:1-10, apps/factory/src/core/sessionIdHydrate.ts:68-99), but the live sample still caught simultaneous agents sessions ... --json children from the extension host. That appears to amplify the incident rather than initiate it.

Reproduction

1. Enable the daemon and Factory extension on a Mac with many long-lived agent/tmux sessions.
2. Leave usage-refresh, fleet-cache-warm, and session-cache-warm active at their shipped cadence.
3. Create enough CPU or I/O pressure for the daemon event loop to miss scheduled fires.
4. Observe daemon catch-up spawning the ephemeral cache routines together while Factory continues session polling.
5. Compare uptime, filtered ps, and the daemon log.

Fix direction

• Make cache warming a coalesced daemon service with one in-flight pass per cache, rather than detached cron-style CLI jobs.
• Do not catch up ephemeral cache refreshes; only the newest snapshot has value.
• Apply backpressure: skip a tick when the previous pass is still active or the host is overloaded.
• Investigate why the shared tmux server remains at one full core with only three attached clients and ten dead panes.
• Verify the Factory subprocess coalescing covers every driver in the installed extension build.

Local profiling artifacts remain on the fleet host and are intentionally omitted from this public issue.