ci: ratchet type errors in apps/sim test files (#8959)

* ci: ratchet type errors in apps/sim unit tests per file

apps/sim's tsconfig excludes *.test.ts(x), so ~3.4k test files are never
type-checked and any new test can add errors. check:test-types runs the
native tsc on apps/sim/tsconfig.tests.json (the app config with tests left
in), counts errors per file, and compares with a shrink-only baseline,
matching check:explicit-any's semantics: a rise or a new file fails, a drop
fails until --update shrinks it, --update never raises, --update --init only
creates a missing baseline. Output it cannot place (global diagnostics, a
crash) fails outright instead of reading as fewer errors.

It type-checks the whole app, so it runs in its own test-types job gated
like test (wired into ci-gate.sh), not in check:audits. On failure the job
uploads counts.json and the compiler output; --counts <file> lets the
baseline be bootstrapped or shrunk from that artifact without a local
compile.

* ci: commit the test type-error baseline from CI and drop the one-time --init path

Baseline from run 38342772080's test-types counts: 3209 errors in 874 files.
--update stays shrink-only; a missing baseline is restored from git.

* ci: ratchet test type errors per file and TS code, with command-level tests

A per-file count let one new error replace a fixed one with the total flat.
The baseline is now { file: { TSnnnn: count } }: any (file, code) count
that rises fails, including a new code in a baselined file, and --update
stays shrink-only per pair. The key stops at the code: line and column
move with edits, and messages print types that change with unrelated
source or compiler versions.

--diagnostics <file> replaces --counts: it reads a saved compiler output
(the CI artifact's diagnostics.txt) through the same parser and undercount
checks, so a fix can shrink the baseline without a local compile. The
baseline was migrated from run 38342772080's diagnostics.txt with the
script's parser: 3209 errors in 874 files across 52 codes, every per-file
total equal to the previous baseline's.

Command-level tests run the real script on fixture diagnostics: unchanged,
a rising count, a new code replacing a fixed error, a new file, a dropped
count, --update refusing a raise, --update shrinking, and unparseable or
inconsistent output. The checks.yml comment now describes the artifact as
input for shrinking the existing baseline.

* ci: verify saved test-type diagnostics against the compile's exit status

--diagnostics read an empty or truncated diagnostics.txt as fewer errors,
so a failed compile's artifact plus --update could erase the baseline.
--report now writes status.json (exit code and printed error count) once
the compile's output and exit code agree, and --diagnostics requires it:
exit 0 must mean no errors, the parse must match the recorded count, and
a missing or malformed status fails. Command-level tests cover each case
and that --update never writes the baseline from such input.

Also passes a route context to the socket-token route test's POST call,
the one new test type error (TS2554) the ratchet caught after #8958.

* ci: read the saved compiler status through toRecord

Author: waleedlatif1

.github/scripts/ci-changes.sh

@@ -6,7 +6,8 @@
 # diff sets all of them.
 #
 #   full          integration, e2e, e2e-build, e2e-mobile: any file not allowlisted below
-#   test          the Vitest shards: content, docs app, and desktop changes, plus `full`
+#   test          the Vitest shards and test-types: content, docs app, and desktop changes,
+#                 plus `full`
 #   build         the app's `next build`: content changes, plus `full`
 #   desktop_live  the live Electron suite: any file outside its own skip list
 #

.github/scripts/ci-gate.sh

@@ -6,7 +6,7 @@
 # A skip passes only for a gated job whose `changes` flag is exactly `false`:
 #
 #   integration, e2e, e2e-build, e2e-mobile  full
-#   test                                     test
+#   test, test-types                         test
 #   build                                    build
 #   desktop-live                             desktop_live
 #
@@ -25,7 +25,7 @@ fi
 
 failed="$(jq -r '
   {"integration": "full", "e2e": "full", "e2e-build": "full", "e2e-mobile": "full", "test": "test",
-   "build": "build", "desktop-live": "desktop_live"} as $gate
+   "test-types": "test", "build": "build", "desktop-live": "desktop_live"} as $gate
   | (.changes.outputs // {}) as $flags
   | to_entries[]
   | select(.value.result != "success")

.github/workflows/checks.yml

@@ -653,9 +653,9 @@ jobs:
       # Every workspace, not just realtime. packages/emcn, packages/utils,
       # apps/desktop and apps/docs had no type check in CI at all; apps/sim's
       # source was covered only as a side effect of `next build` in the separate
-      # `build` job. Note this does NOT cover apps/sim's tests — its tsconfig
-      # excludes *.test.ts(x), and including them today surfaces ~2.2k errors,
-      # so that is its own cleanup rather than a gate to switch on here.
+      # `build` job. This does NOT cover apps/sim's tests — its tsconfig
+      # excludes *.test.ts(x), which carry ~2.2k errors; the `test-types` job
+      # ratchets those per file instead.
       - name: Type-check all workspaces
         run: bunx turbo run type-check
 
@@ -672,6 +672,41 @@ jobs:
           fi
           echo "✅ Schema and migrations are in sync"
 
+  # Type errors in apps/sim's unit tests, ratcheted per file and TS code against a committed
+  # baseline (scripts/check-test-types.ts). It type-checks the whole app with its ~3.4k test files:
+  # lint's uncached apps/sim type-check alone takes ~35 s on this runner, so it gets its own job
+  # beside lint rather than a place in check:audits, which also runs on laptops. Gated like `test`.
+  # On failure the compiler output and its exit status are uploaded, so a fix can shrink the
+  # existing baseline from them (`--diagnostics <file> --update`) without a local compile; nothing
+  # ever raises the baseline.
+  test-types:
+    name: test-types
+    needs: changes
+    if: needs.changes.outputs.test == 'true'
+    runs-on: *runner-4vcpu
+    timeout-minutes: 10
+
+    steps:
+      - *checkout-mirror
+      - *checkout-plain
+
+      - name: Setup workspace
+        uses: ./.github/actions/setup
+        with:
+          provider: ${{ vars.CI_PROVIDER }}
+
+      - name: Ratchet type errors in apps/sim tests
+        run: bun run scripts/check-test-types.ts --report "$RUNNER_TEMP/test-types"
+
+      - name: Upload compiler output and counts
+        if: failure()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: test-types
+          path: ${{ runner.temp }}/test-types/
+          if-no-files-found: ignore
+          retention-days: 14
+
   # The root scripts and every workspace's Vitest suite (`bun run test`), split in two. apps/sim is
   # about 98% of the time, so only its files are sharded; shard 1 also runs the root scripts and
   # the other workspaces. Each shard keeps its own Turbo cache: pass-through args are part of the
@@ -876,7 +911,8 @@ jobs:
   # on a cancelled run would report a cancelled head commit as passing.
   ci:
     name: ci
-    needs: [changes, integration, e2e, e2e-mobile, desktop-live, lint, test, build, e2e-build]
+    needs:
+      [changes, integration, e2e, e2e-mobile, desktop-live, lint, test-types, test, build, e2e-build]
     if: ${{ always() }}
     runs-on: *runner-2vcpu
     timeout-minutes: 5

CLAUDE.md

@@ -22,7 +22,7 @@ This file (also `AGENTS.md`) holds the repo-wide rules. Area detail lives in `.c
   - `compareStrings(left, right)` from `@sim/utils/string` — code-unit ordering for hashes, fingerprints, and cross-process comparisons; never `localeCompare` there
   - `backoffWithJitter(attempt, retryAfterMs, options?)` / `parseRetryAfter(header)` from `@sim/utils/retry` — never reimplement exponential backoff inline
 - **Deployment flags in the browser**: client code inside a workspace, organization, or standalone settings surface reads `hosted`, `billingEnabled`, `chatEnabled`, and the enterprise feature set through `useDeploymentShape()` (components) or `getDeploymentShape()` (block conditions, stores, helpers) from `@/lib/core/config/deployment-shape`, never `isHosted`/`isBillingEnabled`/... from `env-flags`. Those constants freeze at module init from the root layout's `NEXT_PUBLIC_*` transport, which Next's bare 404 shell and `global-error` never emit, so a recovered tab would render Sim Cloud as self-hosted; the reader is seeded from the server-resolved workspace host context, organization layout, or standalone settings layout instead. Server code keeps reading `env-flags`.
-- **Type-checking**: `bun run type-check` at the root checks every workspace (`bun run type-check` inside a workspace checks just that one). `apps/sim`'s tsconfig excludes `*.test.ts(x)`, so its tests are not type-checked. Never remove the `@typescript/native` alias from the root `devDependencies`: nothing imports it, but without it a bare `tsc` resolves to the ~10x slower TypeScript 6 compiler. `bun run check:native-typecheck` enforces this; its header explains the bin-resolution trap.
+- **Type-checking**: `bun run type-check` at the root checks every workspace (`bun run type-check` inside a workspace checks just that one). `apps/sim`'s tsconfig excludes `*.test.ts(x)`, so `type-check` skips its tests; CI's `test-types` job ratchets their errors per file and TS code instead (`check:test-types`, whose header says how to fix a finding): a test file may not gain one. Never remove the `@typescript/native` alias from the root `devDependencies`: nothing imports it, but without it a bare `tsc` resolves to the ~10x slower TypeScript 6 compiler. `bun run check:native-typecheck` enforces this; its header explains the bin-resolution trap.
 
 ## Architecture
 

apps/sim/app/api/auth/socket-token/route.test.ts

@@ -41,7 +41,7 @@ const IP_BACKSTOP = 300
 async function mintAs(userId: string | null, ip: string) {
   requestUtilsMockFns.mockGetClientIp.mockReturnValue(ip)
   authMockFns.mockGetSession.mockResolvedValue(userId ? { user: { id: userId } } : null)
-  return POST(createMockRequest('POST'))
+  return POST(createMockRequest('POST'), { params: Promise.resolve({}) })
 }
 
 describe('POST /api/auth/socket-token', () => {

apps/sim/tsconfig.tests.json

@@ -0,0 +1,4 @@
+{
+  "extends": "./tsconfig.json",
+  "exclude": ["node_modules", ".next", "vitest.config.ts", "vitest.setup.ts"]
+}

package.json

@@ -86,6 +86,7 @@
     "check:unused-exports": "bun run scripts/check-unused-exports.ts",
     "check:explicit-any": "bun run scripts/check-explicit-any.ts",
     "check:file-names": "bun run scripts/check-file-names.ts",
+    "check:test-types": "bun run scripts/check-test-types.ts",
     "desktop-bridge-contract:update": "bun run scripts/check-desktop-bridge-contract.ts --update",
     "mship-contracts:generate": "bun run scripts/sync-mothership-stream-contract.ts",
     "mship-contracts:check": "bun run scripts/sync-mothership-stream-contract.ts --check",