feat(cli): let multi-account users act on another account per command [RED-1122] [ship]

[hlenke]

Problem

RED-1122: after a device-flow login (#1484), a user with several accounts gets select_account from every command, including whoami. The only way out the CLI suggests is login --account-id, which stores a default, so an agent doing cross-account work (reproducing in one account, looking up a CI test session in another) switched the default four times. A cross-account lookup just said "No such test session." Separately, an agent had to wait for the user to say "done" after relaying the login code.

The stored login key belongs to the user, so CHECKLY_ACCOUNT_ID=<id> can pick any of their accounts. On main, though, only the stored-login path knew this; everywhere else (login, the inline login, error messages) a CHECKLY_ACCOUNT_ID was treated as half of an API-key pair.

Changes

Accounts and credentials

• Only CHECKLY_API_KEY counts as environment credentials (config.getCredentialSource(): login | account_override | environment). A CHECKLY_ACCOUNT_ID alone picks the account for the user's login, for every command while it is set. The CLI can't tell a one-command prefix from an export or a .env entry.
  • Without a stored login, an authenticated command starts the device login and then uses that account, storing no default. CI keeps an error, which now names both options.
  • login no longer refuses. Without --account-id it finishes without asking for an account, since the variable already names it.
  • An id the login doesn't have fails with the list of accounts it does have.
  • login, switch and whoami say when the variable takes precedence over the stored default. login and switch reason about the stored login, and activateAccount checks the account it activates.
  • switch still asks for a default after an inline login under the variable, which stores none.
  • --help describes CHECKLY_ACCOUNT_ID on its own.
• select_account offers a per-command choice, next to making the account the default (login --account-id <id>). An inline login passes in its own command line.
• switch without --account-id no longer draws a menu nobody can answer (agent mode, CI, no terminal). An agent gets select_account with the stored defaultAccount; CI and terminal-less runs get an error listing the accounts.
• whoami
  • Shows the default account and the user's other accounts, and says where the account comes from.
  • --output json reports user, account, accountSource, defaultAccount and otherAccounts. defaultAccount is null for API key credentials.

Login

• checkly login --wait waits for the approval of the code an earlier run printed, so the user doesn't have to say "done". login_required.next points to it.

  • If the code expires while answers still come from a proxy or captive portal, the login reports network_error instead of a bare expired_token. pollForTokens decides this, so it applies to every login that polls, and a code that simply went unapproved after such answers cleared stays expired_token.
  • If another run used the same code, it waits briefly for that run's key.
  • It only forgets the code it polled.

• Agent success lines carry defaultAccount (the stored default, or null): a login under CHECKLY_ACCOUNT_ID returns logged_in without storing one. The "account no longer available" notice stays in its message.

Not found

• Not-found errors name the account searched and say how to look in another, when the key comes from a login. With --output json, they also carry hint and searchedAccount: {id, name} fields.
• BaseCommand.withNotFoundHint() covers commands with their own not-found message (checks delete, rca run).

Docs

• Skill + initialize reference: --wait, a "Working with several accounts" section (listing, one-command use, CHECKLY_ACCOUNT_ID in the shell or .env, changing the default, not-found lookups), and a note that a name the user gave counts as their choice if it matches exactly one account. skills/checkly/SKILL.md is synced.

Deliberately left out

• Global --account-id flag: the name collides with account usage series --account-id (a filter) and with the persisting flag on login/switch.
• Remembering the account across re-logins: logout clears the config, and a fresh machine has nothing to remember. A re-login under CHECKLY_ACCOUNT_ID after the stored login became unusable stores no default either; an agent login spans two runs, and the first has already dropped the old one.
• Cross-account lookup by ID: a follow-up. Agents probe accounts one at a time today.
• Public docs (checkly-login, checkly-whoami, checkly-switch, authentication): docs(cli): document multi-account use, login --wait and whoami --output json [RED-1122] docs#566, to merge once this ships.

How this was tested

Automated

• Unit:
  • Credential sources (config.getCredentialSource).
  • validateAuthentication by source, including the account list for a wrong CHECKLY_ACCOUNT_ID.
  • login: --wait approval, network failure, both races and the key-wait timeout; the inline and explicit login under CHECKLY_ACCOUNT_ID; refusal only for API keys; the override notices.
  • switch: agent, CI and terminal-less paths; the stored default under the override; the warning; choosing a default after an inline login under the override.
  • pollForTokens: network_error only while unexpected answers continue at expiry.
  • whoami: text and JSON for all sources.
  • AuthCommand.init in CI with only CHECKLY_ACCOUNT_ID.
  • AuthCommand.catch and CommandStyle.longError hints in text and JSON, checks delete, rca run.
• e2e (login-device-flow.spec.ts, fake Auth0 + API, 9 cases):
  • login --wait.
  • CHECKLY_ACCOUNT_ID per command.
  • A project .env pinning the account on a fresh machine.
  • The inline login under the variable.
  • A wrong id listing the accounts.
  • CI without a key.
• Local: pnpm test (4262 passed), pnpm lint.

Scenario matrix

12 scripted cases were run against both a main build and this PR, using a fake API. A login key works with all of the user's accounts; an API key works with one.

Case	main	PR
Not logged in, .env pins an account	"CHECKLY_API_KEY is not set"	Starts the login; then uses that account
CI, CHECKLY_ACCOUNT_ID only	"CHECKLY_API_KEY is not set"	Names both: log in, or add CHECKLY_API_KEY
Logged in, CHECKLY_ACCOUNT_ID=x checkly login	Refuses (env_credentials)	already_logged_in plus a note that the variable wins
Logged in, mistyped CHECKLY_ACCOUNT_ID	NotFoundError: Account not found	Lists the accounts the login works with
switch (agent mode or CI)	Draws an arrow-key menu, exits 0	Agent: select_account with defaultAccount; CI: error listing the accounts
switch --account-id under CHECKLY_ACCOUNT_ID	Switches silently	Switches and warns that the variable takes precedence
Session in another account, --output json	{"error","detail"}	Adds hint and searchedAccount
API key credentials	login refuses	Unchanged

Blind agent benchmark

Each fresh Claude agent got a realistic user request, a sandbox shell, the skill of the build under test, and a fake Auth0 + Checkly API. The fake "user" approves a login 20 s after the code is issued and never replies. Each run had its own server, port and HOME, and both builds were installed from packed tarballs outside any git repo. Every run was graded by a script against the fake API's request log and the CLI's saved config, plus a check that the project's .env was left unchanged.

Scenario	Build	Pass	Avg commands	Avg time
U1: fresh machine, project .env pins the account: "three most recent sessions for this project"	main	5/5	9.2	64 s
	PR	5/5	6.2	36 s
U2: .env pins account A, default is B: "make Staging my default, then show its newest sessions"	main	5/5	15.2	111 s
	PR	5/5	5.8	44 s
U3: wrong account id given: "look up ts-ci-123 in acc-e2e-old"	main	5/5	15.0	81 s
	PR	5/5	5.6	28 s
U4: API key credentials (regression)	main	3/3	4.7	11 s
	PR	3/3	5.0	16 s
Earlier scenarios (cross-account lookup, fresh login, slow approver, change default; regression)	PR	8/8	—	—

On main the agents got there through workarounds:

• U1: they blanked CHECKLY_ACCOUNT_ID or ran login outside the project to get past "CHECKLY_API_KEY is not set", waited with sleep, and 2 of 5 stored a default as a side effect.
• U2: they hit the menu and read help pages.
• U3: they guessed account ids.

With the PR, every agent followed the CLI's next entries and messages.

Agent feedback led to the last two commits:

• 5/5 U1 runs stopped at select_account after login --wait, even though .env already named the account. A rerun of U1 (5/5) after the fix no longer hit it.
• whoami nudged agents towards storing a default they didn't need.

🤖 Generated with Claude Code

[sorccu]

Thanks, this is a clear improvement for agents working across accounts, and the blind-test write-up is convincing. I ran a multi-lens review (correctness, intent, simplicity) with adversarial verification of the major findings. The inline comments cover the code-level items. The summary:

Major (verified)

• CHECKLY_ACCOUNT_ID override vs. existing auth branches (see config.ts): only whoami and the not-found hint know about account_override. Error messages in api.ts, the inline-login trigger and login's env_credentials refusal still treat the prefix as API-key env credentials. These paths predate this PR, but the new skill guidance sends agents straight into them.
• Public docs not updated or tracked: docs/cli/checkly-login.mdx (no --wait), checkly-whoami.mdx (no --output json, default, or other accounts), checkly-switch.mdx (no non-TTY/agent behaviour) and authentication.mdx (presents CHECKLY_ACCOUNT_ID only as half of an API-key pair) all describe the old behaviour. Please add a docs PR or a follow-up ticket and link it here.

Minor: the inline comments cover the default reported under the override in switch/whoami, races in --wait, a missing onUnexpectedAnswers, test gaps, and duplicated hint formatting.

PR description:

• Please replace the linear.app link with the bare RED-1122 ID. This repo is public, and PR bodies stay public.
• The test list still mentions config.hasAccountOverride, which the last commit replaced with getCredentialSource.

[sorccu]

Major: account_override is only consulted by whoami and notFoundHint(). Every other auth branch still uses hasEnvVarsConfigured(), so the CHECKLY_ACCOUNT_ID=<id> prefix that the skill now recommends is still handled as "API-key credentials from the environment" there:

1. rest/api.ts:85 blanks the hint when the id is mistyped or belongs to an account the user isn't in. The agent then gets a bare Forbidden/NotFound instead of Account "x" is not available with the stored login. That is exactly the cross-account case this PR targets.
2. With no stored key (after logout, or on a fresh machine), authCommand.ts:32 skips the inline login and rest/api.ts:66 reports "CHECKLY_API_KEY is not set". The agent is pointed at API keys instead of the device flow.
3. login.ts:270 refuses with env_credentials and says to delete the variables, although the override is just a one-command prefix.

All three behaviours already exist on main, but the new guidance makes them much easier to hit. Suggest routing at least the api.ts hint and missing-key logic through getCredentialSource(), or documenting these cases in the skill.

[hlenke]

Done in 8e1e0c8 and c3514a2. Only CHECKLY_API_KEY now counts as environment credentials; config.getCredentialSource() (login | account_override | environment) is the single check. A CHECKLY_ACCOUNT_ID alone picks the account for the user's login, for every command while it's set, since the CLI can't tell a prefix from an export or .env.

1. api.ts: an id the login doesn't have now fails with the accounts it does have: "Account "x" from CHECKLY_ACCOUNT_ID is not available with your login. This login works with: …".
2. No stored key: an authenticated command starts the inline device login, then uses the variable's account without storing a default. CI keeps an error, now naming both options (log in, or set CHECKLY_API_KEY as well).
3. login only refuses for CHECKLY_API_KEY; with the variable it finishes without asking for an account. login/switch reason about the stored login and warn that the variable takes precedence over the default they store; whoami says where the account comes from.

[sorccu]

Under CHECKLY_ACCOUNT_ID=B npx checkly switch, this.account is the override (B), so this line says The default account is "B" and currentAccount is B, while the stored default is still A. An agent may then skip switch --account-id B or report the wrong default to the user. Read the default from config.data.store as whoami does, or word this as the account in use for this command.

[hlenke]

Fixed in 8e1e0c8: the default now comes from config.data.store, and the field is defaultAccount (null when none is stored). Under CHECKLY_ACCOUNT_ID the message also says the variable takes precedence. Covered by "reports the stored default, not the account CHECKLY_ACCOUNT_ID picks for this command" in switch.spec.ts.

[sorccu]

Text output hides the default when accountSource === 'environment' (env credentials ignore it), but JSON still reports a stale stored defaultAccount. With CHECKLY_API_KEY=k CHECKLY_ACCOUNT_ID=B, the output is accountSource: "environment" alongside defaultAccount: {id: A}, which contradicts the skill's description of defaultAccount. Suggest emitting null in that case, matching the text branch.

[hlenke]

Fixed in 8e1e0c8: defaultAccount is null for accountSource: "environment", matching the text output. Test added in whoami.spec.ts.

[sorccu]

pollForTokens is called here without onUnexpectedAnswers, unlike the interactive path at L577. If a proxy or captive portal answers the token endpoint, --wait blocks silently for the whole expires_in and then fails with expired_token, giving no hint that the network is the cause. Suggest passing the callback and emitting a JSON progress/network_error line, or stopping early.

[hlenke]

Fixed in 8e1e0c8: --wait now passes onUnexpectedAnswers. It keeps polling, like the interactive path, since a captive portal can clear. If the code expires after such answers, it reports network_error naming the answer and a likely proxy or captive portal, instead of a bare expired_token. Test: "names a proxy or captive portal when the code expires while answers come from something else".

[sorccu]

Two races that --wait now makes much more likely, because a poller is held open for minutes:

• Stale failure: another run (an inline login using #collectPendingAuthorization) consumes the code first and is still exchanging tokens for the API key. This poller's next request gets invalid_grant, config.getApiKey() is still empty, and it prints an error for a login that succeeds a moment later. Consider re-checking for the key over a short bounded window before rethrowing.
• Deleting a newer code: the deletes here and at L655 are unconditional. If another run replaced the near-expiry code X with a new code Y (the MIN_RELAY_TIME_MS path), this run's eventual expired_token deletes Y, which the user may be approving at that moment. Only delete when the stored deviceCode matches the one polled.

[hlenke]

Both fixed in 8e1e0c8:

• Stale failure: on invalid_grant with no key yet, it waits up to 15 s for the other run's key before failing.
• Deleting a newer code: the stored code is only forgotten if its deviceCode is the one this run polled (#forgetPendingAuthorization), on every path, including the near-expiry one.

Tests cover both, plus the case where no key turns up in time.

[sorccu]

This is the path most commands depend on for the hint (e.g. test-sessions get on the wrong account), but no spec covers it. Removing it would leave the suite green. Suggest a spec that calls catch() with a NotFoundError, both with and without environment credentials.

[hlenke]

Added AuthCommand.catch specs in auth-command-login.spec.ts: the hint is added for a login key, and the error is left alone for API key credentials and for errors other than not-found.

[sorccu]

This hand-built [msg, this.notFoundHint()].filter(Boolean).join(…) is repeated in rca/run.ts, with a different separator (' ' here, '. ' there) and no test for the rca one. A withNotFoundHint(message) helper on BaseCommand would remove the duplication, and future custom not-found messages would be less likely to forget the hint.

[hlenke]

Done in 8e1e0c8: BaseCommand.withNotFoundHint(message) is now used by catch(), checks delete and rca run, with one separator. The new rca-run.spec.ts covers the rca message.

[sorccu]

In JSON mode the hint is English prose appended to detail, so an agent has to parse a sentence to learn which account was searched. A structured field (e.g. searchedAccount: {id, name}, or a separate hint) would be a stable contract. It would also fit the structured choices/next in select_account, and the planned follow-up of listing other accounts.

[hlenke]

Done in 8e1e0c8: JSON errors now carry hint and searchedAccount: {id, name} as their own fields, and detail stays the API message. Both are left out when there's nothing to hint. The skill documents them.

[hlenke]

Thanks for the thorough review, @sorccu. I addressed all items; the replies are on each thread. Summary:

Major

• CHECKLY_ACCOUNT_ID vs. the other auth branches. Only CHECKLY_API_KEY counts as environment credentials now. A CHECKLY_ACCOUNT_ID alone picks the account for the user's login, on every command while it's set, whether it comes from a prefix, the shell or .env. There are four guardrails:
  • CI keeps an error, now naming both options.
  • login and switch warn that the variable takes precedence over the default they store.
  • whoami says where the account comes from.
  • A login under the variable stores no default.
• Public docs: docs(cli): document multi-account use, login --wait and whoami --output json [RED-1122] docs#566 updates checkly-login, checkly-whoami, checkly-switch and authentication. It's marked to merge after this ships.

Minor: all fixed. See the threads.

PR description: the linear.app link is replaced with the bare ID, and the test list is current.

Verification: described in the PR description. It covers:

• a scripted 12-case matrix against main and this PR;
• 49 blind agent runs on new multi-account use cases (all passed; with this PR, roughly a third of the commands and time, and no workarounds);
• regressions for the API key and earlier scenarios.

Two follow-ups came from those agent runs: c3514a2 (explicit login under CHECKLY_ACCOUNT_ID no longer stops at select_account) and 8654192 (whoami no longer nudges agents towards an unneeded default).

[sorccu]

Thanks for the thorough follow-up, @hlenke. Every thread is addressed, and the CHECKLY_ACCOUNT_ID handling now follows one clear rule. I re-reviewed 4d1fb714..34c28c8b (correctness, intent and simplicity lenses). There are no critical or major findings. These minors remain:

1. switch under CHECKLY_ACCOUNT_ID with no stored login. The inline login picks no default, but the loggedInInline early return in switch.ts prints "Logged in to B" and exits. The default the user asked to set is never chosen, so they have to run switch a second time.
2. login --wait reports a non-approval as a network problem. unexpectedAnswer is set once and never cleared. A brief captive-portal blip early on therefore turns a code that simply expired unapproved into network_error ("a proxy may be in the way").
3. The skill says a login success means the account "is now the default". A login under CHECKLY_ACCOUNT_ID returns the same line but stores no default, and the JSON gives an agent no way to tell the two cases apart.
4. Logging in again under CHECKLY_ACCOUNT_ID drops the previous default. It also drops the "no longer available" notice from the agent-mode message.
5. Smaller items:
   • No test checks that CI with only CHECKLY_ACCOUNT_ID set keeps failing instead of starting a device login.
   • checkly --help still describes CHECKLY_ACCOUNT_ID only as half of an API-key pair.
   • The initialize reference doesn't mention pinning an account through .env.
   • config.ts reads the stored accountId in three places.
   • The match-then-delete in #forgetPendingAuthorization is best-effort, not atomic. That's fine, but worth a comment.

The branch is also behind main.

I'll take the PR from here: rebase it, make these final changes myself, and update the description when I push.

[github-actions]

Auto-approved: ship/show PR from a same-repo branch.