GLOOK-50: stop the merged-PR search from manufacturing false skips - #71
Conversation
The 2026-09-09 dev run aborted at 21%:
ABORT (GLOOK-13): 21 of 100 engineers couldn't be fetched (21%).
Most failures are GitHub search timeouts (21 of 21)
The guard was right to fire on what it was told, but ~19 of those 21 skips were
false and the previous commit caused them. Verified against GitHub:
sl-chromatic-bot, sl-data-team-jenkins, magdalenastaller, keser, flaksie,
gkim-smartling and viakivchuk-smartling all have GENUINELY zero merged PRs —
bots and non-engineers — and an EMPTY issue search routinely reports
incomplete_results=true while being perfectly correct.
Three mistakes, all mine, in how the merged-PR path was hardened:
- it got the raise but not the retry ladder the commit path has, so one
transient timeout went straight to a SKIP;
- it had no partial/uncertain outcome, only "trusted" or "skipped";
- it assumed incomplete_results carries the same meaning on issue search as
on commit search. It does not.
This is the failure the PR #70 reviewer predicted on the abort-pressure thread
— that hardening these paths would make the abort cliff more likely, not less.
I argued the partial-data channel had dissolved that tension. It had not,
because I never extended the channel to this path.
The endpoints are now deliberately asymmetric:
- Commit search stays strict: retry, narrow the window, then raise. There an
empty timed-out page really did hide 43 commits.
- Merged-PR and review-count searches retry, then raise ONLY on a
self-contradicting page (total_count > 0 with empty items) — the shape that
would report a developer's 42 merged PRs as 0, seen in the same run for
ksoloviov-smartling (3) and kbroadrick-smartling (2). A persistent
timed-out EMPTY result keeps the zero and records `prsUnverified` through
integrity.recordError, so the member stays in the report, the doubt is
visible in run_metadata.errors, and the abort gate is not touched.
The retry ladder is now one shared helper rather than duplicated per endpoint,
with the per-endpoint policy at the call site where the asymmetry is explained.
CLAUDE.md records the asymmetry and says not to unify the two policies, since
that is exactly the tidy-up that would reintroduce this.
126 suites / 1231 tests pass; npm run build clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
msogin
left a comment
There was a problem hiding this comment.
Automated review — 3 reviewers, findings triaged and de-duplicated
Ran three independent reviewers over e71f4f6..7509aab (a Smartling standard fullstack review plus two personas: distributed-systems failure-mode correctness, and data-integrity/observability). Reviewers did not see each other's findings. Everything below was re-verified against the source before posting; findings that did not survive verification were dropped, and pre-existing issues outside the diff are noted as context rather than raised as defects.
Verified locally: 126 suites / 1231 tests pass, and the new pr-search-false-skips.test.ts passes 5/5 — the PR's test claims hold exactly.
Verdict: with fixes. The asymmetry itself is well-reasoned and correctly motivated, and keeping isContradictoryPage strict is right. The gap is that the doubt is only half-plumbed, and all three reviewers converged on it independently:
| endpoint | value kept | doubt recorded |
|---|---|---|
| commit search | partial data | structured SearchShortfall |
| merged-PR search | the zero | prose string → errors |
| review-count search | the zero | nothing |
And the terminal channel has no consumer: evaluateIntegrity is typed Pick<RunMetadata, 'skipped' \| 'expectedCount' \| 'thresholds'>, so it structurally cannot read errors, and IntegrityBadge returns null at state === 'ok'. So the PR's central claim — "the member stays in the report, the doubt is visible in run_metadata.errors" — holds for the first half and, at the operator level, not the second.
The headline finding (report-runner.ts:208) is structural rather than a line fix: the naive patch, thresholding on errors.length, would cry wolf because errors already mixes per-commit sha-merge-check and unmerged-commit-detail noise. That one is worth a design decision before merge rather than a quick edit.
Two smaller items are self-contained regressions worth fixing regardless: the raise message no longer matches the brownout-attribution regex #70 added (github.ts:745), and a page-2 timeout now truncates a paginated PR set where it previously raised (github.ts:753).
🤖 Generated with Claude Code
| if (activity.prsUnverified) { | ||
| log(`@${member.login}: UNVERIFIED PR count — ${activity.prsUnverified}`); | ||
| integrity.recordError({ | ||
| context: 'other', |
There was a problem hiding this comment.
🔴 Recorded doubt has no consumer — an org-wide issue-search brownout now ships a green report
recordError is the only channel this takes, and nothing downstream can act on it:
evaluateIntegrity(skip-classifier.ts:110) is typedPick<RunMetadata, 'skipped' | 'expectedCount' | 'thresholds'>—errorsis not in its parameter type, so no number of unverified members can move the state offok.IntegrityBadge.tsx:19returnsnullwhenstate === 'ok', and it is the only consumer ofmetadata.errors.- On the org page it cannot render even at
degraded:getOrgReport(report/org.ts:20) does not SELECTrun_metadata, so the badge receivesnullthere. (pre-existing, but it closes the last surface this doubt could have reached)
Failure scenario: GitHub's issue index browns out while commit search stays healthy — the 2026-09-09 shape, but org-wide. All 100 merged-PR searches return {total_count: 0, items: [], incomplete_results: true}. Zero skips → state: 'ok' → no badge, no abort. Every developer row reads 0 merged PRs, every impact score loses the min(totalPRs/10,1)*2.7 term — the largest single weight in impact-score.ts — and the ranking the report exists to produce is reshuffled. "The doubt is visible in run_metadata.errors" is true of the DB blob and false of every human surface.
Related: isContradictoryPage does not cover the case the description cites as still-protected. A search that times out having found nothing reports total_count: 0, so the developer with 42 real merged PRs whose search genuinely timed out lands on the lenient branch as a confident 0. The contradiction guard fires only when GitHub happened to compute the total before failing to deliver items.
Suggested shape (this is a shape change, not a line fix): give unverified-ness its own typed, counted channel — unverified: UnverifiedMember[] on RunMetadata, distinct from errors; feed its count into evaluateIntegrity with its own threshold so a correlated brownout goes degraded; render at state === 'ok' when it is non-empty; add run_metadata to the getOrgReport SELECT.
Do not simply threshold on errors.length — errors already mixes sha-merge-check and unmerged-commit-detail entries that can run to hundreds on a healthy run, so that patch would cry wolf and get switched off.
There was a problem hiding this comment.
Fixed in f1740d7, and this is the finding that mattered most — the PR's merge argument rested on a claim that was true of the DB blob and false of every human surface. Leniency without a consumer is worse than the raise it replaced, and I shipped exactly that.
Took the shape you suggested rather than the errors.length patch, for the reason you gave — errors already carries hundreds of per-commit entries on a healthy run:
RunMetadata.unverified: UnverifiedMember[]—{login, field, kept, reason}, member-scoped and deduped bylogin:field.evaluateIntegritynow takes'unverified'in itsPickand counts distinct logins, downgrading atdegradedUnverifiedPct(15%), with allowlisted members out of the denominator as elsewhere. It can never abort on them — those members are in the report.IntegrityBadgerenders atstate === 'ok'when the list is non-empty.getOrgReportnow SELECTsrun_metadata.
Your org-wide brownout is now a test: 100 of 100 unverified → degraded, and separately asserted never failed.
On your second point — you're right and it's worse than I wrote. I claimed the contradiction guard still protects the 42-merged-PRs developer. It does not: a search that times out having found nothing reports total_count: 0, so that developer lands on the lenient branch as a confident zero. The guard only fires when GitHub happened to compute the total before failing to deliver. I've corrected that claim in the PR description; the honest position is that the unverified channel — not the contradiction guard — is what covers this case now, by making it visible rather than by rejecting it.
| 'but delivered none', | ||
| ); | ||
| } | ||
| // A timed-out EMPTY review search is routine and usually correct — same |
There was a problem hiding this comment.
🔴 countReviewedPRs keeps the zero but records no doubt at all — a strict regression against pre-PR behaviour
Before this PR, any isSuspectSearchResult raised, and report-runner.ts:235 caught it into integrity.recordError — so a timed-out review count at least left a forensic trace. Now the only raise is isContradictoryPage, and the timed-out-empty case falls through to return res.data.total_count (= 0) with no unverified return value, no recordError, and no callback. The signature is Promise<number>; there is nowhere for the doubt to go.
The comment at :998-1004 states the harm exactly — "a timed-out review count reads 0 — which halves a scoring factor (weight 0.5, min(reviews/15, 1)) and moves the developer down the ranking" — and the code then ships that outcome silently. In the org-wide brownout described on report-runner.ts:208, all 100 review counts read 0 and the only trace is a [search] reviewed-prs … SUSPECT line in a progress store that keeps the last 200 lines. Unlike the merged-PR path, this loses even the after-the-fact record.
Separately, the comment retained at :1000-1001 says the "counts matched but delivered nothing" arm cannot fire with per_page: 1 — and this diff makes that arm the sole raise condition. If the comment is right, the branch is dead and this endpoint now trusts everything. One of the two should change.
Fix: give it the same doubt channel as the PR path — Promise<{ reviews: number; unverified?: string }> — and record it in report-runner.ts alongside prsUnverified.
There was a problem hiding this comment.
Fixed in f1740d7. You're right that this was a strict regression — pre-PR it at least raised into recordError, and I removed that without replacing it.
countReviewedPRs now returns { reviews: number; unverified?: string }, and report-runner records it via integrity.recordUnverified({field: 'reviews'}) alongside the merged-PR one. Provider interface and mock updated.
On the contradiction being dead here: you're right, and I resolved it the way you implied rather than by keeping a branch that cannot fire. With per_page: 1 the arm is effectively unreachable, so it is gone — !trustworthy now routes straight to the doubt channel. The endpoint no longer has a raise it can never reach, and no longer trusts everything either.
The comment that stated the harm and then shipped it now states the behaviour.
| if (isContradictoryPage(data)) { | ||
| throw new Error( | ||
| `GitHub merged-PR search counted ${data.total_count} PRs for @${user} ` + | ||
| `but delivered none (page ${page})`, |
There was a problem hiding this comment.
🟡 This new message breaks the brownout-attribution heuristic #70 added
formatIntegrityAbortReason (report-runner/types.ts:119) classifies a skip as a search timeout with:
/no trustworthy result|under-delivered/i.test(s.reason)The old merged-PR raise said gave no trustworthy result for @user and matched. This one — counted 3 PRs for @user but delivered none (page 1) — does not.
Failure scenario: a brownout where the contradictory-page shape dominates, i.e. exactly the ksoloviov-smartling (total_count=3) and kbroadrick-smartling (2) shape this PR cites as observed in the incident run. Those skips now all fail the regex → searchTimeouts = 0 → the abort banner reads "Likely upstream auth/permission regression" and sends on-call to rotate the PAT during a GitHub search brownout. The comment at types.ts:112-116 exists specifically to prevent that lead.
Fix: extend the pattern (/no trustworthy result|under-delivered|but delivered none/i) — or better, stop pattern-matching prose and tag the skip with a cause discriminator at recordSkip time. Either way, pin the thrown message against the matcher in a test; nothing in the 1231-test suite couples them today.
There was a problem hiding this comment.
Fixed in f1740d7. Good catch — I reworded a raise without noticing it was load-bearing for a matcher I'd added myself one PR earlier.
Rather than widen the regex, I made the messages conform: every merged-PR raise now begins GitHub merged-PR search gave no trustworthy result for @user: …, which the existing pattern matches. That keeps one phrase meaning "this was a search failure" across all endpoints instead of accumulating alternatives.
Your stronger suggestion — tag the skip with a cause discriminator at recordSkip time instead of pattern-matching prose — is the right end state and I've recorded it on GLOOK-50. It touches the SkippedMember shape and the persisted run_metadata, so I didn't want it riding along here.
Took the test suggestion, since that's what makes the fragility survivable: thrown search messages stay matched to the brownout attribution feeds each real raise string through formatIntegrityAbortReason and asserts it attributes to a search brownout — and asserts a genuine Validation Failed still attributes to auth. Nothing coupled these before.
| // correct — true for bots and non-engineers with genuinely zero merged | ||
| // PRs. Treating it as a failure produced 19 false skips in one run and | ||
| // aborted the report at 21%. So keep the zero and record the doubt. | ||
| const unverified = |
There was a problem hiding this comment.
🟡 A page-2 timeout silently truncates a paginated PR set — the GLOOK-50 undercount, moved from page 1 to page N
This returns whatever was collected so far regardless of page. The justification above it ("an EMPTY issue search routinely sets incomplete_results while being correct") is a page-1 observation; by page 2 the loop has already learned total_count from page 1, and the stateless per-page isContradictoryPage predicate cannot see it.
Failure scenario: author with 250 merged PRs in the window. Page 1 → total_count: 250, items: 100, trusted, loop continues. Page 2 times out → {total_count: 0, items: [], incomplete_results: true} → not contradictory → lenient return of 100 PRs of 250, labelled "count kept as 100 but unverified" as though it were a routine correct zero. Pre-PR this raised. Nothing says 150 are known missing.
Adjacent (pre-existing, same fix): :775 re-reads the current page's total_count in prs.length >= Math.min(res.data.total_count, SEARCH_RESULT_CAP). A trusted {total_count: 0, items: []} on page 2 makes 100 >= 0 true and breaks with 100 of 250 and no signal whatsoever. collectCommits:588-592 documents fixing precisely this for commits; this loop never got it.
Fix: hoist a monotonic expectedTotal = max(expectedTotal, page.total_count) as collectCommits does, gate the leniency on prs.length === 0, and emit a structured SearchShortfall {expected, collected, detail} rather than a bare string when the loss is quantified.
There was a problem hiding this comment.
Fixed in f1740d7. "The GLOOK-50 undercount, moved from page 1 to page N" is exactly right — I applied a page-1 observation to every page, and the stateless predicate had no way to know better.
The loop now mirrors collectCommits:
expectedTotalis monotonic (Math.maxacross pages), so page 1's promise survives a later page reporting zero — which also fixes the adjacent:775read you flagged, since the break now testsexpectedTotal, not the current page.- Leniency requires nothing collected AND nothing promised (
prs.length === 0 && expectedTotal === 0). Anything else raises withunder-delivered N of M (page P). - The walk reconciles against the total at the end, as the commit path does.
Your 250-PR scenario now raises rather than returning 100 as a routine zero.
One deviation: I kept the raise rather than emitting a structured SearchShortfall for the mid-pagination case. For commits a partial set is still useful data; for merged PRs the count feeds prPercentage and the impact score directly, so a silently-100-of-250 count is a wrong number rather than an incomplete list. If you'd rather have the shortfall shape here too, it's a small follow-up — but I didn't want to introduce a third partial-data path in the same PR.
| stub(TIMED_OUT_EMPTY); | ||
| const activity = await drain(fetchUserActivity('Smartling', 'sl-chromatic-bot', SINCE)); | ||
| expect(activity.prs).toEqual([]); | ||
| expect(activity.prsUnverified).toMatch(/timed out on an empty result/); |
There was a problem hiding this comment.
🟡 The suite asserts the internal field, not the integrity outcome the PR is justified by
All 5 tests pass and exercise only github.ts — expect(activity.prsUnverified).toMatch(...) asserts a field on a returned object, and expect(calls.issues).toBe(3) asserts a call count. There is no import of report-runner anywhere in the file.
The merge argument is "the member stays in the report, the doubt is visible in run_metadata.errors, and the abort gate is untouched." None of those three is under test. The first refactor that renames the field, drops the if (activity.prsUnverified) block, or reorders the recordError call keeps this suite green while the doubt disappears — the same silent drift COUNTABLE_SKIP_CLASSIFICATIONS was created to prevent.
Also uncovered: the changed countReviewedPRs behaviour (zero tests, despite being half the behaviour change), the mid-pagination case (page 1 full, page 2 untrustworthy-empty), and the aggregate case (many members unverified in one run).
Fix: add a runner-level test — stub a provider returning prsUnverified for a member with commits, then assert (a) the member has a developer_stats row, (b) run_metadata.errors contains an entry for that login, (c) run_metadata.skipped does not. integrity-guard-regression.test.ts is the natural home.
There was a problem hiding this comment.
Fixed in f1740d7. You're right, and the framing — "the merge argument is X, and none of X is under test" — is the part I'd want applied to my work more often.
Added to integrity-guard-regression.test.ts, asserting the outcome rather than the field:
- an org-wide brownout (100 of 100 unverified) →
degraded, notok - it crosses at the 15% gate (14 → ok, 15 → degraded)
- it never aborts, at any volume
- a login with two unverified figures counts once
- allowlisted members leave the denominator, as with the skip gate
Plus the message/matcher coupling suite on the other thread.
Not yet covered, and I'd rather say so than imply otherwise: a true end-to-end runReport test asserting developer_stats row + run_metadata contents in one pass. That needs a DB fixture and a stubbed provider, and report-runner has no test harness today — the closest existing coverage is all at the unit boundary. Recorded on GLOOK-50. What's here now does cover the drift you described: renaming the field or dropping the recordUnverified call breaks the integrity-level tests, because they run through evaluateIntegrity.
countReviewedPRs and the mid-pagination case are both covered now via the other threads' fixes.
| - Progress store and stop-signal store use `globalThis` to survive Next.js HMR module reloads | ||
| - `@octokit/rest` is ESM-only — any test file that imports from `github.ts` (directly or transitively) must mock it before the import, and it must be the **factory** form: `jest.mock('@octokit/rest', () => ({ Octokit: jest.fn().mockImplementation(() => ({})) }))`. A bare `jest.mock('@octokit/rest')` does **not** work — auto-mocking still loads the real module, so the suite dies with "Jest encountered an unexpected token" pointing at `dist-src/index.js`, which reads like a transform-config problem rather than a mocking one. | ||
| - **A GitHub search that succeeds can still be lying (GLOOK-50).** `incomplete_results: true` means the query timed out and GitHub returned only what it had found by then — so an **empty** page with that flag is an unreliable zero, not a real one. That is how six developers were recorded with 0 commits while GitHub held 60 between them, with no skip and no integrity warning. The rules, all in `github.ts`: a timeout with **items** is deliberately NOT distrusted (GitHub warns a timeout "does not necessarily mean that search results are incomplete", and on the run that produced this code 5 of 6 flagged pages were fine); an untrustworthy **empty** page is retried twice, then the date window is halved (`splitDateWindow`, max 2 levels, second half keeps an open upper bound so nothing falls off the end), and only then does it **raise** — a raise becomes a counted SKIP, which is the goal, not a workaround. Collected hits are reconciled against `total_count` **clamped to** the 1000-result cap, never gated on it. Partial-but-real data is **kept** and reported as a `SearchShortfall` → `integrity.recordError` (member-kept partial data), never discarded — throwing away 90 of 100 commits would be worse than shipping them slightly low. Pagination stops at 1000 results because paging past it returns `Only the first 1000 search results are available`. | ||
| - **A GitHub search that succeeds can still be lying (GLOOK-50).** `incomplete_results: true` means the query timed out and GitHub returned only what it had found by then — so an **empty** page with that flag is an unreliable zero, not a real one. That is how six developers were recorded with 0 commits while GitHub held 60 between them, with no skip and no integrity warning. The rules, all in `github.ts`: a timeout with **items** is deliberately NOT distrusted (GitHub warns a timeout "does not necessarily mean that search results are incomplete", and on the run that produced this code 5 of 6 flagged pages were fine); an untrustworthy **empty** page is retried twice, then the date window is halved (`splitDateWindow`, max 2 levels, second half keeps an open upper bound so nothing falls off the end), and only then does it **raise** — a raise becomes a counted SKIP, which is the goal, not a workaround. Collected hits are reconciled against `total_count` **clamped to** the 1000-result cap, never gated on it. Partial-but-real data is **kept** and reported as a `SearchShortfall` → `integrity.recordError` (member-kept partial data), never discarded — throwing away 90 of 100 commits would be worse than shipping them slightly low. Pagination stops at 1000 results because paging past it returns `Only the first 1000 search results are available`. **The endpoints are deliberately asymmetric, and this is the thing most likely to be "tidied" into a bug:** commit search stays strict (an empty timed-out page really did hide 43 commits), but for **issue/PR search an empty result routinely reports `incomplete_results: true` while being correct** — true for bots and anyone with genuinely zero merged PRs. Raising there produced 19 false skips in one run and aborted the report at 21%. So merged-PR and review-count searches retry, then raise **only** on a self-contradicting page (`total_count > 0` with empty `items`); a persistent timed-out *empty* result keeps the zero and records `prsUnverified` via `recordError`. Do not unify the two policies. |
There was a problem hiding this comment.
🟡 This paragraph asserts the opposite of what the review-count path does
merged-PR and review-count searches retry, then raise only on a self-contradicting page (
total_count > 0with emptyitems); a persistent timed-out empty result keeps the zero and recordsprsUnverifiedviarecordError.
The review-count path records nothing — countReviewedPRs returns a bare number and has no doubt channel (see the comment on github.ts:1014). A maintainer trusting this sentence will believe the doubt is captured on both endpoints when it is captured on one.
Either narrow the sentence to the merged-PR path, or (better) make the code match the doc. The rest of the paragraph — especially "Do not unify the two policies" with the why attached — is the right instinct and worth keeping.
There was a problem hiding this comment.
Fixed in f1740d7 — and I made the code match the doc rather than narrowing the sentence, since the review-count path genuinely should record its doubt (see the github.ts:1014 thread).
The paragraph now describes the actual contract: raise only when the loss is provable (contradiction, or mid-pagination after page 1 set an expectation); a timeout with nothing collected and nothing promised keeps the zero; and any kept-but-unproven figure must land on the typed unverified channel, never only in errors — with the reason why errors can't be thresholded. It also states what the chain buys, so the next reader knows what breaks if they remove a link.
Kept "Do not unify the two policies" with the why attached, per your note.
| * on that produced 19 false skips in one run and aborted the report, so the | ||
| * caller only raises on a self-contradicting page. | ||
| */ | ||
| async function searchPageWithRetry<T extends SearchAccounting>( |
There was a problem hiding this comment.
🔵 The ladder was extracted, but collectCommits still carries its own verbatim copy
The PR description says the retry ladder "is now one shared helper (searchPageWithRetry) instead of duplicated per endpoint." The commit path was never migrated: :596-618 still has the same loop, the same SEARCH_TIMEOUT_RETRIES/SEARCH_TIMEOUT_RETRY_MS handling, and the same log wording inline. So there are now two copies rather than one, which is the exact drift risk the extraction was meant to remove — and the one CLAUDE.md calls "the thing most likely to be tidied into a bug."
It is mechanically replaceable: const { data, trustworthy } = await searchPageWithRetry('commits', user, page, …), then branch on !trustworthy for the window-split/raise decision instead of the data === null sentinel.
Also, this new doc block landed between the collectCommits doc comment (:527-534, "Collect a user's commits for one date window…") and the function it describes — so that comment now reads as documentation for searchPageWithRetry, and collectCommits at :574 is left undocumented. Same pattern in report-runner.ts:196-200, where the pre-existing commitsShortfall paragraph now sits above the new prsUnverified block it does not describe. Given the CLAUDE.md warning, comments pointing at the wrong function are precisely the mechanism by which the bad tidy-up happens.
There was a problem hiding this comment.
Both fixed in f1740d7. The PR description claimed an extraction I hadn't finished — there were two copies, which is worse than the one I started with.
collectCommits now calls searchPageWithRetry and branches on !trustworthy instead of the data === null sentinel. One ladder, three call sites.
The orphaned comment is the more interesting half, because I reproduced the exact failure you described while fixing it. My first attempt moved the doc block the wrong way and left collectCommits's documentation sitting above searchPageWithRetry — the same drift, one function further along. Corrected, and verified by reading the final ordering rather than assuming the edit did what I meant.
The report-runner.ts case is fixed too: the commitsShortfall paragraph now sits with its own block, and the prsUnverified block has its own comment explaining why it is a kept-member signal rather than a skip.
Given the CLAUDE.md warning you quoted, a comment pointing at the wrong function really is the mechanism for the bad tidy-up — and it nearly got me twice in one sitting.
| await sleep(SEARCH_TIMEOUT_RETRY_MS); | ||
| } | ||
| } | ||
| return { data: last as T, trustworthy: false }; |
There was a problem hiding this comment.
🔵 Minor, non-blocking — grouped so they don't compete with the findings above
- Retry amplification lands hardest during the brownout. Both issue-search call sites previously failed fast on the first suspect response; they now take
1 + SEARCH_TIMEOUT_RETRIESattempts with a 5s sleep between, each a freshwithRetryladder with a reset budget. GitHub fetches are sequential (p-limitwraps only the LLM work atreport-runner.ts:404), so an org-wide issue brownout across 100 members adds ~400 extra requests against a 30 req/min search limit and ~33 minutes of pure sleep — to reach a result the code keeps regardless. Consider skipping the retry entirely for theisUntrustworthyEmptyshape you have already decided to accept, and retrying only the contradictory shape. last as Tcasts away anullthe loop bound makes impossible today; it becomes a silentundefinedifSEARCH_TIMEOUT_RETRIESever changes. Initialise from the first call or assertlast !== null.const res = { data };at:758and:1005is a diff-minimising shim that exists only so the untouched lines below can keep sayingres.data. Usedatadirectly.prsUnverified?: stringis free-form prose where its siblingcommitsShortfallis a structuredSearchShortfall. The test already has to assert withtoMatch(/timed out on an empty result/). A structured shape would also let the aggregate gate suggested onreport-runner.ts:208count it.context: 'other'is now the third distinct use of that tag in the same function (:208,:218,:235).IntegrityError['context']is a closed union — a'prs-unverified'member would let an operator filter these out of the badge list, which truncates at 50.
There was a problem hiding this comment.
Four of five fixed in f1740d7; one deliberately kept, with reasoning.
last as T— nowlet last: T | undefinedseeded on the first pass, with an explicit throw instead of a cast. Your point about it silently becomingundefinedif the retry constant changed is exactly right.const res = { data }— gone from both sites; the surrounding lines usedatadirectly.prsUnverifiedprose — superseded. Both doubts now go through the typedUnverifiedMember {login, field, kept, reason}, which is what letsevaluateIntegritycount them for the gate on thereport-runner.ts:208thread. The string survives only as the human-readablereason.context: 'other'— no longer applies. Unverified figures are offIntegrityErrorentirely now, on their own channel with afielddiscriminator, so an operator can filter them without a union change.
Kept: the retry on the shape we already accept. Your arithmetic is right — ~400 extra requests and ~33 minutes of sleep in an org-wide brownout, to reach a result we keep anyway. I kept it because the retry is what distinguishes the two outcomes: without it we can't tell "GitHub timed out on an empty result" from "GitHub timed out on a result it would have delivered on the second ask", and the 2026-09-09 run showed retries recovering four commit searches. Skipping the retry would make every brownout member unverified rather than merely some.
That said, the cost is real and lands at the worst moment. Recorded on GLOOK-50 alongside the existing fan-out-budget follow-up — a wall-clock deadline threaded through the search paths bounds both, and is a better answer than special-casing this shape.
Review found the central claim of the previous commit was half true. "The
member stays in the report, the doubt is visible in run_metadata.errors" held
for the DB blob and for no human surface:
- evaluateIntegrity is typed Pick<RunMetadata, 'skipped'|'expectedCount'|
'thresholds'>, so it structurally could not read errors;
- IntegrityBadge returns null at state === 'ok';
- getOrgReport never SELECTed run_metadata, closing the last surface.
So an org-wide issue-search brownout — the 2026-09-09 shape, but affecting
every member — produced zero skips, state 'ok', no badge, and every developer
reading 0 merged PRs with the largest single term in the impact score silently
removed. Leniency without a consumer is worse than the raise it replaced.
Unverified-ness is now its own typed, counted channel:
- RunMetadata.unverified: UnverifiedMember[] {login, field, kept, reason},
deliberately NOT an IntegrityError — `errors` already mixes per-commit
sha-merge-check and unmerged-commit-detail entries running to hundreds on a
healthy run, so anything thresholded on errors.length would cry wolf and be
switched off;
- evaluateIntegrity counts distinct unverified logins and downgrades at
degradedUnverifiedPct (15%), with allowlisted members out of the
denominator as elsewhere. It can never abort on them: those members are
present in the report;
- IntegrityBadge renders at state === 'ok' when the list is non-empty;
- getOrgReport selects run_metadata.
Also fixed, all found by review:
- countReviewedPRs kept the zero and recorded NOTHING — a strict regression
against pre-PR behaviour, which at least raised into recordError. It now
returns {reviews, unverified}. Its comment already noted per_page:1 makes
the contradiction arm unreachable, which had made the sole raise condition
dead and the endpoint trust everything.
- A page-2 timeout truncated a paginated PR set: page 1 says 250, page 2
times out, 100 returned as a routine "unverified" zero. That is the
GLOOK-50 undercount moved from page 1 to page N. expectedTotal is now
monotonic as in collectCommits, leniency requires nothing collected AND
nothing promised, and the walk reconciles against the total.
- The contradiction raise no longer breaks the brownout attribution added in
#70: its wording had stopped matching the matcher, so a contradiction-shaped
brownout would have told the on-call to rotate the PAT. Messages and matcher
are now pinned together by tests, which nothing did before.
- collectCommits still carried a verbatim copy of the retry ladder the
previous commit claimed to have extracted; it now uses searchPageWithRetry.
- Test drain helpers stopped at the first tick with no pending timer, which
stranded the promise once another sleep entered the path. Settle-aware now.
- `last as T`, the `const res = { data }` shim, and doc comments that had
drifted onto the wrong functions.
Tests: integrity-level cases assert the OUTCOME rather than the internal field
— brownout downgrades to degraded, never aborts, crosses at 15%, dedupes a
login with two unverified figures, excludes allowlisted from the denominator —
plus message/matcher coupling for every raise site.
126 suites / 1241 tests pass; npm run build clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
src/lib/llm-config/index.ts is an unused barrel re-export — nothing imports `@/lib/llm-config`, and service.ts was already tracked. It was untracked in the working copy and got picked up by a broad `git add -A src/lib`, which put an unrelated file in a GLOOK-50 commit. Untracked again; the file stays on disk exactly as it was. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review addressed —
|
| finding | outcome |
|---|---|
| 🔴 recorded doubt has no consumer | Fixed — typed unverified channel, counted by the gate, rendered at ok |
🔴 countReviewedPRs records nothing |
Fixed — returns {reviews, unverified} |
| 🟡 raise message breaks brownout attribution | Fixed + pinned by tests |
| 🟡 page-2 timeout truncates a PR set | Fixed — monotonic total, reconciliation |
| 🟡 tests assert the field, not the outcome | Fixed — integrity-level cases |
| 🟡 CLAUDE.md contradicts the code | Fixed — code changed to match |
| 🔵 ladder extracted but commits kept a copy | Fixed — one ladder, three call sites |
| 🔵 five minor items | 4 fixed, 1 kept with reasoning |
Two corrections to my own claims
The contradiction guard does not protect the case I said it did. A search that times out having found nothing reports total_count: 0, so the developer with 42 real merged PRs lands on the lenient branch as a confident zero. The guard fires only when GitHub computed the total before failing to deliver. What covers that case now is the unverified channel making it visible — not the guard rejecting it. PR description corrected.
I reproduced the orphaned-comment failure while fixing it. My first attempt at reattaching the doc blocks put collectCommits's documentation above searchPageWithRetry — the same drift, one function along. Caught by reading the result instead of trusting the edit.
The shape
RunMetadata.unverified: UnverifiedMember[] — {login, field, kept, reason}, deduped by login:field, deliberately not an IntegrityError for the reason you gave. evaluateIntegrity counts distinct logins and downgrades at degradedUnverifiedPct (15%), allowlisted members out of the denominator; it can never abort, because those members are in the report. Your org-wide brownout is a test: 100 of 100 → degraded, asserted never failed.
Deliberately kept
The retry on the shape we already accept. Your arithmetic is right — ~400 extra requests and ~33 min of sleep in a brownout. I kept it because the retry is the only thing distinguishing "timed out on an empty result" from "timed out on a result it would have delivered on the second ask"; the 2026-09-09 run showed retries recovering four commit searches. Recorded on GLOOK-50 with the fan-out-budget follow-up — a wall-clock deadline bounds both properly.
Also
ec3506a untracks src/lib/llm-config/index.ts, an unused barrel my git add -A src/lib swept into a GLOOK-50 commit. Separate commit rather than a force-push, so your review anchors stay put.
126 suites / 1241 tests; npm run build clean.
Still true: the unverified path has never fired against real GitHub. Today's dev run (7509aab, deployed pre-merge) was healthy — zero merged-PR timeouts — so it exercised the commit retry ladder and nothing here. The 15% gate is calibrated from reasoning, not observation.
Follow-up to #70, same ticket: GLOOK-50.
What happened
The 2026-09-09 dev run aborted:
The guard was right about what it was told, and the new diagnostic text correctly identified the category. But ~19 of those 21 skips were false, and #70 caused them.
Almost every skip was the merged-PR search, not commits. Checked against GitHub:
sl-chromatic-botsl-data-team-jenkinsmagdalenastallerkeserflaksiegkim-smartlingviakivchuk-smartlingksoloviov-smartlingkbroadrick-smartlingBots and non-engineers with genuinely nothing merged. An empty issue search routinely reports
incomplete_results: truewhile being perfectly correct — I reproduced that against GitHub for these very logins. #70 turned that into a SKIP.Three mistakes, all mine
incomplete_resultscarries the same meaning on issue search as on commit search. It does not.This is the failure the #70 reviewer predicted on the abort-pressure thread — that hardening these paths would make the abort cliff more likely, not less. I replied that the partial-data channel had dissolved that tension. It hadn't, because I never extended the channel to this path. That reassurance was wrong and this PR is the consequence.
The fix: the endpoints are now deliberately asymmetric
total_count > 0with emptyitems). A persistent timed-out empty result keeps the zero and recordsprsUnverifiedviaintegrity.recordError.So the member stays in the report, the doubt is visible in
run_metadata.errors, and the abort gate is untouched.The contradiction protection is worth keeping and still fires:
ksoloviov-smartling(total_count=3, items=0) andkbroadrick-smartling(2) showed exactly that shape in the same run, and it's the case that would report someone's 42 merged PRs as 0.The retry ladder is now one shared helper (
searchPageWithRetry) instead of duplicated per endpoint, with the differing policy at each call site where the asymmetry is explained.Tests
126 suites / 1231 tests pass;
npm run buildclean. New file drivesfetchUserActivitythrough the octokit hook: a timed-out empty PR search keeps the zero and records doubt (not a skip); it retries exactly 3 times first; a recovering retry is used; a self-contradicting page still raises; and a healthy empty result is trusted with no retry at all.Reviewer note
CLAUDE.mddocuments the asymmetry and says explicitly not to unify the two policies — collapsing them into one rule is precisely the tidy-up that would reintroduce this, in either direction.Also confirmed working from the same run:
oshpakloggedtotal_count=1089 exceeds the 1000-result cap — this developer's commits are truncated.🤖 Generated with Claude Code