Skip to content

[sense-lab] blast callers resolved by bare method name - #325

Open
luuuc wants to merge 1 commit into
mainfrom
lab/blast-callers-resolved-by-bare-method-name
Open

luuuc wants to merge 1 commit into
mainfrom
lab/blast-callers-resolved-by-bare-method-name

Conversation

@luuuc

@luuuc luuuc commented Sep 24, 2026

Copy link
Copy Markdown
Owner

Opened by sense-lab for #323.

Closes #323

sense-lab rebenches this PR at its own build and posts the result below. A person merges; sense-lab never does.

A 0.3 usage edge is the resolver saying "I could not type the receiver, this
bind is a guess", and extract's law stamps it below blast's traversal floor
precisely so impact analysis ignores it. But MinConfidence is also the depth
control, so the MCP server's 0.3 (a chain of real 0.7 edges has to survive
three hops) re-admitted every guess: discourse's Admin::BadgesController,
whose whole claim on the word `new` is a two-line empty Rails action, served
60 direct callers of which 59 were spans calling some other class's .new,
risk high and total_affected 357 against a true 6.

The floor on a single edge's own confidence is now separate from the
cumulative-path one, and applies to calls/references only: a 0.3 composes
edge is a weak association and a temporal confidence is a co-change ratio,
neither of them a name guess. That call now returns the six route entries,
"6 direct callers" and verdict complete - the answer min_confidence 0.6
already gave. sense_graph is untouched and still shows the stratum.

Collateral, measured on the same indexes: 20% (discourse) to 27% (rails) of
symbols with any inbound usage edge now report zero direct callers. That
stratum is already invisible at every other default floor - CLI blast,
sense_graph and the summary's hub counts all cut at 0.5 - and 82-86% of it
points at a name several symbols share, i.e. a coin flip among N. Where it
hid a real row it hid it among fakes: maket's Account::Cart::DashboardController
served 102 route helpers from unrelated routes, all bound to one `#show`.
@codecov

codecov Bot commented Sep 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@luuuc

luuuc commented Sep 24, 2026

Copy link
Copy Markdown
Owner Author

Rebench by sense-lab at this PR's build, sense 1.15.3-dev+g6df82a5a (schema v5, embeddings: all-MiniLM-L6-v2-ctx1):

  • claude opus: WORSE. Sense arm from 1.00 to 0.12, plain arm from 0.42 to 0.00, over 2 draw(s).

Closed by sense-lab: a WORSE rebench does not merge.

@luuuc luuuc closed this Sep 24, 2026
@luuuc luuuc reopened this Sep 24, 2026
@luuuc

luuuc commented Sep 24, 2026

Copy link
Copy Markdown
Owner Author

Correction from sense-lab: the WORSE result above was not a measurement of this change. The Claude arm ran claude-opus-5-5, a newer model than the claude-opus-5 the bench was banked on, and it never called Sense in either draw, so both draws measured grep. sense-lab should not have scored them or closed the PR. It now leaves such draws out, runs a rebench on the model the bench was banked on, and never closes a PR. A rebench on claude-opus-5 follows.

@luuuc

luuuc commented Sep 24, 2026

Copy link
Copy Markdown
Owner Author

Rebench by sense-lab at this PR's build, sense 1.15.3-dev+g6df82a5a (schema v5, embeddings: all-MiniLM-L6-v2-ctx1):

  • claude claude-opus-5: SAME. Sense arm from 1.00 to 1.00, plain arm from 0.42 to 0.33, over 2 draw(s).
  • claude opus: VOID. Sense arm from 1.00 to 0.00, plain arm from 0.42 to 0.00, over 0 draw(s). 2 draw(s) never called Sense and are left out.

Nothing was measured that can judge this PR. It stays open.

@luuuc

luuuc commented Sep 24, 2026

Copy link
Copy Markdown
Owner Author

Correction from sense-lab: the "claude opus VOID" line above is left over from the earlier claude-opus-5-5 run and is not part of this rebench. This rebench, on claude-opus-5 (the model the bench was banked on): SAME. Sense arm 1.00 before and 1.00 after, plain arm 0.42 before and 0.33 after, over 2 draws, and no row the finding says Sense already carried was lost. The Sense arm was already at 1.00 on this question, so this rebench shows no regression; it cannot show a gain. Ready for a person's review.

@luuuc

luuuc commented Sep 24, 2026

Copy link
Copy Markdown
Owner Author

Rebench by sense-lab at this PR's build, sense 1.15.3-dev+g6df82a5a (schema v5, embeddings: all-MiniLM-L6-v2-ctx1):

  • codex gpt-5.6-sol: WORSE. Sense arm from 1.00 to 0.89, plain arm from 0.67 to 0.33, over 3 draw(s).

sense-lab measured this PR WORSE. A person decides whether to close it; sense-lab never does.

@luuuc

luuuc commented Sep 24, 2026

Copy link
Copy Markdown
Owner Author

Rebench by sense-lab: this PR's build sense 1.15.3-dev+g6df82a5a (schema v5, embeddings: all-MiniLM-L6-v2-ctx1) against Sense main sense 1.16.0-dev+g89b062c0 (schema v5, embeddings: all-MiniLM-L6-v2-ctx1), side by side on the same model:

  • codex gpt-5.6-sol: SAME. Sense arm 0.89 on the base build and 0.89 on this one, plain arm 0.28 and 0.56, over 3 draw(s).

Ready for a person's review. sense-lab never merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[sense-lab] blast callers resolved by bare method name

1 participant