[haggs]

I found that sentiment screening was embedding and classifying the same text repeatedly. In one 10,000-trace workload, 81,729 message occurrences contained only 28,319 distinct encoder inputs. At a batch size of 64, that means 1,278 embedding requests instead of 443.

I changed screen_user_messages to embed and classify each distinct effective input once per screening call, then copy the result back to every original occurrence. Matching uses the existing text[:20000] encoder limit and preserves whitespace. Every message keeps its position and trace association, and each occurrence gets an independent result object. The original message evidence stays intact. Reuse is limited to that call; there is no persistent cache. Timing instrumentation is in PR #127.

I estimate that the deduplication structures could add roughly 50–150 MB of peak memory for a 10,000-trace workload with 97,251 message occurrences and 69,034 unique embedding inputs. This is an estimate from the object structures and input counts; I have not measured memory usage. The extra memory holds lookup tables, unique embeddings, and cached classifications. Short message strings remain shared with the original input, while truncating long messages creates new strings that can add more: 1,000 distinct 20,000-character ASCII strings would use about 20 MB for their text alone. The per-occurrence output results were already required before this change.

I tested complete Compass runs with normal extraction, applicable signals, validation, and compilation. Each pair used the same input and model configuration:

Comparison	Before	After	Observed result
Dataset A, 10,000 traces	1h43m30s	1h22m53s	19.9% shorter total runtime
Dataset B, 532 traces	31m21s	60m06s	No end-to-end improvement established

Dataset A used the earlier combined optimization and tracing revision. Sentiment took 46m46s before and 30m16s after. Both runs produced six insights, but their findings and evidence differed. This was one sequential pair on a shared machine, and oversized exports dropped telemetry batches, so I cannot attribute the entire runtime difference to deduplication.

Dataset B used this PR's final deduplication-only revision. The after run crossed about 11 minutes of machine sleep and spent substantially longer in compilation. Its active timer was also slower: 48m54s versus 31m21s. Sentiment fell from 9m44s to 6m08s, but ethos remained the slowest signal. The baseline produced one valid insight; the after run produced three, including one invalid insight with only one supporting trace. These results do not establish an end-to-end speedup or equivalent insight quality.

Verified locally: The full suite passed on the final revision. Separate fixed-input screening comparisons preserved all message labels and trace flags. For Dataset B's identical extracted messages, requests fell from nine to two and screening took 4.1–4.3s versus 7.8–11.5s in two pairs with reversed execution order. Those measurements isolate screening savings; the complete-run results above describe the end-to-end behavior.