[SuperJaba]

Overview

This PR introduces NLP capabilities to Osintgram with zero external dependencies:

1. Sentiment & Toxicity Analysis (src/sentiment_analyzer.py): Computes valence scores (-1.0 to +1.0), toxicity ratings (0–100%), and flags coordinated comment raids / brigading.
2. Topic & N-gram Cloud Extraction (src/topic_extractor.py): Cleans post captions and comments, filters multilingual stopwords, and generates weighted topic clouds, bigrams, and hashtag frequencies.

---

1. Sentiment & Toxicity Analyzer (src/sentiment_analyzer.py)

• Multilingual Valence Lexicon: Analyzes English, Polish, Spanish, and Italian sentiment polarity.
• Emoji Polarity & Hostility Engine: Parses positive emojis (❤️, 🔥, ✨) vs hostile/degrading emojis (🤮, 💩, 🤡, 🖕).
• Aggression Heuristics: Evaluates CAPS LOCK ratios, punctuation hysteria (???, !!!), abusive phrase matching, and crypto/forex promotional spam.
• Brigading & Raid Detection: Flags potential coordinated commenting attacks when hostility surges (>=40% negative comments with >=20% average toxicity).
• Detailed Output: Provides overall sentiment, positive/neutral/negative distribution progress metrics, and extracts top toxic/positive comments.

2. Topic & N-gram Cloud Extractor (src/topic_extractor.py)

• Text Normalization: Strips URLs, @mentions, and emojis while retaining core keywords.
• Stopword Filtering: Comprehensive multilingual stopword filtering across English, Polish, Italian, and Spanish.
• N-Gram Phrase Extraction: Detects common 2-word bigrams and 3-word trigrams with occurrence counts.
• Weighted Word Cloud: Normalizes keyword weights (0–100) for variable font size visualization.

3. Service, Tools & UI

• Service Methods: analyze_sentiment() and extract_topics() added to OsintgramService.
• Tools: analyze_sentiment and extract_topics registered for AI and manual modes.
• Web UI:
  • Sentiment gauge cards, toxicity index indicator, positive/neutral/negative progress bars, and flagged toxic comments list.
  • Interactive weighted pill word cloud, common 2-word phrases, and hashtag pills.
• REST Endpoints:
  • POST /api/target/{target}/nlp/sentiment
  • POST /api/target/{target}/nlp/topics
  • Enforced _require_local_origin on all endpoints.

---

Verification & Testing

• Full test suite passed: 184 passed.
• Flake8 lint clean: 0 errors.
• Added dedicated test suite in tests/test_sentiment_nlp.py covering sentiment scoring, toxicity heuristics, raid detection, stopword filtering, n-grams, and API endpoints.

[Datalux]

Sentiment and topic extraction are nice additions, but a few things need fixing before merging.

Correctness

1. TOXIC_KEYWORDS is matched as a substring (phrase in text_lower), so "die" matches audience, diet, studied, soldier, dieci, etc. (+50 toxicity, −1.0 sentiment). Please match whole words (\b...\b).
2. The "❤️" key in EMOJI_VALENCE is two code points (U+2764 + U+FE0F), but the loop iterates single characters, so it never matches. Please match emoji sequences (or normalize by stripping U+FE0F).
3. Lexicon issues: "super" is defined twice (the second wins), "kot" (Polish "cat") is +0.7, and "die" is counted both as a lexicon word and as a toxic phrase (and is also the German article). Negation ("not good") isn't handled. Please clean up the duplicates, and mention the limits (no negation, sarcasm, or language detection) in the docs and UI.
4. Italian/Spanish support is very thin (about 18 lexicon words, partial stopwords: nel, alla, dei, gli, hanno, delle, las, los... are missing), so the topic cloud for Italian captions will be mostly function words. Please extend it, or say the support is partial.
5. Spam patterns add +20 toxicity; spam and toxicity are different things, so they shouldn't be mixed in the same score.

"Raid detection"
The alert only checks "≥40% negative, ≥20 average toxicity, n≥5". It doesn't use timestamps, repeated authors, duplicate text or account age, which is what makes a raid coordinated, so a plain controversial post will trigger it. (created_at is read but never used, and analyze_sentiment drops it.) Please either add real coordination signals or rename it (e.g. "hostility spike") and say it's an indicator, not evidence. Since the UI lists real @Handles as "toxic", it should also carry a clear disclaimer.

Cost and robustness

• Cap limit_posts server-side (and don't allow None from the AI tool), and cap comments per post. Each post's comments are fully fetched, which can mean many paid requests.
• posts_scanned reports the requested limit, not the posts actually scanned.
• Validate source (captions / comments / all).

Tests
Please add tests for the substring case ("great audience"), the heart emoji, an Italian comment, and a controversial but non-coordinated thread.

Thanks again!

[SuperJaba]

Addressed all review feedback in commit af1929b:

1. Whole-Word Matching for Toxicity:

   • Replaced substring matching with whole-word regex () to avoid false positives (e.g., "great audience" no longer matches "die").

2. Emoji Normalization:

   • Stripped U+FE0F variation selectors to reliably match composite emoji sequences such as "❤️" (U+2764 + U+FE0F) against valence weights.

3. Lexicon & Heuristics Cleanup:

   • Deduplicated entries (removed duplicate "super", removed "kot"), cleaned overlap between lexicon and toxicity phrases.
   • Separated spam detection from toxicity scoring — spam comments are flagged via is_spam: True without inflating toxicity score.

4. Expanded Italian & Spanish Support:

   • Expanded sentiment lexicons with positive and negative Italian & Spanish vocabulary.
   • Added comprehensive Italian & Spanish stopword sets (nel, nella, della, questo, hanno, las, los, del, sobre, etc.) ensuring topic clouds are not dominated by function words.

5. Hostility Spike & Coordination Indicator:

   • Renamed "raid detection" to "hostility spike indicator" (hostility_spike_detected).
   • Added coordination analysis inspecting repeated authors and identical comment text to distinguish between organic controversy (diverse authors) and coordinated raids (repeat accounts / duplicate text).
   • Added clear triage disclaimers in UI, docs (doc/web-ui.md), and API responses.

6. Server-Side Guardrails & Reporting:

   • Clamped limit_posts server-side (1–30) and capped comments fetched per post (default 50, max 100).
   • Fixed posts_scanned to report the actual number of posts scanned rather than the requested upper bound.
   • Validated source parameter (captions, comments, all).

7. Test Coverage:

   • Added tests covering: substring isolation ("great audience"), emoji sequence matching ("❤️"), Italian sentiment & stopword filtering, and organic controversy vs. coordinated raids.

[Datalux]

Two things before merging:

1. get_comment_data now has max_comments_per_post=50 as a default, and the existing "Comments" tool calls it without that argument, so that command now silently returns at most 50 comments per post instead of all of them (it's also used by the graph export and the cost estimate). Please keep the old behavior by default (e.g. max_comments_per_post=None) and pass the cap only from analyze_sentiment.
2. #2705 is now merged into development, so this PR conflicts in src/web/static/index.html. Please rebase on development.

Once the CI is green I'll merge it. Thanks!

[SuperJaba]

Addressed both items in commit 69734d7:

1. get_comment_data Default Behavior:

   • Restored default max_comments_per_post=None in get_comment_data so all comments are fetched as before when called by standard tools (such as "Comments", graph export, and cost estimation).
   • The cap is now passed explicitly only from analyze_sentiment and extract_topics.

2. Rebased on development:

   • Rebased cleanly onto origin/development on top of PR feat: profile snapshot, diff tracker, and webhook alerts #2705 (20fc1aa), resolving the merge conflict in src/web/static/index.html.

[Datalux]

Thanks for the rebase, it applies cleanly on development now, and the get_comment_data default is back to None.

One thing left before merging: the try/except TypeError around get_comment_data is still there, in analyze_sentiment and in extract_topics. It isn't needed, since the method is in the same class and already accepts max_comments_per_post. If a TypeError is raised for any other reason, it would re-fetch every comment without the cap, doubling the cost and ignoring the limit. Please remove both fallbacks and call get_comment_data(limit_posts=..., max_comments_per_post=...) directly.

[SuperJaba]

Removed both try/except TypeError fallbacks in analyze_sentiment and extract_topics in commit 65be33c and calling get_comment_data(limit_posts=..., max_comments_per_post=...) directly as requested. All test suites pass cleanly.