[SuperJaba]

Overview

When bulk scraping paginated endpoints (followers, followings, comments, post feeds), standard bot traffic triggers Instagram's rate limiting heuristics, resulting in HTTP 429 and feedback_required temporary action blocks.

This PR introduces an authentic human browsing emulation & anti-throttling pacing engine into Osintgram.

Key Changes

1. Gaussian Jitter Engine (src/human_pacing.py):
   • Replaces hardcoded fixed/uniform sleeps with realistic Gaussian distribution pauses bounded by natural reading intervals.
   • Pacing modes: normal (0.8s - 2.4s), stealth (1.6s - 4.8s + reading breaks), fast (minimal ~0.05s delay), and custom.
2. Periodic Reading Breaks:
   • In stealth mode, automatically pauses for 12–22 seconds every 20 pages to mimic natural user feed/profile inspection breaks.
3. Adaptive Backoff & Auto-Decay:
   • When encountering rate-limiting signals or action block warnings (feedback_required, 429), the engine dynamically scales delays up to 4x, protecting sessions from bans, and gradually decays back to normal on successive requests.
4. Service & Backend Integration:
   • Integrated seamlessly into OsintgramService._iter_items and OsintgramService._get_comments.
   • Unified pagination delays in src/instagrapi_backend.py.
5. Web UI & REST API:
   • Added GET /api/pacing/settings and POST /api/pacing/settings with origin protection.
   • Dynamic mode selector and active status badge in the Web UI Settings panel.
   • Configurable via SCRAPE_PACING / OSINTGRAM_PACING env vars or [Pacing] section in config/credentials.ini.
6. Tests & Docs:
   • Full test coverage in tests/test_human_pacing.py (12 tests) verifying distribution clipping, reading breaks, adaptive backoff, and Web UI endpoints.
   • Updated doc/web-ui.md configuration reference.

Verification

• 186/186 tests passing (python -m pytest -q).
• Flake8 0 errors (flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics).

[Datalux]

A few things need fixing before merging.

Naming and docs
The module/docs describe this as "evading bot heuristics" and the mode is called stealth. What the code does is rate limiting with backoff, and that's how it should be presented. Please rename it (e.g. conservative, "adaptive rate limiting") and drop the "evade / emulate a human" wording from the docstring, UI and docs. Please also add a short note in doc/web-ui.md about the risk of an account block with instagrapi, especially in fast mode.

Behavior

1. Pacing is now applied to every backend in _iter_items / _get_comments, including HikerAPI, where there was no inter-page delay before; this slows every paginated scan by about 1.4s per page by default. Please apply it only to the instagrapi backend (or make it opt-in).
2. With instagrapi, both the service and the backend create their own HumanPacer and both call pace(), so delays add up and the break counters are separate. Please share one pacer.
3. The instagrapi backend never calls record_throttle_or_error() / record_success(), so the backoff isn't used where it matters most.
4. A new service (and pacer) is built on every request, so the throttle level is lost after a 429. Please keep the backoff state across scans (module-level, or per account).
5. The default normal mode (mean 1.4s) is faster than the previous 1.2–2.5s delay on instagrapi. Please keep the default at least as conservative as before.

Config

• adaptive_backoff is written to credentials.ini but never read in resolve_pacing_config.
• A single invalid value makes float() raise inside try/except Exception, and the rest of the [Pacing] section is silently ignored. Please parse each option separately and report invalid ones.
• POST /api/pacing/settings doesn't validate the numbers (negative or inverted values, min_delay=0 in custom). Please enforce sane ranges and min <= max.

Thanks!

[SuperJaba]

Rebased on latest development and addressed all review feedback in commit b817f7a:

1. Naming & docs:

   • Renamed mode to conservative (with stealth retained as backwards-compatible alias). Dropped human emulation / bot evasion claims throughout docstrings, UI, and documentation.
   • Added documentation in doc/web-ui.md explaining rate limiting behavior along with an explicit warning note regarding the risk of feedback_required action blocks when using instagrapi (especially in fast mode).

2. Behavior:

   • Pacing delays are scoped to the instagrapi backend; HikerAPI has 0 inter-page delay overhead.
   • Pacer is shared between InstagrapiClient and OsintgramService, with inter-page delays unified in _iter_items and _get_comments to eliminate duplicate delays.
   • InstagrapiClient calls record_success() and record_throttle_or_error() across API methods.
   • Backoff / throttle state is preserved across scans via get_shared_pacer().
   • Default normal pacing range set to 1.2s–2.6s (mean 1.85s), maintaining conservative timing matching baseline instagrapi behavior.

3. Config & validation:

   • adaptive_backoff is now read via sec.getboolean('adaptive_backoff') in resolve_pacing_config.
   • Options are parsed independently with warnings logged for invalid values so valid configuration fields are preserved.
   • POST /api/pacing/settings validates ranges (min_delay <= max_delay, non-negative values, and min_delay > 0 in custom mode).

All 233 unit/integration tests pass cleanly with 0 flake8 warnings.

[Datalux]

One real bug, and a few small things:

1. In POST /api/pacing/settings, the numeric values (min_delay, max_delay, break_*, adaptive_backoff) are only written to credentials.ini when remember is true, and then the shared pacer reloads its config with resolve_pacing_config(), which reads the file and the environment. So with remember=false only mode is applied and the other values are silently dropped. Please apply them directly to the shared pacer's config.
2. _call counts every exception except "not found" as a throttle, so a login error, a private profile or a validation error also increases the backoff. Please limit it to rate-limit style errors (429, feedback_required, "please wait", challenge).
3. The stealth alias isn't needed since this feature hasn't been released, so there are no existing configs to keep working. Removing it also removes the last "stealth" wording from the code.
4. Minor: type(api_client).__name__ == "InstagrapiClient" is fragile; getattr(api_client, "pacer", None) is enough.