Refuse the crawlers that take bandwidth and send nobody (#234)

[openipc-ai]

Closes #234.

The test is not "is it a bot" — Googlebot is a bot and is welcome. It is whether a crawler has ever sent a reader. Measured over the fourteen days of log the host retains, referrals against requests:

sends readers	referrals		sends nobody	requests	referrals
Google	18,170		AhrefsBot	1,018	0
Yandex	813		SemrushBot	1,478	0
bing	650		MJ12bot	2,285	0
Baidu	576		DotBot	224	0
DuckDuckGo	437		SofyaBot	610	0
			VelenPublicWebCrawler	96	0

The left column is why robots.txt and #179's search-console work exist. The right is five thousand requests for nothing, from tools that sell backlink reports to other people.

Also refused: the Open Wall scraper, which is the largest of the lot and not a crawler at all. 2,795 distinct addresses share the single agent Mozilla/5.0 (compatible; crawler) in one day, fetching the ActiveStorage snapshot URLs #146 moved to plain files. A residential-proxy population, so no address rule reaches it; the agent string is all they have in common. The pattern is anchored so it cannot swallow every string with the word in it.

The issue's own list included our build host

#234 names "a headless Chrome at 2,568" among the crawlers worth blocking. Those 2,636 requests come from 62.238.81.167, which is this project's build host — tools/shot.mjs, tools/whatnext-check.mjs and every browser check in the repository, loading real pages with their assets exactly as a reader's browser would, which is the entire point of them.

Blocking it would have broken all of them silently, because a 403 is a perfectly good response and the checks would simply have started reporting a different page. There is a test asserting it stays allowed.

Verified live

AhrefsBot   403     Googlebot        200     ClaudeBot   200
SemrushBot  403     bingbot          200     PetalBot    200
MJ12bot     403     YandexBot        200     our headless browser 200
scraper     403     Baiduspider      200

Attributable to this change rather than assumed — 403 rate before and after the reload:

AhrefsBot             0 of 325  ->  1 of 1
compatible; crawler   0 of 811  ->  4 of 4

(Sogou and zgrab also appear in the 403 log. Those predate this — 17 of 140 and 2 of 11 before the reload — and come from elsewhere.)

Two decisions named rather than taken

The LLM crawlers, as #234 asks: Anthropic 1,006/day, OpenAI 591, Perplexity 785. They do send some traffic back and the trade belongs to whoever owns the project's position on it.

The Chinese crawlers, which #234 does not discuss: YisouSpider 7,944 requests over fourteen days, Sogou 3,519, Bytespider 3,161, PetalBot 1,643 — against 5 referrals from sogou, 6 from huawei (PetalBot's parent), 2 from uc.cn, 1 from toutiao and none from yisou. By the referral test they belong in the map. But Chinese is the largest locale by traffic here, these are the search engines that audience uses, and Baidu — kept, and sending 576 readers — shows the mechanism works there. Foreclosing a market the project has barely entered is the same shape of decision as the LLM one.

Both are written into the config with their numbers, so either answer is one line, and there is a test that the reasoning is not quietly deleted.

Verification

750 tests. Five guards, each confirmed by reverting it — after the first version of the anchoring test listed ClaudeBot, PetalBot and Googlebot as the agents that must not be swallowed, none of which contain the word "crawler", so replacing the anchored pattern with a bare ~*crawler left it green.

The referral figures were then confirmed a second time by an independent slower scan of the same fourteen days, which agreed everywhere material and corrected one number: huawei had sent 6 referrals where the config said essentially none.

Already on the origin

Applied via push-nginx.sh, per #144. Backups at *.bak.20260921-101248; one reload to undo.

Noticed while doing this, and much larger

collectd/5.12.0.git is 56,965 requests a day — 22% of all site traffic and 91% of front-page requests. It is the mirrors' own monitoring (openipc.kz and openipc.eu, roughly one request every three seconds each) fetching the entire 25 KB home page as a health check. Before #233 that was 71 minutes of Rails CPU and 1.36 GiB of egress a day. /up exists and answers 2 bytes in 13 ms.

It is cached now so it costs a file read, and it is mirror-side configuration rather than anything in this repository — but it is by a wide margin the biggest thing on this site that takes bandwidth and sends nobody, and it is ours.

[qodo-free-for-open-source-projects]

PR Summary by Qodo

Block non-referring crawlers at the nginx edge

✨ Enhancement 🧪 Tests ⚙️ Configuration changes 🕐 20-40 Minutes

AI Description

• Block high-volume crawlers that generated no reader referrals.
• Precisely reject the distributed Open Wall scraper by its exact user agent.
• Protect beneficial search crawlers and project browser checks with regression tests.

Diagram

graph TD
  A["Incoming Request"] --> B["nginx Edge"] --> C["User-Agent Map"] --> D{"Blocked?"}
  D -- "Yes" --> E["403 Response"]
  D -- "No" --> F["OpenIPC Site"]
  G["Crawler Tests"] -. "validates" .-> C

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Rely on robots.txt

• ➕ Uses the standard crawler-control mechanism
• ➕ Avoids rejecting requests at the HTTP layer
• ➖ Does not stop crawlers that ignore robots.txt
• ➖ Cannot reliably control the distributed scraper

2. Rate-limit crawler traffic

• ➕ Reduces bandwidth without fully excluding agents
• ➕ Can accommodate crawlers whose referral value changes
• ➖ May still permit substantial low-value traffic
• ➖ Can throttle legitimate search indexing or shared-address users
• ➖ Residential proxies can evade address-based limits

3. Block source addresses

• ➕ Does not depend on truthful user-agent strings
• ➕ Can reject abusive hosts before application processing
• ➖ Cannot cover the scraper's residential-proxy population
• ➖ Requires continual address maintenance
• ➖ Risks blocking unrelated users on reassigned or shared addresses

Recommendation: Keep the PR's nginx user-agent map. Edge rejection avoids application and bandwidth costs, while explicit bot names and an anchored scraper signature minimize collateral blocking. The allow-list regression coverage is essential because robots.txt, rate limits, and address rules do not adequately handle the observed traffic.

Files changed (2) +211 / -4

Tests (1) +137 / -0

crawler_block_test.rbGuard crawler blocking and intentional allow decisions +137/-0

Guard crawler blocking and intentional allow decisions

• Adds tests that parse the nginx map and exercise representative user agents. Coverage verifies targeted blocks, scraper-pattern anchoring, continued search-engine and HeadlessChrome access, and retention of documented maintainer decisions.

test/deploy/crawler_block_test.rb

Other (1) +74 / -4

openipc-crawler-block.confReject non-referring crawlers and the Open Wall scraper +74/-4

Reject non-referring crawlers and the Open Wall scraper

• Expands the nginx user-agent map with six SEO crawler patterns and an exact, anchored signature for the distributed scraper. It documents referral-based policy evidence and explicitly preserves build-host, LLM, Chinese-market, and reader-referring crawlers.

deploy/nginx/conf.d/openipc-crawler-block.conf

[qodo-free-for-open-source-projects]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

Tip of the day

💡 Did you know, you can add REVIEW.md to your repo root and Qodo follows it on every PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗