network: per-host pressure in the navigation rate limiter

[krichprollsch]

The goal of the PR is to space out top-level navigations to the same host. The delay starts
near zero, grows while you keep hitting the host, and comes back down over
time. Only top-level navigations are affected. Iframes, scripts, images, XHR
and fetch are never delayed, and localhost, 127.0.0.1 and [::1] are
exempt.

The bucket is keyed by hostname, ignoring port and scheme.

The rule

Each host carries a pressure counter. Pressure sets the spacing:

spacing = min(20ms * (1 + pressure / 10), 1200ms)

Each navigation books the next free slot:

start = max(now, tat - tau)
tat   = max(tat, start) + spacing

tat is the time the host's booked slots run out. tau is the burst
allowance, (--http-nav-burst - 1) * 20ms, which is 180ms by default. It lets
an idle host run ahead before anything waits.

What you actually see

The first 10 navigations to an idle host go out back to back. After that the
spacing appears and grows:

enforced gap	first at navigation	after
0ms	1 to 10	0s
20ms	11	0.02s
40ms	12	0.06s
60ms	27	0.68s
80ms	52	2.20s
100ms	100	6.06s

It stops at 100ms and stays there. That is not the cap, it is where the two
sides balance: one navigation adds a unit, 100ms of elapsed time removes one.
So a sustained crawl of one host runs at 10 navigations per second, the
same rate --http-nav-delay 100 gives you.

The burst allowance is not a one time budget. A host left alone long enough
gets its 10 free navigations again.

What raises the delay

event	pressure added	effect
one navigation	+1	+20ms every 10 navigations
429 or 503	+50	+100ms at once
other 5xx, or no answer	+10	+20ms at once

A Retry-After header on a 429 or 503 pushes the next slot directly, honoured
up to 60s. Only the delay-seconds form is read; an HTTP-date is ignored.

Pressure stops at 590, so the spacing never exceeds 1200ms. You reach that
with twelve consecutive 429s, or by submitting hundreds of navigations at once.

What lowers it

One unit of pressure is forgiven every 100ms. This counts wall clock, not idle
time: a host under constant load drains too, which is why the spacing settles
instead of climbing to the cap.

Draining is computed lazily, when the host is next touched, and the clock moves
in whole 100ms steps so nothing is lost to rounding.

from the 100ms plateau (pressure 41)  ->  drained after ~4s
from the cap           (pressure 590) ->  drained after ~59s

Measured

400 navigations per phase against one host, with demo/rate-limit:

  mode      phase  reqs  span (s)  req/s  gap p50  gap p95  gap max
  off           1   400      2.29  174.5      5.5      7.7     10.5
  fixed100      1   400     39.00   10.2    100.0    100.8    101.2
  adaptive      1   400     36.06   11.1     99.8    100.7    101.3
                2   400     36.06   11.1     99.8    100.8    101.2

Phase 2 runs after 8s of no traffic and repeats phase 1 exactly. The spacing
falls back to zero and climbs the same curve again.

The gaps the server saw at the start of a phase, in ms:

 nav          2    3    4    5    6    7    8    9   10   11    12    13    14
 gap (ms)   4.5  4.4  4.6  4.6  4.7  5.0  4.7  4.4  4.5  4.6  14.8  40.4  39.1

Eleven go at page load speed rather than the ten the arithmetic predicts,
because the allowance refills while you spend it.

Flags

--http-nav-delay 0 turns the limiter off. --http-nav-delay N gives a fixed
N ms spacing with no ramp and no response feedback. Leaving it unset gives the
adaptive behaviour above.

--http-nav-burst sets how many navigations an idle host absorbs at once,
default 10. It buys a head start and nothing else: the ramp and the sustained
rate are the same at any burst setting.

Constants

In src/network/RateLimiter.zig:

constant	value	role
ADAPTIVE_INTERVAL_MS	20	base interval
RAMP_BURSTS	10	navigations per extra base interval
COOLDOWN_INTERVALS	5	base intervals draining one unit; sets the sustained rate
MAX_INTERVALS	60	spacing cap, in base intervals
OVERLOAD_INTERVALS	5	base intervals added on 429 or 503
FAILURE_INTERVALS	1	base intervals added on other 5xx or no answer
RETRY_AFTER_MAX_MS	60000	longest Retry-After honoured

[karlseguin]

I think this discourages using rate-limiting.

Running RUNS=100 node puppeteer/cdp.js against a ./lightpanda serve --http-nav-delay 100

main: 10103ms
rate-limit-xp: 40106ms

I don't understand why a user would opt into this. And not just because it's slower. Imagine a user wants to crawl a site every 1 second (maybe it's their own network and they're doing testing, maybe they know the site advertises with a Crawl-delay: 1). With this PR there is exactly 2 ways to achieve it:

1 - don't use the rate limiter
2 - set the the delay to be 10x smaller so that you ramp up to your target (forcing the first ~90 pages to exceed the desired limit)

[karlseguin]

Maybe we should take out the log.debug(.http, "navigation delayed", ....) in HttpClient then.

[krichprollsch]

My idea was to find a way to set the rate limiter by default.
So I tough about having a delay very low at first (something like 20ms) but having it growing fast when need.

I get your point, it's not very compatible w/ the 100ms delay you configure.

If we decide to enable the rate limter by default, maybe we could disable the dynamic adjustment if you set explicitly the delay, so setting it to 100ms would keep the same as now, setting it to 0 would disable rate limiter.

[karlseguin]

I agree with this. The issue is that it tramples the configured setting. I also agree that --http-nav-delay 0 should be supported, there are many legitimate use cases where that's the desired/correct/safe/respectful behavior.

I'm not sure you need this AND #3332. If you can get the right feedback, then I think this only gets in the way.

[karlseguin]

Actually, I asked Claude to math this out for me. On my httpbin example in #3332 it says I'd need a pressure of 180 before it serves any purpose. It's the @max:

@max(pressure, TTFB_FACTOR * h.ttfb_ms)

It's one or the other. A fast site being crawled ignores the ttfb. A slow site ignores the pressure. It suggested making them additive (using a different formula, mind you).

[krichprollsch]

I changed the PR:

• by default the rate limiter mode is adaptive.
• you can disable w/ --http-nav-delay 0
• --http-nav-delay 100 applies a fixed rate limit only
• I added a warn message in case of navigation delayed
• we observe the response status to adjust the rate limit in case of error (for adaptive only)
• no rate limit for localhost
• in case of redirection, reserve the new host

[karlseguin]

This is my only complaint. RUNS=100 node puppeteer/cdp.js prints this message 97 times. Obviously, your call, but you do have 2 other options

1. use the new logConfigTips in main.zig
2. look at log.warnDisabledWorker for a first-time-only

(you could use both approaches, a tip + first time it actually happens).

[krichprollsch]

I added a bool into Host to notify once per host the message when delayed, I have already hold the lock.
I prefer logging only when the rate limit applies, I expect it to be invisible for most usages.