Adaptive back-off for rate-limited hosts via the queue stream

[dpol1]

Stage 2 of #867, closes #1106. #1973 handled the explicit case, a 429/503 carrying a Retry-After header. This covers the one we probably see most in the wild: servers that rate-limit without sending any header. Today we just fetch the host again and collect another 429.

A headerless 429/503 arriving on the queue stream now blocks the host's queue in URLFrontier for a growing duration: 60s the first time, doubled per new incident, capped at a day, forgiven after 30 quiet minutes. Same curve as Nutch since 1.19 (NUTCH-2946), but the state lives with the queue owner: a blocked host never reaches the fetcher at all, so the stale fetching Nutch had to patch around (NUTCH-3114) can't happen here.

Everything from the #867 discussion is in:

• a random fraction on every computed block (urlfrontier.backoff.jitter, default 0.1) so hosts blocked on the same schedule don't all come back at once. Thanks @sebastian-nagel, the daily error spike was exactly the failure mode I had dismissed.
• configurable trigger codes (urlfrontier.backoff.status.codes, default 429 and 503; 403 stays out since it so often just means "forbidden"). They now gate the Retry-After path of Block the URLFrontier queue on Retry-After via the queue stream #1973 as well, with one behaviour change: a malformed or zero Retry-After escalates as a headerless incident instead of being ignored.
• fetch exceptions as incidents behind urlfrontier.backoff.on.exceptions, off by default: a timeout is a noisier signal than a clean 429, I'd flip the default once this has seen some real crawls.
• URLs already in the fetcher's internal queues just fail and reschedule. Failures during an active block count as the same incident, so a draining backlog escalates at most once per window instead of jumping to the cap.

One rename against what I posted on #867: the config prefix is urlfrontier.backoff.*, not fetcher.backoff.*. The fetcher plays no part in this, and it sits next to Phase 1's urlfrontier.max.retry.after.

How it's built:

• the decision logic is a package-private HostBackoff, same split as RetryAfterParser in Block the URLFrontier queue on Retry-After via the queue stream #1973; the bolt keeps the RPC and nothing else.
• state is one Caffeine cache with per-entry expiry at block end + decay: memory stays bounded and the 24h cap stays reachable through escalation.
• a block is only ever extended, never shrunk by a later, shorter signal. A signal suppressed by that guard or by the anti-burst one re-asserts the stored block with an idempotent RPC, so a lost fire-and-forget call heals on the next signal instead of leaving a phantom block.
• HostBlockBolt becomes QueueRegulatorBolt in a separate mechanical commit: with Propagate robots.txt crawl-delay to URLFrontier via setDelay #1979 lined up next, blocking is one of the actions this bolt takes on a queue, not the only one.
• same metadata.persist caveat as Block the URLFrontier queue on Retry-After via the queue stream #1973 (fetch.statusCode, plus fetch.exception when the flag is on); the bolt warns at startup when they wouldn't survive the filter.

Tests: escalation math against a controllable ticker and a seeded RNG for the jitter, regressions for the burst and only-extend guards, and two Testcontainers runs against a real frontier (explicit Retry-After and headerless 429).

For all changes

• Is there a issue associated with this PR? Is it referenced in the commit message?
• Does your PR title start with #XXXX where XXXX is the issue number you are tryin
• Has your PR been rebased against the latest commit within the target branch (typically main)?
• Is your initial contribution a single, squashed commit? (mechanical rename kept as idiff stays reviewable)
• Is the code properly formatted with mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false?

For code changes

• Have you ensured that the full suite of tests is executed via mvn clean verify?
• Have you written or updated unit tests to verify your changes?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0? (no new dependencies)
• If applicable, have you updated the LICENSE file, including the main LICENSE file? (n/a)
• If applicable, have you updated the NOTICE file, including the main NOTICE file? (n/a)

[rzo1]

All of these need to land int the docs :)

[rzo1]

Thanks for the PR. Aside from the docs thing already mentioned, I have two things while skimming through the PR:

1. HostBackoff imports com.github.benmanes.caffeine.cache.*, but external/urlfrontier/pom.xml doesn't declare caffeine as dependency it; it only compiles via the transitive compile-scope dep from stormcrawler-core. Add com.github.ben-manes.caffeine:caffeine to the urlfrontier pom.
2. backoff.max.secs doesn't cap the first block since the first-incident branch (backoffSecs == 0 ? baseSecs : min(...)) skips the cap, so with max.secs < base.secs the first block uses the full base while later ones are capped lower; a negative max.secs blocks once at base and then never again. I would clamp the first step against maxSecs too (or clamp maxSecs >= baseSecs in the constructor), consistent with the clamping already done for factor, decay and jitter.