Troubleshooting#

crawl() reports problems as strings in manifest.warnings (and per page in page.warnings). A page-level failure reads [depth N] <url>: <kind>: <message>, where <kind> is a fetch error kind. fetchSource() returns the same kinds in result.error.kind, except the replay-only no-snapshot, snapshot-corrupt, and snapshot-store-error.

A crawl returns no pages#

Read manifest.warnings for the [depth 0] <seed>: line that names an error kind. Fail-open robots.txt warnings and sitemap warnings can come before it. The usual causes:

• egress-denied: the guard refused the seed. The message names the code, for example Server egress rejected (DENIED_ADDRESS) for internal.example. See Egress denied.
• robots-denied: robots.txt disallows the seed path for your user agent's product token, the part of userAgent before the first space or /.
• http-error: the seed answered with a non-2xx status.
• no-snapshot: in mode: "replay", the store has nothing for this page. Replay ids are built from the seed URL, so replay with the exact seed URL you crawled.

Egress denied#

testOnlyAllowedLoopbackOrigins entries must be exact origins such as http://127.0.0.1:4173: http:, an explicit port, and a host of 127.0.0.1 or [::1]. In crawl(), anything else, including localhost, is an InvalidCrawlConfigError; fetchSource() returns it as a network error. egress: { guarded: false } turns the guard off for plain HTTP requests (rendering stays guarded); do not use it for URLs you do not control.

A redirect is refused#

fetchSource() follows redirects itself. It refuses one to another host, or from https: to http:, with a network error such as redirect rejected: cross-host target other.example. More than 5 hops is too-many-redirects. These limits are fixed.

The manifest is truncated#

truncated: true and the warning maxPages cap (N) reached; M further URL(s) discovered but not fetched mean the frontier still had URLs. maxPages counts every attempt, including failures, and is capped at 500. Raise maxPages, lower maxDepth, or use discovery: "sitemap" to reach listing pages directly.

Policy fields that only warn#

These warnings mean the field was accepted but not acted on:

• shouldFollow scoring is deferred in this MVP; the deterministic host/depth policy was used
• politeness.concurrency is deferred in this MVP; the frontier ran sequentially

Rendering does not happen#

render failed; kept plain-fetch snapshot (...) means the plain snapshot was used. When the cause is render unavailable: optional peer dependency 'playwright' is not installed, install playwright and its Chromium browser. With render: "on-shell", a page is rendered only if it is HTML and looks like a JavaScript shell; a page with real text content is not rendered. Use render: "always" to force it.

render: networkidle fallback used after Nms means the page never went quiet on the network, so forage captured it at domcontentloaded instead.

robots.txt warnings#

robots.txt for <origin> unreachable (...); proceeding (fail-open), and the matching proceeding (fail-open) warnings for a 429, a 5xx, or a robots.txt over 256 KiB, mean forage could not read the rules and crawled as if none existed. A 4xx robots.txt is treated as allow-all without a warning.

Snapshot store errors#

A citation does not resolve#

resolveSnapshotSourceRef() reports why in error.kind:

A text capture made by 1.0.0 or later has a different sourceRef, and possibly a different bodyHash, than an earlier-format capture of the same response. Older references still resolve against the records they were made from.