Reference#

The public surface of @kontourai/forage 1.x. The package has three entry points, defined by exports in package.json, and no CLI.

@kontourai/forage#

crawl()#

crawl(seed: Seed, policy?: CrawlPolicy): Promise<CrawlManifest>

Throws only InvalidCrawlConfigError (a TypeError). Everything else is reported in CrawlManifest.warnings.

Seed#

CrawlPolicy#

Numbers must be finite and non-negative. Sitemap discovery runs only when maxPages is above 1 and maxDepth is at least 1, and accepts at most maxPages page URLs. crawl() has no response-size option.

CrawlManifest and Page#

CrawlManifest has seed (the normalized seed URL), pages, truncated (true when URLs were still queued at the maxPages cap), warnings, and, when discovery is "sitemap" or "both", sitemap: { documentsRead, urlsDiscovered } (zeros when discovery was skipped).

Each Page has url (after redirects), status, body, snapshot, sourceRef, depth, rendered, and its own warnings.

EgressPolicy#

Snapshot stores#

createInMemorySnapshotStore(): ExactSnapshotStore & PrunableSnapshotStore
createFilesystemSnapshotStore(options: {
  root: string;
  maxHistoryFiles?: number; // 1 to 10,000; default 10,000
}): VerifiedHeadSnapshotStore & PrunableSnapshotStore

The SnapshotStore interface:

Filesystem store limits and behaviour:

• At most maxHistoryFiles records per source. At the cap, put() rejects with SnapshotHistoryFullError (code: "history-full") until you prune(). maxHistoryFiles cannot change after a source directory is initialized.
• Each record is at most 96 MiB, and a durable body at most 64 MiB.
• Reads are fail-closed. A malformed, mismatched, or foreign record makes list, get, and latest reject with SnapshotStoreReadError, whose code is "snapshot-corrupt", or "snapshot-store-error" for storage and race failures. reason gives the detail, such as "malformed-record" or "store-busy".
• put() and prune() take a per-source lock file, source.lock. A writer waits up to 90 seconds for it, then fails with reason store-busy. Every process writing to one store must run a forage version that takes this lock.

isSnapshotStoreReadError() and isSnapshotHistoryFullError() are type guards for the two error classes.

Verified source-head witnesses#

Only the filesystem store has these two methods:

readVerifiedHead(sourceId, limits?): Promise<ReadVerifiedHeadResult>
compareHeadWitness(witness, limits?): Promise<HeadWitnessComparisonResult>

readVerifiedHead fingerprints the source's record and index files, authenticates the latest record, and fingerprints them again. On success it returns { kind: "found", headSnapshotRef, witness }, where witness is a forage.source-head-witness/v1 object with an opaque token. Otherwise it returns missing, unavailable, corrupt, or unsupported.

compareHeadWitness reads only metadata and returns matches, changed, missing, unavailable, corrupt, or unsupported. A changed result does not name the new head.

limits can only lower the defaults: maxEntries (the store's maxHistoryFiles), maxIndexBytes (8 MiB), and maxVerifiedBodyBytes (96 MiB). The token is local to the physical store, so a copy or restore can invalidate it.

Text bodies#

Rendering#

crawl() and fetchSource() load the renderer on their own when rendering is requested. These exports are for callers that drive a browser themselves:

@kontourai/forage/egress#

createGuardedFetch(options?: {
  resolver?: EgressResolver;
  testOnlyAllowedLoopbackOrigins?: readonly string[];
  maxRedirects?: number; // default 5
}): typeof fetch

A fetch-shaped function. For each request and each redirect hop it validates the URL, resolves the hostname once, requires every answer to be a public address, and connects to the first validated IP while keeping the hostname for Host and TLS SNI. With the default redirect: "follow" it follows redirects itself and refuses a redirect to another origin or from https: to http:. With redirect: "manual" it returns the redirect response.

evaluateEgressUrl(url, deps?) runs the same checks without a request and returns { url, addresses }. classifyAddress(address, safeHost?) checks one IP literal and returns { address, family }. All three throw EgressUrlPolicyError with one of these code values:

@kontourai/forage/fetch#

fetchSource()#

fetchSource(config: SourceConfig, options?: FetchSourceOptions): Promise<FetchResult>

Returns { snapshot?, error?, warnings? } and never throws for an operational failure. It reads options.store but never writes to it.

SourceConfig#

FetchSourceOptions#

FetchErrorKind#

Provenance references#

resolveSnapshotSourceRef returns { ok: true, reference, snapshot, integrity }, where integrity is "snapshot-envelope" for current references and "body-and-identity" for references from the 0.3 grammar (no snapshotSha256). On failure it returns { ok: false, error } with error.kind one of invalid-reference, snapshot-not-found, snapshot-mismatch, snapshot-corrupt, or snapshot-store-error. A store without findExact gives snapshot-store-error.

The subpath also re-exports decodeTextBody, parseDeclaredCharset, snapshotHashBasis, the snapshot and store types, and the store error classes.

Fixed limits#