@kontourai/lookout#

Cheap, honest drift detection for content you re-check over time — "did this source change since we last looked?"

A small source registry and non-throwing drift-check runner built on Forage snapshots. It composes Forage for fetching and snapshot storage, provides deterministic proposal diffing, and emits neutral, typed drift — its own vocabulary, in its own dependency-free package. It does not implement acquisition or extraction, author trust-layer records, project to Surface, review claims, notify, crawl, or schedule. Operational failures are returned as typed results.

Quick start#

Register one source in a JSON file (defaults to <cwd>/lookout.sources.json; see Registry for every field):

{
  "version": 1,
  "sources": [
    {
      "id": "example-home",
      "kind": "web-page",
      "url": "https://example.com/",
      "targetSchema": [{ "path": "title", "type": "string", "required": true }],
      "cadenceHint": "daily",
      "renderPolicy": "never"
    }
  ]
}

Then check it with the CLI:

lookout check example-home --registry ./lookout.sources.json

The first run has nothing to compare against, so it persists a baseline snapshot and reports changed/initial:

{"sourceId":"example-home","sourceUrl":"https://example.com/","checkedAt":"2026-07-20T14:12:48.248Z","warnings":[],"kind":"changed","priorSnapshotRef":null,"currentSnapshotRef":"forage-snapshot:example-home?...","changeBasis":"initial"}

Run the same command again and, if the source hasn't changed, Forage's conditional request returns 304 with zero body transfer — no re-download, just confirmation:

{"sourceId":"example-home","sourceUrl":"https://example.com/","checkedAt":"2026-07-20T14:12:53.089Z","warnings":[],"kind":"unchanged-304","snapshotRef":"forage-snapshot:example-home?..."}

Exit code is 0 for both runs — a per-source result, even a fetch failure, is not a CLI failure. See CLI for the full exit-code contract and Check results for what each kind means.

Why it's different#

The naive way to answer "did it change?" is to re-crawl the source and diff the bytes. That's expensive (full re-download every time), noisy (a changed ad or timestamp reads as "changed"), and fragile (a thrown error mid-run looks the same as "no change"). Lookout is the opposite on all three:

So the point isn't "diffing" — it's cheap + honest + semantic + review-ready change detection, so a periodic re-check surfaces only the real delta (this provider is new, that listing changed) instead of re-reviewing everything.

Where it sits#

One layer in a four-verb stack; each repo owns one verb and is usable alone:

• forage — CRAWL: fetch / frontier / SSRF-safe egress / snapshots
• traverse — EXTRACT: content + schema → reviewable proposals
• lookout — CHANGE: did this registered source drift since last look?
• survey — the SHAPE: what reviewed truth looks like (claims / review / resolution)

Lookout composes Forage's /fetch surface for cheap 304-aware re-checks, and Traverse's ExtractionProposal identity for the semantic diff. Dependency arrows point only downward — no cycles. Lookout itself depends on nothing in the trust layer: its events are already Hachure-evidence-shaped (snapshotRef / locator / excerpt / fieldPath), so a consumer or product lifts them into a Hachure TrustBundle via @kontourai/surface's TrustBundleBuilder — the same pattern Traverse uses to match Survey's shape without importing Survey.

SSRF-safe egress. Registered source URLs are operator- / aggregator-supplied and not fully trusted, so lookout routes its default fetch transport through forage's SSRF-pinned guarded fetch (@kontourai/forage/egress). A registered source pointing at a private, link-local, loopback, or cloud-metadata host (e.g. 169.254.169.254) is refused before any connection — a drift check can never be turned into an SSRF vector. A caller that injects its own fetchSource or fetchOptions.fetch (e.g. tests) owns its transport and opts out of the default guard.

Requirements#

• Node.js >= 22.
• @kontourai/forage >= 0.4.0. Exact durable-reference replay requires the resolver first released in 0.4.0; trustworthy unchanged-304 classification also depends on Forage's validator-scoped revalidation. Lookout exact-pins Traverse separately for schema and extraction-proposal contracts.

Registry#

Sources live in a versioned JSON document (default <cwd>/lookout.sources.json; override with the library path argument or the CLI --registry flag):

{
  "version": 1,
  "sources": [
    {
      "id": "docs-home",
      "kind": "web-page",
      "url": "https://example.com/docs",
      "targetSchema": [{ "path": "title", "type": "string", "required": true }],
      "cadenceHint": "daily",
      "renderPolicy": "never"
    },
    {
      "id": "published-results",
      "kind": "structured-file",
      "format": "yaml",
      "url": "https://raw.githubusercontent.com/example/project/0123456789abcdef0123456789abcdef01234567/results.yml",
      "cadenceHint": "weekly"
    }
  ]
}

Fields per source:

Structured-file entries retain raw fetched bytes and use the same guarded fetch, snapshot, comparison, and replay path as every other source. Parsing and domain normalization stay downstream. Prefer immutable, commit-pinned artifact URLs so an upstream branch move cannot silently redefine the cited source location.

Validation reports every deterministic issue at once (with index/id context) and never reads the network. Lookup is exact-id; listing preserves file order — no merging, discovery, or remote registries.

Check results#

A check classifies each source into exactly one of four kinds. Every result carries sourceId, the registered sourceUrl, checkedAt, and Forage fetch warnings.

unchanged-hash asserts same-resource continuity, so it requires both an identical bodyHash and the same resource URL as the prior snapshot. If a source has moved — the prior snapshot's URL differs from this fetch's — the result is changed even when the bytes are byte-identical, and the fresh snapshot is persisted as the new baseline. (The unchanged-304 path is already resource-scoped by Forage's validators, so only the hash path needs this guard.)

A fetch that repeats the prior capture exactly is not appended to snapshot history: same URL, status, body bytes, text decoding, redirects, render state, and etag / last-modified validators, with only the fetch time differing. Its unchanged-hash result names the stored capture as both priorSnapshotRef and currentSnapshotRef, and checkedAt records the check. Other response headers of the repeat, such as Date, are not kept. A stable source therefore adds no records, however often it is checked. The text decoding is the encoding the declared Content-Type charset resolves to, so the same bytes under another charset (other text for an extractor) are a new capture, while utf8 and utf-8 are the same decoding.

Changed captures still grow history. Pass retention to bound it with Forage's prune: after each stored capture, the runner keeps the newest keepLast snapshots, the latest, the prior capture its result names, and every reference cited(sourceId) returns. Cite what your observations, review rounds, and exported receipts still reference. If cited throws or returns anything but snapshot references, nothing is pruned and the result carries a warning. cited is read just before each prune, outside the store's lock, so record a citation before handing its reference out.

const runner = createCheckRunner({
  store,
  retention: { keepLast: 20, cited: (sourceId) => loadCitedSnapshotRefs(sourceId) },
});

Upgrading to 0.7 (Forage 1.0). New text captures hash their exact bytes. A text page whose bytes are not plain UTF-8 (a non-UTF-8 charset, a byte-order mark, or invalid UTF-8) therefore reports changed once, on its first check after the upgrade, and is extracted again. Drift comes from proposal diffs, so a page whose extracted content is identical emits nothing. Plain UTF-8 pages keep their stored capture.

error results preserve provenance: origin: "forage" carries Forage's discriminated FetchError verbatim (its kind, and status when present); origin: "lookout" carries a typed kind — prior-read, persistence, history-full, dependency-contract, or unexpected. history-full means the source holds Forage's maxHistoryFiles records; with retention the runner prunes once and retries before reporting it.

Snapshot references (snapshotRef / priorSnapshotRef / currentSnapshotRef) are portable logical refs from Forage's buildSnapshotSourceRef — never filesystem paths. Snapshots are stored via Forage's filesystem store, rooted by default at <cwd>/.kontourai/lookout/snapshots (override with the CLI --snapshot-root flag or by injecting a store in library use). Lookout adds no custom filenames or storage format; createLookoutSnapshotStore(root, { maxHistoryFiles }) passes Forage's per-source record ceiling through and returns a store with Forage's prune. resolveLookoutSnapshot() replays one exact reference through an injected store or Lookout snapshot root, authenticates its body and replay metadata, and never fetches. References emitted before Forage's replay-envelope digest remain resolvable with integrity: "body-and-identity"; new references report integrity: "snapshot-envelope".

Schema coverage#

Drift isn't only "the bytes changed" — a source can reformat so that a field your schema declares silently stops being produced. The byte/proposal diffs above are temporal (they need a prior and say nothing on a first look), so they can't catch this. checkSchemaCoverage is the static complement:

import { checkSchemaCoverage } from "@kontourai/lookout";

if (source.kind === "structured-file") throw new Error("structured sources are parsed downstream");
const { covered, gaps } = checkSchemaCoverage(source.targetSchema, proposals);
// covered: declared paths at least one proposal produced (schema order)
// gaps:    declared paths that produced none, each with its `required` flag
for (const gap of gaps) {
  // escalate a missing required field harder than a missing optional one
}

It reduces one declared TargetFieldSchema[] and one produced ExtractionProposal[] set into { covered, gaps } by exact proposal.fieldPath === field.path. It is pure and total — no prior, no network, no throw — and answers on the very first observation. It measures the produced set you pass (it runs no post-verification filtering itself), reports only against the declared surface (an undeclared proposal path is neither covered nor a gap), and preserves schema order. Escalation — warning, review item, or hard failure — is the consumer's call.

CLI#

lookout check <id> [--registry <path>] [--snapshot-root <path>]
lookout check --all [--registry <path>] [--snapshot-root <path>]
lookout emit-drift <id> --observation <path|-> [--registry <path>] [--observation-root <path>] [--snapshot-root <path>]

• Emits exactly one compact JSON object per checked source, per stdout line (JSON Lines), in registry order. Snapshot bodies and resolved secrets are never serialized. Human/fatal diagnostics go to stderr.
• <id> and --all are mutually exclusive.
• Exit codes: 0 when the requested checks were attempted — including per-source error results (a network failure on one source is a normal, emitted result, not a CLI failure). Non-zero only when an invocation or registry problem prevents producing the requested result set at all (unknown id, invalid arguments, or an unreadable/invalid registry).

This exit/output contract is what makes lookout check --all safe to drive from an external scheduler.

emit-drift is a separate composable command: its input is a JSON object with observation (a ProposalSetObservation) and its matching check anchor. Extraction is supplied by the caller. The first successful input commits a baseline fact and returns priorObservationId: null with empty events; a genuine later change returns non-empty events and facts plus a priorObservationId pointing at the prior observation it was diffed against. State defaults to .kontourai/lookout/observations, uses immutable digest-addressed records and an atomic per-source pointer, and retains the latest two valid observations. The emitted events are already Hachure-evidence-shaped; a consumer or product lifts them into a Hachure TrustBundle with @kontourai/surface's TrustBundleBuilder when it wants Surface projection — lookout itself authors nothing in the trust layer.

An observation id is the SHA-256 of the record's canonical JSON: object keys and proposals in UTF-16 code-unit order, never the host locale's collation. New records are version: 2. Records written before this change are version: 1, whose digest used the writing host's locale collation. They still load on a host whose locale collates their keys the same way, and the next commit replaces a version: 1 prior with a version: 2 record. Under a different locale, a version: 1 record with non-ASCII or mixed-case keys or values can read as corrupt-state.

Verified proposal head witnesses#

The concrete filesystem observation store additionally exposes readVerifiedHead(sourceId, limits) and compareHeadWitness(witness, limits). The first returns an authenticated observation id and snapshot reference with an opaque, versioned witness; the second reads only bounded head metadata and returns matches, changed, missing, unavailable, corrupt, or unsupported. It never reads proposal bytes during comparison.

A witness is an as-of fence, not a freshness guarantee: it binds the source, current head, store scope, pointer, and filesystem metadata observed at capture. It detects ordinary older writers without requiring them to write a sidecar. Restarting the same store preserves a witness; copying or restoring the store can invalidate it. It does not prove proposal-body bit-rot or provide an atomic transaction across stores. Limits are finite hard caps on enumeration, pointer, and authenticated-record reads; callers may only narrow the defaults. Store paths refuse symbolic links. An existing source lock is not automatically broken: after confirming no writer is active, an operator may remove an abandoned .lock file and retry.

Before an observation can be diffed or committed, emit-drift resolves its current and selected-prior Forage snapshot references against the configured snapshot store. This is an explicit capability (--snapshot-root defaults to .kontourai/lookout/snapshots); missing, corrupt, malformed, or insufficiently bound captures produce no facts and no continuity commit.

Library#

import {
  loadRegistry,
  createCheckRunner,
  createLookoutSnapshotStore,
} from "@kontourai/lookout";

const registry = await loadRegistry(); // <cwd>/lookout.sources.json
const runner = createCheckRunner({ store: createLookoutSnapshotStore() });
const results = await runner.checkAll(registry.list());

createCheckRunner accepts injected seams — store, fetchSource (defaults to Forage's SSRF-guarded fetcher), fetchOptions, and clock — so checks run with no live network or timers in tests. Injecting either fetchSource or fetchOptions.fetch overrides the default guarded transport.

Using a Traverse snapshot store#

store is a Forage snapshot store. A Traverse store has the same method names but a different record: it requires a resolved contentType, keeps a binary body on bodyBytes, and its bundled stores refuse a record that lacks them. Passed directly, it refuses every capture (an error result of kind persistence, nothing stored), and a Traverse record read as the prior is refused as prior-read rather than compared. Wrap it instead:

import { createFilesystemSnapshotStore } from "@kontourai/traverse/fetch";
import { createCheckRunner, fromTraverseSnapshotStore } from "@kontourai/lookout";

const store = fromTraverseSnapshotStore(createFilesystemSnapshotStore({ root }));
const runner = createCheckRunner({ store });

The adapter adds contentType from the captured Content-Type header, moves a binary body to bodyBytes and back, and implements exact lookup, so captures read back unchanged, repeats classify as unchanged-hash, and references resolve through resolveLookoutSnapshot(ref, { store }). Traverse stores cannot prune, so retention is skipped with a warning. findExact reads the source's whole history, so its cost grows with that history.

Give Lookout a dedicated Traverse store, or source ids no Traverse fetcher writes. Under a shared source id, a Traverse capture hashed over decoded text (a rendered page, a fetch without arrayBuffer(), a record from before Traverse 5) makes the next Lookout check report changed when the response is not valid UTF-8, and a binary body Traverse does not classify as binary (for example a GIF) cannot be prepared by Traverse: replay through fetchAndExtract returns a binary-content error.

Bring your own source store#

The registry file is a convenience, not the contract. What check classification, drift emission, and observation lineage actually consume is source identity — enumerate sources, resolve one by exact id — and that contract is the SourceStore interface: list() and get(id), each returning its value directly or as a promise. LookoutRegistry (what loadRegistry returns) is the file-backed implementation; nothing downstream can tell implementations apart.

Two application shapes motivate the seam. An application whose canonical source of truth is its own database — reviewed rows in Postgres or D1, not a JSON file — builds LookoutSource values from those rows and should not have to adopt a second store of record to inherit lineage, continuity, and CHECK classification. And a host that owns its own acquisition and run state wants to hand Lookout sources it already holds in memory. Both were previously re-implementing identity handling around the registry; either can now do:

import { inMemorySourceStore } from "@kontourai/lookout";

const store = inMemorySourceStore(sourcesBuiltFromYourRows);
const results = await runner.checkAll(await store.list());

inMemorySourceStore applies exactly the validation loadRegistry applies to a file — duplicate ids, URL shape, per-kind field rules, every issue reported at once — so source-id uniqueness is enforced structurally rather than by review convention. A database-backed application can also implement SourceStore directly (both methods may be async) and query lazily; runCli's loadRegistry option accepts any SourceStore. Keep ids stable either way: snapshot identity and observation lineage key off id, so changing one starts a new history.

Observe changed sources without re-extracting unchanged ones#

createObserveExtractDiff is an optional library composition for callers that want one typed observation per check. Supply acquisition, extraction, and recording capabilities; Lookout does not choose a content-preparation method, provider, or observation database.

import { createObserveExtractDiff } from "@kontourai/lookout";

const composition = createObserveExtractDiff({
  acquisition: { check: runner.check },
  // The store acquisition persists to; used to compare captures by reference.
  snapshots: store,
  extraction: {
    async extract({ source, snapshotRef }) {
      // Resolve `snapshotRef`, prepare content, and invoke a caller-selected
      // extraction implementation. Return its public Traverse result.
      return extractChangedSource(source, snapshotRef);
    },
  },
  recorder: {
    async record(observation) {
      // Caller-owned continuity and durable storage.
      return saveObservation(observation);
    },
    async lastExtractedSnapshotRef(source) {
      // The newest non-null extractedSnapshotRef(observation) you recorded.
      return loadLastExtractedSnapshotRef(source.id);
    },
  },
});

const result = await composition.observe(source);

unchanged-304 and unchanged-hash are recorded as unchanged and never call the extraction capability, so they make zero preparation and provider calls, when the capture has the same URL, body hash, and text decoding as the recorder's lastExtractedSnapshotRef. Both references are resolved through snapshots; one that does not resolve (for example, pruned) counts as a different capture and is extracted. That is the last snapshot an observation fully handled: the current snapshot of a completed or unchanged observation, or of a partial one whose every loss would recur on the same capture (the content cap, the chunk cap, or the provider-call ceiling), as extractedSnapshotRef(observation) returns. Otherwise the capture was persisted but never extracted (for example, a provider failed on the check that first saw it), so it is extracted now against that baseline instead of being reported as unchanged forever. Changed observations retain source and snapshot references, Traverse's prepared artifact identity, the proposal set, and the current/prior observation identities returned by the recorder. partial, provider-failure, mixed partial-provider-failure, and extraction-failure remain distinct typed outcomes. Provider failures are reduced to provider-neutral kind and retryable classifications; provider names, messages, native diagnostics, free-form extraction warnings, raw responses, and thrown-error text are not copied into the durable observation. Traverse's per-chunk coverage is kept on the attempt. Every outcome other than completed also marks its proposal set incomplete (Traverse's partial reason, or extraction-error, plus coverage). When that proposal set is diffed or passed to createDriftEmitter, a proposal the prior had that the incomplete run lacks may sit in text that was never read, so it is reported under unobserved* facts, never as a removal, and drift facts carry the same incomplete marker. A run whose loss could go differently next time (an output cap, an unusable answer, a provider failure, a token budget, a cancellation) never replaces an existing baseline in the observation store (committedObservation is then null), so the next run is diffed against the last baseline: a value that run missed is not re-reported as new, and a real removal is still reported. A value change such a run did see is reported by it and again by the next run that is diffed against the same baseline. A run whose every loss recurs on the same capture (the content cap, the chunk cap, the provider-call ceiling) and a first observation are stored with their marker, so a page that stays capped reports each change once. What a later run has that an incomplete baseline lacked is listed under newlyObserved* facts rather than as additions or events; a value removed from text the cap hid is not reported.

Coverage ranges and locators are offsets into each capture's own prepared text, so an insertion earlier in the page shifts them and offsets cannot tell a new entity from one that moved into the read window. To report added entities on a page that stays capped, give createDriftEmitter a priorText preparation:

const emitter = createDriftEmitter({
  store,
  snapshotStore,
  priorText: {
    // Prepare a stored capture exactly as your extraction did. No provider call.
    prepare: ({ snapshot, preparedArtifact }) => prepareText(snapshot, preparedArtifact.preparationMode),
    maxChars: 4_000_000, // default; larger priors are not rebuilt
    timeoutMs: 30_000, // default; a slower prepare is abandoned
  },
});

Against an incomplete prior, the emitter reads the prior's snapshot from the snapshot store, has the caller re-prepare it, and verifies the text against the prior's stored prepared-artifact digest. A current entity none of whose exact excerpts occur anywhere in that text was not in the prior's capture, so it raises new-entity-appeared. An entity with an excerpt in text the prior read is a proposer difference, and one with an excerpt only in text the prior never read (for example shifted into the read window) is still newly observed; newlyObservedEntityAnchors says which. An excerpt that is empty or was not placed by Traverse's exact occurrence resolver is unanchorable. The fact's priorText is verified; not-needed when no entity needed anchoring, in which case the prior's snapshot is not read at all; or unavailable with a reason (not-configured, no-prepared-artifact, preparation-changed, too-large, snapshot-unresolved, preparation-failed, preparation-timeout, text-mismatch), and then every entity the prior lacks stays newly observed. maxChars is checked against the prior's prepared-text length before any read, but a snapshot store cannot report a body's size before returning it, so the body-length check bounds preparation, not the snapshot read. A prepare that exceeds timeoutMs is abandoned, not cancelled. The prior needs its preparedArtifact, which createObserveExtractDiff puts on each proposal set. Limits: an entity the prior never read whose every excerpt also changed is reported as new; a short excerpt that happens to occur elsewhere keeps a real addition newly observed; and a field added to an entity the prior already had stays newly observed. Treat newlyObserved* as reviewable: buildSemanticReviewWork turns each newly observed proposal into proposal-newly-observed review work.

A proposal's confidence is optional: when absent it stays absent in stored observations, diff evidence, and review candidates, and a retained field whose confidence appeared or disappeared is listed in confidenceChanges (a fact, not an event; numeric differences between two scores are not listed). A first changed observation has priorObservationId: null; it is a baseline observation, not a fabricated list of additions or removals.

The composition validates that the check identifies the requested registered source and that Traverse validates the prepared artifact whose snapshot anchor matches the current check. This prevents mismatched capability results from being recorded as one observation. Proposal excerpts, source URLs, and acquisition check warnings can still contain sensitive source material; callers must apply their own retention and access policy before durable storage.

The existing createCheckRunner and createDriftEmitter entrypoints remain available. Use createDriftEmitter when the caller wants deterministic proposal-diff events with its own identity callbacks.

Route semantic transitions to review#

buildSemanticReviewWork turns one genuine prior/current proposal transition into structurally Survey-compatible ReviewItem resources without adding a runtime dependency on a review product. The caller supplies entity/field identity callbacks and claim meaning; Lookout supplies deterministic transition and item identities.

const work = buildSemanticReviewWork({
  prior,
  current,
  observationIdentity: { prior: priorId, current: currentId },
  schema: source.targetSchema,
  selectEntities,
  entityIdentity,
  proposalsFor,
  fieldIdentity,
  claimTarget(change) {
    return describeClaim(change);
  },
});

Added, removed, moved, provenance-changed, and value-changed proposals become distinct work items. One field change is one item: a value change that also moved is a single proposal-value-changed item carrying both locators. Provenance is compared on the occurrence the resolver settled on, so a run that differs only in occurrence.selection / hintUsed (whether the provider sent an optional hint) creates no work. Match count, selected index, selected span and ambiguity are still compared. An excerpt that was only narrowed around a value that stayed put also creates no work: the current span lies inside the prior span with the same text, only whole paragraphs were dropped (a blank line separates them from what is kept), and the equal string value sits at the same offsets on token boundaries. The diff lists it under facts.excerptBoundaryChanges with the dropped text. Widening, shifting, dropping text not set off by a blank line, a value inside a longer word or number, or a value the excerpt does not literally contain is still one proposal-moved or proposal-provenance-changed item. Consumers that asserted two items for one changed field will now see one. A proposal an incomplete prior lacked (newlyObserved*) becomes proposal-newly-observed work: it may be new, or may have sat in text the prior never read. An occurrence of a field the prior did read, found by the current run at another offset, is one proposal-moved item, never also newly observed. Pass priorPreparedText (the prior's verified full prepared text) to report an entity whose text is absent from it as added instead. New coverage or exact-provenance gaps are also reviewable. Each available side retains its exact snapshot reference, observation time, locator, excerpt, and extractor. An absent side is explicit and anchored to the corresponding observation snapshot, so a removal remains reviewable even when the new extraction emits no value. Identical proposal sets produce no semantic work; replaying the same pair of observation identities produces byte-identical resources.

Review resolution, escalation, persistence, and authority policy stay with the consumer. Evidence excerpts and caller-supplied claim targets can contain sensitive data; apply retention, redaction, and access controls before storing or forwarding review work. Only compact typed identities cross the producer metadata boundary; no provider messages, native diagnostics, or raw responses are copied.

Non-goals#

Acquisition and extraction implementations, Surface projection, notifications, crawling, review/escalation or authority policy, rendered-fetch wiring (renderPolicy is inert), and scheduling. Forage retains all fetch politeness, robots, redirects, retries, timeouts, headers, user-agent, and rendered-fetch behavior.

Development#

pnpm install
npm run verify   # content-boundary + decisions + typecheck + test + pack sanity

The pnpm version is pinned in package.json (packageManager). Dependency install scripts are blocked by default; the only packages allowed to run one are listed under allowBuilds in pnpm-workspace.yaml, pinned by version. Scripts are still run with npm run … — that only invokes package.json scripts and does not depend on which tool installed node_modules.

Individual gates: npm test, npm run typecheck, npm run check:pack, npm run check:decisions, npm run check:content-boundary.

License#

Apache-2.0.