Agent Usage Feedback Loop#

This document defines the usage feedback loop for comparing agent runs across repositories, runtimes, profiles, prompts, skills, and future setups. The measurement schema is runtime-neutral: every runtime should import or emit normalized session, event, and outcome records before reporting. Codex and Claude Code can both flow through the shared telemetry envelope; Kiro uses the same canonical telemetry script when hooks are installed.

The loop has three parts:

1. Import or read runtime telemetry into normalized usage records.
2. Sync quality outcomes from task artifacts, evals, release gates, or humans.
3. Report usage and outcome metrics by repo, runtime, profile, prompt, skill, or task type.

Quality outcomes are not inferred from raw telemetry alone. The CLI can automatically derive coarse outcomes from task artifacts such as .kontourai/flow-agents/<slug>/*.md: delivered and complete become success, failed becomes failure, and optionally open artifacts can be recorded as not_verified. It does not invent quality_score or human_minutes_saved; those remain human/eval/release-gate facts.

Retrospective run observations#

usage-feedback compile-observation compiles one exact correlated run from a manifest of producer-owned records. The manifest names logical source IDs and the exact runtime event, Builder state, trust bundle, Flow state, Flow evidence manifest, economics, and terminal-outcome files. Compilation never joins by path, timestamp, task slug, cwd, process ancestry, or content similarity.

The output is a versioned kontour.flow-agents.retrospective-observation record under .kontourai/flow-agents/feedback/observations/<observation-hash>.json by default. One atomically replaced, locked file per run avoids cross-run lost updates. It contains stable identity, process and verification status, measured usage, explicit missing dimensions, and content hashes plus logical IDs for local drill-down. It contains no source paths, prompts, tool payloads, source content, or secrets. Repeat compilation of unchanged sources is byte-identical and upserts the same observation ID.

Process completion and Flow verification are not independent task quality. Until an external evaluation is joined through the eval-owned boundary, every compiled observation reports quality as NOT_VERIFIED with reason not_independently_evaluated. The compiler emits no causal claim about a Kit, model, prompt, or workflow.

npm run usage-feedback -- compile-observation \
  --manifest .kontourai/flow-agents/run-observation-manifest.json \
  --record-root .

To inventory real local usage without crawling arbitrary home directories, pass each operator-configured record root explicitly:

npm run usage-feedback -- compile-corpus \
  --record-root /path/to/runtime-or-repository-root \
  --record-root /path/to/another-configured-root

compile-corpus inspects only known producer locations under those roots: .kontourai/telemetry, .kontourai/flow-agents, and .kontourai/flow/runs. It never joins records by path similarity, timestamps, working directory, or process ancestry. Records without an exact producer run_correlation remain counted as uncorrelated.

Each configured source is parsed once per invocation. Explicitly correlated records and exact Flow/trust secondary records are retained in one bounded snapshot and reused across that invocation's per-run projections; report facts are not restored from an unsigned cache. JSONL producers may append while the inventory runs: the compiler double-hashes the opening-length prefix, records that accepted prefix in the watermark, and verifies it again before publication. Appends after the accepted prefix are deferred to the next generation. Prefix mutation, truncation, replacement, invalid UTF-8, and resource-limit exhaustion fail closed.

The command writes:

• feedback/corpus/current.json: the atomically replaced pointer to the current complete generation;
• feedback/corpus/generations/<watermark>/watermark.json: content hashes and stable root-local file identities for the observed producer snapshot;
• feedback/corpus/generations/<watermark>/report.json: coverage, correlation, compilation, completeness, ambiguity, and quarantine counts;
• feedback/corpus/generations/<watermark>/manifests/*.json: exact root-relative source declarations for compiled runs;
• feedback/corpus/generations/<watermark>/observations/*.json: privacy-safe per-run observations.

Reports always keep causal and quality effects NOT_VERIFIED. Observational coverage can generate hypotheses for Evals, but it cannot establish that a Kit caused an outcome.

The manifest uses the retrospective-observation-manifest/1.0 schema. Every individual source is optional so an incomplete historical run can still compile honestly, but at least one producer record must carry the exact manifest correlation ID. Source files are relative to the caller-selected --record-root; traversal, absolute paths, empty or dot path segments, and symbolic links are rejected. The record root and output directory are operator trust boundaries, not data supplied by a producer. Their directory chains must be non-group/world-writable, except for sticky shared ancestors such as the system temporary directory, and each final boundary directory must be owned by the current operator. Source files must be owned by the current operator or root and must not be group/world-writable. Concurrent replacement or mutation by another process running as the same OS principal is outside this local compiler's filesystem trust boundary; the accepted-prefix checks detect such changes during an invocation:

{
  "schema_version": "1.0",
  "correlation_id": "run-correlation-id",
  "sources": {
    "runtime_events": {
      "source_id": "runtime-events",
      "file": ".kontourai/telemetry/full.jsonl"
    },
    "terminal_outcome": {
      "source_id": "terminal-outcome",
      "file": ".kontourai/flow-agents/task/workflow-outcome.json"
    }
  }
}

Runtime activity, Flow history, trust, economics, and terminal status are projected only when their complete correlation envelopes equal the canonical run envelope. Sharing a correlation_id alone is insufficient. A trust dimension additionally requires a Surface-valid bundle and one exact Flow-manifest attachment whose embedded bundle and SHA-256 match the supplied bundle.

Usage describes a present run delta only when terminal usage and economics agree on timestamp and token totals. CONFIRMED additionally requires an authenticated runtime binding and exactly matching model, duration, delegation, and cost attributes. When token totals agree but another attribute is unavailable or disagrees, the compiler retains the token counts under NOT_VERIFIED and emits null for each unmatched attribute rather than fabricating or discarding useful consumption data. Fixture authority is always NOT_VERIFIED.

Malformed input and partial reports#

Read-only reports, telemetry-source imports, and retrospective compilation quarantine malformed JSONL records and continue over valid records. JSON reports expose a measurement object with total, valid, and malformed counts plus content-free diagnostics; Markdown reports show the same counts under Measurement State. The CLI also emits one stderr warning per affected logical source. Diagnostics include only a logical source name, line number, SHA-256 content hash, and parse error class. They never include the malformed record or an absolute path.

Retrospective compiler diagnostics also classify schema-valid JSON that fails a producer contract as ProducerValidationError; counts remain separate from syntax failures and source bytes never enter the observation.

Corpus compilation keeps malformed and producer-invalid counts on every affected source reference and in the corpus report, but does not copy the same source-level diagnostic into every observation that shares a multi-run source. Run-specific projection failures still appear on that run's observation.

This tolerance applies only to source analysis. A destination that an import, sync, or upsert operation would rewrite remains strict: one malformed existing record aborts before any write. The command does not silently discard corrupt state.

Storage Defaults#

Repo-local telemetry defaults to .telemetry/:

• .telemetry/full.jsonl: raw runtime telemetry events.
• .telemetry/analytics.jsonl: summarized analytics emitted by telemetry hooks.
• .telemetry/sessions/: per-session runtime artifacts, when the runtime writes them.
• .telemetry/outcomes.jsonl: human/eval outcome records for completed work.
• .telemetry/reports/: generated Markdown or JSON usage reports.

Locations are configurable. CLI commands should accept --telemetry-dir, and shell workflows may set TELEMETRY_DATA_DIR. User-level or global Codex telemetry roots, such as ~/.codex/telemetry or ~/.flow-agents/telemetry, should be treated as configurable installation choices, not hard-coded truth. When comparing multiple repositories or machines, pass each telemetry root explicitly with repeatable --telemetry-dir flags.

Runtime hooks can also mirror redacted telemetry to Console without changing the event shape. Set CONSOLE_TELEMETRY_URL=http://127.0.0.1:3737 for a local Console, or CONSOLE_URL=https://console.kontourai.io for a deployed Console. The transport posts to /api/telemetry/records by default. Hosted Console URLs must use https://; http:// is accepted only for localhost or 127.0.0.1 local development. Use CONSOLE_TELEMETRY_ENDPOINT_URL only when the API path is nonstandard, CONSOLE_TELEMETRY_TOKEN or CONSOLE_AUTH_TOKEN for bearer auth, and CONSOLE_TENANT_ID for hosted tenant routing. If no Console URL is set, telemetry remains local-only.

Flow Agents owns the Console telemetry descriptor at console.telemetry.json. Generated bundles include that root descriptor beside AGENTS.md, scripts, and kit assets. The descriptor maps Flow Agents runtime telemetry and workflow sidecars into generic Console facets for skills, tools, flows, repositories, projects, runtimes, agents, models, statuses, and outcomes. Console consumes those mappings as product-owned display metadata; Flow-owned gate and transition semantics stay in Flow contracts and sidecars. Repository and project metadata are stable cross-user workspace identifiers when present. Local working-directory paths stay out of the descriptor so Console display metadata does not expose usernames or machine-local paths; records without a repository identifier should fall back to their logical product root such as product:flow-agents:.kontourai/flow-agents (the runtime artifact root; #778 corrected the descriptor's own record-source roots to match after they drifted from the .flow-agents -> .kontourai/flow-agents runtime-root migration).

Packaged setup modes are:

• local-files: default local JSONL telemetry only; no Console URL or token. Selecting it explicitly (--telemetry-sink local-files) also pins the install local-only, so an operator Console config on the machine does not apply to it; see "Owner machine mirror" below.
• local-kontour-console: mirror to a separately running local Console, using http://127.0.0.1:3737 unless FLOW_AGENTS_LOCAL_KONTOUR_CONSOLE_URL is set.
• kontour-hosted-console: mirror to Kontour's hosted Console default URL. Pass --console-token-file and --console-tenant for headless hosted setup.
• user-hosted-console: mirror to a self-hosted Console; requires --console-url or --console-endpoint.

Use flow-agents init --yes or --headless with the same flags in CI. The legacy sink names kontour-cloud and hosted-kontour-console are still accepted for existing scripts.

Owner machine mirror#

For a personal machine or repo where an owner wants Claude Code (or other runtime) hook sessions mirrored to a hosted Console, without editing the tracked scripts/telemetry/telemetry.conf default template (that file ships verbatim into every packaged bundle, so writing a personal token/tenant into it would leak to downstream consumers) and without exporting env vars per session, scripts/telemetry/lib/config.sh auto-discovers a gitignored, operator-created conf at either of two conventional paths, no extra wiring required:

1. <workspace>/.kontourai/telemetry-console.conf — repo-scoped, checked first.
2. ~/.flow-agents/telemetry-console.conf — machine-scoped, used when no workspace-scoped conf is present.

Populate either path with the existing preset installer: scripts/telemetry/install-console-config.sh <conf-path> --telemetry-sink kontour-hosted-console --console-token-file <token-file> --console-tenant <tenant>. The installer chmod 600s the file, sets it to be owned by the current user, and never echoes the token. Both conditions matter: config.sh only honors a discovered conf if it is mode 600 and owned by the current user, since that combination can only come from an operator running the installer (or an equivalent manual chmod 600) — it distinguishes an operator-created conf from one that arrived via git clone, a tarball, a PR, or any other supply-chain path, none of which can produce a 600-mode file. A conf that fails that check is treated as if it were absent (fail open) and resolution falls through to the next tier.

Both auto-discovered paths are skipped for an install pinned local-only. An install run with an explicit --telemetry-sink local-files (the flag typed on flow-agents init or the bundle's install.sh) records telemetry_sink_pin=local-files in its installed scripts/telemetry/telemetry.conf, and every reader of the Console config (lib/config.sh and the hooks and relays that source it, the liveness relay pre-gate, board sync, and telemetry-doctor) then reads that file instead, even when a trusted workspace or user-global conf exists. An install without the flag records no pin, so the operator conf keeps applying to it. The local-files default, the interactive wizard's Skip answer, and a sink taken only from FLOW_AGENTS_TELEMETRY_SINK(S) do not pin, and neither does a typed local-files sink when Console settings are supplied with it (a --console-* flag, or a Console variable such as CONSOLE_URL in the installer's environment); init's summary names what blocked the pin.

The pin is the exact line telemetry_sink_pin=local-files (a trailing CR is allowed); a line with spaces, a different value or a byte-order mark is not a pin. Because the installed telemetry.conf is not trust-gated (it can be mode 644 and come from a repository or tarball), a pinned install takes no network destination from it. Instead of filtering out what is known to be dangerous, config.sh holds that file to an allowlist of settings that are local by construction, each with a value grammar, and ignores every other line:

A Console set in the environment still gets the shipped redaction list for a pinned install: CONSOLE_TELEMETRY_REDACT comes from the environment if it is exported there (and not empty), and otherwise is the shipped analytics list itself. Neither the installed telemetry.conf (for example channel.analytics.redact=none) nor an exported TELEMETRY_CHANNEL_ANALYTICS_REDACT can narrow it. The doctor reports the list in use as console.redactFields. Channel endpoints are posted with curl -g, and the loopback grammar refuses [ ] { }, because curl otherwise expands those into many requests from one event.

The value grammars matter beyond networking: the numeric settings reach bash arithmetic, which runs a $(...) written inside the value, and channel names become parts of variable names. Redaction uses field paths passed as data to jq, never executable programs. For pinned installs no path is accepted at all, neither pricing_file nor a channel's log_file: every event is appended to its log file verbatim, so a path chosen by the file would be a write of agent-influenced text into a shell startup file, a git hook, or /dev/tcp/HOST/PORT, which bash turns into a network connection. Channels keep their default log files under the telemetry data directory, and pricing keeps its default source. Channel settings are checked by the variable they become: config.sh uppercases channel.<name>.<key> into TELEMETRY_CHANNEL_<NAME>_<KEY>, so channel.full.Endpoint_Url and channel.analytics_endpoint.url are endpoints too. The shipped telemetry.conf is entirely on the allowlist, and telemetry-doctor lists every line a pinned install ignores (pinIgnored). Standard keys are matched exactly, so a spelling such as Console_Url or CONSOLE_URL in the file is ignored by every install. Only ASCII whitespace is trimmed from keys and values, in config.sh (which runs tr and sed under LC_ALL=C), the liveness relay pre-gate and the doctor alike.

Every config file read by the shell runtime, including unpinned installed, workspace, global and explicit files, also receives value validation before assignment. Invalid settings produce a warning naming the key and file, without printing credentials, and retain the prior environment value or default. Valid operator settings keep the resolution and override order above. Rotation settings accept one to nine ASCII digits, parsed as decimal even with leading zeros; expressions, signs, non-digits and overflow-sized values are rejected before bash arithmetic. Redaction accepts none, an empty list, or comma-separated dotted field paths with letters, digits and underscores (spaces are ignored). A missing or differently shaped field leaves the event intact.

Unpinned channel.<name>.log_file settings must be absolute files beneath TELEMETRY_DATA_DIR. Traversal, symlink components below that directory, directories, devices and FIFOs are rejected. Effective default log and rotation paths are checked again before I/O; an unsafe destination emits a warning and returns failure without writing through it. Detached hooks check active files and existing rotation siblings in the foreground so destination warnings reach the caller without waiting for background processing or network requests. A failed channel does not prevent delivery to other safe channels or the Console sink. Existing custom directories must already exist; use ordinary files in the telemetry data directory. Rejected destinations keep the default local log, so ordinary events are retained. The local-only pin remains stricter and accepts no log path from its file. Explicit environment overrides remain operator-controlled.

Re-running init replaces an installed telemetry.conf that nobody edited with the bundle's copy and applies only that run's sink, so re-running it without --telemetry-sink local-files, or with a Console sink, removes the pin. A telemetry.conf edited after install is preserved instead: init leaves it, and its pin, unchanged and applies no sink to it (pass --force to replace it). Deleting the telemetry_sink_pin line removes the pin. init --global for claude-code and opencode cannot record a pin, because those hooks run from the package's own scripts/telemetry; init warns when the flag is given there. Run directly, install-console-config.sh does not pin the Flow Agents package's own telemetry.conf (a source checkout or the installed npm package). The root install.sh copies that package into its destination and tells the script so, so bash install.sh /path/to/workspace --telemetry-sink local-files from a clone does pin the destination.

Environment variables are not affected by the pin: they are an explicit runtime override. That covers the Console variables (CONSOLE_TELEMETRY_URL, CONSOLE_URL, CONSOLE_TELEMETRY_ENDPOINT_URL), a channel endpoint (TELEMETRY_CHANNEL_<NAME>_ENDPOINT_URL, unless the installed telemetry.conf sets that channel's endpoint itself, as it does for full), a channel log file (TELEMETRY_CHANNEL_<NAME>_LOG_FILE), a pricing URL (TELEMETRY_PRICING_URL, FLOW_AGENTS_PRICING_URL), and the economics and liveness relays (FLOW_AGENTS_CONSOLE_ECONOMICS_RELAY, FLOW_AGENTS_CONSOLE_LIVENESS_RELAY, their ..._ENDPOINT_URL, and FLOW_AGENTS_CONSOLE_URL). When one of them, or TELEMETRY_CONFIG_FILE, still sends a pinned install's data off the machine, telemetry-doctor warns once per destination and names its source, and init's summary says the install is not local-only.

The explicit TELEMETRY_CONFIG_FILE env var still overrides both auto-discovered paths and the local-only pin, and always wins. Wiring it from a login-shell profile (for example ~/.profile on bash -l setups) remains available for exotic setups that need a config path outside both conventional locations, but is no longer required for the common case.

Check an installed telemetry setup without opening an interactive prompt:

flow-agents telemetry-doctor --dest /path/to/installed/flow-agents --json --headless

The doctor reads scripts/telemetry/telemetry.conf, reports active local and Console sinks, the resolved local telemetry files, Console URL/endpoint configuration, and a bounded Console reachability check. JSON output is stable for CI and setup scripts; a missing or unreachable configured Console makes the command exit nonzero while still emitting the report. By default, reachability checks are limited to local endpoints; pass --allow-network to probe a non-local HTTPS Console endpoint explicitly.

Normalized Records#

Session#

A session record represents one agent run after import or aggregation. It should include:

• schema_version
• source_id
• repo
• repo_root
• runtime
• runtime_session_id
• session_id
• agent
• model
• profile_id
• prompt_id
• prompt_variant
• skill_ids
• skill_variant
• started_at
• ended_at
• duration_s
• turns
• tool_invocations
• delegations
• permission_requests
• imported_at

Event#

An event record is a runtime-neutral envelope around a runtime event. It keeps report code from depending on Codex-specific names. Runtime adapters can map native event names like session.start, turn.user, tool.invoke, tool.permission_request, agent.delegate, session.usage, and session.end into a shared event shape.

Outcome#

An outcome record captures quality and effectiveness facts:

• schema_version
• outcome_id
• recorded_at
• session_id
• runtime_session_id
• runtime
• repo
• agent
• profile_id
• prompt_id
• prompt_variant
• skill_ids
• skill_variant
• task_type
• task_slug
• result: success, partial, failure, or not_verified
• quality_score: 1 through 5, or null
• human_minutes_saved
• rework_required
• notes
• evidence

Do not store sensitive prompt text, tool payloads, secrets, or customer data in outcome records. Use evidence paths or stable identifiers instead.

Metrics#

Recommended reports should include:

• Joined-outcome success rate, with the joined outcome count named as its denominator.
• Joined partial, failure, and not-verified outcome counts.
• Session duration, including total and average duration.
• Tool invocations per session.
• Delegations per session.
• Permission requests per session.
• Joined-outcome rework rate.
• Average quality score from recorded outcomes.
• Human minutes saved from recorded outcomes.
• Sessions with joined outcomes versus sessions without joined outcomes.
• Total, joined, and unjoined outcome records, including content-free unjoined reason counts.

Usage-only metrics can be computed from telemetry. Quality and value metrics require outcome records. An outcome contributes to quality metrics only when it resolves to exactly one session through a valid run-correlation ID, an explicit runtime-session ID, or the documented legacy {runtime, session_id} pair, in that precedence order. Missing, invalid, unmatched, and ambiguous identities remain visible as unjoined records and never contribute to a success or rework claim. Reports are partial when source records are malformed, outcomes are unjoined, or recorded sessions lack joined outcome coverage.

Core Commands#

Project Dashboard#

Generate the local HTML dashboard. This syncs terminal task artifacts from .kontourai/flow-agents into .telemetry/outcomes.jsonl before rendering:

npm run usage-feedback -- dashboard --force

The dashboard is written to .telemetry/reports/dashboard.html by default.

Sync artifacts without rendering the dashboard:

npm run usage-feedback -- sync-artifacts \
  --artifact-dir .kontourai/flow-agents \
  --repo flow-agents \
  --profile-id codex-default \
  --prompt-id deliver-v1 \
  --skill-id deliver

Import Codex telemetry from an explicit source and tag it with comparable identifiers:

npm run usage-feedback -- import-codex \
  --input-telemetry-dir ../repo-a/.telemetry \
  --telemetry-dir .telemetry/repo-a \
  --source-id repo-a \
  --repo repo-a \
  --profile-id codex-builder \
  --prompt-id deliver-v1 \
  --skill-id deliver

Import telemetry from any runtime that emits the shared event envelope:

npm run usage-feedback -- import-telemetry \
  --runtime claude-code \
  --input-telemetry-dir ../repo-b/.telemetry \
  --telemetry-dir .telemetry/repo-b \
  --source-id repo-b \
  --repo repo-b \
  --profile-id claude-dev \
  --prompt-id deliver-v1 \
  --skill-id deliver

Record a human or eval outcome for one session:

npm run usage-feedback -- record-outcome \
  --telemetry-dir .telemetry/repo-a \
  --session-id session-123 \
  --runtime codex \
  --repo repo-a \
  --agent dev \
  --profile-id codex-builder \
  --prompt-id deliver-v1 \
  --skill-id deliver \
  --task-type delivery \
  --task-slug usage-feedback-docs \
  --result success \
  --quality-score 5 \
  --human-minutes-saved 35 \
  --evidence .kontourai/flow-agents/agent-usage-feedback-loop/agent-usage-feedback-loop--deliver.md

Generate a report:

npm run usage-feedback -- report \
  --telemetry-dir .telemetry/repo-a \
  --group-by profile_id \
  --output reports/usage-feedback.md

Generate a standalone HTML report without artifact syncing:

npm run usage-feedback -- report \
  --telemetry-dir .telemetry/repo-a \
  --group-by profile_id \
  --format html \
  --output reports/usage-feedback.html

Global Dashboard#

Use a global telemetry root when you want one view across projects. The default global root is ~/.flow-agents/telemetry.

Register the current project once with stable comparison labels:

npm run usage-feedback -- register-project \
  --global-dir ~/.flow-agents/telemetry \
  --repo-root "$PWD" \
  --name flow-agents \
  --profile-id codex-default \
  --prompt-id deliver-v1 \
  --skill-id deliver

Sync registered projects into the global root:

npm run usage-feedback -- sync-projects \
  --global-dir ~/.flow-agents/telemetry

Render the global dashboard:

npm run usage-feedback -- global-dashboard \
  --global-dir ~/.flow-agents/telemetry \
  --group-by repo \
  --force

The global dashboard writes ~/.flow-agents/telemetry/reports/global-dashboard.html.

You can also target projects directly without registering them first:

npm run usage-feedback -- global-dashboard \
  --global-dir ~/.flow-agents/telemetry \
  --repo-root /path/to/project-a \
  --repo-root /path/to/project-b \
  --group-by repo \
  --force

Or discover direct child projects under a parent directory:

npm run usage-feedback -- global-dashboard \
  --global-dir ~/.flow-agents/telemetry \
  --discover ~/dev/github \
  --group-by repo \
  --force

Global project stores live under ~/.flow-agents/telemetry/projects/<project>/. Each project store contains normalized sessions and outcomes, so reports can compare projects, profiles, prompts, and skills from one dashboard.

Comparison Examples#

Compare two repositories:

npm run usage-feedback -- report \
  --telemetry-dir ../repo-a/.telemetry \
  --telemetry-dir ../repo-b/.telemetry \
  --group-by repo

Compare two Codex profiles over the same task family:

npm run usage-feedback -- report \
  --telemetry-dir .telemetry/codex-default \
  --telemetry-dir .telemetry/codex-bedrock \
  --runtime codex \
  --group-by profile_id

Compare prompt variants:

npm run usage-feedback -- report \
  --telemetry-dir .telemetry/prompt-a \
  --telemetry-dir .telemetry/prompt-b \
  --group-by prompt_variant

Compare skill setups:

npm run usage-feedback -- report \
  --telemetry-dir .telemetry/with-deliver \
  --telemetry-dir .telemetry/with-tdd \
  --group-by skill_id

Compare Codex with Kiro or another future runtime after import:

npm run usage-feedback -- report \
  --telemetry-dir .telemetry/codex \
  --telemetry-dir .telemetry/kiro \
  --group-by runtime

The non-Codex runtime must provide or import normalized records with the same session and outcome fields. Codex-specific paths and event names should stay inside the Codex adapter.