Kontour Flow Agents

Agent Usage Feedback Loop

This document defines the usage feedback loop for comparing agent runs across repositories, runtimes, profiles, prompts, skills, and future setups. The measurement schema is runtime-neutral: every runtime should import or emit normalized session, event, and outcome records before reporting. Codex and Claude Code can both flow through the shared telemetry envelope; Kiro uses the same canonical telemetry script when hooks are installed.

The loop has three parts:

  1. Import or read runtime telemetry into normalized usage records.
  2. Sync quality outcomes from task artifacts, evals, release gates, or humans.
  3. Report usage and outcome metrics by repo, runtime, profile, prompt, skill, or task type.

Quality outcomes are not inferred from raw telemetry alone. The CLI can automatically derive coarse outcomes from task artifacts such as .kontourai/flow-agents/<slug>/*.md: delivered and complete become success, failed becomes failure, and optionally open artifacts can be recorded as not_verified. It does not invent quality_score or human_minutes_saved; those remain human/eval/release-gate facts.

Retrospective run observations

usage-feedback compile-observation compiles one exact correlated run from a manifest of producer-owned records. The manifest names logical source IDs and the exact runtime event, Builder state, trust bundle, Flow state, Flow evidence manifest, economics, and terminal-outcome files. Compilation never joins by path, timestamp, task slug, cwd, process ancestry, or content similarity.

The output is a versioned kontour.flow-agents.retrospective-observation record under .kontourai/flow-agents/feedback/observations/<observation-hash>.json by default. One atomically replaced, locked file per run avoids cross-run lost updates. It contains stable identity, process and verification status, measured usage, explicit missing dimensions, and content hashes plus logical IDs for local drill-down. It contains no source paths, prompts, tool payloads, source content, or secrets. Repeat compilation of unchanged sources is byte-identical and upserts the same observation ID.

Process completion and Flow verification are not independent task quality. Until an external evaluation is joined through the eval-owned boundary, every compiled observation reports quality as NOT_VERIFIED with reason not_independently_evaluated. The compiler emits no causal claim about a Kit, model, prompt, or workflow.

npm run usage-feedback -- compile-observation \
  --manifest .kontourai/flow-agents/run-observation-manifest.json \
  --record-root .

To inventory real local usage without crawling arbitrary home directories, pass each operator-configured record root explicitly:

npm run usage-feedback -- compile-corpus \
  --record-root /path/to/runtime-or-repository-root \
  --record-root /path/to/another-configured-root

compile-corpus inspects only known producer locations under those roots: .kontourai/telemetry, .kontourai/flow-agents, and .kontourai/flow/runs. It never joins records by path similarity, timestamps, working directory, or process ancestry. Records without an exact producer run_correlation remain counted as uncorrelated.

Each configured source is parsed once per invocation. Explicitly correlated records and exact Flow/trust secondary records are retained in one bounded snapshot and reused across that invocation’s per-run projections; report facts are not restored from an unsigned cache. JSONL producers may append while the inventory runs: the compiler double-hashes the opening-length prefix, records that accepted prefix in the watermark, and verifies it again before publication. Appends after the accepted prefix are deferred to the next generation. Prefix mutation, truncation, replacement, invalid UTF-8, and resource-limit exhaustion fail closed.

The command writes:

  • feedback/corpus/current.json: the atomically replaced pointer to the current complete generation;
  • feedback/corpus/generations/<watermark>/watermark.json: content hashes and stable root-local file identities for the observed producer snapshot;
  • feedback/corpus/generations/<watermark>/report.json: coverage, correlation, compilation, completeness, ambiguity, and quarantine counts;
  • feedback/corpus/generations/<watermark>/manifests/*.json: exact root-relative source declarations for compiled runs;
  • feedback/corpus/generations/<watermark>/observations/*.json: privacy-safe per-run observations.

Reports always keep causal and quality effects NOT_VERIFIED. Observational coverage can generate hypotheses for Evals, but it cannot establish that a Kit caused an outcome.

The manifest uses the retrospective-observation-manifest/1.0 schema. Every individual source is optional so an incomplete historical run can still compile honestly, but at least one producer record must carry the exact manifest correlation ID. Source files are relative to the caller-selected --record-root; traversal, absolute paths, empty or dot path segments, and symbolic links are rejected. The record root and output directory are operator trust boundaries, not data supplied by a producer. Their directory chains must be non-group/world-writable, except for sticky shared ancestors such as the system temporary directory, and each final boundary directory must be owned by the current operator. Source files must be owned by the current operator or root and must not be group/world-writable. Concurrent replacement or mutation by another process running as the same OS principal is outside this local compiler’s filesystem trust boundary; the accepted-prefix checks detect such changes during an invocation:

{
  "schema_version": "1.0",
  "correlation_id": "run-correlation-id",
  "sources": {
    "runtime_events": {
      "source_id": "runtime-events",
      "file": ".kontourai/telemetry/full.jsonl"
    },
    "terminal_outcome": {
      "source_id": "terminal-outcome",
      "file": ".kontourai/flow-agents/task/workflow-outcome.json"
    }
  }
}

Runtime activity, Flow history, trust, economics, and terminal status are projected only when their complete correlation envelopes equal the canonical run envelope. Sharing a correlation_id alone is insufficient. A trust dimension additionally requires a Surface-valid bundle and one exact Flow-manifest attachment whose embedded bundle and SHA-256 match the supplied bundle.

Usage describes a present run delta only when terminal usage and economics agree on timestamp and token totals. CONFIRMED additionally requires an authenticated runtime binding and exactly matching model, duration, delegation, and cost attributes. When token totals agree but another attribute is unavailable or disagrees, the compiler retains the token counts under NOT_VERIFIED and emits null for each unmatched attribute rather than fabricating or discarding useful consumption data. Fixture authority is always NOT_VERIFIED.

Malformed input and partial reports

Read-only reports, telemetry-source imports, and retrospective compilation quarantine malformed JSONL records and continue over valid records. JSON reports expose a measurement object with total, valid, and malformed counts plus content-free diagnostics; Markdown reports show the same counts under Measurement State. The CLI also emits one stderr warning per affected logical source. Diagnostics include only a logical source name, line number, SHA-256 content hash, and parse error class. They never include the malformed record or an absolute path.

Retrospective compiler diagnostics also classify schema-valid JSON that fails a producer contract as ProducerValidationError; counts remain separate from syntax failures and source bytes never enter the observation.

Corpus compilation keeps malformed and producer-invalid counts on every affected source reference and in the corpus report, but does not copy the same source-level diagnostic into every observation that shares a multi-run source. Run-specific projection failures still appear on that run’s observation.

This tolerance applies only to source analysis. A destination that an import, sync, or upsert operation would rewrite remains strict: one malformed existing record aborts before any write. The command does not silently discard corrupt state.

Storage Defaults

Repo-local telemetry defaults to .telemetry/:

  • .telemetry/full.jsonl: raw runtime telemetry events.
  • .telemetry/analytics.jsonl: summarized analytics emitted by telemetry hooks.
  • .telemetry/sessions/: per-session runtime artifacts, when the runtime writes them.
  • .telemetry/outcomes.jsonl: human/eval outcome records for completed work.
  • .telemetry/reports/: generated Markdown or JSON usage reports.

Locations are configurable. CLI commands should accept --telemetry-dir, and shell workflows may set TELEMETRY_DATA_DIR. User-level or global Codex telemetry roots, such as ~/.codex/telemetry or ~/.flow-agents/telemetry, should be treated as configurable installation choices, not hard-coded truth. When comparing multiple repositories or machines, pass each telemetry root explicitly with repeatable --telemetry-dir flags.

Runtime hooks can also mirror redacted telemetry to Console without changing the event shape. Set CONSOLE_TELEMETRY_URL=http://127.0.0.1:3737 for a local Console, or CONSOLE_URL=https://console.kontourai.io for a deployed Console. The transport posts to /api/telemetry/records by default. Hosted Console URLs must use https://; http:// is accepted only for localhost or 127.0.0.1 local development. Use CONSOLE_TELEMETRY_ENDPOINT_URL only when the API path is nonstandard, CONSOLE_TELEMETRY_TOKEN or CONSOLE_AUTH_TOKEN for bearer auth, and CONSOLE_TENANT_ID for hosted tenant routing. If no Console URL is set, telemetry remains local-only.

Flow Agents owns the Console telemetry descriptor at console.telemetry.json. Generated bundles include that root descriptor beside AGENTS.md, scripts, and kit assets. The descriptor maps Flow Agents runtime telemetry and workflow sidecars into generic Console facets for skills, tools, flows, repositories, projects, runtimes, agents, models, statuses, and outcomes. Console consumes those mappings as product-owned display metadata; Flow-owned gate and transition semantics stay in Flow contracts and sidecars. Repository and project metadata are stable cross-user workspace identifiers when present. Local working-directory paths stay out of the descriptor so Console display metadata does not expose usernames or machine-local paths; records without a repository identifier should fall back to their logical product root such as product:flow-agents:.kontourai/flow-agents (the runtime artifact root; #778 corrected the descriptor’s own record-source roots to match after they drifted from the .flow-agents -> .kontourai/flow-agents runtime-root migration).

Packaged setup modes are:

  • local-files: default local JSONL telemetry only; no Console URL or token.
  • local-kontour-console: mirror to a separately running local Console, using http://127.0.0.1:3737 unless FLOW_AGENTS_LOCAL_KONTOUR_CONSOLE_URL is set.
  • kontour-hosted-console: mirror to Kontour’s hosted Console default URL. Pass --console-token-file and --console-tenant for headless hosted setup.
  • user-hosted-console: mirror to a self-hosted Console; requires --console-url or --console-endpoint.

Use flow-agents init --yes or --headless with the same flags in CI. The legacy sink names kontour-cloud and hosted-kontour-console are still accepted for existing scripts.

Owner machine mirror

For a personal machine or repo where an owner wants Claude Code (or other runtime) hook sessions mirrored to a hosted Console, without editing the tracked scripts/telemetry/telemetry.conf default template (that file ships verbatim into every packaged bundle, so writing a personal token/tenant into it would leak to downstream consumers) and without exporting env vars per session, scripts/telemetry/lib/config.sh auto-discovers a gitignored, operator-created conf at either of two conventional paths, no extra wiring required:

  1. <workspace>/.kontourai/telemetry-console.conf — repo-scoped, checked first.
  2. ~/.flow-agents/telemetry-console.conf — machine-scoped, used when no workspace-scoped conf is present.

Populate either path with the existing preset installer: `scripts/telemetry/install-console-config.sh --telemetry-sink kontour-hosted-console --console-token-file --console-tenant

`. The installer `chmod 600`s the file, sets it to be owned by the current user, and never echoes the token. Both conditions matter: config.sh only honors a discovered conf if it is mode 600 and owned by the current user, since that combination can only come from an operator running the installer (or an equivalent manual `chmod 600`) — it distinguishes an operator-created conf from one that arrived via `git clone`, a tarball, a PR, or any other supply-chain path, none of which can produce a 600-mode file. A conf that fails that check is treated as if it were absent (fail open) and resolution falls through to the next tier. The explicit `TELEMETRY_CONFIG_FILE` env var still overrides both auto-discovered paths and always wins. Wiring it from a login-shell profile (for example `~/.profile` on `bash -l` setups) remains available for exotic setups that need a config path outside both conventional locations, but is no longer required for the common case. Check an installed telemetry setup without opening an interactive prompt: ```bash flow-agents telemetry-doctor --dest /path/to/installed/flow-agents --json --headless ``` The doctor reads `scripts/telemetry/telemetry.conf`, reports active local and Console sinks, the resolved local telemetry files, Console URL/endpoint configuration, and a bounded Console reachability check. JSON output is stable for CI and setup scripts; a missing or unreachable configured Console makes the command exit nonzero while still emitting the report. By default, reachability checks are limited to local endpoints; pass `--allow-network` to probe a non-local HTTPS Console endpoint explicitly. ## Normalized Records ### Session A session record represents one agent run after import or aggregation. It should include: - `schema_version` - `source_id` - `repo` - `repo_root` - `runtime` - `runtime_session_id` - `session_id` - `agent` - `model` - `profile_id` - `prompt_id` - `prompt_variant` - `skill_ids` - `skill_variant` - `started_at` - `ended_at` - `duration_s` - `turns` - `tool_invocations` - `delegations` - `permission_requests` - `imported_at` ### Event An event record is a runtime-neutral envelope around a runtime event. It keeps report code from depending on Codex-specific names. Runtime adapters can map native event names like `session.start`, `turn.user`, `tool.invoke`, `tool.permission_request`, `agent.delegate`, `session.usage`, and `session.end` into a shared event shape. ### Outcome An outcome record captures quality and effectiveness facts: - `schema_version` - `outcome_id` - `recorded_at` - `session_id` - `runtime_session_id` - `runtime` - `repo` - `agent` - `profile_id` - `prompt_id` - `prompt_variant` - `skill_ids` - `skill_variant` - `task_type` - `task_slug` - `result`: `success`, `partial`, `failure`, or `not_verified` - `quality_score`: `1` through `5`, or null - `human_minutes_saved` - `rework_required` - `notes` - `evidence` Do not store sensitive prompt text, tool payloads, secrets, or customer data in outcome records. Use evidence paths or stable identifiers instead. ## Metrics Recommended reports should include: - Joined-outcome success rate, with the joined outcome count named as its denominator. - Joined partial, failure, and not-verified outcome counts. - Session duration, including total and average duration. - Tool invocations per session. - Delegations per session. - Permission requests per session. - Joined-outcome rework rate. - Average quality score from recorded outcomes. - Human minutes saved from recorded outcomes. - Sessions with joined outcomes versus sessions without joined outcomes. - Total, joined, and unjoined outcome records, including content-free unjoined reason counts. Usage-only metrics can be computed from telemetry. Quality and value metrics require outcome records. An outcome contributes to quality metrics only when it resolves to exactly one session through a valid run-correlation ID, an explicit runtime-session ID, or the documented legacy `{runtime, session_id}` pair, in that precedence order. Missing, invalid, unmatched, and ambiguous identities remain visible as unjoined records and never contribute to a success or rework claim. Reports are partial when source records are malformed, outcomes are unjoined, or recorded sessions lack joined outcome coverage. ## Core Commands ### Project Dashboard Generate the local HTML dashboard. This syncs terminal task artifacts from `.kontourai/flow-agents` into `.telemetry/outcomes.jsonl` before rendering: ```bash npm run usage-feedback -- dashboard --force ``` The dashboard is written to `.telemetry/reports/dashboard.html` by default. Sync artifacts without rendering the dashboard: ```bash npm run usage-feedback -- sync-artifacts \ --artifact-dir .kontourai/flow-agents \ --repo flow-agents \ --profile-id codex-default \ --prompt-id deliver-v1 \ --skill-id deliver ``` Import Codex telemetry from an explicit source and tag it with comparable identifiers: ```bash npm run usage-feedback -- import-codex \ --input-telemetry-dir ../repo-a/.telemetry \ --telemetry-dir .telemetry/repo-a \ --source-id repo-a \ --repo repo-a \ --profile-id codex-builder \ --prompt-id deliver-v1 \ --skill-id deliver ``` Import telemetry from any runtime that emits the shared event envelope: ```bash npm run usage-feedback -- import-telemetry \ --runtime claude-code \ --input-telemetry-dir ../repo-b/.telemetry \ --telemetry-dir .telemetry/repo-b \ --source-id repo-b \ --repo repo-b \ --profile-id claude-dev \ --prompt-id deliver-v1 \ --skill-id deliver ``` Record a human or eval outcome for one session: ```bash npm run usage-feedback -- record-outcome \ --telemetry-dir .telemetry/repo-a \ --session-id session-123 \ --runtime codex \ --repo repo-a \ --agent dev \ --profile-id codex-builder \ --prompt-id deliver-v1 \ --skill-id deliver \ --task-type delivery \ --task-slug usage-feedback-docs \ --result success \ --quality-score 5 \ --human-minutes-saved 35 \ --evidence .kontourai/flow-agents/agent-usage-feedback-loop/agent-usage-feedback-loop--deliver.md ``` Generate a report: ```bash npm run usage-feedback -- report \ --telemetry-dir .telemetry/repo-a \ --group-by profile_id \ --output reports/usage-feedback.md ``` Generate a standalone HTML report without artifact syncing: ```bash npm run usage-feedback -- report \ --telemetry-dir .telemetry/repo-a \ --group-by profile_id \ --format html \ --output reports/usage-feedback.html ``` ### Global Dashboard Use a global telemetry root when you want one view across projects. The default global root is `~/.flow-agents/telemetry`. Register the current project once with stable comparison labels: ```bash npm run usage-feedback -- register-project \ --global-dir ~/.flow-agents/telemetry \ --repo-root "$PWD" \ --name flow-agents \ --profile-id codex-default \ --prompt-id deliver-v1 \ --skill-id deliver ``` Sync registered projects into the global root: ```bash npm run usage-feedback -- sync-projects \ --global-dir ~/.flow-agents/telemetry ``` Render the global dashboard: ```bash npm run usage-feedback -- global-dashboard \ --global-dir ~/.flow-agents/telemetry \ --group-by repo \ --force ``` The global dashboard writes `~/.flow-agents/telemetry/reports/global-dashboard.html`. You can also target projects directly without registering them first: ```bash npm run usage-feedback -- global-dashboard \ --global-dir ~/.flow-agents/telemetry \ --repo-root /path/to/project-a \ --repo-root /path/to/project-b \ --group-by repo \ --force ``` Or discover direct child projects under a parent directory: ```bash npm run usage-feedback -- global-dashboard \ --global-dir ~/.flow-agents/telemetry \ --discover ~/dev/github \ --group-by repo \ --force ``` Global project stores live under `~/.flow-agents/telemetry/projects//`. Each project store contains normalized sessions and outcomes, so reports can compare projects, profiles, prompts, and skills from one dashboard. ## Comparison Examples Compare two repositories: ```bash npm run usage-feedback -- report \ --telemetry-dir ../repo-a/.telemetry \ --telemetry-dir ../repo-b/.telemetry \ --group-by repo ``` Compare two Codex profiles over the same task family: ```bash npm run usage-feedback -- report \ --telemetry-dir .telemetry/codex-default \ --telemetry-dir .telemetry/codex-bedrock \ --runtime codex \ --group-by profile_id ``` Compare prompt variants: ```bash npm run usage-feedback -- report \ --telemetry-dir .telemetry/prompt-a \ --telemetry-dir .telemetry/prompt-b \ --group-by prompt_variant ``` Compare skill setups: ```bash npm run usage-feedback -- report \ --telemetry-dir .telemetry/with-deliver \ --telemetry-dir .telemetry/with-tdd \ --group-by skill_id ``` Compare Codex with Kiro or another future runtime after import: ```bash npm run usage-feedback -- report \ --telemetry-dir .telemetry/codex \ --telemetry-dir .telemetry/kiro \ --group-by runtime ``` The non-Codex runtime must provide or import normalized records with the same session and outcome fields. Codex-specific paths and event names should stay inside the Codex adapter.