Fixture Ownership
evals/fixtures/ is canonical eval source, not generic sample data. Keep fixture
groups only when an owning eval or validator names the behavior they protect.
When adding, moving, or deleting a fixture directory, update this inventory and
run npm run validate:source -- and npm run fixture:retirement-audit --.
Ownership Inventory
| Fixture directory | Classification | Owners | Cleanup rule |
|---|---|---|---|
evals/fixtures/assignment-provider |
AssignmentProvider local-file and GitHub render/status fixtures (#290); hostile-effective-state.json is the #291 ensure-session ownership guard’s AC9 sanitization fixture | evals/integration/test_assignment_provider_local_file.sh, evals/integration/test_assignment_provider_github.sh, evals/integration/test_pull_work_assignment_join.sh, evals/integration/test_ensure_session_ownership_guard.sh |
Keep while the assignment provider contract’s claim/release/supersede/status/list operations and the assignment ⋈ liveness join are tested against local-file and GitHub fixture inputs. |
evals/fixtures/backlog-provider-settings |
settings precedence fixtures | evals/integration/test_effective_backlog_settings.sh |
Keep while backlog provider settings resolution supports global defaults and project overrides. |
evals/fixtures/builder-kit-workflow-state |
Builder Kit workflow-state fixtures | evals/static/test_workflow_skills.sh |
Keep while Builder Kit state contract and resume behavior are documented in workflow skill contracts. |
evals/fixtures/console-declared-projection |
delivery/DECLARED accepted-gap projection fixtures (#1267): mixed structured gaps[] / legacy / invalid entries proving the no-prose-scraping fold and its disclosed legacy count | evals/integration/test_console_declared_projection.sh |
Keep while console-declared-projection projects only structured gaps[] entries and discloses legacy/invalid entry counts in its summary. |
evals/fixtures/console-learning-projection |
console learning projection fixtures | evals/integration/test_console_learning_projection.sh |
Keep while learning projection supports correction and open-route examples. |
evals/fixtures/console-process-projection |
console process-state projection fixtures (issue #778): blocked (handoff blockers), needs_decision (next_action.summary), a required-pending critique (review_pending), and a plain in_progress/running session | evals/integration/test_console_process_projection.sh |
Keep while the workflow-state -> Console interactive-session process state mapping (needs_input/review_pending/blocked + blockedReason) is enforced. |
evals/fixtures/change-provider-settings |
ChangeProvider effective-settings fixtures for configured, malformed, and hostile secret-bearing inputs | src/cli/effective-change-provider-settings.test.mjs, evals/integration/test_publish_change_operation.sh |
Keep while effective ChangeProvider settings distinguish configured, absent, malformed, and secret-bearing inputs without fallback or secret disclosure. |
evals/fixtures/codex-legacy-agents |
exact historical Flow Agents-generated Codex AGENTS.md migration fixtures (#563) | evals/integration/test_install_merge.sh |
Keep while the Codex home installer recognizes audited historical payload hashes, refuses before mutation, and provides explicit operator remediation without fuzzy ownership matching. |
evals/fixtures/economics |
per-run kit-economics record fixtures (#349): a transcript with .message.usage blocks, state.json/acceptance.json/critique.json join sources, a session.usage event, and the golden expected kontour.console.economics record. Its run-binding/ subdirectory (#922/#925 phase A, extended by the r1 review fix round) carries seven hand-built, schemas/flow-run.schema.json-validated Flow run stores – routeback-completed (also carries a real-shaped gate_outcome_history mirroring production run kontourai-flow-agents-1206, exercising the PRIMARY gate-fire counting path), active-mid-flight, canceled-with-pause (lifecycle pause/resume), paused-mid-execute (mirrors the real, live kontourai-flow-agents-944 production run: ~99.995% pause inside one phase), zero-duration-boundary (two transitions sharing one timestamp), unknown-status (a status value outside Flow’s own enum), and multi-cursor-active (a non-empty multi_cursor ledger) – each with a hand-computed expected.json, plus a synthetic transcript (incl. one isSidechain:true line) + phase-window JSON for the token-slicer |
evals/integration/test_economics_record.sh, evals/integration/test_economics_run_binding.sh |
Keep while the current kontour.console.economics v0.2 contract and historical v0.1 compatibility (docs/specs/economics-record-contract.md) are enforced; the golden proves cost-from-transcript, defects-from-critique, canonical Flow verdict authority, and the R7 co-required guard. run-binding/ fixtures stay while scripts/telemetry/flow-run-economics.mjs and scripts/telemetry/economics-enrich-tokens.mjs (docs/specs/economics-record-contract.md “flow_run_record mode”) derive phases/iterations/defects/terminal_status/human_wait_s from the canonical Flow run store and token attribution from a transcript, both against exact hand-computed values, incl. the pause-exclusion, null-cost-propagation, unrecognized-status-refusal, and multi_cursor-refusal fixes proven by RED/GREEN evidence in the r1 review fix round. |
evals/fixtures/flow-kit-repository |
Flow Kit repository contract fixtures | evals/integration/test_flow_kit_repository.sh, evals/integration/test_local_flow_kit_install.sh, evals/integration/test_runtime_adapter_activation.sh, evals/integration/test_activate_npx_context.sh, evals/integration/test_flow_kit_install_git.sh, evals/static/test_validate_source_kit_asset_scope.sh, evals/static/test_workflow_skills.sh |
Keep valid and invalid cases paired with the Flow Kit repository contract. |
evals/fixtures/kit-conformance-levels |
K-level conformance and consumer-target derivation fixtures | evals/integration/test_kit_conformance_levels.sh |
Keep while K-level derivation, degradation invariant, and consumer-target badge rules are tested. |
evals/fixtures/kit-observability |
host-neutral Kit observability descriptor and third-party compatibility fixtures (#911) | evals/static/test_kit_observability_contract.sh |
Keep while Builder, Knowledge, absent/future-version, and third-party contribution behavior is validated without a host installation. |
evals/fixtures/hook-influence |
hook influence behavioral cases | evals/integration/test_hook_influence_cases.sh, evals/static/test_workflow_skills.sh, scripts/validate-hook-influence-cases.js |
Keep while hook influence cases define agent guidance behavior. |
evals/fixtures/learning-review-proposals |
learning-review kit/gate tuning proposal fixtures (#352): pattern-present (engineered cost-inflation + gate false-block-rate pattern, with sessions/ trust.bundle + gate-review.inquiries.json joins and a hand-computed expected-aggregates.json), balanced (proportional cost/findings movement -> zero proposals), under-threshold (below LR_MIN_WINDOW_SAMPLE), repeat-window (idempotency), and effect-follow-up (later-window effect-fill pass for a ratified proposal) | evals/integration/test_learning_review_proposals.sh |
Keep while scripts/telemetry/learning-review-proposals.sh/learning-review-decide.sh (docs/specs/learning-review-proposals-contract.md) are enforced: hand-computed aggregates, evidence-cited proposals, zero-mutation, idempotency, and the ratify -> effect-fill trail. |
evals/fixtures/narrative-sources |
narrative source snapshot, offline resolution, integrity, completeness, fail-closed redaction, grounded runtime projection, grounded execution narrative, and trust-isolation fixtures (#613, #617, #618, #619) | evals/integration/test_narrative_source_contract.sh, evals/integration/test_narrative_runtime_projection.sh, evals/integration/test_narrative_grounded_envelope.sh, evals/integration/test_narrative_redaction_failclosed.sh, evals/integration/test_narrative_trust_isolation.sh |
Keep while the narrative source manifest, content-addressed offline resolver, integrity verification, grounded runtime projection, grounded execution narrative composition, capture-completeness disclosure, lifecycle locality, fail-closed redaction, and narrative trust-isolation contracts are supported. |
evals/fixtures/narrative-grounding-validator |
narrative grounding validator adversarial fixtures (#623) | evals/integration/test_narrative_grounding_validator.sh |
Keep while deterministic citation integrity, material-event coverage, epistemic-label integrity, and fail-closed narrative publication are enforced. |
evals/fixtures/narrative-prose-renderer |
model-assisted prose renderer adversarial fixtures (#614): provenance-subset (unsupported-summary), no-op-loop lack-of-progress wording, and prompt-injection fixtures reused from evals/fixtures/narrative-grounding-validator | evals/integration/test_narrative_prose_renderer.sh |
Keep while the fail-closed prose renderer (D3 provenance-subset citation check, provider gating, economics-always, zero-prose-artifact-on-failure) is enforced. |
evals/fixtures/narrative-evals |
grounded narrative faithfulness eval corpus + result-schema package (#612): ten R1 case-class fixtures with frozen-manifest answer keys plus five R3 corruption fixtures whose named eval-check anchors are the scorer-teeth mutation targets | evals/integration/test_narrative_evals.sh |
Keep while the versioned adversarial corpus, its frozen-manifest answer keys, the schema-valid narrative-eval-result output, the declared cross-runtime capability parity, and the R3 scorer-teeth mutation battery are enforced. |
evals/fixtures/pr-body-file-refs |
verbatim GitHub-stored pull request bodies for the #1375 PR-body file-reference contract, each paired with a hand-checked expected-paths answer key: #1366’s fabricated body (the incident — it names src/cli/trust-bundle-verifying-actor.test.mjs, which lived on sibling branch #1368 and appeared 0 times in #1366’s diff), #1366’s corrected body, and three unrelated merged bodies (#1041, #1101, #1354) that must not red |
evals/static/test_pr_body_file_refs.sh |
Keep while scripts/ci/validate-pr-body-file-refs.mjs gates PR bodies. These are historical records, not synthesized inputs: regenerating them would destroy the only proof the check reds on the real failure rather than on a happy-path mock. |
evals/fixtures/pull-work-provider |
work item provider normalization fixtures | evals/integration/test_pull_work_provider.sh |
Keep while provider normalization preserves blockers, artifact refs, board membership, and freshness metadata. |
evals/fixtures/pull-work-wip-shepherding |
WIP shepherding state fixtures | evals/static/test_workflow_skills.sh |
Keep while pull-work documents personal versus global WIP behavior. |
evals/fixtures/reconcile-preflight |
#356 reconcile-preflight shape fixtures not already covered by trust-reconcile-exploits (un-superseded disputed critique, standalone disputed session-local claim) | evals/integration/test_reconcile_preflight.sh |
Keep while the local reconcile-preflight subcommand (#356) is proven against the two shapes trust-reconcile-exploits does not already fixture (un-superseded disputed critique, standalone disputed session-local claim); the other four shapes reuse trust-reconcile-exploits/trust-reconcile-mixed-bundle directly rather than forking near-duplicates. |
evals/fixtures/reviewed-grounding-workflow |
deterministic reviewed-grounding reference telemetry fixture (#715) | evals/integration/test_reviewed_grounding_workflow.sh |
Keep while the pinned public-product composition proves offline extraction, semantic review, portable evidence, action refusal, typed failures, and optional live telemetry. |
evals/fixtures/surface-trust |
Surface trust evidence fixtures | evals/integration/test_workflow_sidecar_writer.sh |
Keep while sidecar writer maps Surface trust evidence into workflow records. |
evals/fixtures/telemetry |
hermetic Stop-hook usage transcript fixture (#defect-2026-07 telemetry-usage-cost-extraction): a multi-model JSONL transcript with .message.model + .message.usage token blocks proving real token/cost/model extraction end-to-end; also reused by the #568 slice 1 per-tool-call usage-enrichment eval (test_telemetry_tool_usage.sh), which adds usage-transcript-oversized-prefix.jsonl to prove the bounded tail-read | evals/integration/test_usage_cost.sh, evals/integration/test_telemetry_usage_pipeline.sh, evals/integration/test_telemetry_tool_usage.sh, evals/integration/test_telemetry_tool_outcome.sh, evals/integration/test_installed_runtime_correlation.sh |
Keep while telemetry.sh’s Stop-hook usage pipeline (usage_parse_transcript / usage_model_from_transcript_usage / usage_last_turn_usage), installed producer reconstruction, and idempotent privacy-safe retrospective observation compilation are proven against hermetic multi-model and oversized-prefix transcripts rather than relying on live runtime data. |
evals/fixtures/trust-reconcile-exploits |
WS8 trust-reconcile anti-gaming exploit fixtures (frozen negative regressions); also reused by the #356 local reconcile-preflight eval (same shapes, no forked copies) | evals/integration/test_trust_reconcile_negatives.sh, evals/integration/test_reconcile_preflight.sh |
Keep while trust-reconcile.js enforces the WS8 iteration-2 soundness properties (no-label test_output, unwaived-assumed, status-misassertion, waiver-on-command); each fixture is a permanent negative regression. |
evals/fixtures/trust-reconcile-mixed-bundle |
WS8 trust-reconcile mixed-evidence end-to-end proof fixture; also reused by the #356 reconcile-preflight eval as its CLEAN-BUNDLE (AC4) case | evals/integration/test_trust_reconcile_mixed_bundle.sh, evals/integration/test_reconcile_preflight.sh |
Keep while the trust-reconcile manifest/classification/waiver contract (ADR 0020) is enforced; proves a mixed test_output + session-local + waived bundle passes the CI anchor. |
evals/fixtures/trust-reconcile-ws3 |
WS8 AC6 backward-compat fixture: real ws3-kit-dependencies-namespacing old-style bundle | evals/integration/test_trust_reconcile_negatives.sh |
Keep while backward compatibility with pre-classification (all-test_output) bundles is asserted; proves an old-style bundle still FAILS the same way (no silent pass). |
evals/fixtures/usage-feedback |
usage feedback import/outcome fixtures | evals/integration/test_usage_feedback_import.sh, evals/integration/test_usage_feedback_outcomes.sh, evals/integration/test_usage_feedback_report.sh |
Keep while usage feedback import, outcome, and report flows accept JSONL fixture input. |
Cleanup Policy
Do not delete a fixture group just because a filename has no obvious direct reference. Many fixture directories are consumed as directories by integration tests. A fixture deletion is safe only when:
- The owning eval or validator no longer needs the behavior.
- This inventory is updated in the same change.
npm run validate:source --and the owning eval pass.- Any contract or docs that described the fixture behavior are updated.
Generated eval output belongs under ignored result directories, not under
evals/fixtures/.
npm run fixture:retirement-audit -- is a read-only retirement pass. It reports
fixture groups as kept when they have documented owners and live eval/script
references. A nonzero retire-candidate count is not permission to delete by
itself; it is a prompt to update or remove the owning eval and fixture together.