# Favur Evals — canonical dataset data dictionary

Machine-validated by `schema.json` (JSON Schema 2020-12) in this directory.
Every file here is a projection of the same canonical artifact; no number is
recomputed between formats. Full normalization bands and component formulas:
https://evals.favur.dev/methodology

## Files

| File | Contents |
|---|---|
| `favur-evals-v<version>.json` | Full canonical dataset, immutable per version |
| `latest.json` | Copy of the current version |
| `runs/<sow>__<run>.json` | One JSON document per run |
| `run_subjects.csv` | Flattened run × subject rows |
| `run_agent_roles.csv` | Flattened run × agent-role rows |
| `schema.json` | JSON Schema for the full dataset |
| `manifest.json` | Version, counts, and per-file sha256 |

## Top-level fields (`latest.json`)

| Field | Type | Meaning |
|---|---|---|
| `dataset_version` | string | Semver of the dataset itself; increments when the rubric changes (see `/data` changelog) |
| `generated_from` | object | Scoring-engine provenance (schema/manifest versions) |
| `rubric` | object | Everything needed to independently recompute a composite: `agent_role_weights` (points per role, sum 100 with completion), `composite_blend` (`role_weight` 0.75 / `completion_weight` 0.25), `cell_max` (10) |
| `sows` | array | Statement-of-work catalog (id, run count, last-seen) |
| `vendors` / `models` / `agent_roles` | arrays | Entity catalogs |
| `runs` | array | One entry per scored run (below) |

## Run fields

| Field | Type / unit | Meaning |
|---|---|---|
| `run_id` | string | `"<sow>/<run-folder>"` — permanent identifier |
| `sow` | string | Statement of Work the run executed |
| `rubric_version` | integer | Rubric version that produced the scores |
| `started_at` / `ended_at` | ISO 8601 | Run wall-clock bounds |
| `roster` | object | role → model seat (`vendor`, `family`, `version`, `model_key`, `model_unavailable`) |
| `composite.score` | number, 0–100 | `0.75 * (100 * Σ(w_r * score_r / 10) / Σ(w_r present)) + 0.25 * completion_score * 10` |
| `composite.coverage` | number, 0–1 | Weight-weighted fraction of the composite backed by real measurement — missing telemetry renormalizes away, it never scores zero |
| `composite.low_confidence` | boolean | True when coverage < 0.75 |
| `composite.completion_score` | number 0–10 or null | Run-level completion metric blended at 25% |
| `composite.pin` / `pin_drift` | object / number | Pin-ledger entry the score was frozen against, and any drift |
| `subjects.<id>` | object | Deprecated presentation grouping (raw/max/pct, component `normalized` values 0–1 with `weight`); retained for continuity — the composite is computed from `agent_roles`, not from subjects |
| `agent_roles.<role>.score` | number, 0–10 | The role's rubric cell |
| `agent_roles.<role>.coverage` | number, 0–1 | Fraction of the role's component weight backed by data |
| `agent_roles.<role>.components.<id>` | object | `normalized` (0–1), `weight`, `effective_weight` (post-renormalization), `available`, `group` (`output`/`efficiency`) |
| `telemetry` | object | Run-wide aggregates: `requests`, `failed_requests`, `duration_min` (minutes), `cost_usd` (USD), `cache_hit_rate` (%), token counts, `reasoning_tokens`, `tool_calls`, `tools_per_response`, `failure_breakdown` (count by type). Known gap: Favur tool-call events carry no agent/role tag, so telemetry is attributed run-wide, never per-role |
| `deliverable` | object | Deliverable manifest and static code metrics |
| `artifacts` | object | Repo/site links to raw artifacts and transcripts |

## CSV column notes

- `run_subjects.csv`: one row per (run, subject); `raw`/`max` are subject
  points, `pct` = 100 * raw/max, `normalized_score` is 0–1.
- `run_agent_roles.csv`: one row per (run, weighted role present);
  `score` is the 0–10 cell, `role_weight` its composite weight,
  `model_*` the seat from the roster. Health-check is excluded (unweighted).
- Empty cells mean the value is genuinely absent (renormalized), never zero.