The dataset
Every number on this site is generated from a validated, versioned canonical dataset — currently v1.0.0 (75 runs, rubric v15). An independent verification script recomputes sampled composites from the dataset's own component values on every publish.
Downloads
Static files on the site's CDN — cache-friendly and stable. Prior dataset versions stay retrievable at their versioned URLs. Checksums for every file are inmanifest.json.
| File | Contents | Size |
|---|---|---|
| favur-evals-v1.0.0.json | Full canonical dataset (immutable, versioned) | 1.5 MB |
| latest.json | Current dataset version (stable URL) | 1.5 MB |
| run_subjects.csv | Flattened CSV — one row per run × subject | 80 KB |
| run_agent_roles.csv | Flattened CSV — one row per run × weighted agent role, with the model seat | 124 KB |
| schema.json | JSON Schema (2020-12) for the full dataset | 10 KB |
| DATA_DICTIONARY.md | Data dictionary — every field, unit, and semantics | 4 KB |
| manifest.json | File manifest with sha256 checksums | — |
| LICENSE.txt | License (CC BY 4.0) | 568 B |
| CITATION.bib | BibTeX citation | 405 B |
runs/<sow>__<run>.json | One JSON document per run (75 files) | 1.5 MB |
Reproduce it yourself
A standard-library-only Python script recomputes every composite from the public download's per-role component values and the published rubric block — no access to the scoring engine — and asserts the top run matches its published figure:reproduce_top_run.py. The same check runs in CI on every publish. Full formulas:methodology.
License & citation
The dataset is licensed underCC BY 4.0 — use it for anything, with attribution. Cite the dataset version and rubric version you used (currently dataset v1.0.0, rubric v15):
@misc{favurEvals,
title = {Favur Evals: a deterministic benchmark of LLMs inside a multi-agent software-development lifecycle},
author = {{Awesoft Solutions}},
howpublished = {\url{https://evals.favur.dev}},
note = {Dataset: \url{https://evals.favur.dev/data}. Cite the dataset version and rubric version you used, e.g. dataset v1.0.0, rubric v15.},
year = {2026}
}Pipeline health
The site republishes itself when new run data lands: validate → score → canonical dataset → fixture regression → public exports → deploy. Everything below derives from the data itself (never a hand-set date) and updates on every publish —health.json.
| Runs published | 75 |
|---|---|
| Latest ingested run | solarSystem/multi-xiaomi-mimo-v2.5-7.26.2026 (started 2026-07-25) |
| Runs skipped by the validation gate | 1 |
| Dataset / rubric version | v1.0.0 / rubric v15 |
Changelog
| Version | Date | Rubric | Changes |
|---|---|---|---|
| 1.0.0 | 2026-08-01 | v15 | Initial canonical dataset: per-run composites, subject component breakdowns, per-role rubric cells with data-coverage fractions, telemetry aggregates, deliverable manifests, and entity catalogs (SoWs, vendors, models, agent roles). Scores identical to the pre-migration site (Sprint 0 baseline snapshot). |
Every run on Favur Evals is scored by the same deterministic engine, on the same Statements of Work. The benchmark is self-funded — no vendor sponsorship, credits, or grants.