Current rubric: v15

Methodology

How the score is calculated

Every number on this page is reproducible. A run’s composite is a 75/25 blend: 75% from a weighted average of per-agent role scores and 25% from a run-level completion metric. Each of the 14 weighted agent roles earns its own 0–10 score, and their weights add up to 100. There is no gate, curve, or pass/fail line — a composite measures how well a run’s agents engineered its Statement of Work and whether the run actually finished cleanly, not whether it “passed”.

Each agent role is scored on two groups of normalized component metrics — an output group (the quality of the artifacts or outcomes the role produced) and an efficiency group (the tokens, cost and error-cleanliness it spent getting there) — measured deterministically from artifacts the run already produced. The 0–10 role scores are multiplied by their weights and renormalized over the weighted roles the run exercised; that role-weighted mean provides 75 points. The remaining 25 points come from a run-level completion metric that is bottlenecked on the weakest of four required components: milestone completion, acceptance-criteria completion, the sprint verdict distribution and the whole-run test pass rate. A score of 5 means every component was fully completed; below 5 means at least one was left undone or failed; above 5 is only possible when every component is complete and a first-time/no-fixes bonus also applies.

If a component has no data for a role (e.g. no telemetry was attributed, or static analysis could not run) it drops out and the role’s remaining weights renormalize; if a whole weighted role never ran, its weight renormalizes out of the composite. A run is never penalized for data the harness did not capture. Every role cell records a coverage fraction — the share of its weight backed by real data — and the composite carries a coverage of its own.

Worked example: what does a 81.8 mean?

Take Gemini Flash 3.5 + Gemini Flash Lite on circles, which scored 81.8. That single number is 75% the weight-weighted average of the 12 agent roles this run exercised and 25% the run-level completion score. Each role’s 0–10 score times its weight is renormalized over the roles present, then the completion contribution is added:

Agent role	Role score	Weight	Contribution	Data coverage
Code Review	7.9 / 10	14	9.1	50%
Code	7.6 / 10	14	8.8	100%
Develop	7.8 / 10	14	9.0	100%
Sprint Plan	8.1 / 10	9	6.0	96%
Pseudocode	6.5 / 10	9	4.9	75%
Sprint Review	8.5 / 10	9	6.3	60%
Architect	8.2 / 10	6	4.0	70%
Orchestrator	7.9 / 10	4	2.6	75%
Test	6.9 / 10	4	2.3	80%
Build	5.5 / 10	3	1.4	100%
Platform	5.6 / 10	3	1.4	75%
Scout	5.8 / 10	2	1.0	100%
Composite			56.8

“Contribution” is each role’s share of the 75-point role portion — 0.75 × 100 × (role score ÷ 10) × weight, divided by the weight of the roles that ran. When a run doesn’t exercise every weighted role, the missing roles’ weight renormalizes away rather than counting as zero, which is why 12 roles plus the completion contribution still add to the full 81.8. “Data coverage” is the share of each role’s own weight that was backed by real measurements. The completion score is 0–10 and contributes up to 25 points. It is bottlenecked on the weakest of the four completion components, so a run with any unfinished milestone, acceptance criterion, rejected sprint or failing test cannot score above 5; above 5 is only possible when every component is fully complete and the work was approved first time with no fixes.

The weighted agent roles

Every role is scored on two groups: an output group (the quality of what it produced) and an efficiency group (the tokens, cost and error-cleanliness it spent). The role’s 0–10 score is the weighted average of both, and its weight below is the most it can contribute to the 75-point role portion of the composite. A separate run-level completion metric contributes the remaining 25 points.

Role	Weight	Responsibility	Output quality	Efficiency
Code Review	14	Independent review — severity-classifies issues and issues a PASS / PASS WITH CHANGES / FAIL verdict.	Review verdict, review impact, issues raised and the test quality of the code under review (verdict-derived terms stay unavailable until favur emits structured review telemetry).	Token volume, output per request, cost, request-error and hallucination rates.
Code	14	Writes the implementation — plan → TDD implementation → self-review — against the pseudocode and architecture.	Static code quality and deliverables of the code it shipped.	Token volume, output per request, cost, context efficiency, request-error and hallucination rates.
Develop	14	Per-task lifecycle coordinator — sequences pseudocode, test-creation, implementation and review for each task and drives the review/verify gates.	Code quality and test quality of the task it owned, acceptance-criteria results and deliverables.	Token volume, cost per request and context efficiency.
Sprint Plan	9	Right-sizes tasks (≤ 8h), maps dependencies and runs a consolidation review with acceptance criteria per task.	Plan quality (task sizing, dependencies, acceptance criteria, sections) and deliverables.	Token volume, output per request, cost, context efficiency, request-error and hallucination rates.
Pseudocode	9	Translates a sprint task into implementable pseudocode plus testing documentation (or an explicit testing boundary).	Quality of the pseudocode/plan it produced and deliverables.	Token volume, output per request, cost, context efficiency, request-error and hallucination rates.
Sprint Review	9	Evaluates sprint deliverables via delegated review/build/test and issues an APPROVE / REJECT / ESCALATE verdict.	Sprint verdict, review completeness and test quality (verdict-derived terms stay unavailable until favur emits structured verdict telemetry).	Token volume, output per request, cost, request-error and hallucination rates.
Director	6	Read-only strategic planner — defines sprint-sized goals with binary acceptance criteria and writes the phase direction document.	Direction quality: goal presence plus open-question closure ratio (resolved or deferred vs found) and validator/FSM performance.	Token volume, output per request, cost, context efficiency, request-error and hallucination rates.
Architect	6	Produces the architecture document and ADRs — interfaces, technology decisions and constraints for the phase.	Architecture completeness (components documented, ADRs recorded, document written) and validator/FSM performance.	Token volume, output per request, cost, context efficiency, request-error and hallucination rates.
Orchestrator	4	Top-level lifecycle manager — directs phases, plans and dispatches the agent fleet, and gates development, sprint review and phase completion.	Whole-run code quality, test quality, deliverables completeness and process discipline (the work its fleet shipped under its direction).	Its own token volume, cost per request and context efficiency.
Test	4	Read-only verification — runs the suite, analyses failures and checks coverage against the requested floor.	Test pass rate, coverage above the 90% floor, structural test quality and deliverables.	Token volume, cost and context efficiency.
Build	3	Iterative compile-fix loop until zero errors — minimal fixes only, the only role allowed to install tooling.	Static code quality and deliverables of the build it produced.	Token volume, output per request, cost, context efficiency, request-error and hallucination rates.
Platform	3	Infrastructure artifacts (CI/CD, Dockerfiles, configs) verified via shell execution before completion.	Static code quality and deliverables of the infrastructure it shipped.	Token volume, output per request, cost, context efficiency, request-error and hallucination rates.
Spec	3	Decision-artifact producer — concrete values only (no TBD/hedging), with a self-check step.	Quality of the decision artifact it wrote and deliverables.	Token volume, output per request, cost, context efficiency, request-error and hallucination rates.
Scout	2	Read-only observer — produces a findings document of observations, insights and concerns (an honest null outcome is valid).	Finding quality of the report it produced and deliverables.	Token volume, output per request, cost, context efficiency, request-error and hallucination rates.
Total	100

The quality axes behind the output scores

A role’s output group is built from the deterministic quality axes below — the same axes shown on each run’s radar. They are computed once from the run’s artifacts and then feed the roles that produced them (with whole-run attribution until Favur tags each file with the role that wrote it). Each axis is a weighted average of the normalized components listed under it.

Code Qualitymax 15

15 × (0.35·lint + 0.25·coverage + 0.20·complexity + 0.20·MI)

Measures the intrinsic quality of the code the run produced — independent of whether the tests happen to pass. It is one of the largest subjects (tied with Test Quality) because durable, readable, low-defect code is the primary deliverable of an SDLC run.

Up to 15 points. Clean lint, high coverage, low complexity and a high maintainability index push this toward 15; messy or unmaintainable code pulls it toward 0.

Component	Weight	What it measures	Normalized to 0–1
Lint cleanliness	35%	Number of ruff violations when the produced source is linted against one uniform ruleset (the same standard_ruff.toml for every run, so the bar is identical across models).	Linear decay: 0 violations → 1.0, each violation costs 0.1, 10+ → 0.0 (no step cliffs — one extra violation always moves the score by the same small amount). If the linter itself failed to launch, the component is treated as unavailable.
Test coverage	25%	Line coverage percentage reported by the run’s own pytest run over the code it wrote.	coverage% ÷ 100, clamped to 0–1 (90% → 0.90).
Cyclomatic complexity	20%	Average cyclomatic complexity of the produced source (radon) — how branchy the average function is. Lower is better.	Banded: avg < 2 → 1.0; < 3 → 0.66; < 5 → 0.33; < 8 → 0.16; otherwise 0.0.
Maintainability index	20%	Radon maintainability index (0–100) of the produced source — a blend of volume, complexity and lines of code. Higher is better.	MI ÷ 100, clamped to 0–1.

Test Qualitymax 15

15 × (0.30·structure + 0.25·assertion_density + 0.20·test_density + 0.15·coverage_floor + 0.10·pass_rate)

Measures how good the run’s test suite actually is — not merely that it is green. It leans on structural signals parsed from the test code itself — how deeply each test asserts, how dense the suite is relative to the source, and how sophisticated its organisation is. Raw coverage is Code Quality’s component; here it appears only as a term re-centred on the SoW’s requested 90% floor, and pass-rate is a low-weight sanity floor so a broken suite still hurts.

Up to 15 points. A large, deeply-asserting, well-structured suite with fixtures, parametrization, markers and integration tests approaches 15; a thin suite of a few shallow tests scores well below half even when it is fully green and over 90% covered.

Component	Weight	What it measures	Normalized to 0–1
Structural sophistication	30%	Six good-practice signals extracted from the test code by AST parsing: reusable @pytest.fixture definitions, @pytest.mark.parametrize cases, organised grouping (test classes or many modules), a shared conftest.py, use of pytest markers, and dedicated integration/e2e test files.	Mean of the six axes (each 0–1): fixtures and parametrize ramp to 1.0 at 3+, organisation is 1.0 with ≥2 classes or ≥4 files (0.5 with ≥1 class or ≥2 files), and conftest / markers / integration are each 0 or 1.
Assertion depth	25%	Average number of assertions per test — bare assert statements plus pytest.raises/warns context managers — counted across the suite. Deeper assertion per test means each test checks behaviour more thoroughly rather than smoke-testing that code runs.	Banded on assertions-per-test: <1 → 0.2, <1.5 → 0.5, <2 → 0.7, <3 → 0.9, 3+ → 1.0. A suite with no tests scores 0.
Test density	20%	Number of test functions per 100 lines of source code — how broadly the suite exercises the codebase, independent of line coverage.	Banded on tests-per-100-LOC: <4 → 0.3, <7 → 0.5, <10 → 0.7, <14 → 0.85, 14+ → 1.0. Zero tests scores 0.
Coverage (90% floor)	15%	The same line-coverage percentage used in Code Quality, but re-centred on Favur’s 90% floor so it rewards going beyond the requirement instead of saturating at it.	0.5 + (coverage% − 90) / 20, clamped to 0–1: 90% is neutral (0.5), 100% earns the full 1.0 bonus, and 80% or below is a 0.0 penalty.
Test pass rate	10%	Fraction of the run’s own pytest tests that pass, executed headlessly in a faithful sandbox copy of the run’s output.	Already 0–1; used directly (clamped). Low weight — a sanity floor so a broken suite still hurts, without dominating the subject.

Cost Efficiencymax 17

17 × (0.55·2^(−cost_per_request / SoW-median) + 0.25·cache_savings_ratio + 0.2·lean_overhead)

Rewards delivering the SoW economically (time has its own subject, Velocity). It blends a low average cost per LLM request — cheap calls and lean context per step; total run spend is not a metric anywhere — with how effectively caching paid off — the net amount saved by cache reads at the provider’s discounted rate, less any cache-write premium, as a share of the no-cache bill — and with lean fleet supervision: the share of the full bill eaten by the periodic health-check poller, which is strongly model-specific.

Up to 17 points. Cost per request is judged relative to the same SoW’s other runs — the SoW-median run earns half the cost term, a much cheaper run nearly all of it — a run whose caching meaningfully cut its bill earns the savings term, and a run whose supervision overhead stayed small earns the overhead term on top.

Component	Weight	What it measures	Normalized to 0–1
Cost per request	55%	Average cost per LLM request in USD (run cost ÷ requests, from the run’s meta) — rewards cheap calls and lean context per step rather than total spend. (The site never renders the raw dollar figure — only the resulting score.)	SoW-relative: 2^(−cost per request ÷ SoW-median). The SoW’s median run scores 0.5, half the median ~0.71, double ~0.25. When a SoW has too few scored runs for a trustworthy median, the component drops and the subject renormalizes — one SoW’s cost scale is never applied to another.
Effective caching	25%	Net dollars saved by caching as a share of the no-cache bill: cache reads priced at the provider’s discounted read rate save (input − read) per token, and any cache-write premium (write − input, e.g. on models that charge to populate the cache) is netted back out. Deepseek has no write charge, so its writes cost nothing.	cache_savings_usd ÷ nominal (no-cache) cost, clamped to 0–1 — the fraction of the hypothetical un-cached bill that caching avoided.
Lean supervision	20%	Share of the run’s full bill (all roles included) spent on the excluded supervision roles — the periodic health-check poller. Favur supervises every run the same way, so a model whose health-check responses are terse and cheap keeps this share small.	Band: 2% of the bill → 1.0, 25% → 0.0 (linear — a lower share is better).

The caching term needs per-model pricing + cache-token telemetry; a run without it drops the term and the cost term renormalizes to the full subject weight. The term is also dropped when role_model_parity reports a used model with no pricing entry — the savings ratio would be computed on a partial bill — so the subject falls back to the cost-per-request term alone. A run with no request telemetry (zero requests) drops the cost term too.

Speedmax 13

13 × (0.5·norm(tokens/sec) + 0.5·norm(latency))

Measures how quickly the model produced output during the run — the “quick” leg of a clean, quick, low-cost run. It is a 50/50 blend of run-level tokens per second and average request latency, each judged against an absolute reference band rather than other runs. This avoids penalizing a run that legitimately takes more wall-clock time because it did more work.

Up to 13 points. A run that is fast on both tokens/second and latency earns most of it; a run weak on both axes earns a small share.

Component	Weight	What it measures	Normalized to 0–1
Tokens per second	50%	Run-level output throughput from the run’s meta (output tokens ÷ total request time). Higher is better.	Absolute log band: 20 tok/s → 0.0, 200 tok/s → 1.0, with values in between spread logarithmically.
Request latency	50%	Average LLM request latency for the run (request_time_seconds ÷ requests). Lower is better.	Absolute linear band: 2 s/request → 1.0, 35 s/request → 0.0, clamped outside the band.

Runs whose meta lacks timing or request data (legacy exports) drop the missing components and the subject renormalizes away rather than penalizing the run.

Tool Disciplinemax 13

13 × (0.30·req_failures + 0.25·unauth + 0.20·tool_err + 0.15·halluc + 0.10·first_try)

Measures how cleanly the agents used their tools — the behavioral hygiene of the run. Frequent tool errors, hallucinated/invalid calls, retries and unauthorized-tool attempts all signal a model fighting its harness.

Up to 13 points. Clean, first-try, in-bounds tool use approaches 13; noisy or out-of-bounds tool use pulls it down.

Component	Weight	What it measures	Normalized to 0–1
Tool error rate	20%	Share of tool calls that failed for ordinary reasons — excluding schema-invalid (hallucinated) calls and unauthorized-tool attempts, which are scored by their own components. This keeps the Tool Discipline components disjoint so the same failure is never penalized twice ((failures − hallucinations) ÷ total tool calls). Unavailable (renormalized away) when a run has too few tool calls to be reliable.	Band: 0% → 1.0, 8%+ → 0.0 (linear). A plain 1 − rate left every run in the top 6% of the range; the band spreads the observed 1–6% rates.
Request failure rate	30%	Share of LLM requests that failed at the transport level (HTTP errors, empty or missing responses), from the run’s meta — the cleanliness of the run’s provider connection, distinct from tool-call failures. Unavailable (renormalized away) on runs with too few requests to be reliable.	Band: 0% → 1.0, 5%+ → 0.0 (linear). Observed rates cluster under 2.5% with rare pathological outliers, so the tight band keeps the component discriminating instead of saturating.
Hallucination rate	15%	Share of tool calls that were schema-invalid (ToolValidationError — the model invoked an unknown tool or malformed arguments). Distinct from unauthorized attempts.	Band: 0% → 1.0, 5%+ → 0.0 (linear).
First-try hit rate	10%	Share of tool calls that succeeded on the first attempt (success and an explicit zero retry count), over the calls whose retry status was actually recorded. Calls with no retry telemetry are unknown and excluded from both sides rather than assumed clean. Unavailable when too few calls carry retry telemetry to be reliable.	Already 0–1; used directly (higher → better).
Unauthorized-tool attempts	25%	Count of attempts to call a tool the agent was not permitted to use, detected from error events.	1 − 0.2 × attempts, floored at 0 (0 attempts → 1.0).

These are telemetry-derived. Today Favur emits tool-call events without an agent/role tag, so per-agent attribution is unavailable and these metrics are computed run-wide; when a metric has no data it renormalizes away. This is the one documented Favur-app data gap.

Effort Efficiencymax 13

13 × (0.30·token_volume + 0.20·turns + 0.15·token_econ + 0.15·output_disc + 0.05·ctx_window + 0.15·ctx_efficiency)

Measures economy of effort per unit of output — accomplishing the SoW with fewer total tokens (the dominant term), in few turns, with lean token spend per request, terse decisive responses, a context window that is not crowded, and low static system/tool-schema overhead. Tool breadth was retired from scoring: every conformant run clears it, so it carried no signal (it remains a display stat).

Up to 13 points. Efficient, focused runs score high; runs that thrash, over-spend tokens, rely on a single tool, or waste context on schema overhead score low.

Component	Weight	What it measures	Normalized to 0–1
Token volume	30%	Total billable tokens the run spent on the SoW — fresh input (cache reads excluded, so effective caching is rewarded) plus output. The direct “did it accomplish the task with fewer tokens” signal: it is price-independent (cheap per-token rates otherwise mask token hunger — observed 2.9M–35.3M on the same SoW) and per-request economy misses it when a run simply makes many requests.	SoW-relative: 2^(−billable ÷ SoW-median billable) — the SoW’s median run scores 0.5, half the median ~0.71, double ~0.25. Drops (renormalized away) when the SoW has too few scored runs.
Turns to completion	20%	Total agent sessions the run took (floored at 1). Fewer is better.	SoW-relative: 2^(−turns ÷ SoW-median turns) — the SoW’s median run scores 0.5, so a large SoW that needs more sessions is not penalized. Drops (renormalized away) when the SoW has too few scored runs for a trustworthy median.
Token economy	15%	Average tokens per request (total tokens ÷ requests). Fewer is better.	Band: 6,000 tok/req → 1.0, 50,000 → 0.0 (linear).
Output discipline	15%	Average output tokens per request — how tersely the model responds. Verbosity is a stable per-model fingerprint that costs money and time on every request.	Band: 250 output tok/req → 1.0, 1,100 → 0.0 (linear).
Context-window usage	5%	Largest single-request prompt size as a share of the model’s context window (default 200k tokens). Lower means more headroom.	Band: 25% of window → 1.0, 90% → 0.0 (linear).
Context efficiency	15%	Share of the context window not consumed by static system/tool-schema overhead, from the schema v2 context_breakdown. Higher is better.	Direct ratio; 1.0 means all context is productive, 0.0 means all overhead.

These are scale metrics with no natural 0–1 form, so each maps through a manifest normalization band (a “good” value scores 1.0, a “poor” value 0.0, linear between). Context efficiency is a direct ratio and does not use a band.

Process Disciplinemax 8

8 × (0.60·command_success_rate + 0.40·verification_density)

Measures how rigorously the run drove its own engineering work, read from the telemetry event stream: whether the commands it ran actually succeeded and how densely it verified its own work. (The structured-workflow check was dropped in rubric v7 — favur enforces the sprint/phase/delegation workflow, so it echoed harness conformance.)

Up to 8 points. A run with a high command-success rate and dense verification approaches 8; a run that thrashes failing commands and never verifies its work scores low.

Component	Weight	What it measures	Normalized to 0–1
Command success rate	60%	Share of shell commands that exited successfully, over the commands with a captured exit code. Commands whose exit code was never captured (the harness records −1 for streamed/killed/uncaptured commands) are excluded from both sides rather than miscounted as failures, so the rate reflects real command outcomes. Unavailable when too few commands have a captured exit code to be reliable.	Already 0–1; used directly (clamped).
Verification density	40%	Verification events per 100 commands — how often the run checked its own work relative to how much it did.	Band: 20 verifications / 100 commands → 1.0, 0 → 0.0 (linear).

Telemetry-derived. A run with no `.favur/telemetry` event stream drops every component and the subject renormalizes away rather than scoring 0.

Deliverablesmax 6

6 × (0.50·completeness + 0.50·output_volume)

Measures the shipped output recorded in the run’s deliverables manifest — how complete the artifact set is (source, tests, docs) and how substantial the shipped code is relative to the same SoW’s other runs. Deliberately cost-free: what a run spent is Cost Efficiency’s axis, and re-scoring dollars here dragged complete-but-expensive deliverables to near zero.

Up to 6 points. A run that ships source, tests and docs, with high AC completion, passing tests and substantial docs, approaches 6; a thin or incomplete deliverable scores low.

Component	Weight	What it measures	Normalized to 0–1
Completeness	50%	Continuous score for the three artifact kinds: source is scaled by acceptance-criteria completion, test by the run’s test pass rate, and doc by total doc length normalized against the doc-length band.	Average of source_score, test_score and doc_score (each 0–1).
Output volume	50%	Shipped code lines of code — source + test LOC from the manifest, de-duplicated by content hash (docs are excluded so they cannot dominate the volume).	SoW-relative: code LOC ÷ (1.5 × SoW-median LOC), clamped to 0–1 — the SoW’s median run scores ~0.67 and 1.5× the median saturates. Drops (renormalized away) when the SoW has too few scored runs for a trustworthy median.

A run without a `meta.deliverables` manifest drops every component and the subject renormalizes away.

This scoring rubric is specific to Favur’s execution, which is under constant development. As we add or remove telemetry — and refine how each agent role is measured — these numbers may change. Scores are relative rankings under a versioned rubric, not calibrated measurements of model capability; a run’s first score is pinned to the rubric version that produced it so historical numbers stay traceable.

What this benchmark does not measure

Human factors. Nothing here observes how a model behaves with a human in the loop — prompt-recovery skill, clarifying-question quality, or how pleasant the output is to review. Every run is fully autonomous inside Favur's agentic process.
Domains beyond the Statements of Work. Scores generalize only as far as the SoWs do. A model that excels at these web-application briefs may behave differently on embedded, data-engineering, or research workloads.
Per-agent telemetry the platform cannot yet attribute. Favur emits tool-call events without an agent/role tag, so hallucination rate, retry burden, first-try hit rate, and per-agent context usage are measured run-wide, not per role. Affected role cells renormalize the missing weight away (never scoring zero) and carry a reduced data-coverage fraction — shown on every run page.
Latency and cost at your scale. Telemetry reports what these runs cost and how fast they went under our harness and quotas — not what a production deployment would see under different rate limits, caching, or batching.
Safety and security posture. No jailbreak resistance, data-exfiltration, or license-compliance testing happens here; the rubric scores engineering output quality only.

A benchmark that names its blind spots is easier to trust — and easier to check. The full rubric above documents every component, weight, and normalization band that ismeasured.