FAVUR EVALS

Measures the intrinsic quality of the code the run produced — independent of whether the tests happen to pass. It is the largest subject (tied with Test Quality) because durable, readable, low-defect code is the primary deliverable of an SDLC run.

Up to 18 points. Clean lint, high coverage, low complexity and a high maintainability index push this toward 18; messy or unmaintainable code pulls it toward 0.

Measures how good the run’s test suite actually is — not merely that it is green. Favur enforces an all-green suite and a ≥90% coverage floor, so raw pass-rate and coverage saturate near the top for every conformant run and cannot discriminate them. Test Quality therefore leans on structural signals parsed from the test code itself — how deeply each test asserts, how dense the suite is relative to the source, and how sophisticated its organisation is — and keeps coverage and pass-rate only as small, re-centred terms.

Up to 18 points. A large, deeply-asserting, well-structured suite with fixtures, parametrization, markers and integration tests approaches 18; a thin suite of a few shallow tests scores well below half even when it is fully green and over 90% covered.

Rewards delivering the SoW economically. It deliberately has no time term (wall-clock is dominated by external API latency, not the model’s skill) — it blends low absolute dollar cost with how effectively caching paid off: the net amount saved by cache reads at the provider’s discounted rate, less any cache-write premium, as a share of the no-cache bill.

Up to 14 points. The exponential decay means a very cheap run earns almost the full cost term, and a run whose caching meaningfully cut its bill earns the savings term on top — rewarding both spending little and caching well.

The caching term needs per-model pricing + cache-token telemetry; a run without it drops the term and the cost term renormalizes to the full subject weight.

Measures how cleanly the agents used their tools — the behavioral hygiene of the run. Frequent tool errors, hallucinated/invalid calls, retries and unauthorized-tool attempts all signal a model fighting its harness.

Up to 12 points. Clean, first-try, in-bounds tool use approaches 12; noisy or out-of-bounds tool use pulls it down.

These are telemetry-derived. Today Favur emits tool-call events without an agent/role tag, so per-agent attribution is unavailable and these metrics are computed run-wide; when a metric has no data it renormalizes away. This is the one documented Favur-app data gap.

Measures whether the agents followed Favur’s workflow contract and reported their work well — opening and closing work cycles correctly, and producing structured completion reports rather than terse or empty ones.

Up to 12 points, split evenly between following the process and documenting it.

Measures economy of effort per unit of output — reaching completion in few turns, with lean token spend per request, a healthy breadth of tools, and without crowding the context window.

Up to 12 points. Efficient, focused runs score high; runs that thrash, over-spend tokens, or rely on a single tool score low.

These are scale metrics with no natural 0–1 form, so each maps through a manifest normalization band (a “good” value scores 1.0, a “poor” value 0.0, linear between).

Measures how rigorously the run drove its own engineering workflow, read from the telemetry event stream: whether the commands it ran actually succeeded, how densely it verified its own work, and whether it used the structured sprint/phase/delegation process rather than running flat.

Up to 8 points. A run with a high command-success rate, dense verification and a full sprint/phase/delegation structure approaches 8; a run that thrashes failing commands and never verifies or structures its work scores low.

Telemetry-derived. A run with no `.favur/telemetry` event stream drops every component and the subject renormalizes away rather than scoring 0.

Measures the shipped output recorded in the run’s deliverables manifest — whether it produced a complete set of artifacts, how much code it shipped, and how economically (cost per line of code).

Up to 6 points. A run that ships source, tests and docs in healthy volume at a low cost-per-LOC approaches 6; a thin or expensive-per-LOC run scores low.

A run without a `meta.deliverables` manifest drops every component and the subject renormalizes away. The cost-per-LOC term also drops when cost or code LOC is unavailable.