Agent role · weight 6/100 · rubric v15
🎬 Director
Read-only strategic planner — defines sprint-sized goals with binary acceptance criteria and writes the phase direction document.
- 6.74Pooled mean / 10
- ± 1.7795% CI
- 6Appearances (n)
- 5Models tested
Provisional: n < 15 — the confidence interval is wide; treat rankings as unsettled.
What it's scored on
- Output quality
- Direction quality: goal presence plus open-question closure ratio (resolved or deferred vs found) and validator/FSM performance.
- Efficiency
- Token volume, output per request, cost, context efficiency, request-error and hallucination rates.
Full scoring definition in the methodology → · Sample transcript
Model leaderboard
| # | Model | Score / 10 | Coverage | n | Last seen |
|---|---|---|---|---|---|
| 1 | DeepSeek Deepseek Flash v4 † | 8.60 | 68% | 1 | 2026-06-26 |
| 2 | Qwen Plus qwen3.7 † | 7.43 | 96% | 1 | 2026-07-19 |
| 3 | Google Gemini Flash 3.6 † | 7.41 | 96% | 1 | 2026-07-22 |
| 4 | Meta Muse Spark 1.1 | 6.32 | 96% | 2 | 2026-07-19 |
| 5 | Stepfun Step Flash 3.7 † | 4.36 | 96% | 1 | 2026-07-19 |
† low confidence — sparse role or low data coverage (renormalized, never zero-filled).
Per-run history (newest first)
| Date | Run | SoW | Model | Score / 10 |
|---|---|---|---|---|
| 2026-07-22 | the2048__gemini-multi-model-7-22-2026 | the2048 | Google Gemini Flash 3.6 | 7.41 |
| 2026-07-19 | solarSystem__qwen-qwen3.7-plus-7.18.2026 | solarSystem | Qwen Plus qwen3.7 | 7.43 |
| 2026-07-19 | solarSystem__stepfun-step-3.7-flash-7.19.2026 | solarSystem | Stepfun Step Flash 3.7 | 4.36 |
| 2026-07-19 | the2048__meta-muse-spark-1.1-7.19.2026 | the2048 | Meta Muse Spark 1.1 | 4.94 |
| 2026-07-17 | solarSystem__meta-muse-spark-1.1-7.17.2026 | solarSystem | Meta Muse Spark 1.1 | 7.69 |
| 2026-06-26 | the2048__deepseek-deepseek-v4-flash-6.26.2026 | the2048 | DeepSeek Deepseek Flash v4 | 8.60 |
Every run on Favur Evals is scored by the same deterministic engine, on the same Statements of Work. The benchmark is self-funded — no vendor sponsorship, credits, or grants.