WyzScore Methodology
WyzBench evaluates LLMs against your specific task using a composite scoring system called WyzScore. This page documents exactly how scores are computed so you can assess what the numbers mean for your use case.
Score Scale
WyzScore runs from 0 to 100. Higher is better.
Important: WyzScore is relative to how you define the task and what prompts you supply. A score of 82 on a well-specified cold email task and a score of 82 on a vague “general” task are not directly comparable. Always compare scores within the same task type and prompt set.
Formula
Each of the six dimensions is scored 0–100, then multiplied by its weight. The weighted contributions are summed to produce a final composite score, clamped to [0, 100]. Weights are normalized to always sum to 1.0 — if you override them in the evaluation form, the normalization is applied automatically.
The Six Dimensions
The default weights below represent WyzBench's baseline calibration for business-task evaluation. You can override weights per evaluation or set org-level defaults in Settings → Criteria Weights.
Accuracy, coherence, factual correctness, and completeness of the model's output for the given task.
Whether the output achieves the stated task goal — a cold email that would plausibly book a call, a contract summary that correctly identifies risk clauses, etc.
Consistency, predictability, and absence of hallucinations across similar inputs. Lower variance across prompts = higher reliability.
Derived directly from the provider's published token pricing. Score 100 = near-zero marginal cost (open-source); score 50 = GPT-4o baseline ($0.00625/1k tokens); score 0 = approximately 2× the baseline rate.
Based on measured average response latency across all test prompts. Score 90 = 500ms or faster; score 10 = 8,000ms or slower. Linear interpolation between these points. Measured after a warm-up call to eliminate connection setup time.
Appropriate handling of sensitive content, adherence to instructions and constraints, and absence of policy violations. More relevant for regulated-industry task types.
Measured dimensions (Cost, Speed) are computed deterministically from pricing tables and latency data. AI-judged dimensions (Quality, Business Outcome, Reliability, Governance) are scored by an LLM judge running the rubric against the model's output.
Worked Example
Model: gemini-2.5-flash · Task: Cold Email (B2B SaaS) · Default weights
AI Judge Methodology
WyzBench uses its own central evaluation key for judging — your provider keys are not used.
Zero temperature ensures deterministic scoring. The same input to the judge always produces the same score.
Each test prompt produces one output; the judge scores each output independently. Dimension scores are averaged across all prompts in your eval set.
The judge evaluates outputs against rubrics calibrated to the selected task type (cold email, KYC review, etc.), not a generic quality rubric.
Why Does the Same Model Score Differently Across Evaluations?
This is expected behavior, not a bug. WyzScore is task-and-prompt-specific by design. Common causes of variance:
- Different task type: A “cold email” eval and a “general” eval run different rubrics. A model optimised for structured writing may score 86 on cold email and 22 on a vague unstructured task.
- Different prompt quality: Detailed, realistic test prompts produce more meaningful scores than short, context-free ones.
- Speed scoring: Latency-based speed score can swing significantly depending on API load at the time of evaluation. A model that scores 32 on Speed in one run may score 60 in another if it was under higher load.
- Custom weights: If you override dimension weights between runs, the composite will differ even if underlying dimension scores are identical.
Best practice: To compare two models fairly, run them in the same evaluation with the same task type, prompt set, and weights. The dashboard shows the most recent evaluation — not necessarily the most comparable one.
Worked Example — Cold Email, 5 Prompts, 3 Models
To make the formula concrete, here is how WyzScore was computed for a real cold email evaluation. Scores are per-model averages across 5 test prompts. Weights are the defaults.
| Dimension | Weight | GPT-4o mini | Claude Haiku | Mistral Small |
|---|---|---|---|---|
| Quality | 30% | 82 | 91 | 77 |
| Business Outcome | 30% | 78 | 85 | 71 |
| Reliability | 15% | 95 | 97 | 90 |
| Speed | 10% | 74 | 48 | 81 |
| Cost | 10% | 88 | 42 | 79 |
| Governance | 5% | 76 | 83 | 70 |
| WyzScore | 100% | 82.3 | 80.5 | 77.4 |
Questions about methodology or scoring? Contact us at benchmark@wyzbench.ai. This page is updated whenever scoring parameters change.