---
id: CAP-COMPARE-001
title: "Methodology for Comparing AI Systems"
alternative_names:
  - "Comparing LLMs"
  - "Model Comparison Methodology"
  - "Leaderboard Interpretation"
category: "Capabilities → Measurement"
volume: IV
chapter: 25
entry_type: "Standard Article"
level: 3
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "CAP-BENCH-001"
related_entries:
  - "TECH-LLM-001"
  - "CAP-REASON-001"
  - "CAP-LIMITS-001"
  - "CAP-CODE-001"
  - "CAP-AGENTEVAL-001"
  - "CAP-OPEN-001"
  - "AGI-TEST-001"
---

# Methodology for Comparing AI Systems

## 1. Concise Definition
Comparing AI systems is the practice of drawing a conclusion of the form "system A is better than system B for purpose X" from evidence such as benchmark scores, human preference votes and field trials. A valid comparison specifies what is being compared, under what controlled conditions, on what measure, with what uncertainty, and for which purpose; a leaderboard rank by itself specifies few of these.

## 2. Overview & Significance
Procurement, research direction, policy and public understanding all depend on claims that one system outperforms another. These claims are fragile. Benchmark scores depend on prompt format, decoding settings, scaffolding, grading method and sample size. Rankings may reflect which developers tested more variants. Static test sets may have leaked into training data. This article gives a methodology, organised around a checklist, for reading and producing comparisons; it does not give current rankings, which belong in the Living Reference (`MODEL-001`) with their dates and sources. For the underlying measurement concepts (metrics, contamination, saturation) see [CAP-BENCH-001].

## 3. Historical Development
Comparison in machine learning began with shared datasets and held-out test sets. As language models grew, single-task comparisons gave way to multi-task suites, and then to attempts at holistic reporting. Liang et al. (2022) introduced HELM, evaluating language models across many scenarios and multiple metrics beyond accuracy, with the aim of making evaluation transparent and standardised. Open-ended outputs led to model-based judging (Zheng et al., 2023) and to crowdsourced pairwise preference platforms (Chiang et al., 2024). Critiques followed on each: sensitivity of scores to prompt formatting (Sclar et al., 2023), to leaderboard design (Alzahrani et al., 2024), to verbosity in automatic judges (Dubois et al., 2024), to statistical noise (Miller, 2024), to contamination (Oren et al., 2023; Zhang et al., 2024) and to selective disclosure in public arenas (Singh et al., 2025). Raji et al. (2021) criticised the premise that a single "general" benchmark can measure general capability, a validity argument that remains central.

## 4. Technical / Conceptual Explanation

### 4.1 Define the comparand and the purpose
A comparison should state whether it compares **models** (weights and training), **systems** (model plus prompt, tools and scaffold) or **products** (system plus filters and policies, which may change without notice). It should also state the intended use. A coding-agent comparison (see [CAP-CODE-001], [CAP-AGENTEVAL-001]) does not transfer to medical question answering, and a benchmark's construct validity for the purpose is a precondition for using it (Raji et al., 2021; Reuel et al., 2024).

### 4.2 Control the evaluation conditions
Scores change with factors other than the model. Sclar et al. (2023) showed that language models are highly sensitive to spurious features of prompt format, such as separators and casing, with performance differences that can be large even for semantically equivalent prompts; Alzahrani et al. (2024) showed that small changes to multiple-choice benchmark design can change model rankings. Comparisons therefore need: identical prompts and formats for all systems or a reported prompt search procedure applied equally; stated decoding parameters (temperature, number of samples, whether best-of-$k$ selection is used); stated reasoning or compute budget; and stated tool access. Biderman et al. (2024) document practical sources of irreproducibility in language-model evaluation and describe an open harness designed to address them.

### 4.3 Quantify uncertainty
Benchmark accuracy on $n$ questions has sampling error. If a score is a mean of per-question outcomes with sample standard deviation $s$, the standard error is approximately $s/\sqrt{n}$, and a $95\%$ confidence interval is about the mean $\pm\, 1.96\, s/\sqrt{n}$. Miller (2024) argues that evaluation reports should include such error bars and recommends analysing paired differences between models on the same questions, which typically gives tighter comparisons than comparing separate intervals. A difference smaller than the uncertainty cannot support a ranking claim.

### 4.4 Check for contamination and overfitting
Contamination makes scores unrepresentative of generalisation. Oren et al. (2023) proposed a method to detect test-set contamination in black-box models by testing whether a model prefers a particular canonical ordering of benchmark examples, which has no need for access to pretraining data or weights. Zhang et al. (2024) built a new benchmark designed to match the difficulty and style of an existing grade-school mathematics set and found evidence that some model families performed worse on the fresh set, consistent with overfitting to the original. Fresh, private or continually refreshed test sets reduce but do not eliminate the problem (see [CAP-CODE-001] for LiveCodeBench).

### 4.5 Understand human-preference and judge-based measures
Pairwise human votes aggregated into Bradley-Terry-style ratings are used on platforms such as Chatbot Arena (Chiang et al., 2024). The method collects diverse real prompts but measures aggregate preference, not task-specific correctness, and it is subject to the participation and style biases of its user population. LLM judges scale grading but carry biases including verbosity preference (Zheng et al., 2023; Dubois et al., 2024, who propose a length-controlled regression adjustment to AlpacaEval). Scores from judges and from humans should be calibrated against each other on a sample before one is used as a proxy for the other.

### 4.6 Check the governance of the leaderboard
Singh et al. (2025) reported that undisclosed private testing in Chatbot Arena allowed some providers to test multiple variants before release and retract scores, biasing results through selective disclosure; they identified 27 private variants tested by one provider before one release, and estimated that two large providers received 19.2% and 20.4% of all arena data while 83 open-weight models together received an estimated 29.7% (analysis as of the authors' 2025 study; shares will have changed). The general point is that sampling rates, model retirement policy and disclosure rules affect rankings and should be inspected.

## 5. Types, Components & Classifications
| Evidence type | Strength | Typical weakness |
| :--- | :--- | :--- |
| Static benchmark | Cheap, reproducible, comparable over time | Contamination, saturation, narrow construct |
| Fresh or private benchmark | Resists contamination | Opaque; hard to audit |
| Human preference ranking | Broad, real prompts | Style bias; population bias; platform rules |
| LLM-judged evaluation | Scalable | Judge bias; self-preference |
| Agent environment | Interactive realism | Scaffold dependence; cost; reproducibility |
| Controlled field trial | Closest to real use | Small samples; specific setting |
| Developer report | Direct description of method | Selective; not independent (Tier 2 at best) |

## 6. Applications & Concrete Examples
**Reading a developer announcement.** A table showing a new model ahead of a competitor on several benchmarks should be checked for: whether the competitor's number was re-run under identical conditions or copied from another source; whether the new model used a larger sampling or reasoning budget; whether error bars are given; whether any benchmark is likely contaminated; and whether the comparison covers the purpose at hand.

**Selecting a model for deployment.** Assemble a private task set from real, de-identified workload, run candidates under identical conditions with repeated trials, report confidence intervals and cost, and include at least a small human review sample. Public rankings serve only to shortlist.

**Reading a leaderboard.** Identify who runs it, how models are sampled, what is disclosed, and whether submissions can be withdrawn (Singh et al., 2025).

## 7. Advantages, Limitations & Unresolved Questions
The checklist improves defensibility but does not guarantee validity: a perfectly controlled comparison on an invalid benchmark is still uninformative about the intended use (Raji et al., 2021; Reuel et al., 2024 propose criteria for assessing benchmark quality). Unresolved questions include how to compare systems across different compute budgets in a fair way (cost-normalised or budget-matched comparison), how to detect contamination in closed models with certainty (Oren et al., 2023 give a statistical test with stated assumptions, not a proof), and how to aggregate across heterogeneous benchmarks without arbitrary weighting. Whether a single "overall" ordering of general-purpose systems is meaningful at all is contested; HELM's multi-metric design (Liang et al., 2022) is one response in the direction of reporting profiles rather than ranks.

## 8. Current State of Knowledge
As of 2026-10-15, the cited sources establish that: scores are sensitive to evaluation conditions (Well Supported; Sclar et al., 2023; Alzahrani et al., 2024); statistical uncertainty is often unreported and can be quantified with standard methods (Well Supported; Miller, 2024); contamination and overfitting have been demonstrated for some models on some benchmarks (Preliminary to Well Supported; Zhang et al., 2024; Oren et al., 2023); judge-based and preference-based measures have documented biases and design issues (Well Supported; Zheng et al., 2023; Dubois et al., 2024; Singh et al., 2025). There is no single accepted standard for comparing general-purpose systems; HELM and the open harness described by Biderman et al. (2024) are community efforts, not binding standards.

## 9. Future Trajectory & Strategic Outlook
Observed trend (not a forecast): more evaluation reports include confidence intervals, held-out sets and multi-metric profiles, and independent evaluators (academic groups, safety institutes) increasingly report comparisons in addition to developers. Whether independent evaluation becomes a regular precondition for deployment is a governance matter (see [GOV-LAW-001]) that this article does not forecast.

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[CAP-BENCH-001] AI Evaluation, Benchmarking & Contamination](CAP-BENCH-001_ai_evaluation.md): Metrics, benchmark types and contamination.
  - [[TECH-LLM-001] Large Language Models](../vol03_technical/TECH-LLM-001_large_language_models.md): The systems usually compared.
- **Related Methods & Technologies**:
  - [[CAP-REASON-001] Reasoning, Planning and Long-Horizon Capability](CAP-REASON-001_reasoning_planning_and_long_horizon_capability.md): Interpreting reasoning claims.
  - [[CAP-LIMITS-001] Hallucination, Reliability and Calibration](CAP-LIMITS-001_hallucination_reliability_and_calibration.md): Reliability and abstention in scoring.
  - [[CAP-CODE-001] Software Engineering Capability and Evaluation](CAP-CODE-001_software_engineering_capability_and_evaluation.md): Domain-specific comparison.
  - [[CAP-AGENTEVAL-001] Evaluating Autonomous Agents](CAP-AGENTEVAL-001_evaluating_autonomous_agents.md): Agent comparisons.
  - [[CAP-OPEN-001] Open-Weight versus Closed Models](CAP-OPEN-001_open_weight_versus_closed_models.md): Comparisons across release approaches.
- **Advanced Implications & Future Trajectories**:
  - [[AGI-TEST-001] Testing Claims of General Intelligence](../vol05_agi/AGI-TEST-001_testing_claims_of_general_intelligence.md): Comparison methodology applied to generality claims.
  - [[GOV-LAW-001] AI Governance](../vol08_safety_governance/GOV-LAW-001_ai_governance.md): Evaluation as a regulatory input.
  - [[MODEL-001] Model Registry](../reference/factbook/MODEL-001_model_registry.md): Dated, sourced model comparisons.

## 11. References & Foundational Literature
1. Alzahrani, N., Alyahya, H. A., Alnumay, Y., Alrashed, S., et al. (2024). *When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards*. arXiv:2402.01781. [Shows leaderboard rankings are sensitive to small benchmark design changes].
2. Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., et al. (2024). *Lessons from the Trenches on Reproducible Evaluation of Language Models*. arXiv:2405.14782. [Practical sources of irreproducibility and an open evaluation harness].
3. Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., et al. (2024). *Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference*. arXiv:2403.04132. [Crowdsourced pairwise preference platform].
4. Dubois, Y., Galambosi, B., Liang, P., & Hashimoto, T. B. (2024). *Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators*. arXiv:2404.04475. [Length bias in automatic judges and a regression-based adjustment].
5. Liang, P., Bommasani, R., Lee, T., Tsipras, D., et al. (2022). *Holistic Evaluation of Language Models*. Transactions on Machine Learning Research (2023). arXiv:2211.09110. [Multi-scenario, multi-metric evaluation framework].
6. Miller, E. (2024). *Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations*. arXiv:2411.00640. [Statistical recommendations for reporting uncertainty in evaluations].
7. Oren, Y., Meister, N., Chatterji, N., Ladhak, F., & Hashimoto, T. B. (2023). *Proving Test Set Contamination in Black Box Language Models*. arXiv:2310.17623. [Statistical test for contamination without access to training data].
8. Raji, I. D., Bender, E. M., Paullada, A., Denton, E., & Hanna, A. (2021). *AI and the Everything in the Whole Wide World Benchmark*. arXiv:2111.15366. [Critique of the construct validity of "general" benchmarks].
9. Reuel, A., Hardy, A., Smith, C., Lamparth, M., et al. (2024). *BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices*. arXiv:2411.12990. [Framework for assessing benchmark quality].
10. Sclar, M., Choi, Y., Tsvetkov, Y., & Suhr, A. (2023). *Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting*. arXiv:2310.11324. [Prompt-format sensitivity].
11. Singh, S., Nan, Y., Wang, A., D'Souza, D., et al. (2025). *The Leaderboard Illusion*. arXiv:2504.20879. [Analysis of selective disclosure and sampling asymmetries in Chatbot Arena].
12. Zhang, H., Da, J., Lee, D., Robinson, V., et al. (2024). *A Careful Examination of Large Language Model Performance on Grade School Arithmetic*. arXiv:2405.00332. [Fresh benchmark used to probe overfitting to an existing mathematics set].
13. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., et al. (2023). *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena*. arXiv:2306.05685. [LLM judges and their biases].

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Scores depend on prompt format and benchmark design | Well Supported | Sclar et al. (2023); Alzahrani et al. (2024) | High |
| Sampling error should be reported; paired analysis is tighter | Established (statistical principle); recommendation is Preliminary | Miller (2024) | Moderate to High |
| Contamination can be tested in black-box models | Preliminary (method with assumptions) | Oren et al. (2023) | Moderate |
| Some model families overfit an existing math benchmark | Preliminary | Zhang et al. (2024) | Moderate |
| LLM judges and arena rankings carry measurable biases | Well Supported | Zheng et al. (2023); Dubois et al. (2024); Singh et al. (2025) | Moderate to High |
| A single benchmark cannot measure general capability | Contested (validity argument) | Raji et al. (2021) | Active debate |
| Arena data shares and private-variant counts | Preliminary (single study; dated 2025) | Singh et al. (2025) | Moderate |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.