Comparing AI Systems

Comparing AI systems is the practice of drawing a conclusion of the form "system A is better than system B for purpose X" from evidence such as benchmark scores, human preference votes and field trials. A valid comparison specifies what is being compared, under what controlled conditions, on what measure, with what uncertainty, and for which purpose; a leaderboard rank by itself specifies few of these.

Procurement, research direction, policy and public understanding all depend on claims that one system outperforms another. These claims are fragile. Benchmark scores depend on prompt format, decoding settings, scaffolding, grading method and sample size. Rankings may reflect which developers tested more variants. Static test sets may have leaked into training data. This article gives a methodology, organised around a checklist, for reading and producing comparisons; it does not give current rankings, which belong in the Living Reference (MODEL-001) with their dates and sources. For the underlying measurement concepts (metrics, contamination, saturation) see [CAP-BENCH-001].

Comparison in machine learning began with shared datasets and held-out test sets. As language models grew, single-task comparisons gave way to multi-task suites, and then to attempts at holistic reporting. Liang et al. (2022) introduced HELM, evaluating language models across many scenarios and multiple metrics beyond accuracy, with the aim of making evaluation transparent and standardised. Open-ended outputs led to model-based judging (Zheng et al., 2023) and to crowdsourced pairwise preference platforms (Chiang et al., 2024). Critiques followed on each: sensitivity of scores to prompt formatting (Sclar et al., 2023), to leaderboard design (Alzahrani et al., 2024), to verbosity in automatic judges (Dubois et al., 2024), to statistical noise (Miller, 2024), to contamination (Oren et al., 2023; Zhang et al., 2024) and to selective disclosure in public arenas (Singh et al., 2025). Raji et al. (2021) criticised the premise that a single "general" benchmark can measure general capability, a validity argument that remains central.

#4.1 Define the comparand and the purpose

A comparison should state whether it compares models (weights and training), systems (model plus prompt, tools and scaffold) or products (system plus filters and policies, which may change without notice). It should also state the intended use. A coding-agent comparison (see [CAP-CODE-001], [CAP-AGENTEVAL-001]) does not transfer to medical question answering, and a benchmark's construct validity for the purpose is a precondition for using it (Raji et al., 2021; Reuel et al., 2024).

#4.2 Control the evaluation conditions

Scores change with factors other than the model. Sclar et al. (2023) showed that language models are highly sensitive to spurious features of prompt format, such as separators and casing, with performance differences that can be large even for semantically equivalent prompts; Alzahrani et al. (2024) showed that small changes to multiple-choice benchmark design can change model rankings. Comparisons therefore need: identical prompts and formats for all systems or a reported prompt search procedure applied equally; stated decoding parameters (temperature, number of samples, whether best-of-$k$ selection is used); stated reasoning or compute budget; and stated tool access. Biderman et al. (2024) document practical sources of irreproducibility in language-model evaluation and describe an open harness designed to address them.

#4.3 Quantify uncertainty

Benchmark accuracy on $n$ questions has sampling error. If a score is a mean of per-question outcomes with sample standard deviation $s$, the standard error is approximately $s/\sqrt{n}$, and a $95\%$ confidence interval is about the mean $\pm\, 1.96\, s/\sqrt{n}$. Miller (2024) argues that evaluation reports should include such error bars and recommends analysing paired differences between models on the same questions, which typically gives tighter comparisons than comparing separate intervals. A difference smaller than the uncertainty cannot support a ranking claim.

#4.4 Check for contamination and overfitting

Contamination makes scores unrepresentative of generalisation. Oren et al. (2023) proposed a method to detect test-set contamination in black-box models by testing whether a model prefers a particular canonical ordering of benchmark examples, which has no need for access to pretraining data or weights. Zhang et al. (2024) built a new benchmark designed to match the difficulty and style of an existing grade-school mathematics set and found evidence that some model families performed worse on the fresh set, consistent with overfitting to the original. Fresh, private or continually refreshed test sets reduce but do not eliminate the problem (see [CAP-CODE-001] for LiveCodeBench).

#4.5 Understand human-preference and judge-based measures

Pairwise human votes aggregated into Bradley-Terry-style ratings are used on platforms such as Chatbot Arena (Chiang et al., 2024). The method collects diverse real prompts but measures aggregate preference, not task-specific correctness, and it is subject to the participation and style biases of its user population. LLM judges scale grading but carry biases including verbosity preference (Zheng et al., 2023; Dubois et al., 2024, who propose a length-controlled regression adjustment to AlpacaEval). Scores from judges and from humans should be calibrated against each other on a sample before one is used as a proxy for the other.

#4.6 Check the governance of the leaderboard

Singh et al. (2025) reported that undisclosed private testing in Chatbot Arena allowed some providers to test multiple variants before release and retract scores, biasing results through selective disclosure; they identified 27 private variants tested by one provider before one release, and estimated that two large providers received 19.2% and 20.4% of all arena data while 83 open-weight models together received an estimated 29.7% (analysis as of the authors' 2025 study; shares will have changed). The general point is that sampling rates, model retirement policy and disclosure rules affect rankings and should be inspected.

Reading a developer announcement. A table showing a new model ahead of a competitor on several benchmarks should be checked for: whether the competitor's number was re-run under identical conditions or copied from another source; whether the new model used a larger sampling or reasoning budget; whether error bars are given; whether any benchmark is likely contaminated; and whether the comparison covers the purpose at hand.

Selecting a model for deployment. Assemble a private task set from real, de-identified workload, run candidates under identical conditions with repeated trials, report confidence intervals and cost, and include at least a small human review sample. Public rankings serve only to shortlist.

Reading a leaderboard. Identify who runs it, how models are sampled, what is disclosed, and whether submissions can be withdrawn (Singh et al., 2025).

The checklist improves defensibility but does not guarantee validity: a perfectly controlled comparison on an invalid benchmark is still uninformative about the intended use (Raji et al., 2021; Reuel et al., 2024 propose criteria for assessing benchmark quality). Unresolved questions include how to compare systems across different compute budgets in a fair way (cost-normalised or budget-matched comparison), how to detect contamination in closed models with certainty (Oren et al., 2023 give a statistical test with stated assumptions, not a proof), and how to aggregate across heterogeneous benchmarks without arbitrary weighting. Whether a single "overall" ordering of general-purpose systems is meaningful at all is contested; HELM's multi-metric design (Liang et al., 2022) is one response in the direction of reporting profiles rather than ranks.

As of 2026-10-15, the cited sources establish that: scores are sensitive to evaluation conditions (Well Supported; Sclar et al., 2023; Alzahrani et al., 2024); statistical uncertainty is often unreported and can be quantified with standard methods (Well Supported; Miller, 2024); contamination and overfitting have been demonstrated for some models on some benchmarks (Preliminary to Well Supported; Zhang et al., 2024; Oren et al., 2023); judge-based and preference-based measures have documented biases and design issues (Well Supported; Zheng et al., 2023; Dubois et al., 2024; Singh et al., 2025). There is no single accepted standard for comparing general-purpose systems; HELM and the open harness described by Biderman et al. (2024) are community efforts, not binding standards.

Observed trend (not a forecast): more evaluation reports include confidence intervals, held-out sets and multi-metric profiles, and independent evaluators (academic groups, safety institutes) increasingly report comparisons in addition to developers. Whether independent evaluation becomes a regular precondition for deployment is a governance matter (see [GOV-LAW-001]) that this article does not forecast.

• Foundational Concepts:
  • CAP-BENCH-001 AI Evaluation, Benchmarking & Contamination: Metrics, benchmark types and contamination.
  • TECH-LLM-001 Large Language Models: The systems usually compared.
• Related Methods & Technologies:
  • CAP-REASON-001 Reasoning, Planning and Long-Horizon Capability: Interpreting reasoning claims.
  • CAP-LIMITS-001 Hallucination, Reliability and Calibration: Reliability and abstention in scoring.
  • CAP-CODE-001 Software Engineering Capability and Evaluation: Domain-specific comparison.
  • CAP-AGENTEVAL-001 Evaluating Autonomous Agents: Agent comparisons.
  • CAP-OPEN-001 Open-Weight versus Closed Models: Comparisons across release approaches.
• Advanced Implications & Future Trajectories:
  • AGI-TEST-001 Testing Claims of General Intelligence: Comparison methodology applied to generality claims.
  • GOV-LAW-001 AI Governance: Evaluation as a regulatory input.
  • MODEL-001 Model Registry: Dated, sourced model comparisons.

An open-weight model is one whose trained parameters (weights) are published for download, usually under a licence, so that others can run, fine-tune and inspect it locally. A closed model is available only through a service interface (an API or product) controlled by its developer. "Open-weight" is narrower than "open-source": the Open Source Initiative's Open Source AI Definition 1.0 requires, in addition to parameters, sufficiently detailed information about the training data and the complete training and inference code (Open Source Initiative, 2024).

Release strategy determines who can use a model, modify it, audit it and misuse it. The choice therefore bears on research reproducibility, market competition, security and regulation. Public debate often treats "open versus closed" as binary, but release is a spectrum, and the terms are used loosely (and sometimes strategically). This article defines the terms, summarises the main arguments and evidence, and notes where law and policy treat the categories differently. It does not rank current models; for dated model facts, see MODEL-001.

Machine learning research has long released code and trained models. In language modelling, a period in which leading developers withheld weights of their largest models while describing them in papers was followed by releases of weights such as LLaMA (Touvron et al., 2023a) and Llama 2 (Touvron et al., 2023b), the latter described by its authors as open foundation and fine-tuned chat models. Other developers released models under permissive licences, for example Mistral 7B (Jiang et al., 2023), and fully open efforts such as OLMo aimed to release not only weights but also data, code and evaluation tooling (Groeneveld et al., 2024). DeepSeek-AI released the DeepSeek-V3 technical report (DeepSeek-AI, 2024) and the R1 reasoning model, whose report was published in Nature in 2025 (Guo et al., 2025), demonstrating that openly described reasoning-model training was feasible outside the largest closed developers.

Evaluation of openness itself developed in parallel. The Foundation Model Transparency Index scored developers on disclosure of upstream resources, model properties and downstream use (Bommasani et al., 2023). Liesenfeld and Dingemanse (2024) assessed claims of openness in generative AI and argued that "open washing" exists, where partial releases are marketed as open source, with implications for the EU AI Act. The Open Source Initiative published its definition in 2024.

#4.1 A spectrum of release

Seger et al. (2023) describe a gradient of release options from fully closed to fully open, including staged release, hosted access, structured API access with safeguards and full open release, and discuss alternative ways of pursuing the aims of open-sourcing without releasing weights. The relevant artefacts are separable:

The table is a summary of the definitions cited, not a claim about any particular model. The OSI definition lists three components of the preferred form for modification: data information, code and parameters (Open Source Initiative, 2024).

#4.2 What weight access enables

Possessing weights allows local inference without data leaving the user's infrastructure, fine-tuning for specialised tasks, quantisation and compression, inspection of internal activations for research, and long-term availability independent of the developer's service. Kapoor et al. (2024) identify five distinctive properties of open foundation models, including greater customisability and poor monitoring, and argue that these properties lead to both benefits and risks.

#4.3 What closed access enables

A closed service allows the developer to apply safeguards, monitor misuse, update the model, withdraw access and charge per use. The cost is dependence: users cannot audit the weights, are subject to changes and deprecations, and must send their data to the provider.

• Closed / API-only: weights withheld; interaction through a service.
• Open-weight, restrictive licence: weights downloadable but with conditions on use or scale, which fails the Open Source Definition's freedom to use for any purpose.
• Open-weight, permissive licence: weights under a licence such as Apache 2.0 (Jiang et al., 2023, state that Mistral 7B is released under Apache 2.0).
• Open-source AI (OSI): weights, code and data information under OSI-approved terms (Open Source Initiative, 2024).
• Hybrid: smaller models open, largest closed; or weights released after a delay.

Classification of a specific model into these categories requires reading its licence and release materials. The licence of a given model version is a fact to be checked at the source, not inferred from marketing.

Open-weight models are used for on-premises deployment where data cannot leave an organisation, for research on interpretability and training methods, for fine-tuned domain models and for low-cost inference. Closed models are widely used through APIs and consumer products where operators prefer a managed service and developer-provided safeguards. A single organisation may use both. Open-weight releases such as Llama 2, Mistral 7B and OLMo are cited here as representative examples of a type, not as statements about current capability or licensing, which change between versions.

Safety and misuse. Qi et al. (2023) showed that fine-tuning aligned language models on a small number of adversarial examples can remove safety behaviour, and that even benign fine-tuning can degrade it. This was demonstrated for both an open-weight model (Llama-2-7b-chat) and a closed model offered with a fine-tuning API (GPT-3.5 Turbo), so it does not by itself separate the two cases. For open weights, however, the developer cannot recall a release or enforce usage policies technically once weights are downloaded. Seger et al. (2023) assess risks and benefits of releasing highly capable models and discuss alternatives. Kapoor et al. (2024) argue that the evidence on the marginal risk of open models, compared with existing technology and closed models, should be assessed per risk vector, and that much concern lacks empirical backing; they present this as a position paper. Both sides cite the same missing evidence: systematic, replicated studies of real-world misuse uplift are few. The question is therefore Contested.

Competition and concentration. Open weights lower barriers to entry and allow independent audit; closed models allow developers to recoup training cost and apply safeguards. The cost of frontier training is covered in [ECON-COMP-001]; the sources cited here do not quantify the net effect on market concentration.

Evaluation fairness. Comparisons between open and closed models are affected by access asymmetries; for example, Singh et al. (2025) reported that in Chatbot Arena in their study, an estimated 29.7% of data went to a combined 83 open-weight models, against 19.2% and 20.4% for two large proprietary providers each (2025 estimates; see [CAP-COMPARE-001]).

Terminology. Calling a model "open source" when training data or code are withheld contradicts the OSI definition; Liesenfeld and Dingemanse (2024) discuss this as open washing.

As of 2026-10-15: multiple developers have released open-weight models of significant capability, with a range of licences and disclosure levels (Well Supported; Touvron et al., 2023b; Jiang et al., 2023; Groeneveld et al., 2024; Guo et al., 2025). A formal definition of open-source AI exists and is stricter than weight release (Established as a published definition; adoption is voluntary). The magnitude of any capability gap between the best open-weight and best closed models varies with date, domain and measurement; this article gives no figure and defers to MODEL-001 and the cautions in [CAP-COMPARE-001]. The balance of risks and benefits of open weights for highly capable models is Contested.

Law and policy (checked against the Act's text as published on a mirror of the Regulation, see references). Regulation (EU) 2024/1689 (the EU AI Act) states in Article 2(12) that it does not apply to AI systems released under free and open-source licences unless they are placed on the market or put into service as high-risk systems or are covered by certain other provisions. Article 53(2) exempts providers of general-purpose AI models released under a free and open-source licence, with publicly available parameters, architecture information and usage information, from some documentation obligations, but this exception does not apply to general-purpose AI models with systemic risk. Chapter V on general-purpose AI models applies from 2 August 2025 under Article 113(b) as shown at the cited source; the Act has been subject to amendment proposals and the current application dates should be checked at the official EU source [VERIFICATION NEEDED: current in-force text and dates of Articles 2, 53 and 113 against EUR-Lex]. On United States policy, Executive Order 14110 was revoked on 2025-01-20 [VERIFICATION NEEDED: primary source for revocation; stated in the project brief]. See [GOV-LAW-001].

Observed pattern, not a forecast: release strategies are mixed, with developers releasing some models open and keeping others closed, and with the definitions of "open" contested (Open Source Initiative, 2024; Liesenfeld & Dingemanse, 2024). The direction of regulation, which treats openness as a partial exemption but not for systemic-risk models in the EU, is documented above; whether other jurisdictions adopt comparable carve-outs is open. This article registers no forecast.

• Foundational Concepts:
  • TECH-LLM-001 Large Language Models: The model class to which release choices apply.
  • CAP-GENAI-001 Generative AI: Open and closed generative systems across modalities.
• Related Methods & Technologies:
  • CAP-COMPARE-001 Methodology for Comparing AI Systems: Fair comparison across release types.
  • ECON-COMP-001 Compute Economics: Training cost and release incentives.
  • SAFE-SEC-001 AI Security: Prompt Injection, Poisoning, Extraction: Security consequences of weight access.
• Advanced Implications & Future Trajectories:
  • GOV-LAW-001 AI Governance: Legal treatment of open models.
  • SAFE-ALIGN-001 The AI Alignment Problem: Safety behaviour persistence after release.
  • MODEL-001 Model Registry: Dated model facts and licences.