The Turing Test and How AI Is Measured Today

For the encyclopedia's full treatment see AI evaluation and benchmarks and Testing claims of general intelligence. Terms are defined in the glossary.

#Quick reference

#What is the Turing test?

In "Computing Machinery and Intelligence", published in Mind in October 1950, Turing opens: "I propose to consider the question, 'Can machines think?'" He then sets it aside as "too meaningless to deserve discussion" and replaces it with a game. An interrogator questions two hidden players by text, one human and one machine, and tries to say which is which.

Turing also made a dated forecast. He believed that in about fifty years computers could be programmed to play the game so well that an average interrogator would have no more than a 70 per cent chance of making the right identification "after five minutes of questioning." That points to around 2000. See Alan Turing for his wider work.

#Has any AI passed the Turing test?

One study says yes. In a March 2025 preprint, Cameron Jones and Benjamin Bergen ran two randomised, pre-registered three-party tests with five-minute conversations. GPT-4.5, prompted to adopt a humanlike persona, was judged to be the human 73% of the time, more often than the real human participants. LLaMa-3.1-405B with the same prompt reached 56%; ELIZA and GPT-4o scored 23% and 21%. The authors call this "the first empirical evidence that any artificial system passes a standard three-party Turing test."

Treat this as Preliminary: it is a single, non-peer-reviewed study, and the result depended on the persona prompt. It also shows the test's weakness. Passing it says that people were fooled, which is not the same as showing general capability.

#What are the limits of the Turing test?

• It rewards imitation, not ability. Joseph Weizenbaum's ELIZA (1966), a scripted chatbot with no knowledge of the world, was still picked as the human 23% of the time in the 2025 study. A system can score by seeming human, including by imitating human mistakes.
• It depends on the judge and the setup. Five minutes with a naive interrogator is a different test from an hour with an expert. Turing gave no fixed protocol for "passing."
• It says nothing about understanding. John Searle's "Chinese room" argument (1980) holds that a program could produce the right answers without understanding anything. Whether behaviour can ever show understanding is Contested; see Consciousness and machine understanding.
• It is one number for a many-sided question. It does not separate knowledge, reasoning, coding or planning, which is what developers and regulators need to know.

These limits are why "Is it AGI?" is now asked with task-based definitions; see What is AGI? Definitions compared.

#What replaced the Turing test?

Benchmarks: published sets of tasks with automatic scoring, so different systems can be compared on the same items.

MMLU (Measuring Massive Multitask Language Understanding; Hendrycks et al., posted September 2020, ICLR 2021) covers 57 subjects, from elementary mathematics to law. At launch most models were near random chance, and the largest GPT-3 model beat chance by almost 20 percentage points.

ARC-AGI comes from François Chollet's 2019 paper "On the Measure of Intelligence," which defines intelligence as "skill-acquisition efficiency" and argues that skill on known tasks can be bought with data. Its puzzles require inferring a rule from a few input-output grids. ARC Prize reports that the best score on its private evaluation set rose from 33% to 55.5% during 2024, against a prize target of 85%. ARC-AGI-2 (May 2025) adds harder tasks, and ARC-AGI-3 is an "interactive reasoning benchmark" built from games that agents must learn by exploring.

SWE-bench (Jimenez et al., October 2023, ICLR 2024) asks a model to resolve 2,294 real GitHub issues from 12 Python repositories, scored by running each repository's tests. The best model at launch, Claude 2, solved 1.96%. SWE-bench Verified is "a human-validated subset of 500" instances, created with OpenAI. Stanford's 2026 AI Index reports that performance on it "rose from 60% to near 100% in a single year." See Software engineering capability and evaluation.

Humanity's Last Exam (Phan et al., January 2025; Nature, January 2026) has 2,500 multiple-choice and short-answer questions written by subject experts, designed so that answers "cannot be quickly answered via internet retrieval." Its authors built it because LLMs "now achieve over 90% accuracy on popular benchmarks like MMLU."

#Why do benchmarks stop working?

Contamination is hard to prove from outside because training data is usually private. Oren et al. (2023) showed a statistical test that can flag it without the data, by checking whether a model prefers a benchmark's original ordering, and Sainz et al. (2023) called for contamination to be measured for every benchmark. The Yang et al. study also found that simple rephrasing gets past n-gram decontamination: a 13B-parameter model trained on rephrased test items matched GPT-4 on benchmark scores. See Training data, contamination and provenance.

The practical rule: read any benchmark score with its date, version, test-time setup and contamination checks. A high score on a saturated or leaked test is weak evidence; a large gain on a fresh, held-out one is stronger. The same caution applies to why AI hallucinates: models that score well can still fail on questions outside the test.

#Frequently asked questions

#What exactly is the Turing test?

A text conversation in which a judge tries to tell a machine from a human. Turing proposed it in 1950 as a replacement for the question "Can machines think?"

#Did ChatGPT pass the Turing test?

A March 2025 preprint reported that GPT-4.5, given a humanlike persona, was judged human 73% of the time in five-minute tests. GPT-4o without that persona scored 21%. The result is Preliminary.

#Why is MMLU no longer enough?

Leading models score over 90% on it, so it no longer separates them, and its public questions may have leaked into training data.

#What is benchmark contamination?

It happens when test questions, or close paraphrases, are in a model's training data. Scores then measure memorisation as well as ability.

#Which benchmark best measures general intelligence?

None is agreed. ARC-AGI targets learning efficiency on novel tasks, but whether any fixed test can measure general intelligence is Contested.

#Sources

• Turing, A. M.: Computing Machinery and Intelligence (Mind, 59(236), 433-460, 1950)
• Jones, C. R., Bergen, B. K.: Large Language Models Pass the Turing Test (arXiv preprint, 2025)
• Weizenbaum, J.: ELIZA: A Computer Program for the Study of Natural Language Communication Between Man and Machine (Communications of the ACM, 9(1), 1966)
• Searle, J. R.: Minds, brains, and programs (Behavioral and Brain Sciences, 3(3), 1980)
• Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., Steinhardt, J.: Measuring Massive Multitask Language Understanding
• Chollet, F.: On the Measure of Intelligence
• Chollet, F., Knoop, M., Kamradt, G., Landers, B.: ARC Prize 2024: Technical Report
• Chollet, F., Knoop, M., Kamradt, G., Landers, B., Pinkard, H.: ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems
• ARC Prize: ARC-AGI-3
• Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., Narasimhan, K.: SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• SWE-bench team: SWE-bench Verified
• Aleithan, R., Xue, H., Mohajer, M. M., Nnorom, E., Uddin, G., Wang, S.: SWE-Bench+: Enhanced Coding Benchmark for LLMs
• Phan, L., Gatti, A., Han, Z., Li, N., et al.: Humanity's Last Exam (arXiv); Center for AI Safety, Scale AI and HLE Contributors: A benchmark of expert-level academic questions to assess AI capabilities (Nature, 649, 2026)
• Yang, S., Chiang, W.-L., Zheng, L., Gonzalez, J. E., Stoica, I.: Rethinking Benchmark and Contamination for Language Models with Rephrased Samples
• Oren, Y., Meister, N., Chatterji, N., Ladhak, F., Hashimoto, T. B.: Proving Test Set Contamination in Black Box Language Models
• Sainz, O., Campos, J. A., García-Ferrero, I., Etxaniz, J., Lopez de Lacalle, O., Agirre, E.: NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
• Stanford HAI: The 2026 AI Index Report

Status: draft, reviewed 26 October 2026 against the primary sources above. Benchmark scores change monthly and are deliberately left out except where a dated source reports them; the Nature version of HLE was not read in full.