Seventy-five years of edits

Descartes argues language use, not action, distinguishes real men from machines

René Descartes

In Part V of Discourse on the Method, Descartes considers hypothetical machines built to resemble and imitate human bodies as closely as possible, and argues two tests would still reveal they were not real men: they could never flexibly arrange words or signs to reply appropriately to whatever was said in their presence, and they would act only from the disposition of their organs rather than understanding, so they would fail outside their built-in repertoire. This is a philosophical precursor to the Turing Test, not a test protocol itself: Descartes proposes flexible, situation-appropriate language use as a marker separating genuine minds from mechanism more than 300 years before Turing's 1950 imitation game.

Of these the first is that they could never use words or other signs arranged in such a manner as is competent to us in order to declare our thoughts to others: for we may easily conceive a machine to be so constructed that it emits vocables, and even that it emits some correspondent to the action upon it of external objects which cause a change in its organs; for example, if touched in a particular place it may demand what we wish to say to it; if in another it may cry out that it is hurt, and such like; but not that it should arrange them variously so as appositely to reply to what is said in its presence, as men of the lowest grade of intellect can do.

René Descartes, Discourse on the Method, Part V, John Veitch translation

A Discourse on Method, Project Gutenberg, trans. John Veitch, 1637

Turing predicts the imitation game will be winnable 30% of the time by 2000

Alan Turing

Turing predicts that in about fifty years (i.e., by roughly 2000), computers will play the imitation game well enough that an average interrogator will have no more than a 70 percent chance of correctly identifying the human after five minutes of questioning. Because this figure is the interrogator's success rate, it is arithmetically equivalent to the machine successfully passing as human in up to 30 percent of five-minute sessions against an average interrogator. This is the paper's own baseline spec: a specific time frame, a specific session length, and a specific success threshold, against which every later claim of the test being "passed" can be compared.

I believe that in about fifty years' time it will be possible, to programme computers, with a storage capacity of about 109, to make them play the imitation game so well that an average interrogator will not have more than 70 per cent chance of making the right identification after five minutes of questioning.

Alan Turing, Computing Machinery and Intelligence, Mind 59(236), Section 6 (Contrary Views on the Main Question)

Computing Machinery and Intelligence, Mind, 1950-10

Turing dismisses "Can machines think?" as too meaningless to discuss

Alan Turing

Turing states plainly that he considers the original question, "Can machines think?", too meaningless to deserve discussion, and offers the imitation game as the substitute question the rest of the paper actually investigates. This line establishes that the imitation game was never meant as a literal test of "thinking" itself, but as a deliberately narrower, more tractable replacement question. Any later claim to have settled whether machines "really" think is answering a question Turing explicitly declined to engage with.

The original question, "Can machines think?" I believe to be too meaningless to deserve discussion.

Alan Turing, Computing Machinery and Intelligence, Mind 59(236), Section 6 (Contrary Views on the Main Question)

Computing Machinery and Intelligence, Mind, 1950-10

Turing explains why the test runs over a teleprinter, not in person

Alan Turing

Turing explains that restricting the game to written/typewritten communication (a teleprinter link between rooms, per Section 1) is a deliberate design choice: it draws a sharp line between physical and intellectual capacities so the machine is judged only on the latter. He illustrates this with the analogy of artificial skin, arguing that even a perfect physical disguise would be beside the point, since the game's condition already prevents the interrogator from seeing, touching, or hearing the competitors. This establishes that the original test was never meant to evaluate embodiment or physical humanlikeness, only text-based conversational performance.

The new problem has the advantage of drawing a fairly sharp line between the physical and the intellectual capacities of a man. No engineer or chemist claims to be able to produce a material which is indistinguishable from the human skin. It is possible that at some time this might be done, but even supposing this invention available we should feel there was little point in trying to make a "thinking machine" more human by dressing it up in such artificial flesh. The form in which we have set the problem reflects this fact in the condition which prevents the interrogator from seeing or touching the other competitors, or hearing -their voices.

Alan Turing, Computing Machinery and Intelligence, Mind 59(236), Section 2 (Critique of the New Problem)

Computing Machinery and Intelligence, Mind, 1950-10

Weizenbaum publishes ELIZA, and some users cannot tell it isn't human

Joseph Weizenbaum

Joseph Weizenbaum published ELIZA, a natural-language program built at MIT that matched key words in a user's typed input against decomposition rules and generated responses through associated reassembly rules. Its best-known script, later called DOCTOR, imitated a Rogerian psychotherapist so the program could hold a conversation without needing any real model of the world. The paper reported that some users found it very hard to accept they were talking to a program rather than a person, which Weizenbaum called a striking, informal form of the Turing test, without claiming the program understood anything said to it.

ELIZA is a program operating within the MAC time-sharing system at MIT which makes certain kinds of natural language conversation between man and computer possible.

Joseph Weizenbaum, "ELIZA—A Computer Program For the Study of Natural Language Communication Between Man And Machine," Communications of the ACM 9(1), abstract

ELIZA—A Computer Program For the Study of Natural Language Communication Between Man And Machine, Communications of the ACM 9(1), pp. 36–45 (scanned reprint, University at Buffalo CSE course mirror; ACM DL copy of record 403s automated fetches), 1966-01

Psychiatrists can't reliably tell PARRY's transcripts from real patients'

Kenneth Colby, Franklin Hilf, Sylvia Weber, Helena Kraemer

In 1972, Kenneth Colby, Franklin Hilf, Sylvia Weber, and Helena Kraemer published a validation study for PARRY, Colby's computer simulation of paranoid patient behavior, using a Turing-like indistinguishability design: experienced psychiatrists conducted teletyped interviews with both real paranoid patients and the PARRY program without being told which was which, and a separate panel of psychiatrists later reviewed transcripts of these interviews. Judges asked to identify which interviewee was the computer program performed at essentially chance level; in one reported test, only 48% of judgments correctly identified the program versus the human patient.

Only 48% made the correct identification, which is also a chance level.

Ayse Pinar Saygin, Ilyas Cicekli & Varol Akman, "Turing Test: 50 Years Later," Minds and Machines 10(4) (2000), p. 501, describing the results of Colby et al. (1972)

Turing Test: 50 Years Later, Minds and Machines 10(4), pp. 463-518 (author's copy, Saygin's UCSD site), 2000

Weizenbaum recoils from how readily people trusted ELIZA

Joseph Weizenbaum

After ELIZA's DOCTOR script drew an intense, emotionally invested response from ordinary users, including his own secretary, who asked him to leave the room so she could talk to it privately, and after some psychiatrists proposed developing it into an automated therapy tool, Weizenbaum grew alarmed rather than encouraged. In his 1976 book Computer Power and Human Reason: From Judgment to Calculation, he argued that a program's ability to produce convincing conversational output is not evidence that it understands anything, and warned against treating conversational fluency as a substitute for human judgment. He went on to become one of the most prominent critics of the idea that fooling people in conversation demonstrates machine intelligence.

What I had not realized is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people

Joseph Weizenbaum, Computer Power and Human Reason (1976), p. 7, as quoted by Smithsonian Magazine

the bar moved
ELIZA's conversational fluency had already impressed ordinary users and some psychiatrists as human-like and potentially therapeutic, and Weizenbaum then argued that surface fluency is not evidence of understanding at all, regardless of how convincing the output seemed.

Why Joseph Weizenbaum Invented the Eliza Chatbot, Smithsonian Magazine, 2023

Searle: passing the Turing test doesn't settle whether a program understands

John Searle

In "Minds, Brains, and Programs" (1980), John Searle proposed the Chinese Room thought experiment: a person who knows no Chinese sits in a room following an English-language rulebook to manipulate Chinese symbols, producing responses indistinguishable from those of a fluent Chinese speaker to anyone outside the room. Searle argues the person still understands no Chinese, and that the same holds for any digital computer running a program with equivalent input-output behavior, because purely formal symbol manipulation (syntax) is not by itself sufficient to produce understanding or meaning (semantics), however convincing the external behavior appears. He extends this directly to the Turing test, arguing that a system's ability to pass it does not settle, and in fact begs, the question of whether it understands.

The only motivation for saying there must be a subsystem in me that understands Chinese is that I have a program and I can pass the Turing test; I can fool native Chinese speakers. But precisely one of the points at issue is the adequacy of the Turing test.

John Searle, "Minds, Brains, and Programs," Behavioral and Brain Sciences 3(3) (1980), pp. 417-457 (unedited penultimate draft)

the bar moved
Before Searle, convincing conversational behavior (imitation-game-style) was widely treated as evidence of understanding or thinking; Searle argued that behavioral indistinguishability is irrelevant because syntax is not semantics, so no purely behavioral test, however well passed, can establish genuine understanding.

Minds, Brains, and Programs, Behavioral and Brain Sciences 3(3), 1980 (unedited penultimate draft, originally circulated via bbsonline.org for BBS open peer commentary; hosted as a course reading by Tufts University), 1980

French: passing proves only "culturally-oriented" intelligence, not general intelligence

Robert M. French

In "Subcognition and the Limits of the Turing Test" (Mind, Vol. 99, No. 393, pp. 53-65), French accepts Turing's philosophical claim (passing the test would prove intelligence) but attacks his pragmatic claim, arguing the test can be passed via 'subcognitive' associative-priming questions that only reveal low-level structure acquired by having lived a human, culturally-embedded life -- not general intelligence.

the Test provides a guarantee not of intelligence but of culturally-oriented human intelligence

Robert M. French, "Subcognition and the Limits of the Turing Test," Mind 99(393), 1990

the bar moved
Where the implicit Turing framing held that behaviorally passing the test would itself be sufficient evidence of general intelligence, French argues passing is achievable by exploiting 'subcognitive' associative shortcuts that only require a culturally human life-history, not general cognition, so a pass demonstrates only culturally-oriented human intelligence rather than intelligence in general.

Subcognition and the Limits of the Turing Test (full text, Robert M. French), Texas A&M University course mirror (CSE/Robotics, cs625), reproducing the Mind (1990) 99(393):53-65 text, 1990-01

The first Loebner Prize restricts topics "to give the computer a better chance"

Hugh Loebner / Cambridge Center for Behavioral Studies

The first Loebner Prize competition was held November 8, 1991 at The Computer Museum in Boston, administered by the Cambridge (MA) Center for Behavioral Studies with funding from Hugh Loebner (President, Crown Industries). Instead of Turing's open-ended format, each of the ten entrant systems (six computer programs vs. two hidden human confederates, with equipment cutting the field to eight terminals on the day) was restricted to a specific topic (e.g., romantic relationships, Shakespeare's plays, Burgundy wines) 'to give the computer a better chance.' Joseph Weintraub's PC Therapist won, fooling 5 of 10 judges; separately, four of the six programs fooled at least one judge, and two judges thought one of the human confederates was a computer.

Instead of being open-ended-the way Turing intended- this first round gave the computer a better chance by limiting "conversations" to topics such as romantic relationships, Shakespeare's plays and Burgundy wines.

The Computer Museum member newsletter, on the 1991 Loebner Prize Competition

the bar moved
Restricting each conversation to a single assigned topic narrowed the test from Turing's open-domain interrogation specifically, and explicitly, 'to give the computer a better chance' -- redefining the test to fit 1991 technology rather than waiting for technology capable of passing Turing's original unrestricted test.

The Computer Museum newsletter coverage of the 1991 Loebner Prize Competition (preview + winner recap), The Computer Museum (Boston) member newsletter, archived by the Computer History Museum, 1991-1992

Shieber: a restricted Turing test "no longer serves its original purpose"

Stuart Shieber

In "Lessons from a Restricted Turing Test" (Communications of the ACM, Vol. 37, No. 6, 1994, pp. 70-78), Shieber -- who observed the Loebner competition -- argues the contest's topic-restricted format has no clear scientific purpose and, by limiting the test, breaks the logical structure ('syllogism') that gave Turing's original unrestricted test its evidential force for intelligence.

By limiting the test, it no longer serves its original purpose (and arguably no purpose at all), as Turing’s syllogism fails.

Stuart M. Shieber, "Lessons from a Restricted Turing Test," Communications of the ACM 37(6), 1994

the bar moved
Shieber argues that restricting topic and conversational scope, as the Loebner contest did, does not yield a harder or easier version of Turing's test but a fundamentally different and much weaker one, so Loebner-style 'passing' results are not valid evidence toward the real, unrestricted Turing Test.

Lessons from a Restricted Turing Test (Stuart M. Shieber, self-archived preprint), arXiv (cmp-lg/9404002), self-archived version of the CACM 37(6):70-78 (1994) article, 1994-06

University of Reading declares Eugene Goostman has passed the Turing Test

University of Reading / Kevin Warwick

At an event held at the Royal Society in London on June 7, 2014 -- organized by the University of Reading's School of Systems Engineering with RoboLaw, and led by Professor Kevin Warwick, on the 60th anniversary of Alan Turing's death -- the chatbot 'Eugene Goostman' (a program simulating a 13-year-old Ukrainian boy, created by Vladimir Veselov and Eugene Demchenko) reportedly convinced 33% of 30 human judges, across five-minute keyboard conversations, that it was human. The University of Reading's press release presented this as clearing the 30% figure it attributed to Turing's own prediction, and quoted Warwick declaring 'Alan Turing's Test was passed for the first time.'

Eugene managed to convince 33% of the human judges (30 judges took part - see more details below) that it was human.

University of Reading press release, June 8, 2014

Turing Test success marks milestone in computing history, University of Reading (archived press release), 2014-06-08

Scott Aaronson publishes his own unconvincing conversation with Goostman

Scott Aaronson

Within days of the University of Reading's announcement, computer scientist Scott Aaronson (UT Austin) published a transcript of his own five-minute conversation with the same Eugene Goostman chatbot on his blog Shtetl-Optimized, under the title 'My Conversation with "Eugene Goostman," the Chatbot that's All Over the News for Allegedly Passing the Turing Test.' The transcript shows Goostman giving evasive, nonsensical answers to simple factual questions (e.g., on how many legs a camel has: 'Something between 2 and 4. Maybe, three?'). Aaronson argued Turing never specified a 30% numerical pass threshold and placed blame for the 'passed' framing on Warwick's promotion and uncritical press coverage.

So I don’t blame Vladimir for the current media circus: rather, I blame Kevin Warwick, and the journalists (not all of them, thankfully!) who uncritically repeated Warwick’s pronouncements.

Scott Aaronson, Shtetl-Optimized blog, June 9, 2014

the bar moved
The University of Reading's press claim treated the 30%/33% threshold as sufficient for 'passing' the Turing Test, but critics immediately argued that Goostman's young, non-native-English-speaking persona artificially lowered interrogator expectations, so clearing the numeric threshold this way didn't count as genuinely passing Turing's test.

My Conversation with “Eugene Goostman,” the Chatbot that’s All Over the News for Allegedly Passing the Turing Test, Shtetl-Optimized (Scott Aaronson's blog), 2014-06-09

Gary Marcus: passing the Turing Test "means little"

Gary Marcus

In The New Yorker (published June 9, 2014), cognitive scientist Gary Marcus (NYU) argued that Eugene Goostman's win reflected clever conversational misdirection descended from the 1960s ELIZA program rather than genuine machine intelligence, calling the chatbot's tactics 'ploys' that mask its limitations, and concluding that clearing the Turing Test threshold does not demonstrate real progress toward artificial intelligence.

In terms of practical significance for artificial intelligence, though, passing the Turing Test means little.

Gary Marcus, "What Comes After the Turing Test?", The New Yorker, June 9, 2014

the bar moved
The University of Reading's press claim treated the 30%/33% threshold as sufficient for 'passing' the Turing Test, but critics immediately argued that Goostman's young, non-native-English-speaking persona artificially lowered interrogator expectations, so clearing the numeric threshold this way didn't count as genuinely passing Turing's test.

What Comes After the Turing Test?, The New Yorker, 2014-06-09

Google commits to AI self-disclosure after the Duplex backlash

Google

At Google I/O on May 8, 2018, Google demoed 'Duplex,' an AI system that places phone calls to businesses (a hair salon, a restaurant) and converses naturally enough -- including speech disfluencies like 'um' and 'mm-hm' -- that the human callee did not appear to realize they were talking to an AI; Google's own blog post framed this as a triumph of natural-sounding conversation, and it drew immediate criticism over the ethics of an AI passing as human without disclosing itself. Two days later, a Google spokeswoman told press that disclosure was being built into the feature and that Duplex would be 'appropriately identified,' though Google said it had not yet decided exactly how that disclosure would work.

We are designing this feature with disclosure built-in, and we'll make sure the system is appropriately identified. What we showed at I/O was an early technology demo, and we look forward to incorporating feedback as we develop this into a product.

Google spokesperson, quoted by TechSpot, May 2018

the bar moved
Duplex was demoed on stage as a success specifically because it was indistinguishable from a human caller (the Turing criterion applied approvingly), but the backlash inverted that framing -- indistinguishability without disclosure became the problem rather than the achievement -- and Google's response established a norm that an AI must identify itself as automated.

Google responds to Duplex backlash: AI voice system will identify itself, TechSpot, 2018-05-10

The Loebner Prize holds its last contest

AISB

The Loebner Prize ran annually from 1991. Founder Hugh Loebner died in December 2016, and AISB, which had run the contest since 2014, held what became its final edition on September 12–15, 2019 at Swansea University, folded into the AISB-X public event, with Steve Worswick's Mitsuku winning for a fifth time. No contest was held in 2020 or in any year since, and AISB's own Loebner Prize announcements page has posted nothing past that 2019 result.

Mitsuku wins 2019 Loebner Prize and Best Overall Chatbot at AISB X, AISB (The Society for the Study of Artificial Intelligence and Simulation of Behaviour), 2019-09-15

Blake Lemoine claims Google's LaMDA is sentient

Blake Lemoine

In June 2022, Google engineer Blake Lemoine publicly claimed the company's LaMDA chatbot was sentient, based on his own conversations with it. He defended leaking transcripts to the Washington Post on the grounds that he could recognize personhood regardless of whether it came from a brain or code. Google placed him on administrative leave shortly after and fired him that September. Google's review of his concerns “per our AI Principles” concluded the evidence did not support his claims.

I know a person when I talk to it. It doesn’t matter whether they have a brain made of meat in their head. Or if they have a billion lines of code.

Blake Lemoine, quoted in The Washington Post, June 11, 2022

Google's Artificial Intelligence App LaMDA Believes It's Human With Real Thoughts & Feelings, 1053rnb.com, reproducing Nitasha Tiku, “The Google engineer who thinks the company's AI has come to life,” The Washington Post, June 11, 2022 (washingtonpost.com 403s automated fetches), 2022-06-11

Nature: “ChatGPT broke the Turing test”

Celeste Biever / Nature

In this Nature News Feature (25 July 2023, Nature 619, 686-689), science journalist Celeste Biever reports that ChatGPT-class large language models have broken the classic Turing test -- their conversation is now indistinguishable from a human's to many observers -- and that AI researchers, unsettled by this, are scrambling to build new tests that probe reasoning and understanding instead of mere conversational mimicry.

ChatGPT broke the Turing test — the race is on for new ways to assess AI

Celeste Biever, Nature News Feature, July 25, 2023

the bar moved
Declares LLMs have outright broken/passed the conversational Turing test, then surveys a research community that responds not by accepting this as proof of intelligence but by racing to invent stricter replacement benchmarks -- moving the goalposts for what counts as evidence of real machine reasoning rather than conceding the original test's verdict.

ChatGPT broke the Turing test — the race is on for new ways to assess AI, Nature 619, pp. 686–689, 2023-07-25

Johnson-Laird and Ragni: passing the test doesn't mean machines think

Philip Johnson-Laird and Marco Ragni

Cognitive scientists Philip Johnson-Laird (Princeton) and Marco Ragni (Chemnitz University of Technology) argue in this commentary, published in the open-access AAAS journal Intelligent Computing on November 10, 2023, that although chatbots and other AI tools pass the Turing test, this does not mean the machines think. They propose replacing it with a test that asks whether a program reasons the way humans reason, run like a psychological study of the program's inferences and self-understanding.

Despite their success in passing the Turing test, these machines do not think.

Philip Johnson-Laird and Marco Ragni, "What Should Replace the Turing Test?", Intelligent Computing, November 10, 2023

the bar moved
Concedes that chatbots now pass the conversational Turing test, then declares that passing insufficient to establish thought, and proposes a new three-step test of human-like reasoning as the real bar an 'intelligent' program must clear.

What Should Replace the Turing Test?, Intelligent Computing (AAAS), 2023-11-10

Bayne and Williams: whether LLMs passed the test is "moot"

Tim Bayne and Iwan Williams

Philosophers Tim Bayne and Iwan Williams argue in this Nature Human Behaviour piece (volume 7, pages 1806-1807, 2023) that the Turing test cannot serve as a meaningful benchmark for thought in large language models, because Alan Turing's original description left too many of the test's operational parameters unspecified to ever yield a decisive verdict.

The question of whether LLMs have passed the Turing test is moot, as Turing left many of the game’s parameters unspecified — including how many questions the interrogator is allowed to ask (50? 100? 1,000?)

Tim Bayne and Iwan Williams, Nature Human Behaviour 7, pp. 1806–1807, 2023

the bar moved
Rather than concede the test as originally described has been passed, argues Turing's original specification was too underspecified (question count, interrogator's background knowledge, etc.) to ever count as decisively 'passed' -- shifting the goalpost to demand a level of procedural rigor the original 1950 imitation game never had.

The Turing test is not a good benchmark for thought in LLMs, Nature Human Behaviour 7, pp. 1806–1807, 2023-11

People cannot distinguish GPT-4 from a human in a Turing test

Cameron R. Jones and Benjamin K. Bergen

Jones and Bergen ran a randomized, controlled, preregistered two-party Turing test in which human participants had a 5-minute text conversation with either a human or an AI and then judged whether their interlocutor was human. GPT-4, prompted with a humanlike persona, was judged human 54% of the time, ahead of ELIZA (22%) but behind actual human participants (67%). This is explicitly a two-party design (one judge, one hidden interlocutor per session) rather than the three-party, simultaneous-comparison format Turing's 1950 paper specifies and that the 2025 Jones & Bergen study below uses; the abstract itself frames the result as the first demonstration of passing "an interactive 2-player Turing test," a narrower claim than a three-party pass.

We evaluated 3 systems (ELIZA, GPT-3.5 and GPT-4) in a randomized, controlled, and preregistered Turing test. Human participants had a 5 minute conversation with either a human or an AI, and judged whether or not they thought their interlocutor was human. GPT-4 was judged to be a human 54% of the time, outperforming ELIZA (22%) but lagging behind actual humans (67%). The results provide the first robust empirical demonstration that any artificial system passes an interactive 2-player Turing test. The results have implications for debates around machine intelligence and, more urgently, suggest that deception by current AI systems may go undetected. Analysis of participants' strategies and reasoning suggests that stylistic and socio-emotional factors play a larger role in passing the Turing test than traditional notions of intelligence.

Cameron R. Jones and Benjamin K. Bergen, "People cannot distinguish GPT-4 from a human in a Turing test," arXiv:2405.08007, abstract

People cannot distinguish GPT-4 from a human in a Turing test, arXiv (preprint, 2405.08007), 2024-05-09

Large Language Models Pass the Turing Test

Cameron R. Jones and Benjamin K. Bergen

In a follow-up study, Jones and Bergen ran two randomized, controlled, pre-registered three-party Turing tests -- matching Turing's original 1950 format, where an interrogator judges a human and a hidden competitor simultaneously rather than one interlocutor at a time. GPT-4.5, prompted with a humanlike persona, was judged human 73% of the time, significantly more often than the real human participants were judged human; LLaMa-3.1-405B with the same prompt was judged human 56% of the time, not significantly different from the human baseline; ELIZA and un-prompted GPT-4o both scored significantly below chance (23% and 21%). This is presented as the first result matching Turing's original three-party format in which a system was judged human more often than an actual human -- exceeding the human baseline itself, not merely clearing a fixed 30% threshold. The 73% figure is specific to the persona-prompted condition: without the persona prompt, GPT-4.5's win rate dropped to 36% and LLaMa-3.1's to 38%, and the paper's own analysis found GPT-4.5 without the persona prompt was not significantly better than the ELIZA baseline (23%).

We evaluated 4 systems (ELIZA, GPT-4o, LLaMa-3.1-405B, and GPT-4.5) in two randomised, controlled, and pre-registered Turing tests on independent populations. Participants had 5 minute conversations simultaneously with another human participant and one of these systems before judging which conversational partner they thought was human. When prompted to adopt a humanlike persona, GPT-4.5 was judged to be the human 73% of the time: significantly more often than interrogators selected the real human participant. LLaMa-3.1, with the same prompt, was judged to be the human 56% of the time -- not significantly more or less often than the humans they were being compared to -- while baseline models (ELIZA and GPT-4o) achieved win rates significantly below chance (23% and 21% respectively). The results constitute the first empirical evidence that any artificial system passes a standard three-party Turing test. The results have implications for debates about what kind of intelligence is exhibited by Large Language Models (LLMs), and the social and economic impacts these systems are likely to have.

Cameron R. Jones and Benjamin K. Bergen, "Large Language Models Pass the Turing Test," arXiv:2503.23674, abstract

Large Language Models Pass the Turing Test, arXiv (preprint, 2503.23674), 2025-03-31

Large language models pass a standard three-party Turing test

Cameron R. Jones and Benjamin K. Bergen

This is the peer-reviewed publication of the 2025 preprint's line of work, in PNAS, carrying the same core two-study result (GPT-4.5 at 73%, LLaMa-3.1-405B at 56%) plus a newly reported third study replicating the finding in 15-minute (rather than 5-minute) conversations, where two persona-prompted models scored 56% and 59%. Notably, the published title differs from the arXiv preprint's title: the preprint is titled "Large Language Models Pass the Turing Test" (arXiv:2503.23674), while the PNAS version of record is titled "Large language models pass a standard three-party Turing test" -- the published title adds the qualifier "a standard three-party," and the published abstract also drops the preprint's superlative "first" framing ("the first empirical evidence that any artificial system passes..." becomes "constitute empirical evidence that artificial systems can pass..."). Stated here as a factual observation about how the claim was worded differently between preprint and peer-reviewed publication, not itself classified as a goalpost move. As in the preprint, both persona-prompted models performed significantly worse without that prompt (the published abstract's own figures), and the 73%/56% headline numbers hold only under the persona-prompted condition -- see the 2025 preprint event above for the exact per-model no-persona breakdown, independently confirmed against that paper's full text.

The Turing test has been widely discussed as a test of machine intelligence, but it also provides a measure of how humans distinguish other humans from machines. We evaluated 4 systems (ELIZA, GPT-4o, LLaMa-3.1-405B, and GPT-4.5) in two randomized, controlled, and preregistered Turing tests on independent populations. Participants had 5 min conversations simultaneously with another human participant and one of these systems before judging which conversational partner they thought was human. When prompted to adopt a humanlike persona, GPT-4.5 was judged to be the human 73% of the time: significantly more often than interrogators selected the real human participant. LLaMa-3.1, with the same prompt, was judged to be the human 56% of the time—not significantly more or less often than the humans it was being compared to. Without these prompts, however, the same models performed significantly worse (38% and 36%), and did not consistently outperform baseline models, ELIZA and GPT-4o (23% and 21%, respectively). A third study replicated these results in 15-min games: two PERSONA-prompted models achieved pass rates of 56% and 59%. The results constitute empirical evidence that artificial systems can pass a standard three-party Turing test. Interrogators’ reasoning focused more on stylistic and socio-emotional aspects of human behavior rather than more traditional notions of intelligence. The results have implications for debates about what kind of intelligence is exhibited by large language models, the social impacts these systems are likely to have, and the aspects of human behavior that people continue to see as unique.

Cameron R. Jones and Benjamin K. Bergen, "Large language models pass a standard three-party Turing test," PNAS 123(21):e2524472123 (2026), abstract

Large language models pass a standard three-party Turing test, Proceedings of the National Academy of Sciences 123(21):e2524472123 (PMC open-access mirror, PMCID PMC13214042), 2026-05-19