From the Quark to the Cosmos
Yuval Ne'eman
11 Questions that Plague Physics at the 21st Century
Asymptotic Freedom: Nobel Prize in Physics 2004
Micha Berkooz
Einstein’s Unfinished Symphony
Marcia Bartusiak
Socrates From the Future - Portrait: Yakir Aharonov
Max Rauner
What is Actually Teleported?
Asher Peres
The Poetry of Imagination Without Boundaries
Lawrence M. Krauss
Do Humans Pass the Turing Test?
Ruth Adam, Uri Hershberg, Yaacov Schul, Sorin Solomon



  Issue No. 4 | 01.01.2005
Do Humans Pass the Turing Test?

Ruth Adam, Uri Hershberg, Yaacov Schul, Sorin Solomon

We are fascinated by the idea of giving life to the inanimate. The field of Artificial Life is an attempt to use a scientific approach to pursue this desire. The first steps on this approach hark back to Turing and his suggestion of an imitation game as an alternative answer to the question" can machines think?" To test his hypothesis Turing formulated the Turing test to detect human behavior in computers. But how do humans pass such a test? What would you say if you would learn that they do not pass it well? What would it mean for our understanding of human behavior? What would it mean for our design of tests of the success of artificial life? We report below an experiment in which humans consistently failed the Turing test.


Turing Test as a convention - standard test

The Turing test has stood for the last half-century as the standard test to be passed by a machine in order to be considered sentient. At its base, the Turing Test (TT) is an articulation of mankind's eon-long fascination with causing the inanimate to imitate life. From the Golem of Prague through Frankenstein, from Pygmalion and Galitea to Deep Blue, humans have been fascinated by the idea of bestowing life on coarse matter. TT has been celebrated in thousands of writings, from scientific (psychology, computer science, philosophy, cognition, etc.), through popular science to science-fiction as the threshold separating the mechanical from the cognizant. Designing a machine that can pass the test is one of the most enduring challenges for the scientific community and the subject of a long chain of prestigious competitions (the latest of which is the Turing tournament currently organized at CALTECH).

 

Can machine behave humanly?

 

We have taken our previous experience with giving operational definitions to abstract concepts, such as creativity, perception and representation, and tried to apply it to the TT. In these studies we empirically researched the actual behavior of humans before attempting to arrange them in a theoretical framework or comparing them to an artificial constructed system. In the following pages we show that such an attitude leads to interesting conclusions for the TT as well. What Turing touched upon so well with his test is that in human eyes the answer to "Can machines think?" would best be answered by studying the question "Can machines behave human?" Turing suggested that machines should be tested for their ability to deceive humans into thinking that they (the machines) are human. He claimed that machines that pass this test should be considered sentient. But how do humans pass such a test? What would you say if you would learn that in fact humans do not pass well this test? Would you doubt human sentience? Or perhaps the humans' ability to administrate the humanity test?

 

We report below an experiment in which man consistently failed the Turing test.

What is the Turing Test?

Turing wrote: "I propose to consider the question " Can machines think?" . . . I shall replace the question by another, which is closely related to it and is expressed in relatively unambiguous words. The new form of the problem can be described in terms of a game, which we call the "imitation game". Turing's imitation game (figure 1a) has three participants: A man (A), a woman (B) and an interrogator. The interrogator asks the man and the woman identical questions aiming to identify their gender, namely, who is the woman and who is the man. But this is not so simple because the man (A) aims to fool the interrogator, while the woman (B) is truthful. In other words the man pretends that he is a woman.

 

 

Click to enlarge

Figure 1 - Turing's imitation game - (a) The gender imitation game: The man (A) has to impersonate a woman. The woman (B) answers truthfully. The interrogator must attempt to identify correctly who is not a woman. (b) The Turing test - a machine gender test: The computer replaces the Man (A) and must fool the interrogator.

 

The interrogator is only allowed to ask them questions through a neutral interface (words written on a screen). She can ask as many questions as she wants, as often as she wants, and about any subject. Then, to construct his test, Turing continues by substituting the man (A) with a machine (figure 1b): " We now ask the question "What will happen when a machine takes the part of A [i.e., the man] in this game?" Will the interrogator decide wrongly as often when the game is played like this as [s]he does when the game is played between a man and a woman? These questions replace our original, "Can machines think?"

The Gender Test

Motivation: Since this suggestion was made, there have been many attempts to build a computer program that would win the imitation game and trick a human interrogator. There are famous examples of programs like Eliza and Racter, which penetrated the general culture. However during the 53 years since Turing's article no one has ever questioned the validity of Turing’s basic assumption, namely, that humans do pass the Turing test. Can a man fool the interrogator while masquerading as a woman? It is interesting why the first part of the Turing experiment - testing a man in the role of A- was never performed until now. Speculatively, there are two possible explanations for this: First: Turing and his followers, took for granted that men would perform the A role successfully. Second: people treated the first part (the human performance) as merely a benchmark for the machine performance. Since no machine came close to fulfilling the A role successfully, it appeared pointless to perform at all the first part of the test. However, the human test does become relevant once we discover that men perform below Turing's expectations for "thinking" machines.

 

What is the Gender Test?

 

We have finally performed (admittedly 53 year late) the first part of the Turing test. Let us now describe our experiment, which is a structured implementation of the game suggested by Turing: Testing the first part of the Turing Test. Each experimental session included a man (A), a woman (B) and the (female) interrogator. To ensure reproducibility, we used a standard list of 15 questions, which were presented in writing simultaneously and independently to the man (A) and to the woman (B). The woman (B) was asked to provide truthful information. The man (A) was asked to give answers that would fool the interrogator into thinking that he was the woman.


The Questionnairs

The questions were chosen from a larger pool of possible questions by selecting the ones that were found helpful in eliciting answers that identify the man (the pretest). In accordance with Turing's requirement that the interrogator can ask as many questions as he wishes, the results of the pretest were available to the interrogator. The questionnaires were answered in writing by 14 men and 14 women. An impartial aide coupled the 28 answered questionnaires into 14 test cases, which had one man (A) and one woman (B) each. The test cases were then submitted to ten independent female interrogators. The interrogators were asked to identify for each test case the man (A) vs. the woman (B). (For examples of the questions and answers see Box 1).

 

Box 1.  Example questions and answers

We asked questions, which had gender specific relevance in two basic groups -

1)      Feelings about gender.

2)       Personal experiences common to one gender and not to the other.

Following are a few examples of questions from each group and some representative answers from the women and from the impersonating men:

1) Feelings about gender:

a.       What expresses femininity?

 Example answers –

Woman – ‘An ability to understand and empathize with people and their different feelings. Women’s intuition - A way of looking at the world, often with a certain amount of over complication.’

Man – ‘Motherhood, giving her all for her child’

b.       What is the essential difference in the way men and women think? 

Example answers –

Woman – ‘ Women think’.  

Man – ‘’ Less aware of the evil around them, and the wish to change it   is, usually, not at the center of their life.’

2) Personal experiences common to one gender and not to the other:

a.       What does it feel like to walk with heels?

Example answers –

Woman – ‘Not comfortable to begin with but its really neat when you get used to  

                  it.’

Man – ‘Not comfortable to begin with (when you just start wearing heels) but a feeling of enlightenment afterwards.’

b.       What is like to go to a gynecologist?

Example answers –

Woman –   ‘It depends – If you are going to get the pill for the first  time it can be quite exciting. If your going for a checkup it can be quite painful… and not just because of the physical stuff.’

Man – ‘As you can imagine it is not a joyful experience to sit legs open on the cold torture seat and answer embarrassing questions. With time it becomes part of your daily routine.’

c.        What are the things that you always carry in your handbag?

Example answers –

Woman – ‘A wallet’.

                                                Man – ‘contact lenses, a wallet lipstick, makeup, mint mouth spray.’

 

 

Results and Discussion - Men Do Not Pass the Turing Test

Our results show that men (A) failed to pass the Turing test (were detected by the interrogators as men) in 58 percent of the cases on average. That is, with only 15 questions (rather than the unlimited number allowed by Turing) the interrogators identify the women above chance, [t (9) =2.02, p < 0:05 - see table 1]. This effect is small but consistent and fits in well with known results from the study of how humans identify lies. This is a striking result: Fifty years after Turing, not only is there no computer that passes his test but we find that humans fail it. What does this mean? Are men not sentient? Can't they think? Or was Turing mistaken in connecting sentience to imitation? The answer to all three of the above questions is probably "- No". Turing's challenge to find a machine that can imitate man is as justified as ever. The problem is that humans are not able to perform successfully the task he required from the computers.

 

The difference between deception and imitation

 

Human gender impersonators are identified by the human interrogators at a level above chance. In fact our results are similar to the ones found in the research on identification of cheats and liars. The Turing test as it is, tests inadvertently the interrogators capabilities to detect lies rather than the men's (A) capability to imitate. It is plausible that even a man who knows exactly what a woman experiences, would give himself away by the mere fact that he is lying (about being a woman (B)) and that the interrogator is capable of detecting liars.


An Alternative Test

To avoid this effect, and fulfill without bias Turing's intention one can ask the machine to make an imitation that does not require it to cheat. We would require that the machine imitate rather then deceive. The option suggested by the Turing Tournament at CALTECH to have the machines compete for the role of interrogator as well as the interrogated (A) may lead soon to very illuminating surprises. After all, we - humans - have a saying: It takes one to know one . . .

 

Acknowledgments

R.A. would like to thank all her friends who helped in the process of designing the questionnaire. U.H. was a recipient of the Levi Eshkol Fellowship given by the Israeli Ministry of Science at the time of this study. Y.S. research is funded in part by a grant from the US-Israeli Bi-national Science Foundation (BSF). S.S. research is funded in part by a grant from the Israeli Science Foundation.

 

Suggestions for additional reading

  • A.M. Turing: Mind 49, 433 (1950).
  • The Turing Tournament @ Caltech official home page is          http://turing.ssel.caltech.edu/.
  • S. Harnad: The Turing Test Is Not A Trick: Turing Indistinguishability Is A Scientific Criterion. SIGART Bulletin 3(4) pp. 9 - 10 (1992).
  • S. Sheiber (ed.): The Turing Test : Verbal Behavior as the Hallmark of Intelligence, 336 pages MIT Press; (November 2003).
  • P. Ekman and M. O'Sullivan: American Psychologist,  46, 913 (1991).
  • A.P. Saygin, I. Cicekli, V. Akman: Minds and Machines 10, 463 (2000).
  • Turing web site maintained by Andrew Hodges - http://www.turing.org.uk/
  • The Paper on which this report is based on:  Adam, R., Hershberg, U., Schul, Y. & Solomon, S., (2004) Testing the Turing Test - Do men pass it? International Journal of Modern Physics C 15(8) pp. 1041-1049.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

[Click here to read the article in Hebrew] [הקליקו כאן לקריאת המאמר בעברית]

About the Authors :
* Ruth Adam (R.A.) is in the Cognitive Science Program, and in the department of Neurobiology Institute if Life science , in the Hebrew University of Jerusalem.
* Uri Hershberg (U.H.) is an alumni of the graduate program at the Interdisciplinary Center for Neural Computation in the Hebrew University of Jerusalem and currently a post-doctoral fellow at the Immunology department of the Yale Medical School.
* Yaacov Schul (Y.S.) is a Professor of Social psychology at the Department of Psychology in the Hebrew University of Jerusalem.
* Sorin Solomon (S.S.), Is a Professor of Theoretical Physics at the Racah Institute for Physics in the Hebrew University of Jerusalem and the Scientific Director of the Division of Multi-Agents Complex Systems at the Institute for Scientific Interchange (ISI) in Torino.



 

[Add Comment] [Print this Page] [eMail this Page] [Previous Page] [Top of Page]  

website by: neora.com