Can Computers Really Think? - Thinking as Computation - page 271

12.3 The Chinese Room
But philosophers are an obstinate lot. They are not going to back down on the issue
of whether real understanding is taking place just because someone like Turing says
they should. Perhaps the most successful attack on Turing's position came from the
philosopher John Searle in the 1980s. He devised a different thought experiment
to argue that there was much more to understanding (or thinking, or being intelli-
gent) than merely getting the external behavior right. Here is (a slight variant of) his
argument.
Imagine that there is already a computer program that is somehow able to pass the
Turing Test, but in Chinese, not in English. Let us call the computer program that
does this chinese.pl , assuming it is written in Prolog. So Chinese characters can be
presented to this program (encoded in some way), and it will produce responses (also
encoded in some way) that cannot be distinguished even over a long period of time
from those of a person who really understands Chinese.
Now imagine that Searle does not know Chinese, but knows Prolog very well. He
is put in a room with a book that contains the text of the chinese.pl program. When
someone outside the room slips him pieces of paper with Chinese written on them, he
does not understand them, but he can follow what chinese.pl would do given these
as input. He traces the behavior of the program, writes on a piece of paper the output
that the program would produce, and hands that back, again without understanding
what any of it means.
So there is a person, Searle, who is receiving Chinese messages and producing Chi-
nese responses no different from those of a native Chinese speaker (since chinese.pl
is assumed to pass the Turing Test). His external behavior, in other words, is per-
fect . And yet Searle does not know Chinese. Searle's conclusion: getting the external
behavior right is not enough, and so Turing is wrong.
A counterargument is that Searle is not producing this behavior, but Searle together
with the topic. Although Searle does not understand Chinese, the system consisting
of Searle and the topic does. So Turing is not wrong. Searle's reply to this objection
is beautiful in its simplicity: Imagine that he memorizes the topic and then destroys it.
Then there is no longer a system to talk about; there is just Searle. So Turing is wrong.
Is this the last word on the topic? Remember, nothing is obvious in philosophy.
Consider this: How can we be so sure that Searle does not come to learn Chinese after
memorizing the topic? If he did, then the argument would collapse. The answer is
that we do not know what happens after he learns the topic simply because we do