We wish to clarify an account of the 2014 Turing Test experiment we conducted at the Royal Society London, U.K., as outlined by Moshe Y. Vardi in his Editor’s Letter “would Turing Have Passed the Turing Test?” (Sept. 2014). Vardi was referring to a New Yorker blog by Gary Marcus, rather than to our experiment directly. But Marcus had no first-hand experience with our 2014 experiment nor has he seen any of our Turing Test conversations.
Our experiment involved 30 human judges, 30 hidden humans, and five machines—Cleverbot, Elbot, Eugene Goostman, JFred, and Ultra Hal; for background and details see http://turingtestsin2014.blogspot.co.uk/2014/06/eugene-goostman-machine-convinced-3333.html. We used social media to recruit judges and a variety of hidden humans, including males, females, adults, teenagers, experts in computer science and robotics, and non-experts, including journalists, lecturers, students, and interested members of the public.
Prior to the tests, the judges were unaware of the nature of the pairs of hidden entities they would be interrogating; we told them only that they would simultaneously interrogate one human and one machine for five minutes and that the human could be a male or female, child or adult, native English speaker, or non-native English speaker. We asked the hidden humans to be themselves, that is, to be human.
The 30 judges, each given an anonymous experiment identity—labeled J1–J30—interrogated five pairs of hidden entities. Likewise each human and machine was given a unique identity—E1–E35. We ran 150 “simultaneous comparison” Turing Tests in which we instructed the judges that their task was to determine which was human and which was machine in the pair, a decision to be made based solely on the responses the hidden entities posted in reply to what a judge said.
Eugene Goostman was not correctly identified as the machine in the pair in 10 of its 30 tests; that is, 10 judges did not recognize it was a machine. Eugene Goostman’s personality is that of a 13-year-old boy from Odessa, Ukraine, a character we do not consider contrary to Alan M. Turing’s vision for building a machine to think. In 1950, Turing said, “Instead of trying to produce a programme to simulate the adult mind, why not rather try to produce one which simulates the child’s?”
The figure here includes one simultaneous conversation from the experiment, showing one of Judge J19’s tests after that judge simultaneously interacted with two hidden entities, in this case E20 and E24. In this test, E20’s responses to the judge were relayed to a message box displayed on the left of the judge’s screen; E24’s answers were relayed on the right. Timings and text are exactly as they were in the test.
So, could you “pass the test” and be able to say which of the two entities—E20 and E24—is the human and which the machine?
Huma Shah, London, U.K., and Kevin Warwick, Reading, U.K.
Author’s Response:
The details of this 2014 Turing Test experiment only reinforces my judgment that the Turing Test says little about machine intelligence. The ability to generate a human-like dialogue is at best an extremely narrow slice of intelligence.
Moshe Y. Vardi, Editor-in-Chief
Bahrain Revokes Masaud Jahromi’s Citizenship
I have written several letters to the editor (June 2012, Jan. 2012, and Nov. 2011) about a citizen of Bahrain, professor and chair Masaud Jahromi of the Telecommunications Engineering Department at the Ahlia University in Bahrain, whose human rights had been violated by his own government. Jahromi was arrested and imprisoned in April 2011 for nearly six months for attending a rally on behalf of freedom. He was eventually tried, convicted, and sentenced by a court to five months in prison and a fine of approximately $1,400. As he had already served five months, the court simultaneously suspended the four months. Following this January 19, 2012 ruling, he was dismissed from his position as professor and chair at Ahlia University only to be reinstated as professor February 20, 2012 and then as chair in March or April 2012.
The Bahrain Ministry of Interior has now revoked Jahromi’s citizenship through a decree issued January 31, 2015. Jahromi was one of 72 Bahrainis, including journalists, activists, and doctors, to be stripped of their citizenship pursuant to a revision of the 1963 Bahrain Citizenship Act.
The Ministry of Interior announced its decree without court process or opportunity to respond, saying it was revoking the citizenship of the named individuals for “terrorist activities,” including “advocating regime change through illegal means.” There is no evidence Jahromi has ever participated in terrorism in any form. His sentence in 2011 was for participating in “unauthorized rallies” during protests. There are no additional allegations or evidence that he violated any law since then.
The summary revocation of citizenship appears to be a result of nonviolent expressive activity that has already been punished and not recurred. International instruments, including the Universal Declaration of Human Rights and the International Convention on Civil and Political Rights, to which Bahrain is a signatory, explicitly protect both the right of individuals to be free from arbitrary deprivation of their nationality and their right to engage in nonviolent expressive activity. Indeed, Article 15 of the Universal Declaration of Human Rights specifically prohibits arbitrary deprivation of anyone’s nationality.
Denial of citizenship without explanation or apparent basis imposes severe damage on an individual who consequently becomes stateless. Moreover, just including Jahromi’s name on a list with the names of obvious terrorists serving with ISIS abroad damages Jahromi’s reputation as an academic.
Jahromi believes wide publicity of his plight through Communications and support from the ACM membership was a positive factor in addressing his previous legal problem. Those who wish to help may write to the following address to request immediate restoration of Jahromi’s Bahraini citizenship.
His Majesty Shaikh Hamad bin Issa
Al Khalifa
King of Bahrain
Office of His Majesty the King
P.O. Box 555, Rifa’a Palace
Isa Town Central,
Kingdom of Bahrain
Jack Minker, College Park, MD
The Case of the Missing Skillset
In their article “Verifying Computations without Reexecuting Them” (Feb. 2015), Michael Walfish and Andrew J. Blumberg reviewed the state of the art in program verification while questioning whether to trust anything stored or computed on third-party servers, as in cloud computing, where companies and consumers alike access remote resources, including data, processing power, and memory, on a rental basis. Walfish and Blumberg proposed the formalism of probabilistically checkable proofs to allow, at least theoretically, a verifier to verify remotely performed computations. Not discussed, however, was an important aspect of verifiable software for the cloud—the general lack of the skillsets needed to write efficient and verifiable parallel programs.
As anyone in the software industry or related academic institutions can attest, training software engineers to use new tools and paradigms can involve an arduous learning curve. Even if able to program in a new language, a developer might still need further training to become a productive member of a team. It takes time and resources to graduate from “Hello World” to programs fulfilling client use cases. Such an investment is perhaps what motivates organizations to outsource their software projects, a practice that risks even failure due to poor-quality software.2
Though developing verifiable software is not typically high on an organization’s must-teach list, hosting an application in a cloud adds further requirements due to the code’s remote service-driven execution architecture. As long as the cloud delivers the service at the promised quality of service, clients are unlikely to be interested in the details of its implementation. For example, Jeremy Avigad and John Harrison concluded in their article “Formally Verified Mathematics” (Apr. 2014), “There is a steep learning curve to the use of formal methods, and verifying even straightforward and intuitively clear inferences can be time consuming and difficult.”1
Yet another aspect of the problem is the absence of the skillset needed to write efficient cloud-based code, as not all code is readily convertible for parallel and cloud-friendly environments. While ample processing power may be available in a rental cloud, it may be difficult to find and train software developers to produce related high-quality cloud-based code.
Formalisms (such as those discussed by Walfish and Blumberg) may be exciting, at least theoretically, but realizing efficient and verifiable cloud code requires bridging the gap between the software industry and the community of formal-methods practitioners.
Muaz A. Niazi, Islamabad, Pakistan
Join the Discussion (0)
Become a Member or Sign In to Post a Comment