Subscribe now

Log in

Your login is case sensitive

I have forgotten my password

close

One Per Cent: A New Scientist Blog

Bot with boyish personality wins biggest Turing test

Celeste Biever, deputy news editor

Picture-2-chatbot.jpg

Eugene Goostman, a chatbot with the personality of a 13-year-old boy, won the biggest Turing test ever staged, on 23 June, the 100th anniversary of the birth of Alan Turing.

Held at Bletchley Park near Milton Keynes, UK, where Turing cracked the Nazi Enigma code during the second world war, the test involved over 150 separate conversations, 30 judges (including myself), 25 hidden humans and five elite, chattering software programs.

By contrast, the most famous Turing test - the annual Loebner prize, also held at Bletchley Park this year to honour Turing - typically involves just four human judges and four machines.

"With 150 Turing tests conducted, this is the biggest Turing test contest ever," says Huma Shah, a researcher at the University of Reading, UK, who organised the mammoth test.

That makes the result more statistically significant than any other previous Turing test, says Eugene's creator Vladimir Veselov based in Raritan, New Jersey. "It was a pretty huge number of conversations," he said, shortly after he was awarded first prize: "I am very excited."

First conceived by Turing in the early 1950s, the test is the most famous evaluation of machine intelligence. Human judges converse via a text interface with both hidden bots and humans - and say in each case whether they are chatting to a human or machine.

Turing said that a machine that fooled humans into thinking it was human 30 per cent of the time would have beaten the test. Just short of this, Eugene fooled its judges 29 per cent of the time. In a close second place, came JFred, the brain child of Robby Garner, and in third place Rollo Carpenter's Cleverbot. The other two bots to compete were Ultra Hal and Elbot.

Unlike several of Eugene's rivals, which put together sentences by imitating people they have spoken to before or by searching through Twitter transcripts for conversational ideas, Veselov has given his bot a consistent and specific personality. "He has created very much a person where Cleverbot is everybody," says Carpenter.

Eugene's character is that of a 13-year-old boy living in Odessa, Ukraine. He has a pet guinea pig and a father who is a gynaecologist. Is 13 about the right age for a chatbot, then? "Thirteen years old is not too old to know everything and not too young to know nothing," explains Veselov.

A veteran of the Loebner prize and the Chatterbox challenge , Eugene was due a win. "We took second place several times but never were we the winners," says Veselov.

Did having a personality give him an advantage? "I think any appearance of a particular personality is likely to have a persuasive effect on judges," says John Barnden, an AI researcher specialising in machine understanding of metaphor at the University of Birmingham, UK, and a fellow judge.

He cautions against concluding that this was Eugene's edge, however - for that you would have to compare two versions of the same bot, but in one case with personality suppressed.

"In my own case it's not so much personality in the abstract that's key as how the system responds to a comment - is the response relevant and non-vacuous?" he adds.

I can sympathise with that: in some cases I knew it was a machine because the entity didn't seem to follow the sense of the conversation. I was however, delighted by how funny, and zany some of the conversations with beings that I labelled as bots (Disclaimer: the best judge award is still to be awarded so I don't actually know how often I was right). They also forced me to consider in a new way just what it is that makes humans human.

Facebook iconDigg iconDelicious iconStumbleUpon iconTwitter iconTechnorati iconReddit iconAddThis icon
Subscribe to New Scientist Magazine

Post a comment

6 Comments

It wasn't Turing who cracked the Enigma guys....

 

Im supposed to be human, possible, but a lot of computers these days are starting to be coded with social functions, the most difficult behaviour to analyze, simulate, that I dont and very likely cant have.

Autistic Spectrum Disorder is right there in Uncanny Valley.

Weird, that kid looks familiar, I think I saw him in a mirror many years ago.

 

Oh, don't get so technical. Turing was an instrumental force in keeping up with the increasingly more difficult Enigma encryption. When the Germans added another wheel, it was because of Turing's work that the Brits could still manage to break it. But was it Turing alone? Of course not! But it's widely documented that Turing was one of the people who played a key role in it.

 

Doesn't anyone remember PARRY, the chatbot that simulated a conversation with a paranoid personality? It was created by Colby in the 1970's.

 

If it wasn't Turing, then who was it? He did work with a team though.

 

In the official Loebner site M. Embar is mentioned as the prize winner. Is he the author of Eugene?

 
Twitter Follow us
Twitter updates
Recent comments

Login

Your login is case sensitive

I have forgotten my password

close

© Copyright Reed Business Information Ltd.