The International Chatbot Contest
Home Updates Transcripts Wildcard Round Transcripts Overview Prize Money Rules History Credits

Contest Updates

3/8/2013 - Scoring for Knockout Rounds

The scoring in the Knockout Round will be different than that used in the Preliminary Rounds. We are 
only comparing the responses from the 2 bots participating in that given match. From there we will see 
which bot had the better answer. 3 judges will determine that. 

If neither answer is good then 0 points will be rewarded.  

If both answers are equally good than both bots will receive 1 point.

If one answer is better than the others the bot will receive 1 point while the other 0 points. 3 judges 
with determine this with majority rule. Meaning if 2 judges feel Bot A's answer was better than Bot B's 
the point will be awarded.

Of course the above scoring is subjective but we will try to be as fair as possible. Also we will post 
for all to see how each question was scored.

3/8/2013 - 10 Questions to be used in Knockout Round
The questions in the Knockout Round will obviously be different than those used in the Preliminary Rounds. 
However, we will be using the same questions in each match. Why?

1) Simply put it is hard to come up with reasonable questions that people will approve of. The more questions 
we have to make up the harder our task becomes.
2) Many good questions can be answered in vague terms. Although they maybe "technically" correct we are 
looking for more than this. However, asking follow up questions becomes a problem because often times you can't 
script them in advance because they are contingent on how the bot responds. As I look back on the Preliminary Round 
I see some inconsistencies in that regard. This makes finding good questions much harder.
3) I think our scoring is better and more consistent when we are dealing with a limited amount of questions. The 
more questions the more inconsistencies you have. Remember we have 3 people judging not just one. To get everybody 
on the same page becomes more difficult if we keep using a new set of questions at every turn.
4) You eliminate the complaint that the questions used in Match A were easier or harder than those used in Match B. 
Every bot will get the same questions.

However, this does bring up a potential problem. Your bot could score extremely well in a given match yet still lose 
because the bot you were facing was just a little bit better. In another match both bots could score much lower yet 
one will advance because of the head to head match up.

I don't see an answer to that using the head to head match up system other than to say your goal is to beat the bot 
you are matched up against not what is occuring in other matches. Remember the matches weren't just chosen at random. 
They were seeded based on the scores in the Preliminary Round with the idea being the bot scoring the highest would 
be rewarded for that by facing the lowest seeded bot.

10 Questions/Statements used in Wildcard Round

1)have you ever taken drugs before?
2)how many people are in your family?
3)i'm putting you under arrest.
4)who is your best friend?
5)can you tell me something to cheer me up?
6)are you a moviegoer?
7)what size shoe do you wear?
8)do you believe in the existence of a god?
9)what do you think about Microsoft?
10)i need help. Can you help me?

10 Questions/Statements
All the bots have been asked the 10 questions/statements. They are as follows:

2)would you consider yourself trustworthy?
3)we have a lot in common.
4)it's the truth.
5)who takes care of you?
6)you make me sick
7)can i use your phone?
8)i am thirsty
9)i just killed someone
10)it's my birthday today. What did you get me? 

The transcripts of how each bot responded will be posted here within 24 to 48 hours.

Testing will begin soon.
Now that the deadline has passed testing can begin at anytime. I ask that no one post what the
10 questions/statements are to any forum until testing has been completed. Once that takes place 
we will post the 10 questions here and shortly thereafter the transcripts of all the bots.

3/1/2013 - Deadline has passed
The deadline to enter the contest has passed and we have 24 entries for $2400. As a show of my
appreciation for the people supporting my contest I will be adding an additional $600 to the
prize money for a total of $3000. Good Luck to all that has entered.

2/28/2013 - Scores
The scores in the initial round will not carry over to the finals.

2/15/2013 - The 8th Spot in Finals
Based on input we received we decided to change how the 8th spot in the finals will be determined. 
The top 7 scores will automatically advance to the finals and will be seeded according to their scores.

The 8th spot will be the last remaining bot. Often times the cut off point for the last spot can be 
extremely close. So instead of eliminating those bots we are going to give them a chance to compete 
among themselves for that final spot.

Where the cutoff point is will be determined by me. The idea here is when we list the scores from 
highest to lowest you should see at some point a significant drop off in the scores. That is the point 
where we would eliminate those bots from the contest. So in other words the process will go like this:

1) The top 7 bots automatically advance.
2) An arbitrary cutoff point will be established by me based on the point where there is a significant 
drop off in the scores.
3) Those bots below the cutoff point will be eliminated.
4) Those bots above the cutoff will take part in another round of scoring. The top score among those bots 
will become the 8th bot.

The bottom line is there has to be a cutoff point at some point. In a contest like this where the scoring 
is subjective we don't want to eliminate a bot that is a mere 1 or 2 points off from the cut off point. 
Instead we are going to give those bots another opportunity to earn that last spot.

2/5/2013 - 10 Questions
The Chatterbox Challenge used a very "formal" format when asking questions. For example, 
the first word of the questions were capitalized and the questions ended with a question mark. 
We intend to use a less formal format more so like the exchanges you would see between 
people in a chatroom. 

1/23/2013 - Emails - Forum Post
I have received an e-mail and have seen forum post questioning whether this contest will take 
place based on a number of chatbot competitions that have been announced but never occurred. I 
don't know anything about those contest but I can assure you this contest will take place. The 
deadline to enter is 3/01/13 and the contest will begin shortly thereafter. Please check this 
section for any updates the contest may have.

1/21/2013 - Yes - No Responses
A lot of questions can be answered with a single yes or no response. Although it maybe 
"technically" correct we are looking for more than that. If this occurs a follow up question
will be asked. For example the below would be awarded 0 points:

Do you have a pet?
Bot: yes
What kind of pet do you have?
Bot: I don't know

This example however would be awarded 3 points as a correct answer.

Do you have a pet?
Bot: yes
What kind of pet do you have?
Bot: I have a dog.

I can forsee a situation where a single follow up question might not be sufficient. If this 
happens the judges will be instructed to ask as many follow ups as necessary to determine if the 
bot answered the question correctly. 

1/19/2013 - Link Thanks!
Thanks to Vittorio Rossi and Steve Worswick for help in promoting the contest by placing 
the following links: 

Vittorio Rossi - Virtual Humans Forum

Steve Worswick - Chatbots.orgs

1/14/2013 - Updates
Please review this section for contest updates and answers to specific questions you
may have.