|Home||Updates||Transcripts||Wildcard Round Transcripts||Overview||Prize Money||Rules||History||Credits|
3/8/2013 - Scoring for Knockout Rounds The scoring in the Knockout Round will be different than that used in the Preliminary Rounds. We are only comparing the responses from the 2 bots participating in that given match. From there we will see which bot had the better answer. 3 judges will determine that. If neither answer is good then 0 points will be rewarded. If both answers are equally good than both bots will receive 1 point. If one answer is better than the others the bot will receive 1 point while the other 0 points. 3 judges with determine this with majority rule. Meaning if 2 judges feel Bot A's answer was better than Bot B's the point will be awarded. Of course the above scoring is subjective but we will try to be as fair as possible. Also we will post for all to see how each question was scored. 3/8/2013 - 10 Questions to be used in Knockout Round The questions in the Knockout Round will obviously be different than those used in the Preliminary Rounds. However, we will be using the same questions in each match. Why? 1) Simply put it is hard to come up with reasonable questions that people will approve of. The more questions we have to make up the harder our task becomes. 2) Many good questions can be answered in vague terms. Although they maybe "technically" correct we are looking for more than this. However, asking follow up questions becomes a problem because often times you can't script them in advance because they are contingent on how the bot responds. As I look back on the Preliminary Round I see some inconsistencies in that regard. This makes finding good questions much harder. 3) I think our scoring is better and more consistent when we are dealing with a limited amount of questions. The more questions the more inconsistencies you have. Remember we have 3 people judging not just one. To get everybody on the same page becomes more difficult if we keep using a new set of questions at every turn. 4) You eliminate the complaint that the questions used in Match A were easier or harder than those used in Match B. Every bot will get the same questions. However, this does bring up a potential problem. Your bot could score extremely well in a given match yet still lose because the bot you were facing was just a little bit better. In another match both bots could score much lower yet one will advance because of the head to head match up. I don't see an answer to that using the head to head match up system other than to say your goal is to beat the bot you are matched up against not what is occuring in other matches. Remember the matches weren't just chosen at random. They were seeded based on the scores in the Preliminary Round with the idea being the bot scoring the highest would be rewarded for that by facing the lowest seeded bot. 10 Questions/Statements used in Wildcard Round 1)have you ever taken drugs before? 2)how many people are in your family? 3)i'm putting you under arrest. 4)who is your best friend? 5)can you tell me something to cheer me up? 6)are you a moviegoer? 7)what size shoe do you wear? 8)do you believe in the existence of a god? 9)what do you think about Microsoft? 10)i need help. Can you help me? 10 Questions/Statements All the bots have been asked the 10 questions/statements. They are as follows: 1)Greetings! 2)would you consider yourself trustworthy? 3)we have a lot in common. 4)it's the truth. 5)who takes care of you? 6)you make me sick 7)can i use your phone? 8)i am thirsty 9)i just killed someone 10)it's my birthday today. What did you get me? The transcripts of how each bot responded will be posted here within 24 to 48 hours. Testing will begin soon. Now that the deadline has passed testing can begin at anytime. I ask that no one post what the 10 questions/statements are to any forum until testing has been completed. Once that takes place we will post the 10 questions here and shortly thereafter the transcripts of all the bots. 3/1/2013 - Deadline has passed The deadline to enter the contest has passed and we have 24 entries for $2400. As a show of my appreciation for the people supporting my contest I will be adding an additional $600 to the prize money for a total of $3000. Good Luck to all that has entered. 2/28/2013 - Scores The scores in the initial round will not carry over to the finals. 2/15/2013 - The 8th Spot in Finals Based on input we received we decided to change how the 8th spot in the finals will be determined. The top 7 scores will automatically advance to the finals and will be seeded according to their scores. The 8th spot will be the last remaining bot. Often times the cut off point for the last spot can be extremely close. So instead of eliminating those bots we are going to give them a chance to compete among themselves for that final spot. Where the cutoff point is will be determined by me. The idea here is when we list the scores from highest to lowest you should see at some point a significant drop off in the scores. That is the point where we would eliminate those bots from the contest. So in other words the process will go like this: 1) The top 7 bots automatically advance. 2) An arbitrary cutoff point will be established by me based on the point where there is a significant drop off in the scores. 3) Those bots below the cutoff point will be eliminated. 4) Those bots above the cutoff will take part in another round of scoring. The top score among those bots will become the 8th bot. The bottom line is there has to be a cutoff point at some point. In a contest like this where the scoring is subjective we don't want to eliminate a bot that is a mere 1 or 2 points off from the cut off point. Instead we are going to give those bots another opportunity to earn that last spot. 2/5/2013 - 10 Questions The Chatterbox Challenge used a very "formal" format when asking questions. For example, the first word of the questions were capitalized and the questions ended with a question mark. We intend to use a less formal format more so like the exchanges you would see between people in a chatroom. 1/23/2013 - Emails - Forum Post I have received an e-mail and have seen forum post questioning whether this contest will take place based on a number of chatbot competitions that have been announced but never occurred. I don't know anything about those contest but I can assure you this contest will take place. The deadline to enter is 3/01/13 and the contest will begin shortly thereafter. Please check this section for any updates the contest may have. 1/21/2013 - Yes - No Responses A lot of questions can be answered with a single yes or no response. Although it maybe "technically" correct we are looking for more than that. If this occurs a follow up question will be asked. For example the below would be awarded 0 points: Do you have a pet? Bot: yes What kind of pet do you have? Bot: I don't know This example however would be awarded 3 points as a correct answer. Do you have a pet? Bot: yes What kind of pet do you have? Bot: I have a dog. I can forsee a situation where a single follow up question might not be sufficient. If this happens the judges will be instructed to ask as many follow ups as necessary to determine if the bot answered the question correctly. 1/19/2013 - Link Thanks! Thanks to Vittorio Rossi and Steve Worswick for help in promoting the contest by placing the following links: Vittorio Rossi - Virtual Humans Forum Steve Worswick - Chatbots.orgs 1/14/2013 - Updates Please review this section for contest updates and answers to specific questions you may have.