Viewing a single comment thread. View all comments

AnArgumentativeFool t1_izf55tb wrote

As a diplomacy player who plays a lot online, is there any chance that you plan on testing how the AI would do in a more standard setting (for example 24 hours per turn, instead of the 5 minute games)?

The ability to win in the games it has is an incredible achievement but I am still under the impression there is a long way to go before the machines achieve super human ability in the formats of the game that require much more human connection and communication against the best players. Thanks

1

MetaAI_Official OP t1_izfod3v wrote

We were originally targeting 24hr-turn games, but ended up pivoting to 5min-turn games due to the inability to gather a sufficient number of samples in the 24hr-turn format (as playing a single game can sometimes take months)! Playing 24hr-turn games would indeed pose additional challenges from a language generation perspective — while human players tend to send a similar number of messages in each format, messages in 24hr turns tend to be signficantly longer (and likely more complex). Moreover, human players would have more time to interrogate mistakes from the bot, which could potentially lead to the agent making further mistakes. -ED

1

MetaAI_Official OP t1_izfq8y9 wrote

[Goff] There's also some interesting "anti-weirdness" steps that the team worked on that would need to be put in place - an AI that responds to messages within five minutes 24 hours a day would not feel right at all. I think the most intense timeframe is probably 15 to 30 minutes, as then you will need longer, more complex communications but also the rapid tactical back-and-forth - that pivot would be a cool challenge.

2