The Dialog-based Language Learning dataset, designed to measure how well models can perform at learning as a student given a teacher¡¯s textual responses to the student¡¯s answer (as well as potentially receiving an external real-valued reward signal). Details and baseline results on this dataset can be found in the paper:

Jason Weston. Dialog-based Language Learning, arXiv:1604.06045.

Here is an example dialog, the last number (0 or 1) is the external reward:

1 Mary moved to the bathroom.           0
2 John went to the hallway.             0
3 Where is Mary?        bathroom        0
4 That's right.         1
5 Daniel went back to the hallway.              0
6 Sandra moved to the garden.           0
7 Where is Daniel?      office  0
8 No, they are downstairs.              0
9 John moved to the office.             0
10 Sandra journeyed to the bathroom.            0
11 Where is Daniel?     office  0
12 No, they are downstairs.             0
13 Mary moved to the hallway.           0
14 Daniel travelled to the office.              0
15 Where is Daniel?     office  0
16 Correct!             1
17 John went back to the garden.                0
18 John moved to the bedroom.           0
19 Where is Sandra?     garden  0
20 No, they are upstairs.               0