Datasets and Simulators

CIFF comes with the following tasks.

1. Blocks: Manipulating Blocks on a Map

Block world task and dataset were introduced by Bisk et al., 2016. In Misra et al., 2017 we introduced an additional simulator developed using Unity3D for interactive experiments. The dataset and simulator can be downloaded with CIFF (please see README.md) for more details.

Dataset Files:

The train, test and dev files can be accessed at:

1. Training data Json file

2. Development data Json file

3. Test data Json file

Dataset File Format:

The block world files are in JSON format. You can think of JSON as similar to python dictionary. If you are unfamiliar with JSON then please see this link. Each file contains a list of JSON strings where each string contains the following fields:

• decoration: Either logo or digits. Logo problems have logos of various companies and digits have digits on them.

• shape_params: color or pattern on different sides of a single block.

• notes: Contains the data for a task involving creating MNIST digits from a completely scrambled position. This itself contains several instructions. Each note further consists of:

  • type: Is either A0, A1 or A2. A0 includes instructions that move only block from one position to another. It consists of a single step in the process of creating the MNIST digit pattern. A1 includes moving several blocks and consists of a sequence of steps. A2 contains instruction for the entire task of creating the MNIST digit pattern. We only experimented with A0 instructions. A1 and A2 instructions are much more challenging and remain an open problem.

  • start: Agent starts at this index, at the beginning of instruction, in the sequence of steps leading to the creation of the MNIST digit pattern.

  • finish: Agent ends at this index in the sequence of steps leading to the creation of the MNIST digit pattern. For A0 tasks, finish = start + 1 since the agent completes only a single step.

  • notes: Contains a sequence of instructions that take the agent from the start step to the finish step of that task. They can be thought of as paraphrase to an extent although they might use a different trajectory with the same common goal state.

  • users: M-Turk ID of the users who wrote the instructions (in that order).

Block World Simulator:

The simulator file and support files are available for the Linux system here. The main build file is this. The CIFF code launches these builds internally and connects them to the agent using socket connection so you don't have to worry about it.

The block world build was created using Unity3D game engine. Please find the Unity3D source file for the block world here. We use the Unity3D community edition which is made publicly available for free.

2. LANI: Navigation in a 3D Landspace

LANI dataset was introduced in Misra et al, EMNLP 2018. It contains instructions for navigation in an open 3D land space. There are several landmarks (chosen from 63 unique landmarks) that are placed in the land space and the instruction places constraints on how the agent should navigate.

Dataset Files:

The train, test and dev files can be accessed at:

1. Training data Json file

2. Development data Json file

3. Testing data Json file

These files are in JSON format and together contain 6000 natural language paragraphs (a paragraph is a sequence of instructions). A single entry in the JSON contains the following key:

• id: a unique problem id for this paragraph.

• instructions: A paragraph that has been segmented into separate instructions. In order to get the original paragraph, simply concatenate the instructions in this list.

• num_tokens: Number of tokens in each segment.

• start_x and start_z: x-coordinate (z-coordinate) of agent's start position at the beginning of segments. The list follows the same order as segments in the instructions.

• end_x and end_z: x-coordinate (z-coordinate) of agent's final position at the beginning of segments. The list follows the same order as segments in the instructions. From definition it follows, end_x[i] = start_x[i+1].

• start_rot and end_rot: The horizontal rotation angle (yaw) of agent's start (final) position at the beginning of segments. The list follows the same order as segments in the instructions. Rotations are measured using an absolute coordinate system (i.e. not in the agent's frame of reference).

• moves: Contains gold execution for all segments. L means left turn, R means right turn and F means forward. All demonstrations terminate by taking the stop action.

• path_file: The file name of the file containing the agent's path information.

• config_file: The file name of the file containing the land space configuration (choice of landmark and their position and rotation).

• valid: Valid is a validation key that we use for filtering the data. Valid being set to False means a mechanical tech worker flagged the instruction as being spam and didn’t complete the trajectory. We don't use invalid instructions.

LANI Simulator:

LANI simulator was built using Unity3D game engine. It can be found here and the build file is here. CIFF launches these build programmatically so you don't have to worry about how to run these builds. If you are interested in the Unity3D source code then please send us an email.

3. CHAI: Navigation and Manipulation in a 3D House

See: https://lil.nlp.cornell.edu/resources/Misra-EMNLP-2018/

4. Touchdown: Navigation in Google Street View World

See: https://github.com/lil-lab/touchdown