Pre-exploration in the unseen environment

[chijames]

Hi,

Can you please share the augmented path file in the unseen environment? If I understand correctly, the aug_paths.json is only for the back translation in the seen environment. If it is not possible, some statistics about the unseen augmented path would also be very helpful. Thanks.

[airsplay]

Sure. test and valid unseen. Test data and valid unseen data have been carefully removed from these aug paths. For these experiments, I actually only allow the agent to explore the test environment but not give it testing instructions (as in RCM).

I have not tested the performance with new PyTorch version but the result should be reproducible by replacing --aug tasks/R2R/data/aug_paths.json in bt_envdrop.bash with appropriate json files (especially on unseen valid because you could see the local evaluation). I would update github after verification. Could you please leave this issue open until then?

[chijames]

Thanks for the timely response, I will leave this issue open. By "not give it testing instructions", you mean the instructions will be generated by the trained speaker model with environmental dropout, right? Thanks.

[airsplay]

Yep. And it would never touch the paths/instructions in testing data and validation unseen data.

[chijames]

May I know if there are some rules you use for generating augmented paths? Since I want to sample more paths to see the upper bound of the pre-exploration method. Currently, I randomly sample two viewpoints with their distance larger than 5 and a start heading angle. Then I add them to the original aug_path file, but the result gets worse. Do you have any suggestion? Thanks.

[airsplay]

Sorry for late replying (due to ACL).

The file is generated by exploiting all viewpoint pairs which have action length from 4~6. I also exclude the val/test data. Thus it is a complete set of all available short-distance paths. The initial heading angle are randomly sampled. I visualize the headings in training data and I believe that the initial headings in training are uniformly sampled.

If you want to verify the upper bound, I suggest to try:

1. Sampling more initial headings.
2. Generate more instructions for each paths. (currently one for each)

I am also notified (by Peter) that the speaker model trained with PyTorch 1.0 might be weaker than with PyTorch 0.4. Since the pre-exploring results highly depends on the performance of speaker, I doubt whether the results are still the same. Have you achieved any similar result by adding pre-exploring paths? If not, I would definitely take more time on fixing the speaker issue.

[yestinl]

Sure. test and valid unseen. Test data and valid unseen data have been carefully removed from these aug paths. For these experiments, I actually only allow the agent to explore the test environment but not give it testing instructions (as in RCM).

I have not tested the performance with new PyTorch version but the result should be reproducible by replacing --aug tasks/R2R/data/aug_paths.json in bt_envdrop.bash with appropriate json files (especially on unseen valid because you could see the local evaluation). I would update github after verification. Could you please leave this issue open until then?

May I know how to allow the agent to explore the test environment but not give it testing instructions?

[airsplay]

It is completed via the trick of back-translation. The paths are first randomly sampled and the instructions are then generated from the speaker.

[amoudgl]

@airsplay Does aug_paths.json also contain original R2R train + val seen data?