About the back-translation performance and trigger of pre-exploration

[Zihan-Liu-00]

Hi,

We tried to replicate the pre-exploration part in ablation study to verify its performance. However, the only trigger we found is the function utils.add_explorations(paths). In this function, we notice that a new file "tasks/R2R/data/exploration.json" that is not provided is loaded as something like data augmentation, and this function is used nowhere in the total project. Is the correct pre-explore trigger provided?

One another question is that, as we've replicated the back-translation part, the performance of bt-agent is 48% success rate for val-unseen, lower than the expected at least 50%. Sincewe did it following README.md, we guess the problem is the pytorch version as you mentions. Could you pls share the version number you used? Or if there is any other settings we should change to replicate the project, could u pls share them as well?

[airsplay]

Thanks.

• I think that utils.add_exploration is used to beam-search instead of pre-exploration. For pre-exploration, replacing the augmentation file would be fine.

• Since the PyTorch version was at 0.4.0 when I started this project, the torch library was under development and thus unstable. My actual PyTorch version is a nightly update of PyTorch 0.4.0, 0.4.0a0+f988247. I used this version to include some new functions (in my memory, it's distribution.entropy() which is not utilized before) of Categorical Distribution here. I personally think that the difference is the performance of speaker. However, I haven't located it yet. With the PyTorch version 1.1, I have tested this Github release on my server and this code should get some results near 50%. However, I am not sure whether the CUDA versions/CUDNN version would affect the results. They should be not!

  BTW, I have all the original code (before code cleaning)/snapshots frozen here: Release of pretrained snapshots #11 (comment), which could reach the (almost) exact number shown in my paper, where the snapshots should be compatible with the current GitHub codebase.

  So another possibility besides the PyTorch version is that I might mis-delete some code when I do the code cleaning. I have checked it multiple times but I might still miss something.

[Zihan-Liu-00]

Hi,

Thanks a lot for kind answering!

About beam search, we was applying the beam search test to the agent but only see an increase in path length. Later I download the original code of this paper, but got confused that we found no difference in the beam-search part. Also we find the beam search function is not called in the original code, therefore the trigger '-beam' actually cannot work. May I ask what operations to do if we want to apply beam search?

BTW, thanks to your advise, we're now trying to run with PyTorch 0.4.0 to see the influence of pytorch version. Hopefully it will get better.

Best

[airsplay]

The beam-search is not officially supported by this code by intention since I do not consider BS as a correct setup of navigation.

To get the BS results, I provide the script and snapshots here:
#11

In general, it would first calculate the best ratio between speaker and agent, then do beam-search.