-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Release of pretrained snapshots #11
Comments
Hi, The single-run model could be achieved by the code in this Github. However, the beam-search part is still ineffective for now... It seems that I misedit some code thus the speaker score is not accurate. It only affects the beam-search inference and would not change the results of agent/speaker training. I am still looking into this issue but haven't located it. So I share my code before cleaning here, which could achieve the results I reported in the paper: https://drive.google.com/file/d/1ML98KBE5MkGt987vc0KUhoOPbB6vGNPY/view?usp=sharing. I hope it helps! Thanks. |
Thank you for your response.
Yes, I have been able to reproduce the results of the single-run model using the codebase. Thanks a lot for sharing the code and the snapshots. Could you also mention the values of the hyperparameters used in beam search, mainly Thanks! |
In my memory, the beam size is set to 20. Following the paper `speaker and follower', alpha (the ratio between speaker score and agent score) is tuned based on the validation set, where I select the one with the highest accuracy from [0.00, 0.01, 0.02, ..., 0.99, 1.00]. |
Thank you. Could you also please provide me access to the candidate features |
The file is available here: https://drive.google.com/file/d/1I3o_lp13zmM1ohIbPGqljSRN2GLpBjbt/view?usp=sharing. Since the candidate locations are not always centered at the camera if only 36 views are used. I rotate the camera to the specific angel and capture new images for each candidate; the features in this file is the ImageNet features of these images. It's historic design but I remove it from both this Github version and the code I shared. The shared code will still load this tsv file but would not use it when creating the features. |
Thanks a lot. I have tried the new codebase that you provided in the Google Drive link. I have been able to reproduce similar numbers as reported in the paper for the single-run model using the snapshots you provided.
My results were (logfile):
But, for the beam search part, I was not able to get close to 69% Success Rate for Val Unseen and 75% for Val Seen. I have tried using both Pytorch 0.4.1 and 1.1.
And here is the logfiles for param search (using --paramSearch) and results for Pytorch 0.4.1 and I am not sure what the issue is. Did I miss passing any arguments? Please let me know if you have any ideas. Also, for the beam size 20, the trajectory length is around 400-450 and the results reported in the paper have trajectory length close to 660-700. So, I tried running beam search with beam size 30, and I got trajectory length 700-750. Maybe, you had set the beam size to a number higher than 20?
Please let me know if you have any ideas. Thanks a lot! |
Thank you a lot for verifying it!!! I tested my scripts under PyTorch 1.2.0 and PyTorch 0.4.0 on my server and the logs are the same:
I do not remember whether the code/snapshots are the same. In case I did anything differently before, I re-upload everything to the google drive (including the source files, running scripts, and snapshots). Could you help download and retest it? Please let me know if there is any problem. Here are some other specifications of the running environments, which might be related to the results. However, I do not think that they would cause this problem for now...
|
By the way, since the feature loading costs a lot of time, adding |
Thanks a lot for providing the new checkpoints. I have been able to verify the results for beam search using the new weights. My results are exactly the same as yours. Thanks, closing the issue now. |
Thanks for verifying it! I will look into the difference between these codes and update this Github repo to support beam-search. |
Hi,
Could you please release the snapshot (model weights) of the best model for single run and beam search that led to the numbers reported in the paper?
Thanks!
The text was updated successfully, but these errors were encountered: