-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
unable to reproduce repllama performance #129
Comments
for reproducing repllama training, I'd suggest using the main branch and using the latest code
you can either use the mistral as initialization for a little higher effectiveness or use llama2 for reproducing. Notice that the training data |
Hi, Thank you for the reply!
|
I would suggest keep the accumulation steps same as before. |
Hi @MXueguang @amy-hyunji, Yes I tried a similar training config as @amy-hyunji which set gradient_accumulation_steps to 1 and it won't work well..., adding back gradient_accumulation_steps to 4 will reproduce the results. Note gradient_accumulation_steps will affect the number of negatives per training example, This is odd to me as well as the experience from the literature is more negative is better... |
Traceback (most recent call last): why i am getting this error , during saviung a checkpoint |
Hi @riyajatar37003 , is your DenseModel a class of DenseModel(EncoderModel):? The save only support tevatron EncoderModel here, if the DenseModel is your own implemented model then you may need to add into the supported model list. |
what is mean by train group size and how it important |
@MXueguang I also had a similar experience with failing to reproduce using that script. Using your suggested config above I get the 72.95 nDCG for DL19 and 70.6 nDCG@10 for DL20 (compared to the paper's 74.3 and 72.1). I used the gradient steps parameters (batch size of 8, gradient accumulation of 4) suggested by @ArvinZhuang. Are you able to reproduce it with that code or did the recent updates to |
hi @orionw, which base llm were you using? llama2? |
Yes, llama2 Full config (if helpful):
|
do you have msmarco dev set results of the ckpt you get, does it matches? |
I didn't evaluate dev, I stopped at these two for compute reasons. |
I see. I'll schedule a training to see how it goes on my end... |
Thanks for looking into it! |
one difference I observed is the lora_r parameters, it was set to 32 in original experiment, now the default one is 8. I am seeing if this would affect trec dl results. |
Hmm, could be @MXueguang! Let me know if that fixes it! I was curious about potential discrepancies with the number of GPUs/batch size? I don't know the exact command you used, but your paper said it was trained with 16 V100 GPUs. Perhaps having the batches distributed larger makes it better (like @ArvinZhuang was saying). Could also be related to the cross device negatives/group size (did that logic change in the recent version?). I unfortunately don't have access to a node with 16 GPUs to test it out on. |
yeah @orionw , I. am running it, but still need a day or so to get a number due to limited compute... |
Hi @orionw, my reproduce with lora_r=32, others keeps same, gives: dev/dl19 a bit higher and dl20 a bit lower than original experiments. |
Awesome, thank you so much @MXueguang! Those differences could easily be due to random seeds. Really appreciate you looking into it :) |
Hello, thanks for sharing great work!
I tried trying repllama myself with the repllama branch but failed to reproduce the numbers.
Could you check whether any of my hyperparameters are wrong? I add the training script underneath
I am currently running on 8 A100 and the result I got is NDCG@10: 0.3959, NDCG@100: 0.4515
When I download the released model from hf I get the number in the paper, which I assume the issue is from the training not evaluation.
Thanks :)
The text was updated successfully, but these errors were encountered: