Replicating the results of the pre-trained models.

[weihaosky]

Thanks for releasing this amazing work!
However, I cannot replicate the results of the pre-trained models using the provided code.
The results after training with the provided code are

• RVQ Reconstruction:

FID: 0.032, conf. 0.000                                                                                     
Diversity: 9.676, conf. 0.054                                                                       
TOP1: 0.507, conf. 0.003, TOP2. 0.697, conf. 0.002, TOP3. 0.794, conf. 0.002                                
Matching: 3.024, conf. 0.009                                                                                
MAE:0.042, conf.0.000

• Text2motion Generation

FID: 0.148, conf. 0.006
Diversity: 9.576, conf. 0.079
TOP1: 0.504, conf. 0.003, TOP2. 0.699, conf. 0.003, TOP3. 0.795, conf. 0.002
Matching: 3.062, conf. 0.010
Multimodality:1.327, conf.0.044

May I ask where is the problem? Many thanks!

[Murrol]

Hi, thanks for your interest!

We are happy to figure out the problem together with you. Could you please provide the configures for training (opt file) and testing (scripts) for each stage?

[weihaosky]

I just use the provided code and commands without any change.
vq training: opt.txt
mtrans training: opt.txt
rtrans training: opt.txt
vq test: rvq_nq6.log
mtrans test: evaluation.log

[Murrol]

I just use the provided code and commands without any change.
vq training: opt.txt
mtrans training: opt.txt
rtrans training: opt.txt
vq test: rvq_nq6.log
mtrans test: evaluation.log

Hi wenhao, thanks for your info.

I've checked the configures and corrected the scripts in our README. We used --gamma 0.05 to train the rvq. I just re-trained the rvq and got the following results:

Hope you will find it useful.

[weihaosky]

I just use the provided code and commands without any change.
vq training: opt.txt
mtrans training: opt.txt
rtrans training: opt.txt
vq test: rvq_nq6.log
mtrans test: evaluation.log

Hi wenhao, thanks for your info.

I've checked the configures and corrected the scripts in our README. We used --gamma 0.05 to train the rvq. I just re-trained the rvq and got the following results:

Hope you will find it useful.

Hi, after setting --gamma 0.05 I got a worse result:

net_best_fid.tar final result, epoch 36
        FID: 0.040, conf. 0.000
        Diversity: 9.598, conf. 0.097
        TOP1: 0.506, conf. 0.002, TOP2. 0.696, conf. 0.002, TOP3. 0.793, conf. 0.002
        Matching: 3.027, conf. 0.008
        MAE:0.039, conf.0.000

[aszxnm]

I meet the same problem and get the same results with @weihaosky .
Have you solved the problem？ Thanks.

[weihaosky]

I meet the same problem and get the same results with @weihaosky . Have you solved the problem？ Thanks.

No. I still cannot replicate the results.

[Seoneun]

Have you solved the problem now? @weihaosky @aszxnm
Thanks.

[JHang2020]

the same for me...

I meet the same problem and get the same results with @weihaosky . Have you solved the problem？ Thanks.

No. I still cannot replicate the results.

[wang-zm18]

The same for me, and another confusing problem is the MPJPE of the reconstruction setting on the kit dataset:

The command I run is python eval_t2m_vq.py --gpu_id 0 --name rvq_nq6_dc512_nc512_noshare_qdp0.2_k --dataset_name kit --ext rvq_nq6, and I used the default pretrained checkpoint.

Similarly, when I test it on the Humanml dataset, the results is :

It is obvious that the MPJPE results on these two datasets differ by several orders of magnitude.
Does anyone else have the same problem?

[HitBadTrap]

I just use the provided code and commands without any change.
vq training: opt.txt
mtrans training: opt.txt
rtrans training: opt.txt
vq test: rvq_nq6.log
mtrans test: evaluation.log

Hi wenhao, thanks for your info.
I've checked the configures and corrected the scripts in our README. We used --gamma 0.05 to train the rvq. I just re-trained the rvq and got the following results:

Hope you will find it useful.

Hi, after setting --gamma 0.05 I got a worse result:

net_best_fid.tar final result, epoch 36
        FID: 0.040, conf. 0.000
        Diversity: 9.598, conf. 0.097
        TOP1: 0.506, conf. 0.002, TOP2. 0.696, conf. 0.002, TOP3. 0.793, conf. 0.002
        Matching: 3.027, conf. 0.008
        MAE:0.039, conf.0.000

Have you solved the problem now? @weihaosky @aszxnm

Thanks.

[aszxnm]

I just use the provided code and commands without any change.
vq training: opt.txt
mtrans training: opt.txt
rtrans training: opt.txt
vq test: rvq_nq6.log
mtrans test: evaluation.log

Hi wenhao, thanks for your info.
I've checked the configures and corrected the scripts in our README. We used --gamma 0.05 to train the rvq. I just re-trained the rvq and got the following results:

Hope you will find it useful.

Hi, after setting --gamma 0.05 I got a worse result:

net_best_fid.tar final result, epoch 36
        FID: 0.040, conf. 0.000
        Diversity: 9.598, conf. 0.097
        TOP1: 0.506, conf. 0.002, TOP2. 0.696, conf. 0.002, TOP3. 0.793, conf. 0.002
        Matching: 3.027, conf. 0.008
        MAE:0.039, conf.0.000

Have you solved the problem now? @weihaosky @aszxnm

Thanks.

No. I still cannot replicate the results.

[Murrol]

Thank you all for your attempts to replicate the results. I just got some time to re-train the masked-transformer and res-transformer using our released code. Here are some results for your reference:

I used the rvq checkpoint I obtained here #27 (comment). I used the following scripts to train the m-trans and r-trans:

python train_t2m_transformer.py --name mtrans_replicate --gpu_id 1 --dataset_name t2m --batch_size 64 --vq_name rvq_replicate

python train_res_transformer.py --name rtrans_replicate --gpu_id 2 --dataset_name t2m --batch_size 64 --vq_name rvq_replicate --cond_drop_prob 0.2 --share_weight

evaluation script:

python eval_t2m_trans_res.py --res_name rtrans_replicate --dataset_name t2m --name mtrans_replicate --gpu_id 1 --cond_scale 4 --time_steps 10 --ext evaluation_replicate

The above results are obtained from these scripts without any modification to this code base. The replicate experiments were done on a single RTX 2080 Ti GPU, torch==1.7.1. For the processed dataset cloned from the original HumanML3D project, please send inquiry to cguo2@ualberta.ca or ymu3@ualberta.ca

[wang-zm18]

#27 (comment)
Does anybody knows the problem?

[Murrol]

The same for me, and another confusing problem is the MPJPE of the reconstruction setting on the kit dataset:

The command I run is python eval_t2m_vq.py --gpu_id 0 --name rvq_nq6_dc512_nc512_noshare_qdp0.2_k --dataset_name kit --ext rvq_nq6, and I used the default pretrained checkpoint.

Similarly, when I test it on the Humanml dataset, the results is :

It is obvious that the MPJPE results on these two datasets differ by several orders of magnitude. Does anyone else have the same problem?

The scale is different. Check the Mean.py files.

[wang-zm18]

I re-download the KIT-ML dataset from Link directly. While it still remains this problem. Maybe it is not the mean problem？Thank you in advance! @Murrol

[wang-zm18]

#27 (comment)

Any other suggestions? Thanks!

[Murrol]

I re-download the KIT-ML dataset from Link directly. While it still remains this problem. Maybe it is not the mean problem？Thank you in advance! @Murrol

Scale difference means that the scale of AMASS(HumanML3D) and kit data is different. Take an example, inch vs cm.

"the MPJPE results on these two datasets differ by several orders of magnitude" should not be an issue.

[wang-zm18]

I re-download the KIT-ML dataset from Link directly. While it still remains this problem. Maybe it is not the mean problem？Thank you in advance! @Murrol

Scale difference means that the scale of AMASS(HumanML3D) and kit data is different. Take an example, inch vs cm.

"the MPJPE results on these two datasets differ by several orders of magnitude" should not be an issue.

Oh, thanks! I checked the range of means from these two datasets, the mean of KIT datasets is three orders of magnitude higher than that of HumanML3D on average. Thank you! @Murrol

[Suzixin7894946]

Thank you all for your attempts to replicate the results. I just got some time to re-train the masked-transformer and res-transformer using our released code. Here are some results for your reference:

I used the rvq checkpoint I obtained here #27 (comment). I used the following scripts to train the m-trans and r-trans:

python train_t2m_transformer.py --name mtrans_replicate --gpu_id 1 --dataset_name t2m --batch_size 64 --vq_name rvq_replicate

python train_res_transformer.py --name rtrans_replicate --gpu_id 2 --dataset_name t2m --batch_size 64 --vq_name rvq_replicate --cond_drop_prob 0.2 --share_weight

evaluation script:

python eval_t2m_trans_res.py --res_name rtrans_replicate --dataset_name t2m --name mtrans_replicate --gpu_id 1 --cond_scale 4 --time_steps 10 --ext evaluation_replicate

The above results are obtained from these scripts without any modification to this code base. The replicate experiments were done on a single RTX 2080 Ti GPU, torch==1.7.1. For the processed dataset cloned from the original HumanML3D project, please send inquiry to cguo2@ualberta.ca or ymu3@ualberta.ca

Hello, I have some doubts regarding the reproduction of results. Currently, the minimum FID value I am getting while replicating the results on t2m is 0.059, and other metrics are normal. However, after adding certain modules, the FID value is 0.047, but in your paper, the FID value is 0.045. I would like to know if this is an improvement? I am a bit unclear whether my modifications are effective, or if, in fact, my reproduced results are not correct.

[Tiooo111]

My rvq replicated FID is also 0.032, has it been solved?

[cr8br0ze]

My rvq replicated FID is also 0.032, has it been solved?

Same question, my rvq replicated FID is similar, (0.33).
Using this rvq and given terminal command here to train mask transformer and residual transformer leads me to a generation fid only around (0.182).
But if I use the given vq checkpoint to train both transformers, I can achieve a more similar result to here
Hope we can have a solution soon since rvq quality matters a lot for generation here.