Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ICON+SCANimate #204

Open
Bill-WangJiLong opened this issue Apr 21, 2023 · 10 comments
Open

ICON+SCANimate #204

Bill-WangJiLong opened this issue Apr 21, 2023 · 10 comments

Comments

@Bill-WangJiLong
Copy link

Hi, thank you very much for the great work, it's cool. I want to implement ICON+SCANimate now, but I have two questions I would like to ask you.

  1. SMPL fit
    The smpl parameter output of ICON is a .npy file with a size of a few kb, while the SMPL fit file format of SCANimate is a .npz file with a size of several hundred kb. Please tell me in the infer.py
    np. save(
    f"{args.out_dir}/{cfg.name}/obj/{data['name']}_smpl.npy", smpl_info, allow_pickle=True
    )
    Is it enough to change the file format to npz in this sentence? Do I need to do anything else to generate the parameter npz file of SMPL?
  2. Remove untextured meshes
    Can I modify the obj file when infer.py is saved, please give me some details if it is convenient, thank you!
    Thanks again for your great work!
@Bill-WangJiLong
Copy link
Author

Bill-WangJiLong commented Apr 21, 2023

Further, I seem to have solved the problem of the npz file. The problem now is that the size of the pose array in the npy file saved by ICON is different from the size of the CAPE data set used by SCANimate. After flattening, the ICON is 138 (23x6) or I It can be modified to 207 (23x3x3), but the CAPE is 72(24x3), which leads to the following error, how should I solve it, thank you.
image
image

@Bill-WangJiLong
Copy link
Author

Now I seem to have solved this problem, but the result with SCANimate is very poor, can't see that it is a human shape, but I haven't tried to remove all invisible faces. Is this related to the problem, or should I make some changes to SCANimate?
image

@YuliangXiu
Copy link
Owner

YuliangXiu commented Apr 25, 2023

Hi, thank you very much for the great work, it's cool. I want to implement ICON+SCANimate now, but I have two questions I would like to ask you.

  1. SMPL fit
    The smpl parameter output of ICON is a .npy file with a size of a few kb, while the SMPL fit file format of SCANimate is a .npz file with a size of several hundred kb. Please tell me in the infer.py
    np. save(
    f"{args.out_dir}/{cfg.name}/obj/{data['name']}_smpl.npy", smpl_info, allow_pickle=True
    )
    Is it enough to change the file format to npz in this sentence? Do I need to do anything else to generate the parameter npz file of SMPL?
  2. Remove untextured meshes
    Can I modify the obj file when infer.py is saved, please give me some details if it is convenient, thank you!
    Thanks again for your great work!
  1. The raw output from ICON cannot be directly used for SCANimate, but it should contain all the info used for SCANimate.
  2. Just remove the invisible faces, you can find the related functions from query_color

@Bill-WangJiLong
Copy link
Author

Hi, thank you very much for the great work, it's cool. I want to implement ICON+SCANimate now, but I have two questions I would like to ask you.

  1. SMPL fit
    The smpl parameter output of ICON is a .npy file with a size of a few kb, while the SMPL fit file format of SCANimate is a .npz file with a size of several hundred kb. Please tell me in the infer.py
    np. save(
    f"{args.out_dir}/{cfg.name}/obj/{data['name']}_smpl.npy", smpl_info, allow_pickle=True
    )
    Is it enough to change the file format to npz in this sentence? Do I need to do anything else to generate the parameter npz file of SMPL?
  2. Remove untextured meshes
    Can I modify the obj file when infer.py is saved, please give me some details if it is convenient, thank you!
    Thanks again for your great work!
  1. The raw output from ICON cannot be directly used for SCANimate, but it should contain all the info used for SCANimate.
  2. Just remove the invisible faces, you can find the related functions from query_color

Thank you for your reply. I have revised the first question by referring to other answers. Thank you for your reminder on the second question.
Another question I have is, do I need to change any parts of SCANimate besides changing the output of ICON? Such as training parameters or LOSS functions, except for training epochs.
Although I was able to run through the process, the results I got were very poor, but this may also be related to the fact that I used too little data during the experiment. I want to make sure that my method is correct.

@YuliangXiu
Copy link
Owner

YuliangXiu commented Apr 25, 2023

Hi, thank you very much for the great work, it's cool. I want to implement ICON+SCANimate now, but I have two questions I would like to ask you.

  1. SMPL fit
    The smpl parameter output of ICON is a .npy file with a size of a few kb, while the SMPL fit file format of SCANimate is a .npz file with a size of several hundred kb. Please tell me in the infer.py
    np. save(
    f"{args.out_dir}/{cfg.name}/obj/{data['name']}_smpl.npy", smpl_info, allow_pickle=True
    )
    Is it enough to change the file format to npz in this sentence? Do I need to do anything else to generate the parameter npz file of SMPL?
  2. Remove untextured meshes
    Can I modify the obj file when infer.py is saved, please give me some details if it is convenient, thank you!
    Thanks again for your great work!
  1. The raw output from ICON cannot be directly used for SCANimate, but it should contain all the info used for SCANimate.
  2. Just remove the invisible faces, you can find the related functions from query_color

Thank you for your reply. I have revised the first question by referring to other answers. Thank you for your reminder on the second question. Another question I have is, do I need to change any parts of SCANimate besides changing the output of ICON? Such as training parameters or LOSS functions, except for training epochs. Although I was able to run through the process, the results I got were very poor, but this may also be related to the fact that I used too little data during the experiment. I want to make sure that my method is correct.

I just use the default setting for training, but firstly you should make sure the inputs of SCANimate are correct.

Convert the ICON's pose parameters into the right format, which are compatible with SCANimate's input. And only use the visible vertices to train all the networks (skinning, geometry, texture) of SCANimate.

@Bill-WangJiLong
Copy link
Author

Hi, thank you very much for the great work, it's cool. I want to implement ICON+SCANimate now, but I have two questions I would like to ask you.

  1. SMPL fit
    The smpl parameter output of ICON is a .npy file with a size of a few kb, while the SMPL fit file format of SCANimate is a .npz file with a size of several hundred kb. Please tell me in the infer.py
    np. save(
    f"{args.out_dir}/{cfg.name}/obj/{data['name']}_smpl.npy", smpl_info, allow_pickle=True
    )
    Is it enough to change the file format to npz in this sentence? Do I need to do anything else to generate the parameter npz file of SMPL?
  2. Remove untextured meshes
    Can I modify the obj file when infer.py is saved, please give me some details if it is convenient, thank you!
    Thanks again for your great work!
  1. The raw output from ICON cannot be directly used for SCANimate, but it should contain all the info used for SCANimate.
  2. Just remove the invisible faces, you can find the related functions from query_color

Thank you for your reply. I have revised the first question by referring to other answers. Thank you for your reminder on the second question. Another question I have is, do I need to change any parts of SCANimate besides changing the output of ICON? Such as training parameters or LOSS functions, except for training epochs. Although I was able to run through the process, the results I got were very poor, but this may also be related to the fact that I used too little data during the experiment. I want to make sure that my method is correct.

I just use the default setting for training, but firstly you should make sure the inputs of SCANimate are correct.

Convert the ICON's pose parameters into the right format, which are compatible with SCANimate's input. And only use the visible vertices to train all the networks (skinning, geometry, texture) of SCANimate.

Thank you very much for the guide, I will give it a try

@Bill-WangJiLong
Copy link
Author

Hi, thank you very much for the great work, it's cool. I want to implement ICON+SCANimate now, but I have two questions I would like to ask you.

  1. SMPL fit
    The smpl parameter output of ICON is a .npy file with a size of a few kb, while the SMPL fit file format of SCANimate is a .npz file with a size of several hundred kb. Please tell me in the infer.py
    np. save(
    f"{args.out_dir}/{cfg.name}/obj/{data['name']}_smpl.npy", smpl_info, allow_pickle=True
    )
    Is it enough to change the file format to npz in this sentence? Do I need to do anything else to generate the parameter npz file of SMPL?
  2. Remove untextured meshes
    Can I modify the obj file when infer.py is saved, please give me some details if it is convenient, thank you!
    Thanks again for your great work!
  1. The raw output from ICON cannot be directly used for SCANimate, but it should contain all the info used for SCANimate.
  2. Just remove the invisible faces, you can find the related functions from query_color

Thank you for your reply. I have revised the first question by referring to other answers. Thank you for your reminder on the second question. Another question I have is, do I need to change any parts of SCANimate besides changing the output of ICON? Such as training parameters or LOSS functions, except for training epochs. Although I was able to run through the process, the results I got were very poor, but this may also be related to the fact that I used too little data during the experiment. I want to make sure that my method is correct.

I just use the default setting for training, but firstly you should make sure the inputs of SCANimate are correct.

Convert the ICON's pose parameters into the right format, which are compatible with SCANimate's input. And only use the visible vertices to train all the networks (skinning, geometry, texture) of SCANimate.

Hello, sorry to bother you, I tried according to your method, including modifying the pose parameters and removing the invisible faces, but in SCANimate, neither the cano model nor the reposed model I trained had legs, and the final result still appeared Some redundant points, where is the problem? Should I modify the result generated by ICON or should I modify SCANimate somewhere?
cano
image
reposed
image
final output
image

@tao-he
Copy link

tao-he commented May 8, 2023

@Bill-WangJiLong did you get the correct avatar by SCANimate?

@Bill-WangJiLong
Copy link
Author

@Bill-WangJiLong did you get the correct avatar by SCANimate?

The problem I mentioned above seems to be caused by inconsistent global scales, but I ultimately failed to achieve good results. I am not sure if my method is incorrect or the data is insufficient

@2579690686
Copy link

此外,我似乎已经解决了npz文件的问题。现在的问题是,ICON 保存的 npy 文件中 pose 数组的大小与 SCANimate 使用的 CAPE 数据集的大小不同。展平后,ICON为138(23x6)或I可以修改为207(23x3x3),但CAPE为72(24x3),导致以下错误,我该如何解决,谢谢。 image image

hello, how did you solve this problem, thank you very much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants