Whether the inference speed can be optimized

[NickJony]

Great job! I do inference on 3090, it takes about 0.7 seconds to calculate the time of two images, and what other operations can be used for inference acceleration, in addition to reducing the image resolution.

[Parskatt]

You can try fp16, see my code in roma for some examples.

I haven't gotten torch.compile to work, but it could improve things.