Ensure the alternative ND manager can use GPUs #3138
Conversation
| alternativeManager = engine.newBaseManager(device); | ||
| } catch (RuntimeException | UnsatisfiedLinkError ignore) { | ||
| // Use the default device instead. | ||
| alternativeManager = engine.newBaseManager(); |
There was a problem hiding this comment.
It's arguable using the same device is the best. For simplicity, I prefer use default device directly.
The UnsatisfiedLinkError error you hit is a caching issue (only affect development time), not related here. You can just need to clear your rust JNI cache:
rm -rf ~/.djl.ai/tokenizer
gradle :extensions:tokenizers:clean
This makes a massive performance difference when using OnnxRuntime, which delegates to PyTorch for some NDArray operations.
frankfliu
force-pushed
the
alternative-engine-use-gpu
branch
from
012dd8d
to
3cf7bf3
Compare
|
@frankfliu - on my tests with my application (which does a lot of non-GPU work too), my original change ran in 6m 8s, but with the change not to keep the device, this went to 6m 25s. So it is a significant performance hit on multi-GPU systems, and for other workloads the discepancy will be much higher. I also saw GPU 0 having way higher (too high) utilisation compared to the other GPUs since this is the default device set for the alternative ND manager, for all my worker processes (16 on my test). Can't we keep my original change without the catch for UnsatisfiedLinkError? Otherwise this puts OnnxRuntime further away from PyTorch again, which won't have this issue. |
|
You are right, in multiple GPU case, it does make sense to keep original device. I just doesn't like try catch. Let me think a bit, there should a simple way. |
|
@frankfliu - any ideas with this? I am pretty keen to get that multi-GPU performance back. 😉 |
|
I created a PR: #3146 |
|
Awesome - thank you. That looks like a better way of achieving it too. |
This makes a massive performance difference when using OnnxRuntime, which delegates to PyTorch for some NDArray operations. This almost makes OnnxRuntime run as fast compared to PyTorch. Without this change, OnnxRuntime was over 3 times slower.