Pytorch preset missing torch::jit related functions #1068
Comments
|
BTW, if your use case is about exporting an existing PyTorch model, you may want to try to export it to ONNX instead since that's well supported by ONNX Runtime: https://github.com/bytedeco/javacpp-presets/tree/master/onnxruntime |
|
I managed to get my model into Java using (https://github.com/pytorch/java-demo). My use case is solely inference using my model but I needed to be able to call other helper methods attached to the model in addition to the forward method. I will take a look at whether the model exported in ONNX can provide the same functionality. |
|
It seems like the pytorch_java_only library underpinning the java-demo only supports CPU and VULKAN (= mobile android GPUs) inference, not NVIDIA GPU inference: Device.java has only CPU and VULKAN enum values. ONNX looks workable and like it supports GPU, but a lot more complicated than |
|
Ok, it's done! We can now load TorchScript models like this: |
|
Thank you! I've taken the code for a spin and got it to produce an output. There's a couple of question marks:
val javacpp = "1.5.7-SNAPSHOT"
libraryDependencies += "org.bytedeco" % "pytorch-platform" % s"1.9.0-$javacpp"
libraryDependencies += "org.bytedeco" % "pytorch-platform-gpu" % s"1.9.0-$javacpp"
libraryDependencies += "org.bytedeco" % "cuda-platform-redist" % s"11.4-8.2-$javacpp"
libraryDependencies += "org.bytedeco" % "mkl-platform-redist" % s"2021.3-$javacpp"in my build.sbt and a GPU in my win10 machine. I do have CUDA drivers of most likely the wrong version installed, so depending on the load order priority of various DLLs, it may have tried the wrong ones?
|
You'll need the latest version of the drivers for CUDA 11.4 to be able to load, yes. Mapping a C++ API to Java isn't an easy task, and JavaCPP doesn't try to do anything too fancy, so there are going to be rough edges. Are there any issues that you were not able to work around? I would like to prioritize working on issues that prevent users from getting anything done at all, first. Once those issues are fixed, we can revisit usability issues. Thanks for testing! |
|
Reading https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html it says that for I did however have multiple versions of CUDA installed & in the path. I have removed these from the path, but I still get The code does seem to work in the sense that it can run on CPU and produce a plausible output, but I have been unable to get it to run on GPU. Here's a dump running it with |
|
There's more in the log as the code runs - it seems like JavaCPP retries loading torch_cuda as it's accessed again. I can add more if you like, the dump above was quite long as it was ;) Here's my simple test script: object PytorchJavacppSpike {
def main(args: Array[String]): Unit = {
System.setProperty("org.bytedeco.javacpp.logger.debug", "true")
/* try to use MKL when available *//* try to use MKL when available */
System.setProperty("org.bytedeco.openblas.load", "mkl")
System.out.println("loading module...")
var module: JitModule = null
try // Deserialize the ScriptModule from a file using torch::jit::load().
module = load("demo-model.pt1")
catch {
case e: Exception =>
System.err.println("error loading the model")
throw e
}
System.out.println("ok")
// Create a vector of inputs.
val inputs = new IValueVector()
//inputs.push_back(new IValue(ones(1, 3, 224, 224)))
val in0: Tensor = reshape(arange(new Scalar(1), new Scalar(7)), 2, 3)
println("in0: ")
print(in0)
// val p = new Pointer(FloatBuffer.wrap(Array[Float](1,2,3,4,5,6)))
//val p = new FloatPointer(1,2,3,4,5,6)
val p = new FloatPointer(FloatBuffer.wrap(Array[Float](1, 2, 3, 4, 5, 6)))
val in1: Tensor = from_blob(p, 2, 3)
println("in1: ")
print(in1)
inputs.push_back(new IValue(in1))
inputs.push_back(new IValue(new Scalar(3.0)))
// Execute the model and turn its output into a tensor.
val output: Tensor = module.forward(inputs).toTensor
System.out.println("sizes: " + output.sizes().vec())
System.out.println("output: " + output)
System.out.println("pytorch output ---- ")
print(output)
System.out.println("pytorch output ---- ")
val device = output.device()
System.out.println("device type: " + device.`type`())
System.out.println("device is_cpu: " + device.is_cpu())
System.out.println("device is_cuda: " + device.is_cuda())
System.out.println("device is_hip: " + device.is_hip())
System.out.println("device is_xpu: " + device.is_xpu())
System.out.println("torch.cuda_is_available() " + torch.cuda_is_available())
System.out.println("torch.cuda_device_count() " + torch.cuda_device_count())
System.out.println("torch.cuda() " + torch.cuda())
System.out.println("module.parameters(): " + module.parameters().size())
System.out.println("module.named_parameters(): " + module.named_parameters().size())
System.out.println("module.attributes(): " + module.attributes().size())
System.out.println("module.named_attributes(): " + module.named_attributes().size())
val methodOpt: MethodOptional = module.find_method("forward")
System.out.println("methodOpt: " + methodOpt)
val method = methodOpt.get()
System.out.println("method: " + method)
System.out.println("method.num_inputs: " + method.num_inputs())
System.out.println("method.name: " + method.name().getString)
System.out.println("method.function().pretty_print_schema: " + method.function().pretty_print_schema().getString())
val schema = method.function().getSchema
System.out.println("method.function().getSchema.arguments: " + schema.arguments())
def printArguments(label: String, root: Argument): Unit = {
System.out.println(s" ------ ${label} ------")
System.out.println("root.type(): " + root.`type`())
System.out.println("root.type(): " + root.`type`().str().getString)
System.out.println("root.`type`().kind(): " + root.`type`().kind())
if (root.name() != null && !root.name().isNull) System.out.println("root.name().getString: " + root.name().getString)
System.out.println("root.N(): " + root.N())
System.out.println("root.N().isNull: " + root.N().isNull)
// System.out.println("root.N().get(): " + root.N().get())
// for (i <- 0 to root.N().get()) {
// val arg = root.position(i)
//
// System.out.println(s"arg_${i}.type(): " + arg.`type`())
// System.out.println(s"arg_${i}.name().getString: " + arg.name().getString)
// }
}
printArguments("args", schema.arguments())
printArguments("returns", schema.returns())
}
with build.sbt: name := "pytorch_spike"
version := "1.0"
scalaVersion := "2.12.13"
resolvers += "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"
val javacpp = "1.5.7-SNAPSHOT"
libraryDependencies += "org.bytedeco" % "pytorch-platform" % s"1.9.0-$javacpp"
libraryDependencies += "org.bytedeco" % "pytorch-platform-gpu" % s"1.9.0-$javacpp"
libraryDependencies += "org.bytedeco" % "cuda-platform-redist" % s"11.4-8.2-$javacpp"
libraryDependencies += "org.bytedeco" % "mkl-platform-redist" % s"2021.3-$javacpp"The |
You're going to need |
|
Installed CUDA 11.4 and got the missing dll you referenced, and now For future reference: https://developer.download.nvidia.com/NsightVisualStudio/2.2/Documentation/UserGuide/HTML/Content/NVIDIA_Tools_Extension_Library_NVTX.htm |
|
Should that DLL be included in the CUDA presets for windows? I reinstalled the NVIDIA driver first and didn't get the DLL, I had to install CUDA itself. |
Hum, ok, so it gets installed with CUDA. It looks like a deprecated API, but since PyTorch needs it, sure, let's bundle it! I've added it in commit 02c27e4, and the snapshots have been updated, so please give it a try with |
|
Success! I removed the recently installed CUDA libs from my PATH, and re-fetched Looked like you added |
|
The import of torch visions and other torch-script models works like a charm (using mostly CPU+MKL). I noticed there is also the torch.trace function available, but based on the signature that seems not to be the same as Is there a way to create a torch-script JitModule from a regular Module defined in Java? So something like: |
@jbaron According to issue pytorch/pytorch#17614 (comment) it doesn't appear possible, no. But it's still possible to train a |
|
Thanks. I am a bit surprised to see that tracing is not done by recording the native C++ operations since I would expect it to be almost the same as a forward pass in training. But seems a lot of the tracing logic resides in Python, which is a shame for re-usability. |
|
Again, see issue pytorch/pytorch#26086 about that:
|
|
Did something happen to the snapshots? I'm re-importing my project: resolvers += "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"
// https://github.com/bytedeco/javacpp-presets/tree/master/pytorch
val javacpp = "1.5.7-SNAPSHOT"
libraryDependencies += "org.bytedeco" % "pytorch-platform" % s"1.10.0-$javacpp"
libraryDependencies += "org.bytedeco" % "pytorch-platform-gpu" % s"1.10.0-$javacpp"
libraryDependencies += "org.bytedeco" % "cuda-platform-redist" % s"11.4-8.2-$javacpp"
libraryDependencies += "org.bytedeco" % "mkl-platform-redist" % s"2021.4-$javacpp"And get a bunch of errors: |
|
Can't find the bytedeco pytorch libraries on https://oss.sonatype.org/#nexus-search;quick~pytorch ? |
|
Ah, yes, there's a long list of them there, but the specific ones that I get error messages for aren't for some reason. So there are many windows snapshots, but no There is a |
|
@jxtps I've fixed the snapshots, please give it a try again! As for GPU support, please use LibTorch for now: |
|
Not seeing the platform-specific jars in https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/pytorch-platform/1.10.1-1.5.7-SNAPSHOT/ ? |
|
They're present for 1.10.0 ( https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/pytorch/1.10.0-1.5.7-SNAPSHOT/ ) but then I get a weird linking error for jnitorch: The file is present on disk, but |
|
Yes, that means there's some function definitions missing from the DLLs you are giving it. It's probably just because the versions of LibTorch you're trying to use isn't the right one. If you're sure your versions of PyTorch match, please try to use something like the Dependencies tool to find out what exactly is missing: |
Are you looking for https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/pytorch/1.10.1-1.5.7-SNAPSHOT/ ? |
|
The specific SBT errors I'm getting with 1.10.1: I can see the relevant files on https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/pytorch/1.10.1-1.5.7-SNAPSHOT/ (sorry I got the paths mixed up), but I tried adding |
|
It sounds like you're trying to use a very old version of sbt. IIRC, all these bugs have been fixed a few years ago. |
|
Ah, right, I had reused an old project without looking at the SBT version, that resolved it, thanks! The linking issue remains unfortunately. I am using 1.10.1 for both
When I hover over the red icon it says |
|
Click on it to see the missing imports... |
|
I've clicked on it plenty, but I don't see any missing imports? The window has four sections:
These seem to contain: So presumably the missing imports would be shown in the top right window. When I scroll through every single import for When I scroll through every single import for Cross-matching a handful of the imports in jnitorch.dll with the exports in torch_cpu.dll, they seem to be there. It uses (tens of?) thousands of functions, so cross-checking each and every one by hand is not quite reasonable. ??? |
|
Well, something's missing. Someone will need to debug this, might as well be you. :) I find that Dependency Walker is usually still better at spotting missing things even though it hasn't been updated for many years. Please give that one a try as well: |
|
Found the bastard. Apparently I missed it when scrolling through the list in the original dependencies tool. Anyway, here it is in all its glory: |
|
Ok, so Visual Studio doesn't like that definition it seems. I've tweaked this in commit 4b18d83. Let's see if it likes it better that way. |
|
@jxtps The snapshots have been updated! Please let me know if this is still happening. Thanks for the help! |
|
Ok, so even after updating to the new snapshot I'm still getting the same error: That's from Separately it turns out you can sort the imports in Dependencies by clicking the little headers, and that also goes for the first header that marks the missing functions in magenta/red (fuchsia?), so a single click on that header shows you the missing methods quickly & easily. |
|
Yeah, ok, it looks like Visual Studio wants to import inline functions no matter what: |
|
It linked & |
|
I've now actually run an exported model, and it is successfully running on GPU, thanks! I'm a little fuzzy on the precise memory ownership semantics - do I need to explicitly call float[] floats = /* image loading code */
FloatPointer p = new FloatPointer(FloatBuffer.wrap(floats));
Tensor t = from_blob(p, 1, 3, h, w);(yes, this should presumably be in a try-with-resources block) Getting the data out of the resulting tensor is actually a little tricky. Below is the crude recipe I've got so far. Note that TensorAccessor is unmapped - I don't know what idiomatic usage should be, saw some references to it. Tensor tensor = module.forward(inputs).toTensor(); // Model internally transfers tensor to CPU, otherwise need .to(...)
FloatBuffer fbuf = tensor.data_ptr_float().limit(tensor.numel()).asBuffer(); // Requires that tensor.has_storage()==true & probably that tensor.is_contiguous()==true
// Calling fbuf.array() barfs, so get() the contents instead
fbuf.rewind(); // Probably unnecessary
float[] floats = new float[fbuf.remaining()];
fbuf.get(floats);The |
|
Also, any chance for a non-snapshot |
I don't think Tensor takes ownership of a pointer like that, let's check the docs: If you want to be sure you're doing it right, just call close() on everything, that's fine, but that's what PointerScope is for:
TensorAccessor is only useful because it's more efficient than Tensor itself, but it's not going to be fast when used with JNI anyway, so if you're looking for something fast, what you want to use is an Indexer: If you have already tried the Indexer interface and found it to be lacking though, please do mention it. I can't guess that you've already tried it just from what you're saying here! Please be more specific in your questions.
We can't force a Buffer to take ownership of a random pointer, that's part of the docs too:
In 2 or 3 months, probably. You're still requesting fixes every other week, so I don't see the point of clogging the central repository with binaries no one is ever going to use! But if you mean that you would like to do it yourself, please let me know your account name on Sonatype OSSRH, and I'll give you deploy access, that's fine. |
I was unaware of the Indexer interface, will take a look at that when I implement the actual production code.
Sounds good, fair point on the rate of change, hopefully things will stabilize soon - I think it's close now that I have an end-to-end proof of concept.
Those look great!
We both know who needs to thank whom here, so thanks! ;) |
|
These new features and fixes have all been released with version 1.5.7 alongside full builds for PyTorch 1.10.2. Enjoy! |
|
@jxtps I've updated a couple of more things, so let me reply to all your initial questions to see if there's anything still missing for your application.
With the factory methods added in commit 392c34c, we can easily create tensors from Java arrays, for example in this case, something like
Those functions seem to be available only from the Python API, in which case we can't map them directly. If there are any functions available from C++ that you need to access from Java though, please let me know.
The C++ API relies on a lot of wrapper objects, but the user doesn't need to deal with them explicitly since the C++ language itself can do implicit conversions. The Java language cannot do that, so we need to wrap and unwrap a lot of things like that manually. I don't see an easy way to work around this, in general. In this case, yes,
Those functions have been available, anything in particular missing?
I've added an
I've added |
|
That all sounds great! Maybe we could add a public long[] shape() {
long[] out = new long[(int) ndimension()];
for (int i = 0; i < out.length; i++) out[i] = size(i);
return out;
}(Note that In scala I'd make that a private long[] _shape;
public long[] shape() {
if (_shape == null) {
// Fill-then-assign to guarantee that concurrent calls always return a filled array.
// (Assign-then-fill would risk returning an unfilled array in a parallel call).
long[] out = new long[(int) ndimension()];
for (int i = 0; i < out.length; i++) out[i] = size(i);
_shape = out;
}
return _shape;
}(I don't think it'd be worth it to add synchronization to ensure only a single Happy to do a pull request with whichever version you prefer. |
|
We can add a If |
Add `shape()` convenience method, similar to `sizes().vec().get()`. See bytedeco#1068 for discussion.
|
That can indeed happen, with a call to an in-place Created a pull request with the simpler version: #1161 |
Hi there - I'm trying to use the pytorch preset to load a model saved to TorchScript. Having looked at the 'Loading a TorchScript Model in C++' Tutorial (https://pytorch.org/tutorials/advanced/cpp_export.html), I thought this would be done by calling org.bytedeco.pytorch.jit.load("torch_script_model_name.pt") but it looks like the classes/functions related to torch::jit are missing.
The text was updated successfully, but these errors were encountered: