Sending a request to the forward method of a model that expects more than one vector

[mehmetcalikus]

Hi,

I'm trying to make a request to the bert model whose trace I have taken. I've tried this in different ways, I've also tried with K value args but I always get an error.

As you know, the Bert model expects 3 input vectors. Ids, attention_mask and token_type_ids. I'm making a test request as follows.

IValueVector inputs = new IValueVector();
inputs.push_back(new IValue(ones(1,128)));
inputs.push_back(new IValue(ones(1,128)));
inputs.push_back(new IValue(ones(1,128)));

Tensor output = module.forward(inputs).toTensor();

And I am getting this error. I think I got the trace of the model correctly. Because I can use the model on cpp.

Exception in thread "main" java.lang.RuntimeException: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/transformers/models/bert/modeling_bert.py", line 21, in forward
    _2 = torch.to(extended_attention_mask, 6)
    attention_mask0 = torch.mul(torch.rsub(_2, 1.), CONSTANTS.c0)
    _3 = (embeddings).forward(input_ids, token_type_ids, )
          ~~~~~~~~~~~~~~~~~~~ <--- HERE
    _4 = (encoder).forward(_3, attention_mask0, )
    return (_4, (pooler).forward(_4, ))
  File "code/__torch__/transformers/models/bert/modeling_bert.py", line 49, in forward
    _6 = torch.slice(position_ids, 0, 0, 9223372036854775807)
    input = torch.slice(_6, 1, 0, _5)
    _7 = (word_embeddings).forward(input_ids, )
          ~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    _8 = (token_type_embeddings).forward(token_type_ids, )
    embeddings = torch.add(_7, _8)
  File "code/__torch__/torch/nn/modules/sparse.py", line 10, in forward
    input_ids: Tensor) -> Tensor:
    weight = self.weight
    inputs_embeds = torch.embedding(weight, input_ids, 0)
                    ~~~~~~~~~~~~~~~ <--- HERE
    return inputs_embeds

Traceback of TorchScript, original code (most recent call last):
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/torch/nn/functional.py(2183): embedding
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/torch/nn/modules/sparse.py(158): forward
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/torch/nn/modules/module.py(1098): _slow_forward
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/torch/nn/modules/module.py(1110): _call_impl
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py(214): forward
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/torch/nn/modules/module.py(1098): _slow_forward
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/torch/nn/modules/module.py(1110): _call_impl
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py(989): forward
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/torch/nn/modules/module.py(1098): _slow_forward
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/torch/nn/modules/module.py(1110): _call_impl
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/torch/jit/_trace.py(958): trace_module
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/torch/jit/_trace.py(741): trace
<ipython-input-7-6f0adf2a89a3>(1): <module>
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/IPython/core/interactiveshell.py(3417): run_code
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/IPython/core/interactiveshell.py(3337): run_ast_nodes
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/IPython/core/interactiveshell.py(3145): run_cell_async
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/IPython/core/async_helpers.py(68): _pseudo_sync_runner
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/IPython/core/interactiveshell.py(2922): _run_cell
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/IPython/core/interactiveshell.py(2876): run_cell
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/ipykernel/zmqshell.py(536): run_cell
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/ipykernel/ipkernel.py(306): do_execute
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/tornado/gen.py(209): wrapper
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/ipykernel/kernelbase.py(543): execute_request
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/tornado/gen.py(209): wrapper
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/ipykernel/kernelbase.py(268): dispatch_shell
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/tornado/gen.py(209): wrapper
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/ipykernel/kernelbase.py(365): process_one
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/tornado/gen.py(748): run
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/tornado/gen.py(787): inner
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/tornado/ioloop.py(743): _run_callback
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/tornado/ioloop.py(690): <lambda>
/home/mehmet/anaconda3/envs/training/lib/python3.8/asyncio/events.py(81): _run
/home/mehmet/anaconda3/envs/training/lib/python3.8/asyncio/base_events.py(1859): _run_once
/home/mehmet/anaconda3/envs/training/lib/python3.8/asyncio/base_events.py(570): run_forever
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/tornado/platform/asyncio.py(149): start
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/ipykernel/kernelapp.py(612): start
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/traitlets/config/application.py(845): launch_instance
/home/mehmet/anaconda3/envs/training/lib/python3.8/site-packages/ipykernel_launcher.py(16): <module>
/home/mehmet/anaconda3/envs/training/lib/python3.8/runpy.py(87): _run_code
/home/mehmet/anaconda3/envs/training/lib/python3.8/runpy.py(194): _run_module_as_main
RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got CPUFloatType instead (while checking arguments for embedding)

	at org.bytedeco.pytorch.JitModule.forward(Native Method)
	at src.main.java.com.example.JavacppSampleApplication.main(JavacppSampleApplication.java:71)

It works fine when I request a model with a forward method that expects a single variable. But it doesn't work on bert. Can you share an example usage for models with multi-variable forward method? Or what is the reason for the error I get?

Thanks in advance.

[saudet]

If you call, for example, inputs.get(1).toTensor(), are you getting what you would expect?

[mehmetcalikus]

Yes. When I call inputs.get(1).toTensor() I get a 1 by 128 vector filled with 1. The type and size of the vector looks like [ CPUFloatType{1,128} ].

I think this is not the problem. I actually have another model that starts with the bi-lstm layer. It works fine when I send a request to its forward. Because its forward method expects a single variable. But bert wants three. Otherwise, the data types of the requested vector are the same.

[saudet]

Maybe there's some incompatibility between versions of PyTorch. Are you sure you're using the same exact version of PyTorch?

[mehmetcalikus]

I'm not sure I understand what pytorch versions there would be incompatibility between. I am using maven distribution on linux. From what I've read on README.md I guess there is no need to download libtorch on linux and give it its path.

In addition, if there was such a problem, I think it would not work at all, but I only cannot send requests to models with forward method that require more than one variable.

[saudet]

I mean, you're not using the same binaries from C++ and from Java. There could be incompatibilities between say PyTorch 1.10.x and 1.11.x. JavaCPP is just calling the C++ API as is, and you're saying that the content of std::vector<c10::IValue> that you get in your tests in C++ is the same that what you're getting with IValueVector, so I don't see what else could go wrong with JavaCPP.

[mehmetcalikus]

I understand, but I don't think there is a problem with binaries because I can already use the library on my own model. Actually, my reason for opening an issue is as I wrote in the first message, I am requesting a usage example for the following method in the JitModule class.

    @ByVal
    public native IValue forward(@ByVal IValueVector var1, @Cast({"const torch::jit::Kwargs*"}) @ByRef(nullValue = "torch::jit::Kwargs()") IValueIValueMap var2);

forward can be called in 2 different ways as you know. It works when I call it as below. Because my own model expects a single forward variable. But the Bert model expects 3 different variables. So I need an example of the usage above.

     @ByVal
    public native IValue forward(@ByVal IValueVector var1);

Thanks again for trying to help.

[saudet]

Those 2 methods map to the same function in C++. It's just a way to emulate the default arguments that the Java language doesn't support. So you're saying you're having issues with IValueIValueMap? Could you show me how you're calling this from C++?

[saudet]

Oh, that's not right. That Kwargs is supposed to be std::unordered_map<std::string, at::IValue>. I'll have to fix that.

[saudet]

I've fixed the kwargs argument in 87b01a0. Please give it a try with the snapshots: http://bytedeco.org/builds/

Other than that, what you proposed at first using IValueVector.push_back() seems like the way to go, see pytorch/pytorch#18337, but apparently in some circumstances we need to cast the input values, for example, see huggingface/transformers#2952, so you should try to do something like that as well.

[saudet]

The fix for this has been released with version 1.5.8! Thanks for reporting