UnicodeDecodeError

[xlxlcyRuSReXxlxl]

Hi all! Running run_placesCNN_basic.py in this repository I run into this following error:
Traceback (most recent call last):
File "", line 4, in
File "/usr/local/lib/python3.5/dist-packages/torch/serialization.py", line 231, in load
return _load(f, map_location, pickle_module)
File "/usr/local/lib/python3.5/dist-packages/torch/serialization.py", line 379, in _load
result = unpickler.load()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 875: ordinal not in range(128)

Any advice? Thanks in advance!

[zhoubolei]

No idea. maybe you could try useGPU = 1. Model is cached using python2.7.

[LCorleone]

i met the same problem! Did you solve it?

[LCorleone]

i use python 3.6, i think the version may be the key!

[zhoubolei]

Indeed, it is the pickle library in python which cannot correctly load the CNN model trained in python2.7 into python3.6. Please let me know if there is some good solution for that.

[LCorleone]

yep, i just use virtualenv to create a pure environment for python2.7 and all is ok.
thank you for your reply! this is an excellent work!

[paffell]

I solved it with a small change in torch code in serialization.py lines 376-377 by adding the encoding there:
_sys_info = pickle_module.load(f,encoding='latin1')
unpickler = pickle_module.Unpickler(f,encoding='latin1')

I know it is not the cleanest solution, but it works :) To make it cleaner, it should be at least wrapped it by Try - Catch. But I do use Torch for this purpose only

[xlxlcyRuSReXxlxl]

I solved in the same way as paffel did, and it works pretty well.

[soravux]

Thanks @paffell for the trick.
For those of you who don't want to modify PyTorch's source code, you can do the following in your code just before loading the model:

from functools import partial
import pickle
pickle.load = partial(pickle.load, encoding="latin1")
pickle.Unpickler = partial(pickle.Unpickler, encoding="latin1")
model = torch.load(model_file, map_location=lambda storage, loc: storage, pickle_module=pickle)

This is for the run_placesCNN_unified.py example. For run_placesCNN_basic.py, replace the model_file in the last line with model_weight.
Note that your pickle namespace will be polluted after this operation. A better idea would be to copy the module first and modify the methods of this copy.

[paffell]

Thanks @soravux for cleaning the solution. I started to use it as well :)

[Fangyh09]

Thanks @soravux, it works for me too.

[Qinxianshen]

@soravux Thanks so much!!! it's work for me

[mrgloom]

https://discuss.pytorch.org/t/unicodedecodeerror-ascii-codec-cant-decode-byte-0xc3-in-position-918-ordinal-not-in-range-128/15356/7

[sakshijain20]

UnpicklingError: invalid load key, '\x5c'.

Getting the same error.Can anyone help?

[zhangzhengde0225]

_pickle.UnpicklingError: invalid load key, '\x04'.

Getting the same error. Tried Soravux's method, still report error.

[sanhok99]

I solved in the same way as paffel did, and it works pretty well.

will you please mention the line numbers and the file name. Thanks.

[LCorleone]

您的邮件已收到，我会尽快回复，祝好！Your email has been received, I will reply as soon as possible！

[carandraug]

The hack suggested in this issue, i.e., the following line:

pickle.Unpickler = partial(pickle.Unpickler, encoding="latin1")

completely breaks toch.load since version 1.11 because in that version, PyTorch subclasses pickle.Unpickler but after this, pickle.Unpickler is a function and not a type which leads to the following error:

[...]
  File "lib/python3.9/site-packages/torch/serialization.py", line 820, in _legacy_load
    class UnpicklerWrapper(pickle_module.Unpickler):  # type: ignore[name-defined]
TypeError: the first argument must be callable

To load models saved in Python2, since PyTorch 1.1.0 (released May 2019), do

torch.load(...., encoding='latin1')

[LCorleone]

您好，我是陆鑫益，您的邮件已经收到，我会尽快回复，谢谢。