load_image - decode b64encode and encodebytes strings

[amyeroberts]

What does this PR do?

Updates base64 decoding login to decode base64 string encoded with either base64.b64encode or base64.encodebytes

Fixes #30114

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ydshieh]

Could you explain a bit why b64decode here can't handle the data encoded by encodebytes?

[ydshieh]

So the fix is to make it handle string?

[ydshieh]

But image is always str inside this if branch if isinstance(image, str):, so I feel something is strange here.

[amyeroberts]

Yep, sorry, this is a hangover from when I was testing with my own scripts. I've changed it so it's just image.encode()

[ydshieh]

I still see we have

image.encode() if isinstance(image, str) else image

??

Other than this, LGTM.

[ydshieh]

Hi @amyeroberts Let me know your thoughts on my comments whenever you get some bandwidth

[amyeroberts]

@ydshieh OK, so the solution I put in here was a hacky one I found worked without digging too much into it - sorry for the rushed solution.

So, base64.decodebytes can decode the bytes string from both base64.b64encode and base64.encodebytes, and base64.decode can decode the string of base64.b64encode and base64.encodebytes. However, this will fail if we have validate=True:

import base64
from io import BytesIO
from PIL import Image

import torch

buffered = BytesIO()
im = torch.rand((256, 256,3))
image = Image.fromarray(im.numpy().astype('uint8'), 'RGB')
image.save(buffered, format="JPEG")

base64.b64decode(base64.encodebytes(buffered.getvalue()), validate=True)

This is because encodebytes inserts newline characters into the encoded bytes string after every 76 bytes if output and ensures there's a trailing new line. This fails validation with Error: Non-base64 digit found

The solution to enable these images would be to remove validate=True from the current b64decode call. I'm not sure we should do this, as the bytes string from base64.encodebytes(s) can contain arbitrary binary data. WDYT?

[ydshieh]

The solution to enable these images would be to remove validate=True from the current b64decode call. I'm not sure we should do this, as the bytes string from base64.encodebytes(s) can contain arbitrary binary data. WDYT?

This is a good question. I tried to feed some bytes string without. Without adding extra arguments to load_image, not sure if we can keep an option about the check. Do think think it is worth a new argument?

[ydshieh]

I tried to decode some arbitrary bytes and decodebytes could still fail. It will succeed only if the bytes is a base64 encoded data.
So I think we can safely use decodebytes as you do in this PR.

import base64


s = b'data to be encoded'
print(type(s))

data = base64.decodebytes(s)
print(data)

[amyeroberts]

So I think we can safely use decodebytes as you do in this PR.

Awesome. Thanks for looking into this! I'll tidy up the code and let you know when it's ready for re-review.

[amyeroberts]

@ydshieh Ready for re-review 🤗

[ydshieh]

Thank you for the fix, the iterations and the explanations!