Insert from iPhone/iPad doesn't work.

[SuperKenVery]

Steps to reproduce

Under macOS, in the text edit box,

1. right click (ctrl-click or two-finger) and select Insert from iPhone or iPad
2. Choose Take a photo (other ones not tested but should be the same)
3. Take a photo on iPhone (or iPad)
4. The photo does not gets sent.

Outcome

What did you expect?

The picture gets sent or I get a prompt like that I get when pasting a picture

What happened instead?

Nothing.

Operating system

macOS 13.3.1

Browser information

Safari 16.4 (18615.1.26.110.1)

URL for webapp

app.element.io

Application version

Element version: 1.11.30 Olm version: 3.2.12

Homeserver

pi.tail32664.ts.net Synapse from docker sha256:8d4aedabc178e248d707e660a062825ec598a1774ccbf2ca62cc4763a08bde83

Will you send logs?

Yes

[t3chguy]

I can't seem to find any docs online about how Apple is actually inserting this data. Does it work in other contenteditable fields? Does it work in the comment field on this issue? Does it work in http://madebyevan.com/clipboard-test/ - I don't have access to a device to be able to test this.

[SuperKenVery]

I can't seem to find any docs online about how Apple is actually inserting this data.

Oops, that's so sad... But I would suggest trying registering all event listeners possible and see what happens, if this thing is worth that effort :D

Does it work in other contenteditable fields? Does it work in the comment field on this issue? Does it work in http://madebyevan.com/clipboard-test/

None of these places did the Insert from iPhone/iPad show up... But it magically showed up in element web app's text edit field. I thought you specifically did this 😆

[t3chguy]

Does it work on the contenteditable div in https://developer.mozilla.org/en-US/docs/Web/HTML/Global_attributes/contenteditable

But I would suggest trying registering all event listeners possible and see what happens, if this thing is worth that effort :D

We already bind all relevant handlers, but this thing Apple is doing is unspecced so difficult to know how to solve without having a device to test against.

[SuperKenVery]

Does it work on the contenteditable div in https://developer.mozilla.org/en-US/docs/Web/HTML/Global_attributes/contenteditable

Ohhh it works!!!

https://developer.apple.com/documentation/appkit/supporting_continuity_camera_in_your_mac_app?language=objc

Related, but not web... Hope that gives some hint...

Apple is doing is unspecced so difficult to know how to solve without having a device to test against.

These two problems are a real pain!

[t3chguy]

Related, but not web... Hope that gives some hint...

None at all, we can't interface with AppKit and we have to rely on the browser translating it to Web events. I'm guessing it might be a paste event which is only present on contenteditable elements, but without knowing the underlying format it sends the data with the contribution would need to come from someone else.

Not seeing anything in Safari's changelog about this feature either - https://developer.apple.com/documentation/safari-release-notes/safari-16_4-release-notes

Could you try using that feature into https://jsbin.com/tebuxibate/edit?console,output @SuperKenVery and share what the output shows afterwards?

e.g.

[SuperKenVery]

Nothing shows up in the console. I guess it might be normal editing like typing text?

[t3chguy]

@SuperKenVery that might be a hint to how it works, might be using insertFromPaste

[SuperKenVery]

?

Well I can't quite understand what that means, from my understanding if it uses insertFromPaste then it's a paste... I don't have much experience in the front end.

But I would be happy to do more tests!

[SuperKenVery]

Ohh you were right!

[SuperKenVery]

Some experiments:

I think we're there! It's a blob.

[t3chguy]

@SuperKenVery could you try https://pr10844--matrix-react-sdk.netlify.app/

[SuperKenVery]

Doesn't seem to work :(

[t3chguy]

The issue will likely be due to it being a blob -

[SuperKenVery]

You might need to call getData with "text/html" I guess? Did not see text/html anywhere in your pr though.

[t3chguy]

@SuperKenVery what is the value of dataTransfer.types in your experiment above? Just ["text/html"]? Adding HTML parsing for inserting images to work will be a bit more work

[SuperKenVery]

Only text/html seem to work.

The issue will likely be due to it being a blob -

Probably... It seems it's now mostly processing it as files

what is the value of dataTransfer.types in your experiment above

I wish I could send you javascript objects lol

EDIT: Tried another time, getData("text/plain") returns empty string. So I guess we'll have to parse HTML :(

[t3chguy]

So I guess we'll have to parse HTML :(

I don't think its a feasible approach until the composer gains support for inline images, if the HTML contains a mix of images + text there'd be no sane way to handle that with today's composer.

[SuperKenVery]

I think we can still extend pasting (handle it like pasting a picture). When Insert form iPhone or iPad triggers the event, the data can't contain mixed text/pictures, but only one picture. It's different from pasting a picture into the text editor with text, I think.

[SuperKenVery]

Oh I think I know what you mean. You want to handle all kinds of pasted content rather than specifically support Apple Safari's single feature?

Well... Totally understandable but that would lead to a sad truth that we won't be able to use Insert from... until a much more rarely needed function gets supported :(

[t3chguy]

Contributions welcome, though it wouldn't be the most trivial thing given it gives it in an unspecced img tag format with a blob src which you'd need to download & re-upload to Matrix

Feel free to use my PR as a basis, it'd be fine to just handle Safari usecases as long as it is well written enough to not break on more exotic attempts