[Tracking Issue] Implement HTML -> Markdown conversions in Expensimark

[roryabraham]

Problem

Expensify.cash uses the Expensimark library in the expensify-common repo to convert markdown to HTML. That way, a user can compose a message using markdown, and it can be displayed + stored using HTML. However, when a user later goes to edit the message, we currently have no way to take the HTML and display it to them as markdown. We also don't want to store the original markdown in our database.

Why this is important

Editing comments is an important usability feature in Expensify.cash. However, we need to make sure that the edit-comment experience is as similar as possible to the original message composition experience.

Solution

Implement bi-directional HTML <---> Markdown conversion in Expensimark for all of the elements we currently support for Markdown ---> HTML conversion:

• Code fence - Implement Code Fence HTML to markdown conversion [Hold for payment on August 13th] #4184
• Inline code blocks - [Hold for payment 2020-08-09] Implement Inline Code Block HTML to markdown conversion #4185
• Email links - [Hold for Payment 13 August] Implement Hyperlink and Email link HTML to markdown conversion #4187
• Hyperlinks - [Hold for Payment 13 August] Implement Hyperlink and Email link HTML to markdown conversion #4187
• Bold
• Italics
• Strikethrough
• BlockQuotes - Implement Block quote HTML to markdown conversion #4188
• Newline / <br /> - [HOLD for payment July 22] Update line break regex for HTML->Markdown to support break tags with styles #4052
• Special characters (& currently looks like &amp when you go to edit a previously sent message with & in it) - [HOLD for payment August 4) Fix message Editing and pasting issues. #4169

[roryabraham]

HOLDing this on a POC PR over here. We might need to do an internal PR to update our other repos that also use Expensimark, depending on how we change the Expensimark interface.

[jasperhuangg]

@roryabraham I was just thinking about this–and sorry if this has already been considered–but why don't we just store both the HTML and markdown versions of the report comment, and switch between the two for editing/copying report comments?

To minimize space, we can only store the HTML if the markdown and the HTML are identical (and default to it if the markdown version doesn't exist).

Feel like this would be a whole lot simpler than doing a bunch of string manipulation on HTML to make it look like markdown–we'd probably also avoid a bunch of bugs this way.

[parasharrajat]

@jasperhuangg has raised a very good point. But I think it in another way. I would say that we should save the MD as I believe converting HTML to markdown will loose original MD markup which user entered.

And we can always convert the MD to HTML uniformly.

[roryabraham]

I don't necessarily disagree @jasperhuangg, but this was discussed in slack here. If you'd like to reopen the discussion, I think Slack would be the best place for that so it gets better visibility. The reason we didn't go with that solution is that it will nearly double the amount of data we need to store in our largest table.

But @parasharrajat has introduced an interesting solution I don't believe was discussed before – only store the plain-text (markdown) version of a comment, then convert it to HTML only when needed.

[roryabraham]

Raised the proposal to store only markdown in this slack thread.

[roryabraham]

Moving forward with the original plan, I'm taking this off hold.

[roryabraham]

For reference, this PR was merged as a POC, and it's being integrated into E.cash here.

[parasharrajat]

@Jag96 We can mark finished tags on the list.

[Jag96]

Marked off the finished tags, left <br> unchecked and linked the bug we found

[parasharrajat]

@Jag96 Let's open another issue for other tags. I see many complaints about it.

[isagoico]

Make hyperlinks editable in chat comments

Action Performed

1. Send this a link using the hyperlink markdown to a conversation
2. Click on edit the comment and save without making any changes

Expected result

Hyperlink markdown should be visible when editing the comment

Actual result

Hyperlink markdown is removed when editing the comment and then it's saved as plain text.

Notes/images/Video

Recording.198.mp4

---

From @joaniew https://expensify.slack.com/archives/C01GTK53T8Q/p1626810543044700

When I edit a chat that has a link to it already, the link turns into plain text and removes the hyper link, and I can’t edit it again to make it a link

[Jag96]

@parasharrajat it sounds like your proposal for #4169 will solve Hyperlink markdown is removed when editing the comment and then it's saved as plain text. right? If so, I'd like to hold off on adding the other HTML->MD conversions until the ones we currently support are behaving the way we expect them to.

[parasharrajat]

@Jag96 No. That proposal fixes the content pasting by excluding not-needed tags. script & Style. Then it fixes the Html to MD parsing as unescaping the escaped Chars. Last point on the checklist Special characters (& currently looks like &amp when you go to edit a previously sent message with & in it) - Message Editing and pasting issues. #4169.

We are still lacking Link parsing. Until we have HTML to MD parser for links, Hyperlink markdown is removed when editing the comment and then it's saved as plain text will not be fixed. When we edit the message, Html is converted to MD. But currently it does not convert [link](url) links or email links.

[Jag96]

Ok sounds good, I'll make new issues then for each of the items in the checklist

[parasharrajat]

@Jag96 Links and emails links can go into one. There will be one regex for both.

[roryabraham]

• Quote support implemented here, integrated into E.cash here
• Inline code blocks added here, not yet integrated into E.cash, PR is here
• We've had some proposals but no movement yet on hyperlinks and email links.

[roryabraham]

Looks to me like everything except #4188 is done or almost done

[parasharrajat]

That is also done.

[roryabraham]

Looks like all the issues on our checklist are done here. I think I am going to ask Applause to do some extra QA here before we close this out and call it done.

[roryabraham]

Additional QA steps I would like to have done:

1. Create a message with each of the markdown features in the issue description above.
2. Attempt to edit the message. Verify that the markdown you entered when creating the message reappears in the edit box.
3. Perform additional exploratory QA around editing messages containing markdown in New Expensify. Try to break it and let us know how it goes. 🙃

[roryabraham]

QA requested in Slack

[roryabraham]

Bumped Applause in slack here

[roryabraham]

Cool, actually Applause completed the exploratory testing we requested and exported these follow-up issues:

• [HOLD for payment 2021-09-27] Chat - Text and URL inside a strikethrough markdown isn't working #4897
• Chat - Additional lines are shown when editing a text with markdown type Quote #4898
• Chat - Markdown entered in email responses don’t render in the app correctly #4899

Since those are all being processed by our engineering pipeline independently, I think we can close this out.