Local outbox

[ChristophWurst]

Feature Request

Instead of sending outgoing messages directly to SMTP and IMAP we should store them in a local outbox and try to push them from there afterwards.

This gives us new abilities

• At a temporary server issue the message can remain in the outbox, a background job can try to resend the message later automatically. The user doesn't have to convert to draft and send the message themselves.
• We can implement Undo sending an email #5438 in a way that messages are still sent when the user closes their tab before the expiration of the fake undo timer because then the message is on the server and a background job will pick it up.
• We can implement Send scheduled / time-controlled mail #1626 if we add an optional send time for a message. A background job will pick up the message when the time is reached.

It should be sufficient to have one outbox per user account. We don't need a separate outbox for every Mail account.

Summary

Store messages before sending them to SMTP.

Implementation

• Create database structure and REST API for an outbox #5885
• Create outbox back-end implementation #6139
• Add background job that sends outbox messages #5886
• Create frontend for an outbox #5887

Critical bugs and other follow-ups

• Outbox message is sent while it's edited #6156
• Saving a draft reports success even if it fails #6157
• PageController uses LocalMessageMapper, should use OutboxService #6161
• Outbox doesn't set \\answered to cross-account replies #6189
• Avoid double sending of an outbox message #6194
• Disallow scheduled sending for instances using AjaxCron #6195
• Follow up for "scheduled sending" and "undo sending" features  #6242
• Disallow scheduled sending for instances using AjaxCron #6195

These chunks do not make an atomic change. Let's use a feature branch for this whole outbox feature where the changes from above are merged into and then one final PR that integrates it back into main.

[miaulalala]

Some things:

1. remove is_mdn
2. Getting the local mailbox messages - separating headers and body as we did with envelopes because it could be a lot of data
3. Attachments handling is super unclear atm.
4. Replied to message - get message from DB
5. add alias_id field to migration

[miaulalala]

Save endpoint will remove message from draft mailbox (if possible).
Undo send endpoint that will convert message back to imap draft.

[st3iny]

Some more thoughts:

1. Continue to use old MDN endpoint -> skip outbox

[ChristophWurst]

Attachments handling is super unclear atm.

What part of it? As per #5885 we either have to serialize or normalize the data into two tables.

[miaulalala]

Attachments handling is super unclear atm.

What part of it? As per #5885 we either have to serialize or normalize the data into two tables.

More the frontend part and the way we offer different attachments in the first place. I am using the localAttachments table for local attachments in the backend.

[ChristophWurst]

@jancborchardt @nimishavijay we found a small issue with this feature.

For context, this is the base work for undo sending and scheduled sending. Moreover it will make sending emails more snappy, allow automatic retries when sending fails and also do a large chunk of work that we'll need for the revised draft handling.

What we realized is that we do have two options to handle cloud or forwarded attachments of messages that might not be sent immediately.

1. Keep only a reference to the cloud file or forwarded attachment. Then we will run into the issue that the attachment a user picks might not be the one that is actually sent, because the file can change until the message gets out. Even worse, the cloud file or the original message with the attachment could be deleted and then we can't send the message.
2. Copy/download the files when the message is added to the outbox. This makes us independent of time, because the file doesn't depend on changes in your cloud files, nor the existence of the original message of which you forward an attachment. The "downside" is that if a user wants to always attach the latest version this won't be possible. So maybe this needs some hints in the UI. But then users could share that same cloud file via link.

From a technical PoV we'll have an easier life if we go with 2). But for completeness I wanted to have you in the loop on this design decision.

[nimishavijay]

1. Keep only a reference to the cloud file

If someone is sending an attachment using a cloud file link, then the sender would know that it could be changed by the time the recipient opens it, right? It is expected behavior when you attach, for eg. a link to a Google Doc, so it should be all good.
If you want to send the current state, you can explicitly attach a saved copy.
So option 1 works :)

[ChristophWurst]

I am not talking about share links. It's only about file that get attached. The question is when we should take the copy of the attached file when a message is sent at a later point in time.

With 1) we take a copy at the time of the actual sending. With 2) we take a copy at the time of writing.

I've updated my post above to reflect that we'd strongly lean towards option 2) because 1) has too many potential side effects and errors to catch.

[jancborchardt]

@ChristophWurst so Nimisha and I just talked about this and cleared up the confusion, and we agree with option 2. :)

The "downside" is that if a user wants to always attach the latest version this won't be possible. […] But then users could share that same cloud file via link.

Exactly that ^, so there’s no downside.

We think it is a bit confusing that there are 2 separate entries to share stuff from Nextcloud, and we’ll open a separate issue about how to best combine that.