Parse inbound emails in "raw" format

[Jericho]

When developers configure the inbound parse feature in their SendGrid account they must specify if they want to use the "raw" format or not. Currently, StrongGrid's webhook parser is not compatible with the raw format but some developers don't realize that and they enable the "raw" format. Also, there's some verbiage in SendGrid's app that implies that it's preferable to enable the "raw" format which understandably leads some developers to enable it.

Over the years this has caused a number of problems such as:

• parsing completes without error but some (or all) properties of the parsed object are null/empty. Sometimes, attachments are missing
  InboundEmail model missing properties (ParseInboundEmailWebhookAsync) #446
  Inbound Parse Message-ID and In-Reply-To #257
  ParseInboundEmailWebhook() is not reading attachments #176
• Value cannot be null. (Parameter 'source')
  Value cannot be null. (Parameter 'source') on ParseInboundEmailWebhookAsync (.NET 5) #417
• Duplicate field in section
  Unable to parse SendGrid Example Default or Example Raw payloads - "Duplicate field in section" error #400
  InboundParse  #375
  "Duplicate field in section" MultipartParseException in Email Parse #346
  can't parse default message from sendgrid #333
  Email parse event error #202

I am wondering if it would make sense to offer a way to parse inbound emails in raw format. Something along the lines of:

[HttpPost]
public async Task<IActionResult> Post()
{
    var parser = new StrongGrid.WebhookParser();
    var inboundEmail = await parser.ParseRawInboundEmailWebhookAsync(Request.Body).ConfigureAwait(false);

    ... do something with the inbound email ...

    return Ok();
}

The developer would be responsible to chose between ParseRawInboundEmailWebhookAsync and ParseInboundEmailWebhookAsync depending on whether they enabled the raw format or not.

I'm looking for feedback, does anybody think it would be valuable? Is there a reason developers prefer one format over the other? If we move forward with this idea, would anybody be able and willing to help beta test this new functionality?

[dandenton]

I think the ideal implementation would be to keep the single Parse method and internally detect which setting was used. That way code doesn't need to change when updating settings on the SendGrid side. It could detect the content by looking for the headers parameter which is only included with Raw turned off, email is also only sent with Raw turned on.

We've been using "Send Raw" disabled but recently noticed that certain attachment types can be dropped. MSG files for example won't come through from SendGrid. We're talking with SendGrid's support on that issue but enabling Send Raw does make the attachment come through. Ironically I've had two different support contacts say

The only option for receiving inbound parse with attachments is to send the email as raw data.

[Jericho]

It could detect the content by looking for the headers parameter which is only included with Raw turned off, email is also only sent with Raw turned on.

hummmmm... Automatically detect the format of the payload based on the presence or absence of the "email" parameter is a very good suggestion: developers wouldn't have the make a decision which method of the StrongGrid parser to invoke, it wouldn't matter which format they configure in the SendGrid app, they could even switch back and forth between the two formats (the SendGrid UI allows you to do this very easily) and StrongGrid would transparently parse the content appropriately without any change in the developer's code. I like it!

We've been using "Send Raw" disabled but recently noticed that certain attachment types can be dropped

Thank you for sharing your experience, The fact that some attachments are missing when you turn off the "Raw" format makes it critical that we support the raw format. I'll start working on it and post a beta as soon as possible.

[Jericho]

Published beta package 0.102.0-parse-raw-inboun0017 to my Feedz.io nuget repository. Looking forward to your feedback.

[dandenton]

Thanks for the fast turnaround on this!

Unfortunately we can't use that beta package (or latest on master) right now because we use it in an Azure Function app (in process) which can't use v7.0.0 versions of Microsoft.Extension packages without moving to isolated. Once we can update to .Net 8 it should also fix the issue. If you could update the .Net 6 version to use v6.0 for Microsoft.Extensions.Logging and possibly System.Text.Json that would help us update to the latest version.

That said I pulled your updates into an extension method to test it in our project and have a couple notes:

• When the email includes an MSG (and I'd assume EML) attachment the cast to MimePart fails because it's really a MessagePart
• After copying the attachment to the memory stream should it be reset back to Position = 0 so it's ready to use?
• Consider changing to using var on the MemoryStream in ParseInboundEmailAsync
• Consider changing to using var on the MimeMessage.LoadAsync in ParseInboundEmailAsync

For the first two items here is my updated version of ConvertMimePartEntityToAttachmentAsync that seems to work with MSG attachments.

        private static async Task<InboundEmailAttachment> ConvertMimeEntityToAttachmentAsync(MimeEntity mimeEntity, CancellationToken cancellationToken)
        {
            var attachment = new InboundEmailAttachment {
                ContentId = mimeEntity.ContentId,
                ContentType = mimeEntity.ContentType.MimeType,
                Data = new MemoryStream(),
            };
            if (mimeEntity is MimePart mimePart) {
                attachment.FileName = mimePart.FileName;
                attachment.Name = mimePart.ContentType.Name;
                await mimePart.Content.DecodeToAsync(attachment.Data, cancellationToken).ConfigureAwait(false);
            } else if (mimeEntity is MessagePart mimeMessage) {
                _ = MimeTypes.TryGetExtension(mimeMessage.ContentType.MimeType, out var extension);
                attachment.FileName = $"{mimeMessage.Message.Subject}{extension}";
                attachment.Name = mimeMessage.Message.Subject;
                await mimeMessage.Message.WriteToAsync(attachment.Data, cancellationToken).ConfigureAwait(false);
            } else {
                throw new NotImplementedException($"Converting a MimeEntity type {mimeEntity.GetType().Name} to an attachment is not supported");
            }
            // reset to the start
            attachment.Data.Position = 0;
            return attachment;
        }

[Jericho]

we use it in an Azure Function app (in process) which can't use v7.0.0 versions of Microsoft.Extension packages without moving to isolated.

Is there a reason to prefer "in process" rather that "isolated"? As I understand it, the "in process" mode condemns you to being at the mercy of the the version of certain nuget packages referenced by the Azure environment (such as System.Text.Json and Microsoft.Extensions.Logging for example) which may conflict with the version of same packages referenced by 3rd party parckages (StongGrid being an example of such a 3rd party but this problem is not limited to StrongGrid).

I'm sure you're aware that a few people have raised issues in this repo to make me aware of this problem and that the conclusion is that switching to "isolated" mode would instantly solve this problem. There's probably a legitimate reason for preferring "in process" that I'm simply not aware of.

Once we can update to .Net 8 it should also fix the issue. If you could update the .Net 6 version to use v6.0 for Microsoft.Extensions.Logging and possibly System.Text.Json that would help us update to the latest version.

Downgrading the version of these packages referenced by StrongGrid could probably solve the problem in the short term but this is a never ending cycle caused by the Azure environment not keeping up with new releases of the referenced packages. As far as I know, this whole problem could be completely avoided by simply using Azure's "isolated" mode. I say "simply" but I realize nothing is ever as simply as it seems!!! 😄

• When the email includes an MSG (and I'd assume EML) attachment the cast to MimePart fails because it's really a MessagePart
• After copying the attachment to the memory stream should it be reset back to Position = 0 so it's ready to use?
• Consider changing to using var on the MemoryStream in ParseInboundEmailAsync
• Consider changing to using var on the MimeMessage.LoadAsync in ParseInboundEmailAsync

Thanks for the suggestions.

For the first two items here is my updated version of ConvertMimePartEntityToAttachmentAsync that seems to work with MSG attachments.

Excellent! Thank you for sharing. I'll try to send myself an email with an attached MSG and/or EML attachment to more closely mimic your scenario. Also, if you could share your payload, that would help me test your scenario as closely as possible,

[Jericho]

@dandenton I sent myself a test email with an attached EML and I experienced the casting problem you alluded to and your ConvertMimeEntityToAttachmentAsync method took care of this problem nicely. I also saved this test inbound email and created a unit test for it which makes it easy to test this functionality.

I have published another beta version 0.102.0-parse-raw-inboun0021 to my Feedz.io nuget repository.

In an effort to help you test this beta version, I have downgraded the references to System.Text.Json and Microsoft.Extensions.Logging but I want to make crystal clear that I will not publish the final version with these downgraded references mainly because it would bring back nasty serialization problems (see #463 and #476 for some details).

[dandenton]

You're absolutely right about In-Proc vs Isolated. Isolated should be the way to go but when you're dealing with projects from < 2020 sometimes it's hard to get time to update, especially since it didn't cause any issues until .Net 7.0. No worries about not being able to revert them. I'll try to get the update to isolated prioritized higher.

Changes look good although I just realized the file name for attached EML/MSG is using the subject which could be blank. It might be a good idea to give it some sane default like var attachmentName = !string.IsNullOrWhiteSpace(mimeMessage.Message.Subject) ? mimeMessage.Message.Subject : "Attachment"

Another question I thought of is if StrongGrid should take a dependency on MimeKit vs MimeKitLite. Lite seems to work for this use and it basically the same as MimeKit just without the crypto dependencies.

[Jericho]

It might be a good idea to give it some sane default like ...

Good idea.

Another question I thought of is if StrongGrid should take a dependency on MimeKit vs MimeKitLite. Lite seems to work for this use and it basically the same as MimeKit just without the crypto dependencies.

I wasn't aware of their "lite" package. Let me look into it...

[Jericho]

First of all, I was referencing MailKit when I should have referenced MimeKit. I made the switch to MimeKitLite which works perfectly for our situation. Thanks for pointing me to this package.

I also implemented your idea of ensuring attachments have a name even when the subject line is blank. I improved upon your suggestion by defaulting to $"Attachment{index}" to ensure the name is reasonably unique even if multiple attachments have a blank subject line.

Published one more beta package 0.102.0-parse-raw-inboun0023 to my Feedz.io nuget repository.

[Jericho]

🎉 This issue has been resolved in version 0.102.0 🎉

The release is available on:

• GitHub Release
• NuGet Package

Your GitReleaseManager bot 📦🚀