Unable to parse SendGrid Example Default or Example Raw payloads - "Duplicate field in section" error

[StuFrankish]

I'm trying to integrate SendGrid inbound parsing into my .Net Core 3.1 web app but I'm unable to get the request body to work.
This line of my code:
var inboundEmail = await parser.ParseInboundEmailWebhookAsync(Request.Body);
constantly fails with the exception "Duplicate field in section" when using either of the example payloads shown here;
https://docs.sendgrid.com/for-developers/parsing-email/setting-up-the-inbound-parse-webhook

I'm using version 0.82.0 of the StrongGrid nuget package.

If I remove what appear to be the duplicate fields from the "headers" section I just get an "Unexpected end of stream", even when adjusting the string length in the payload.

How do I get this to work because I don't know if this is an error with StrongGrid and/or the examples provided by SendGrid in their documentation can be trusted.

For example, the first line, should "Date" not be "Data"?
Is it valid to have multiple "Received" fields?

I'm using Postman to test this locally with the body of the request set to raw and the example below pasted as the body.
I have also tried setting the body to use form-data and setting the fields manually by copying out the data from each, but this results in a new error - "Value cannot be null. parameter ('source')" - here's the screenshot;

[Date] array(16) {
  ["headers"]=>
  string(1970) "Received: by mx0047p1mdw1.sendgrid.net with SMTP id 6WCVv7KAWn Wed, 27 Jul 2016 20:53:06 +0000 (UTC)
Received: from mail-io0-f169.google.com (mail-io0-f169.google.com [209.85.223.169]) by mx0047p1mdw1.sendgrid.net (Postfix) with ESMTPS id AA9FFA817F2 for <example@example.comom>; Wed, 27 Jul 2016 20:53:06 +0000 (UTC)
Received: by mail-io0-f169.google.com with SMTP id b62so81593819iod.3 for <example@example.comom>; Wed, 27 Jul 2016 13:53:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sendgrid.com; s=ga1; h=mime-version:from:date:message-id:subject:to; bh=DpB1CYYeumytcPF3q0Upvx3Sq/oF4ZblEwnuVzFwqGI=; b=GH5YTxjt6r4HoTa+94w6ZGQszFQSgegF+Jlv69YV76OLycJI4Gxdwfh6Wlqfez5yID 5dsWuqaVJZQyMq/Dy/c2gHSqVo60BKG56YrynYeSrMPy8abE/6/muPilYxDoPoEyIr/c UXH5rhOKjmJ7nICKu1o99Tfl0cXyCskE7ERW0=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=DpB1CYYeumytcPF3q0Upvx3Sq/oF4ZblEwnuVzFwqGI=; b=Sq6LVHbmywBdt3sTBn19U8VOmelfoJltz8IcnvcETZsYwk96RBxN+RKMN5fOZSKw4j 15HrgdIFfyDmp67YK0ygvOITlTvZ6XY5I0PtnvDtAQt79kS3tKjI3QKJoEp/ZjIjSzlL KG7agl6cxFgBbIN0yHWBOvy3O+ZXY8tZdom1yOvULjmjW1U9JkdOs+aJ6zq4qhZX/RM/ tIgLB461eJ5V95iQDDc5Ibj9Cvy4vJfXLQRO0nLVQAT2Yz58tkEO1bDZpWOPAyUNneIL yhIWp+SpbuqhMA68mq0krG1PjmWalUbpVcGJIGuOKB9mQFFo/MqdrUCjvYnyo1jPLPeX psdQ==
X-Gm-Message-State: AEkoousvdxmDoxLlTUYJ1AOmCGJv77xRBBlfKv6YrthH0M2NueMwlOxUD6t8nidE9uonXbdJ/DQy/chmHUnN//a4
X-Received: by 10.107.6.101 with SMTP id 98mr38024553iog.41.1469652785829; Wed, 27 Jul 2016 13:53:05 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.107.48.17 with HTTP; Wed, 27 Jul 2016 13:53:05 -0700 (PDT)
From: Sender Name <example@example.com>
Date: Wed, 27 Jul 2016 14:53:05 -0600
Message-ID: <CAN_P_JMvV7ZpAQhOnDienypLrJmuhN=LQWweu4yScw4jQyXY2w@mail.gmail.com>
Subject: Different File Types
To: example@example.comom
Content-Type: multipart/mixed; boundary=001a113f8ad03e85160538a4343c
"
  ["dkim"]=>
  string(22) "{@sendgrid.com : pass}"
  ["content-ids"]=>
  string(37) "{"ii_1562e2169c132d83":"attachment1"}"
  ["to"]=>
  string(26) "example@example.comom"
  ["html"]=>
  string(479) "<div dir="ltr">Here&#39;s an email with multiple attachments<div><br></div><div><img src="cid:ii_1562e2169c132d83" alt="Inline image 1" width="455" height="544"><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><img src="https://sendgrid.com/brand/sg-logo-email.png" width="96" height="17"><br><div><br></div></div></div>
</div></div>
"
  ["from"]=>
  string(33) "Sender Name example@example.com"
  ["text"]=>
  string(139) "Here's an email with multiple attachments
"
  ["sender_ip"]=>
  string(14) "209.85.223.169"
  ["spam_report"]=>
  string(844) "Spam detection software, running on the system "mx0047p1mdw1.sendgrid.net", has
identified this incoming email as possible spam. The original message
has been attached to this so you can view it (if it isn't spam) or label
similar future email. If you have any questions, see
@@CONTACT_ADDRESS@@ for details.

Content preview:  Here's an email with multiple attachments [image: Inline image
   1] -- [...]

Content analysis details:   (2.6 points, 5.0 required)

 pts rule name              description
---- ---------------------- --------------------------------------------------
 0.8 HTML_IMAGE_RATIO_02    BODY: HTML has a low ratio of text to image area
 0.0 HTML_MESSAGE           BODY: HTML included in message
 1.8 HTML_IMAGE_ONLY_08     BODY: HTML: images with 400-800 bytes of words
 0.0 T_MIME_NO_TEXT         No text body parts

"
  ["envelope"]=>
  string(66) "{"to":["example@example.comom"],"from":"example@example.com"}"
  ["attachments"]=>
  string(1) "2"
  ["subject"]=>
  string(20) "Different File Types"
  ["spam_score"]=>
  string(5) "2.597"
  ["attachment-info"]=>
  string(287) "{"attachment2":{"filename":"DockMcWordface.docx","name":"DockMcWordface.docx","type":"application/vnd.openxmlformats-officedocument.wordprocessingml.document"},"attachment1":{"filename":"MG_2359.jpg","name":"_MG_2359.jpg","type":"image/jpeg","content-id":"ii_1562e2169c132d83"}}"
  ["charsets"]=>
  string(77) "{"to":"UTF-8","html":"UTF-8","subject":"UTF-8","from":"UTF-8","text":"UTF-8"}"
  ["SPF"]=>
  string(4) "pass"
}

[Jericho]

This issue has ben discussed here, here, here and here.

To summarize: you are using a payload in MIME format which StrongGrid does not support. The two samples that SendGrid provides on the page you linked are both in MIME format and therefore cannot be used with StrongGrid. Feel free to use the "EmailFromSendGrid.txt" sample I provided back in 2017 here.

When you are ready to parse actual webhooks from SendGrid, MAKE SURE that you didn't configure SendGrid to send you the "raw" format. In other words, make sure the "POST the raw, full MIME message" setting is NOT checked like in this example:

[StuFrankish]

This is the problem I'm having, I've tried posting form data (Using postman, see the error and screenshot above the SG sample), the additional options checkbox is definitely off and I'm still getting errors raised when the endpoint is called by SendGrid themselves.

[Jericho]

In your original comment you said

This line of my code ... constantly fails with the exception "Duplicate field in section" when using either of the example payloads

I hope I was clear that the samples are the cause for this exception. You cannot use those samples because they are in raw MIME format which cannot be parsed. That's the reason I provided an alternate sample. Did you try it?

Now you are saying that you are getting this same exception when trying to parse a live webhook posted by SendGrid despite the fact that "Post the raw, full MIME message" option is unchecked??? This sounds really strange. I have never witnessed this particular exception that wasn't caused by the raw format. Did you look at the data posted by SendGrid? Does it seem to be in MIME format? If so, you need to let SendGrid know that they are sending you data in raw MIME format despite your configuration. If not, can you provide the data to help me reproduce the problem and investigate further?

[Jericho]

If you visually inspect the data coming from SendGrid, here are a few pointers to help you determine if it's in the desired format:

• each section should be separated by a boundary marker with -- prefix (I marked them with red arrow in the example below).
• the Content-Disposition of each section should be "form-data"
• each section must properly labeled with a name (I marked the names with red rectangles in the example below).

[StuFrankish]

I've been able to work around the problem by implementing my own parser for the incoming FormData.
So for now you can close this issue as we're now receiving posted data from SendGrid without a problem.

For info though, this is what we were getting when we inspect the incoming request;

The content (from both ourselves using Postman & Sendgrid) is multipart/form-data.
But as soon as we step into ParseInboundEmailWebhookAsync() we get the "Value cannot be null" error.

My own parser only needs to be very simple to ingest automated job failure notifications, and looks like this;

[Jericho]

Did you get to the bottom of why you are getting webhook data in MIME format despite configuring your SendGrid data for structured data?

[StuFrankish]

According to SendGrid, the option was enabled for a short period then disabled - most likely one of our other developers trying different options.
But there appeared to be a lag between the option being turned off and the resulting payloads reverting back to formdata instead of raw.

It didn't resolve the issue ParseInboundEmailWebhookAsync() was having even with what appeared to be correctly formatted data, but for right now, we have a solution that works for us.
We'll come back to investigate later once we have the time.

[Jericho]

Ok, good to know that there's a delay between turning off the option and when it takes effect. Thanks for the info.

By the way, my offer still stands: if you give me a copy of the payload causing problem, I'll be happy to investigate and attempt to reproduce the problem.

[Jericho]

@Saldash are you still interested in researching this issue? I need a copy of your payload to investigate further. Otherwise, let me know and I'll close this issue.

[Jericho]

@Saldash I haven't heard back from you in a while so I'll go ahead and close this issue but I'll be happy to reopen it and continue the investigation if you provide additional info.