InternalMessageId on webhook Events not always being returned with .filter in the value

[nelyom]

A recent change in the data being received from Webhooks shows that the IndexOf in the Event.MessageId property just needs to use the first . (dot) to substring the MessageId. We are regularly receiving incoming Events with data like YYYYYYYY.recvd-XXXXX-XXXX-5.0
but not on all events.

StrongGrid/Source/StrongGrid/Models/Webhooks/Event.cs

Line 83 in 4e77c2f

var filterIndex = InternalMessageId.IndexOf(".filter", StringComparison.OrdinalIgnoreCase);

[Jericho]

Can you share the payload you receive to help me reproduce the situation and write unit test.

[nelyom]

Unfortunately we don't store the raw incoming payloads.
However, for real data examples

What we are receiving currently for some InternalMessageId are:

56WoYBdESN6RyEn_7bJRKA.recvd-5b6594c664-csn64-1-65A9F4D1-A.0
kiC2BsplQXGrH6HLUMYlgQ.recvd-5b6594c664-jh9rs-1-65A9F366-11.1
q30rlKhQQ4CCuDhRMZR2_Q.recvd-f4c78f47d-crzxh-1-65A9F4D5-11.0

The previous InternalMessageId looked like.

tC4LGVjZTgi116GN5ZRO6Q.filterdrecv-p3iad2-5dc87598f5-bvh7w-20-5FB44F45-3E.0
pWucfaSHSF6rUvaYz0-Cqw.filterdrecv-p3iad2-6d658b4b48-4cgp5-20-5FB1E16E-1D.0
2H7YeIvAQlOM4OJbtwr92Q.filterdrecv-p3iad2-df8579867-kwdns-18-5F7CDB37-1C.0

SendGrid appear to have dropped the text filterrecv and replaced with just recvd (and some of the other numbers). But only for some messages, not all. We are now receiving both types formats of the InternalMessageId.

Of note, if a Event message gets this new InternalMessageId format, it will always get it for all EventTypes for the message chain (linked by MessageId).

A review of our stored history shows that the "." (dot ) is the separator rather than looking for ".filter" when extracting the shorter MessageId to tie the original Send response back to the Events

[Jericho]

Thanks for providing these samples, they help me better understand and I'll be able to turn them into unit tests.

SendGrid appear to have dropped the text filterrecv and replaced with just recvd. But only for some messages, not all.

Agree. They indeed appear to be changing the format of the message id. I'm guessing they are slowly rolling out this change which would explain why some of your messages still have the old "filter" and some have the new "recvd". Probably all messages will switch to the new format over the next few days/weeks/months.

just needs to use the first . (dot) to substring the MessageId.

My first reaction when I read your original comment is that we should continue looking for .filter and fallback on .recvd if we don't find 'filter'. But the more I think about it, the more I'm starting to think you might have a point: we should simply look for .. This would insulate us from possible future changes. The only risk I can think of is that we would introduce a bug if a message id was to legitimately contain a dot like this hypothetical example: aaaaaa.bbbbbb.ccccc.recvd-5b6594c664-csn64-1-65A9F4D1-A.0. Is this really a legitimate possibility or did I just make up a very unlikely scenario? I don't know. and I don't think we have a way to validate my theory.

How about we combine both my idea and yours like so:

1. look for .filter
2. fall back on .recvd if we don't find "filter"
3. fall back on . if we found neither "filter" not "recvd"

Edit:
What I'm proposing is something like this:

if (string.IsNullOrEmpty(InternalMessageId)) return InternalMessageId;

// Up until January 2024, a typical message id looked like this: tC4LGVjZTgi116GN5ZRO6Q.filterdrecv-p3iad2-5dc87598f5-bvh7w-20-5FB44F45-3E.0
var filterIndex = InternalMessageId.IndexOf(".filter", StringComparison.OrdinalIgnoreCase);
if (filterIndex > 0) return InternalMessageId.Substring(0, filterIndex);

// In January 2024, SendGrid introduced a different format for the message id: 56WoYBdESN6RyEn_7bJRKA.recvd-5b6594c664-csn64-1-65A9F4D1-A.0
filterIndex = InternalMessageId.IndexOf(".recvd", StringComparison.OrdinalIgnoreCase);
if (filterIndex > 0) return InternalMessageId.Substring(0, filterIndex);

// This check should future-proof our logic in case SendGrid changes the format once more
filterIndex = InternalMessageId.IndexOf(".", StringComparison.OrdinalIgnoreCase);
if (filterIndex > 0) return InternalMessageId.Substring(0, filterIndex);

// If all else fails
return InternalMessageId;

[nelyom]

The only risk I can think of is that we would introduce a bug if a message id was to legitimately contain a dot like this hypothetical example: aaaaaa.bbbbbb.ccccc.recvd-5b6594c664-csn64-1-65A9F4D1-A.0. Is this really a legitimate possibility or did I just make up a very unlikely scenario? I don't know. and I don't think we have a way to validate my theory.

Indeed. I can't say I know the answer here either. We are a small operation and only have ~250,000 emails sent, but none of them have a MessageId with a . in it when stored from the Send action and when I look through the Events recorded for those sent emails (1.5m records), I can safely use the first . to get a valid MessageId that matches the MessageId stored from the Send.

How about we combine both my idea and yours like so:

If you feel that is the better compatible option. I'd probably pepper it with a comment as to why and bemoaning the poor documentation or notification from SendGrid/Twilio, but you already know all that.

[Jericho]

I'm on it. I'll try to publish an update this afternoon but, if life gets in my way, I'll work on it over the weekend.

Thanks again for making me aware of this new situation.

[Jericho]

🎉 This issue has been resolved in version 0.104.0 🎉

The release is available on:

• GitHub Release
• NuGet Package

Your GitReleaseManager bot 📦🚀