[$250 Internal] Notifications not being received

[Julesssss]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Problem

Multiple platforms are not receiving notifications in the production/staging environment.

Platform:

Where is this issue occurring?

• Android ✅
• iOS ✅
• Web?
• Desktop?

Version Number: 1.2.33-7 1.2.34-1 1.2.35-0, 1.2.36-2,
Reproducible in staging?: Yes
Reproducible in production?: Yes
Email or phone of affected tester (no customers): ALL
Issue reported by: @parasharrajat
Slack conversation: https://expensify.slack.com/archives/C01GTK53T8Q/p1670408581155829

View all open jobs on GitHub

[melvin-bot]

Triggered auto assignment to @Christinadobrzyn (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[Julesssss]

I tested release builds as far back as 1.2.33-7 and I'm not receiving notifications in any build. 1.2.34-1 is where my Airship config change was merged and this was my first guess, but it seems this is unlikely to be the issue now.

Next guess is a backend change, which I'll look into next.

[Julesssss]

Given that this already is on prod it should not be a deploy blocker, but it is a major issue that needs to be resolved ASAP

[Julesssss]

interestingly I did receive a single push notification from Chronos on the latest prod build. Not receiving messages from other accounts though:

[Julesssss]

I built a debuggable release build and discovered that we're hitting this condition. It makes sense that I'm not seeing notifications, but it does seem odd that this wasn't previously an issue. This is for local notification though which only affects web/desktop. I still need to figure out why Airship notifications have stopped.

[Julesssss]

@robertjchen reports that he isn't receiving notifications on Android or Desktop. Perhaps this isn't just Android, I'll ask in Slack about iOS.

[0xmiros]

Same to me. I haven't received any notifications recently even on web localhost. I remember this was working before.
If this is open to external, I will try to find regression and propose solution.

[Julesssss]

Same to me. I haven't received any notifications recently even on web localhost. I remember this was working before. If this is open to external, I will try to find regression and propose solution.

Thanks, but no this needs to be internal. It's critical, and is likely a backend issue.

[Julesssss]

Focusing on the notification project in the meantime, which is more critical.

[Julesssss]

We discussed this not being a deploy blocker and will fix after the notification doc has been released.

[Julesssss]

Looking into these PRs:

• https://github.com/Expensify/Web-Expensify/pull/35748/files - 'Async pusher notification failed', we recently started to see notifications with no data being sent:
• https://github.com/Expensify/Web-Expensify/pull/35572 - recently changed notification logic
• https://github.com/Expensify/Web-Expensify/pull/35741 - unlikely, but sounded related

I can confirm this is a recent config change issue, as I can receive notifications when directly targeting my device from the Airship console:

[Julesssss]

Next step is to dive into the logs. I found examples of successful and failed notifications.

Here's one of the failed messages. Notice that we route it via Pusher instead of Airship.

Here are logs for a successful push notification send (from Chronos).

---

Theories:

1. Have we rotated keys? I think no, because the test notification works, but I'm running out of ideas
2. Are we somehow accidentally setting shouldFailToSend to true (we recently switched to staging)
3. Send via Pusher async, OR just end via Pusher
   in Web-E because we're using the staging server?! We recently switched from prod to staging...
4. A recent Web-E change. I've been through all PRs from the last week and those suspicious PRs mentioned above don't show an obvious bug
5. Other PRs? A, B, C

[Julesssss]

Just noting that my previous guesses were wrong 😅

I found the issue and a PR is in review

[Julesssss]

Notifications are fixed -- just requires that users sign out and in again. Closing.

[parasharrajat]

@Julesssss Am I eligible for reporting bonus here?

[Julesssss]

Yeah of course. My bad.

@Christinadobrzyn could you please create an UpWork contract to pay a reporting bonus to @parasharrajat? Thanks

[Christinadobrzyn]

Upwork job - Internal - https://www.upwork.com/ab/applicants/1603246366811955200/job-details
External - https://www.upwork.com/jobs/~01cc93fd22b5099ef9

Hired @parasharrajat as reporter for the $250 payment.

@Julesssss or @parasharrajat can you confirm this job is finished and can be paid out/closed? I think it is based on the PR - https://github.com/Expensify/Web-Expensify/pull/35811

[Julesssss]

Hey @Christinadobrzyn, yep the PR was resolved and deployed.

[Christinadobrzyn]

@parasharrajat when you have a moment, can you please accept the offer so I can pay you? Thanks!

https://www.upwork.com/jobs/~01cc93fd22b5099ef9

[parasharrajat]

@Christinadobrzyn Accepted.

[Christinadobrzyn]

Thanks @parasharrajat! I paid you as the reporter, closed the Upwork job and am closing this! Let me know if I've missed anything!