Stuck notifications

[Johennes]

We are experiencing multiple issues in the area of "stuck" notifications and unread markers. Many are related to the way receipts interact with threaded messages.

Symptoms

We are seeing different symptoms of the problem:

• "Stuck" unread dots and notification counters: Even when I have read all messages in a room, the room is still marked as unread or having notifications.
• Unread dots or notification counters returning on app startup: When I restart the app (or refresh the page on Web) rooms which I have already read re-appear as unread.
• Notification counters growing and shrinking spontaneously after entering or scrolling in a room.

Spec-level causes

Message ordering

Fundamentally, in order to interpret the meaning of a receipt that says "I have read everything up to here", we need to know what order messages are in. This is not clear in the spec, and we propose to make it clear and explicit in MSC4033.

In the meantime, Element Web uses a combination of "sync" order (the implicit order of events arriving via a /sync request) and "timestamp" order (using the ts property within events).

Some of the existing bugs are probably caused by this inconsistency, but it is not clear yet how many: we believe there are also bugs in the implementation that cause additional problems, and this theoretical inconsistency is only the cause of a few problems.

Which thread the root belongs to

The spec has what we consider a bug when it talks about which thread the root message belongs to, which has been reflected in client code, making it inconsistent with the server implementation (at least on the Synapse server). We have a proposal to fix this bug in MSC4037.

Identifying which thread any message is in

It is sometimes difficult for clients to identify which thread an event belongs to, meaning that a receipt pointing to it is sometimes ignored. We have begun drafting MSC4023 to address this.

Other

Previously, we believed that MSC3981 (recursive relations) would solve some of the problems, but since that MSC does not solve the event-ordering problem (because the events from the /relations API are returned in "topological" order) we no longer believe it is important, except as a performance optimisation.
Code-level causes

Code-level causes

We have found and fixed several bugs in the Element Web code that were caused by an incomplete understanding of the meaning of threaded and unthreaded read receipts. We anticipate that some more exist.

(We believe that the primary reason why we're not seeing the same problems on mobile is that the apps persist events they've received whereas Element Web has to re-fetch from scratch after every launch. As a result, any issue in the unread state logic, strikes again and again. The apps also use a single timeline whereas Element Web maintains one timeline per thread in addition to the main timeline in every room.)

High-level plan of actions

Tasks

Beta Give feedback

1. Set up POC branch / deployment with threads disabled – Experiment with putting threads back behind a beta toggle #25676
2. Fix read-receipt behavior around thread roots – Fix read receipt sending behaviour around thread roots matrix-org/matrix-js-sdk#3600
3. Fix read-receipt behavior around non-thread relations to thread roots – Fix edge cases around non-thread relations to thread roots and read receipts matrix-org/matrix-js-sdk#3607
4. Fix missing message issues due to replies to unknown events – Fix how Room::eventShouldLiveIn handles replies to unknown parents matrix-org/matrix-js-sdk#3615
5. Fix unread count returning from zero after reload – Message in a thread is read and becomes unread when I refresh #25806
6. Fix notification counters sometimes being doubled – Notification count is doubled in e2ee rooms on startup #25803
7. Fix remaining bugs found in no-threads POC as they likely impact the threaded experience as well – Experiment with putting threads back behind a beta toggle #25676
8. Set up unread & notification test suite – Test cases for stuck unreads and notifs #25449
9. Pre-investigate issues listed after the threads model refactoring to see if any of them are fixable before
10. Refactor threads model code and remove serverless / legacy code path to simplify further changes
11. Fix missing notifications – Thread replies sent from Element Android cause neither unread count nor unread dot on Element Desktop #25621
12. Fix incorrect thread reply count – Reply count in thread is wrong #24636
13. Fix notification counters not adding up between spaces and rooms – Notification badges in left panel contradict each other #20372
14. Fix unread counter explosion – Unread count goes from zero through the roof after restarting the app #25479
15. Fix Zombie notifications from old threads – ? issue ?
16. Fix read receipts / fully read marker jumping backwards – ? issue ?
17. Triage existing issues to increase confidence that the root causes will actually be fixed through the MSCs mentioned above.

We believe a lot of progress can still be made without spec changes. So we're slightly deprioritising work on the MSCs.

• MSC4037: Thread root is not in the thread matrix-org/matrix-spec-proposals#4037
  We claim it is implemented in EW
  Synapse will need small fixes when this is merged
• MSC4033: Explicit ordering of events for receipts matrix-org/matrix-spec-proposals#4033
  Needs implementation on server and client but we believe it should mostly only cover edge cases
• MSC4023: Thread ID for 2nd order-relation matrix-org/matrix-spec-proposals#4023
  There is disagreement about the way to specify this and possible interference with MSC3051: A scalable relation format matrix-org/matrix-spec-proposals#3051
• MSC3389: Relation redactions matrix-org/matrix-spec-proposals#3389
  There is likely no good workaround other than implementing this MSC

New issue inbox

The following is a holding area for newly reported issues that require review. Once reviewed, issues should either be moved to one of the other task lists below or, if not applicable, removed from this epic.

Tasks

Beta Give feedback

1. Missing notifications badges vector-im/element-web#25331
   A-Notif-Panel A-Notifications O-Occasional S-Major T-Defect X-Regression +6
2. Stuck notifications due to unread threads not updating rapidly. vector-im/element-web#25420
   A-Notifications A-Threads O-Frequent S-Major T-Defect +5
3. Room stuck as unread despite all messages/threads marked as read (to my knowledge) vector-im/element-web#25481
   A-Notifications T-Defect X-Needs-Investigation +3
4. Stuck notification after redacting thread root vector-im/element-web#25480
   A-Notifications A-Threads O-Occasional S-Major T-Defect +5
5. Unread count goes from zero through the roof after restarting the app vector-im/element-web#25479
   A-Notifications O-Occasional S-Major T-Defect +4
6. https://github.com/matrix-org/element-web-rageshakes/issues/21717
7. https://github.com/matrix-org/element-web-rageshakes/issues/21724
8. Stuck notification case vector-im/element-web#25513
   A-Notifications O-Occasional S-Critical T-Defect +4
9. https://github.com/matrix-org/element-web-rageshakes/issues/21737
10. Stuck notifications due to edits in threads vector-im/element-web#25482
    A-Notifications A-Threads O-Occasional S-Major T-Defect +5
11. Constant unread highlight notifications in one particular room vector-im/element-web#25528
    A-Notifications O-Occasional S-Critical T-Defect +4
12. Starting nightly for the first time seems to trigger notification counter mess in normal Desktop vector-im/element-web#25541
    A-Notifications O-Occasional S-Major T-Defect +4
13. Thread replies sent from Element Android cause neither unread count nor unread dot on Element Desktop vector-im/element-web#25621
    A-Notifications A-Threads O-Occasional S-Major T-Defect +5
14. Room with thread message not shown at top of list when „Show rooms with unread messages first“ is checked vector-im/element-web#24547
    A-Threads O-Frequent S-Minor T-Defect +4
15. Stuck unread when latest message is in a thread vector-im/element-web#25623
    A-Notifications A-Threads O-Occasional S-Minor T-Defect +5
    
16. Errors in console likely linked to stuck unreads vector-im/element-web#25642
    A-Notifications O-Occasional S-Minor T-Defect +4
    
17. Persistent Threads dot when last message in a thread is redacted vector-im/element-web#25670
    A-Notifications A-Redaction A-Threads O-Occasional S-Minor T-Defect +6
18. Experiment with putting threads back behind a beta toggle vector-im/element-web#25676
    1 of 2
    A-Notifications A-Threads T-Task +3
19. Permanent stuck unread-notification when starting a thread with a reply message vector-im/element-web#23976
    A-Notifications A-Threads O-Occasional S-Major T-Defect X-Cannot-Reproduce X-Needs-Info +7
20. Read message is marked as unread vector-im/element-web#25804
    A-Read-Marker O-Uncommon S-Major T-Defect X-Regression +5
21. Message in a thread is read and becomes unread when I refresh vector-im/element-web#25806
    A-Notifications O-Occasional S-Minor T-Defect +4
    
22. Read status of room resetting itself and not shared to other devices vector-im/element-web#24111
    A-Room-List O-Frequent S-Minor T-Defect +4
23. impossible to dismiss notification vector-im/element-web#25408
    A-Notifications O-Frequent S-Minor T-Defect +4
24. New threads/new thread messages always show "new messages" at already 20x times read other threads too vector-im/element-web#25907
    A-Notifications A-Threads O-Occasional S-Major T-Defect +5
25. Audible ping but no badge on notifications for bot messages vector-im/element-web#25904
    A-Notifications T-Enhancement Z-GetYourUpdates Z-Labs +4
26. Stuck unread dot on v1.11.38 vector-im/element-web#25929
    A-Notifications O-Occasional S-Major T-Defect +4
27. https://github.com/matrix-org/element-web-rageshakes/issues/22226
28. https://github.com/matrix-org/element-web-rageshakes/issues/22363
29. https://github.com/matrix-org/element-web-rageshakes/issues/22414
30. Stuck notifications in a room with no posting permissions vector-im/element-web#25975
    O-Frequent S-Minor T-Defect +3
31. Persistent stuck unread in encrypted room #25984
    A-Notifications O-Frequent S-Major T-Defect +4
32. New message count not clearing in DM vector-im/element-web#25950
    A-Notifications O-Occasional S-Minor T-Defect +4
33. Stuck notification in DM #26063
    A-Notifications O-Occasional S-Major T-Defect +4
34. https://github.com/matrix-org/element-web-rageshakes/issues/22526
35. Threads might skip processing some replies from the sync response matrix-org/matrix-js-sdk#3665
    +0
36. Notification for unread redacted message in thread gets stuck #26933
    A-Notifications A-Threads O-Uncommon S-Major T-Defect +5

Tasks not blocked by spec work

Tasks

Beta Give feedback

1. Stuck notifications: Create devtools room stuck notifications debug tool vector-im/element-web#24388
   A-DevTools T-Task +2
   
2. Stuck notifications: Reacting to a not-latest message in a thread causes a stuck notification vector-im/element-web#24000
   A-Threads O-Uncommon S-Minor T-Defect +4
   
3. Stuck notifications: DM stuck as unread vector-im/element-web#23991
   A-DMs A-Notifications O-Frequent S-Major T-Defect Z-MadLittleMods +6
   
4. Stuck notifications: New DMs have stuck notifications #23685
   A-Notifications O-Frequent S-Major T-Defect Z-MadLittleMods Z-Synapse +6
5. https://github.com/matrix-org/element-web-rageshakes/issues/21667
6. Read messages reappear as "new" after restarting the app. vector-im/element-web#24629
   A-Notifications O-Frequent S-Major T-Defect +4
   
7. „Mark as read“ button does not send the correct receipt for the last event in a thread vector-im/element-web#25207
   A-Read-Receipts A-Threads O-Frequent S-Minor T-Defect +5
   
8. Unthreaded read receipts are sent after reading the main timeline vector-im/element-web#25196
   A-Notifications A-Read-Receipts A-Threads O-Uncommon S-Minor T-Defect +6
   
9. Test how we handle receiving an unthreaded read receipt vector-im/element-web#25212
   A-Read-Receipts +1
   
10. Rooms incorrectly marked as unread vector-im/element-web#10954
    A-Read-Marker A-Room-List T-Defect +3
    
11. https://github.com/matrix-org/element-web-rageshakes/issues/21641
12. Cannot mark room as read. vector-im/element-web#25411
    T-Defect X-Release-Blocker +2
    
13. New reaction to old event ends up on the main timeline instead of the thread vector-im/element-web#25450
    O-Uncommon S-Major T-Defect X-Release-Blocker +4
    
14. Test cases for stuck unreads and notifs #25449
    A-Notifications T-Enhancement +2
    
15. Experiment with putting threads back behind a beta toggle vector-im/element-web#25676
    1 of 2
    A-Notifications A-Threads T-Task +3
16. Missing reactions for some messages vector-im/element-web#25596
    A-Reactions O-Uncommon S-Major T-Defect +4
    

Tasks that are related to or dependent on spec work

We've written the following MSCs to try and address the root causes in a reliable and performant way:

• MSC4033: Explicit ordering of events for receipts matrix-org/matrix-spec-proposals#4033
  A client-side workaround for this based on timestamp ordering has been implemented. This is imperfect and easily abused though.
• MSC4023: Thread ID for 2nd order-relation matrix-org/matrix-spec-proposals#4023
  A client-side workaround for this based on calling /event to fetch the parent has been implemented. This is functionally correct but has a noticeable performance impact.
• MSC4037: Thread root is not in the thread matrix-org/matrix-spec-proposals#4037
  The client-side code has already been modeled to reflect the behavior proposed in the MSC (which is also what Synapse already does today).
• MSC3981: /relations recursion matrix-org/matrix-spec-proposals#3981
  We expect this to not help with the ordering problems but it will be a performance improvement.

Tasks

Beta Give feedback

1. Write / shepherd MSC3981 vector-im/element-meta#1350
   A-Notifications T-Task +2
   
2. Implement MSC3981 in Synapse matrix-org/synapse#15377
   A-Threads O-Occasional S-Minor T-Enhancement +4
   
3. Implement MSC3981 in matrix-react-sdk vector-im/element-web#25021
   A-Notifications O-Frequent S-Major T-Task Team: App +5
   
4. Room stuck as unread after mention in a thread #24312
   A-Threads O-Occasional S-Minor T-Defect +4
5. Stuck unread with reaction + start thread vector-im/element-web#24394
   A-Notifications A-Threads O-Frequent S-Major T-Defect +5
6. Can't get rid of unread status of a room vector-im/element-web#24595
   A-Notifications O-Frequent S-Major T-Defect +4
7. Thread not being marked as read vector-im/element-web#24442
   A-Notifications A-Threads O-Occasional S-Major T-Defect +5
8. impossible to dismiss notification vector-im/element-web#25408
   A-Notifications O-Frequent S-Minor T-Defect +4
9. https://github.com/matrix-org/element-web-rageshakes/issues/21679
10. Stuck notifications due to edits in threads vector-im/element-web#25482
    A-Notifications A-Threads O-Occasional S-Major T-Defect +5
11. Write / shepherd MSC4023 element-meta#1714
    App: web +1
12. Implement MSC4023: Thread ID for 2nd order-relation in Synapse matrix-org/synapse#15701
    A-Relations-Endpoint A-Threads O-Occasional S-Minor T-Task roadmap +6
13. Excessive calls to GET /relations and GET /event, with MSC3981 enabled and working vector-im/element-web#25395
    O-Frequent S-Major T-Defect +3
    
14. Stuck notifications when there's no new message after a thread vector-im/element-web#25893
    A-Notifications O-Uncommon S-Major T-Defect +4

Issues that are related but out of scope

• Several places in the code use timestamp to determine event order matrix-org/matrix-js-sdk#3325

Time sheeting

WEB: Stuck notifications

[Johennes]

Descoping #24373 as this doesn't have the same level of severity as the other issues.

[Johennes]

As mentioned in the issue description, the remaining bugs will require a spec change. We're currently in the process of drafting the MSC but it'll, unfortunately, be more involved than just a simple client-side bug fix.

[jplatte]

I don't think that is true. If you look at the comments in #24595, you will find some from rooms where threads were never used and IME clearing cache often fixes these issues.

[Johennes]

We're focusing on the most prominent problems first which are in the interplay of threads and other relations. There may be further issues beyond that.

[AlBundy33]

I had the same issue in some rooms.
Because I don't participate that much in this rooms I tried to write a test-message and it seems that the stuck notifications are gone.

[Johennes]

@8227846265 the threads-related stuck notification issues are blocking on matrix-org/matrix-spec-proposals#3981.

[Johennes]

We've completed initial implementations for it last week and are now starting to prepare for testing it. Unfortunately, it'll still take some more time to get this landed due to the nature of the spec process.

[leonardehrenfried]

@Johennes Thanks for this explanation. If there is an environment where a regular user could help test it, I'd love to hear about it.

[Johennes]

Thanks @leonardehrenfried. We will most likely be enabling it on beta.matrix.org as to not impact regular users but you can sign-up for a separate account there if you want to help testing.

[alexander-potemkin]

@Johennes , thanks for the explanation indeed! How soon it could be expected to land to the self hosted solutions?

[Johennes]

I'd expect the initial PRs (matrix-org/synapse#15315 & matrix-org/matrix-js-sdk#3248) to land this week.

[leonardehrenfried]

For everybody subscribed to this ticket: the backend and frontend PRs have been merged now so I expect a rollout quite soon.

(I'm not an Element/Matrix employee, just an interested user.)

[Johennes]

@8227846265 we have one more fix in the making that doesn't depend on the MSC (#25196) and should land soon. If that one doesn't help your case, we'd be interested to hear more details about your particular error scenario.

[leonardehrenfried]

Even this one chat that I couldn't mark as read for ~ 1 year doesn't have a notification anymore.

[j-nava]

I've been using develop.element.io for quite a while now and I can also attest that it's getting better and better. I can't wait for this to be completely fixed so I can convince more people to join my little community of friends. 😃

Keep up the good work!

[MTRNord]

Seems like since the latest changes I am back to notifications that cant be dismissed on any device. Only a clear cache works. This affects both ele-web and ele-android. So I wonder if its actually synapse causing it this time around 🤔

[benjaoming]

@MTRNord

Seems like since the latest changes

Can you be more clear about which changes you refer to? In the comments immediately before yours, people are reporting success from develop.element.io so it's important to know if you are having a contradicting experience with the exact same version of Element.

[simonmichael]

When will the develop.element.io fixes be rolled out in Element desktop and mobile and Element X production/testflight ?

[MTRNord]

@MTRNord

Seems like since the latest changes

Can you be more clear about which changes you refer to? In the comments immediately before yours, people are reporting success from develop.element.io so it's important to know if you are having a contradicting experience with the exact same version of Element.

I dont really have an exact commit but it happened to start like that after the crypto upgrade on friday(?) on develop while being on fosdem.

[ajkessel]

Still experiencing this problem daily in multiple rooms with latest develop.element.io as well as the regular web/desktop release and on iOS Element from TestFlight, using Synapse 1.101.

What's really perplexing is how random it is. Unread notifications show up in random old threads that haven't been touched for a while. There seems to be no precipitating event or rhyme or reason as to what triggers the notification.

Usually, "mark all as read" clears it now, at least temporarily, which is some progress.

Is there any client- or server-side logging I could do to try to isolate the cause more systematically?

[iyanmv]

Maybe someone can do a quick update about the test cases? @florianduros ?

[florianduros]

@iyanmv #25449 we closed the issue about the test cases. We assume that we fixed the existing tests and we are satisfied with the current tests. Of course, when we are fixing bus around notifications we are adding new tests.

[jo-so]

My original report #8941 was about rooms flagged as having new messages after login or cache reset, but they don't have. This issue was closed in favour of this one, but yesterday, I've logging in on a new session and still got stale room notifications. Does anything happen to this issue?

[mutantcornholio]

After each time I restart Element, I have to go over 10-15 chats, to mark everything as read. A lot of times, the chats are half a year old.

I feel like it's connected to the fact that messages are being loaded when I click on the conversation, not when I open Element, but that's just a guess.

[mutantcornholio]

Also, opening chats for the first time after restarting element, when the messages are loaded, marks some of the loaded ancient messages from threads as unread.

[mutantcornholio]

Also, opening chats for the first time after restarting element, when the messages are loaded, marks some of the loaded ancient messages from threads as unread.

One more thing that I've noticed, is a lot of times the ancient messages are stuff like Alice invited Bob, or Alice changed the power level of Bob from Default to Admin, or Alice made the room invite only. Happens far more often in comparison to active chats, where people actually send messages.

[ajkessel]

Has progress stalled? I've had the same stuck unread notifications for several months now, both on every production release of Element as well as running bleeding edge from develop.element.io. It does seem to be the same rooms/old messages that consistently show as unread. Is there anything I can do as an end-user to troubleshoot the circumstances, particularly where the issue is reproducible at least for some fraction of the time?

[mutantcornholio]

Any update?

[JMoVS]

Is there still work on threads becoming unread again and again all the time? Happens quite frequently to me nowadays

[MTRNord]

Is there still work on threads becoming unread again and again all the time? Happens quite frequently to me nowadays

I also have this. In foundation room I get a loud ping for a mention in a thread each time a new message is added to the room. Its annoying me.

[dreirund]

Is there still work on threads becoming unread again and again all the time? Happens quite frequently to me nowadays

I have this, too.

[HelderFSFerreira]

I have more than 30 rooms stuck notifications.

Mark everything a read does not work. Let me know if I should provide any logs.

[JMoVS]

are there more logs required or anything? This issue has been around for more than a full year now and recently progress seems to have stalled