MSC2762: Allowing widgets to send/receive events #2762

turt2live · 2020-09-02T19:50:23Z

Rendered

Implementations

Send events:

Add an implementation of MSC2762: send/receive events matrix-widget-api#9
Add new widget API actions for changing rooms and sending/receiving events matrix-react-sdk#5385

Read/receive events (from prior MSCs):

Timeline:

Redactions:

Special case redaction event sending from widgets per MSC2762 matrix-react-sdk#6686

proposals/2762-widget-event-receiving.md

MSC: matrix-org/matrix-spec-proposals#2762

Part of MSC2762: matrix-org/matrix-spec-proposals#2762 Requires: matrix-org/matrix-widget-api#9 This is the bare minimum required to send an event to a widget and receive events from widgets. Like the view_room action, this is controlled by a well-known permission key. **Danger**: This allows widgets to potentially modify room state. Use the permissions with care.

turt2live · 2020-11-03T04:48:43Z

This has been implemented in element-web: matrix-org/matrix-widget-api#9 & matrix-org/matrix-react-sdk#5385

KB1RD · 2021-01-26T20:02:26Z

proposals/2762-widget-event-receiving.md

+are introduced, with the same formatting requirements as the `m.send.event` and `m.send.state_event`
+capabilities above (ie: `m.receive.event:m.room.message#m.text`).
+
+For each event type requested and approved, the client sends a `toWidget` request with action `event`


Should local echos be sent to widgets? I.e, if a widget sends an event, should this event be sent back to them?

Current behavior by Element is to echo an event after the send request to the homeserver succeeds. This is sent along with a response to the send message sent by the widget. There is no way for widgets to ignore local echos unless they maintain a list of IDs that they have sent and ignore events with those IDs when they are received.

they shouldn't receive the local echo but they should receive the server's echo (if the widget requested the capability to do so)

I would really like to have local echos for the events. Would it be beneficial to include local echos in this msc? Or can this be achieved easier in some (future) widget-sdk build on top of the matrix-widget-api and we can keep this msc simpler?

turt2live · 2021-05-14T20:01:51Z

ftr this has absorbed #2876 given receiving is essentially the same as reading, so we make them the same action here.

turt2live · 2021-07-23T00:48:43Z

proposals/2762-widget-event-receiving.md

+The `events` array is simply the array of events requested. When no matching events are found, this
+array must be defined but can be empty.


we should define the order of this array too

With the addition of pagination tokes it would make the action more explicit if a dir parameter is introduced. (see: #2762 (comment))

Suggested change

The `events` array is simply the array of events requested. When no matching events are found, this

array must be defined but can be empty.

The `events` array is simply the array of events requested. When no matching events are found, this

array must be defined but can be empty. The order of this array is given by the `dir` parameter of the request.

The widget is informed if the end of the available events is reached by not providing an end token.

proposals/2762-widget-event-receiving.md

kevincox · 2021-07-26T22:48:09Z

proposals/2762-widget-event-receiving.md

+To get around an issue where widgets would not be able to request an event type with `#` in it (because
+it'll be seen as a state key), widgets can use a `\` character to escape the `#`. For example,
+`org.example.\#test#hello` would be parsed as an event type of `org.example.#test` with state key `hello`.
+Clients should be careful to parse `\\#` as `\#` (single escape).


This is confusing and likely insufficient. We should precisely describe the escaping protocol.

Confusing and insufficient how? It feels fairly precise to me.

It may be partly the wording "can use" and "should". It also isn't clear how \ is escaped in general. For example if I see \\\# that presumably means a literal \# in the event type. However you only talk about # \# and \\#. Presumably all slashes need to be handled.

How about:

Literal # and \ characters in the event type MUST be replaced with \# and \\.

I don't know if we need to explicitly spell out the decoding routine but it can be done similarly. It should be an error to have a resulting string that doesn't escape # or \. The other question is do we need to reserve other magic characters in the future? If so we need to escape those as well, use a better format than a string for structured data or just accept that we can never add a new restriction on these permissions.

kevincox · 2021-07-26T22:51:24Z

proposals/2762-widget-event-receiving.md

+
+If the widget did not get approved for the capability required to send the event, the client MUST
+send an error response (as required currently by the capabilities system for widgets). If the widget
+was approved, the client MUST only send the event to the room the user is currently viewing.


"currently viewing" sounds like timing attacks waiting to happen. Should it be the room where the widget was spawned? Or is it expected that there can be "account widgets" that work cross-room? In that case we need protocols to ensure that delayed events don't end up in the wrong room.

There are account widgets too. This MSC should probably do a better job of describing how room widgets work, which is indeed limiting them to where they are added.

The opportunity for a timing attack is so slim it's really not worth the extra overhead. If a stickerpicker sends an event and immediately changes room, the sticker might go to the wrong room, but the user would have had to switch in under 10ms in most cases. This is not a human-approachable response time.

proposals/2762-widget-event-receiving.md

MSC: matrix-org/matrix-spec-proposals#2762 Additions: matrix-org/matrix-spec-proposals@8eedf1d

toger5 · 2021-10-29T17:36:56Z

proposals/2762-widget-event-receiving.md

+single canonical room, however.
+
+To complement the send/receive event capabilities, a single capability is introduced to access the timelines
+of other rooms: `m.timeline:<Room ID>`. The `<Room ID>` can either be an actual room ID, or a `*` to denote


If I am understanding correctly, m.timeline:<Room ID> not only gives the widget access to the rooms timeline events but also to the state events. Because of the name timeline I am still not certain if this actually allows a widget to read the other rooms state events. Although the trick at the end of this paragraph and the calendar example in the introduction imply that this should be possible.
Something like m.roomAccess:<roomId> would be less confusing. Or adding:

... introduced to access the timeline and room state of other rooms: ...

to the spec text.

toger5

This suggestion contains three different proposed changes:

Adding the functionality that a widget can request an event with a specific id.
Adding pagination to the read_events action.
Propose a new format for the m.timeline capability to allow different send/receive permissions in different rooms.

toger5 · 2021-12-29T13:12:32Z

proposals/2762-widget-event-receiving.md

+`state_key` for `m.room.message` requests.
+
+The `type` is simply the event type to go searching for.
+


Suggested change

The `id` specifies an event id that the widget wants to load. If an `id` is present, all the other fields in `data` are not required anymore, but can still be provided.

The client will only send an event if an event with the requested id exists, the event conforms to the approved capabilities and to the other optionally provided fields in `data`.

toger5 · 2021-12-29T20:43:12Z

proposals/2762-widget-event-receiving.md

+The `limit` is the number of events the widget is looking for. The client can arbitrarily decide to
+return less than this limit, though should never return more than the limit. For example, a client
+may decide that for privacy reasons a widget can only ever see the last 5 room messages - even though
+the widget requested 25, it will only ever get 5 maximum back. When `limit` is not present it is
+assumed that the widget wants as many events as the client will give it. When negative, the client
+can reject the request with an error.


Suggested change

The `limit` is the number of events the widget is looking for. The client can arbitrarily decide to

return less than this limit, though should never return more than the limit. For example, a client

may decide that for privacy reasons a widget can only ever see the last 5 room messages - even though

the widget requested 25, it will only ever get 5 maximum back. When `limit` is not present it is

assumed that the widget wants as many events as the client will give it. When negative, the client

can reject the request with an error.

The `limit` is the number of events the widget is looking for. The client can arbitrarily decide to

return less than this limit, though should never return more than the limit. For example, a client

may decide that for privacy reasons a widget can only ever see the last 5 room messages - even though

the widget requested 25, it will only ever get 5 maximum back. When `limit` is not present it is

assumed that the widget wants as many events as the client will give it. When negative, the client

can reject the request with an error.

This action can be paginated. The `from` parameter is a token generated by the client,

from which the widget wants to read events. The direction can be set by the `dir` property.

`f` for chronological order and `b` for reversed order. The token can either be a token from the

client sever api [`/messages`](https://spec.matrix.org/v1.1/client-server-api/#get_matrixclientv3roomsroomidmessages) endpoint or it can be generated by

the client in case the client already has some of the history.

It cannot be assumed, that the pagination token is in any way compatible with the client server api `/messages` endpoint.

toger5 · 2021-12-29T20:48:08Z

proposals/2762-widget-event-receiving.md

+API action:
+
+```json
+{


Suggested change

{

// Read state event

{

toger5 · 2021-12-29T20:53:28Z

proposals/2762-widget-event-receiving.md

+    "type": "m.room.topic",
+    "limit": 25
+  }
+}


Suggested change

}

}

// Read room events

{

"api": "fromWidget",

"widgetId": "20200827_WidgetExample",

"requestid": "generated-id-1234",

"action": "read_events",

"data": {

"msgtype": "m.text",

"type": "m.room.message",

"limit": 50,

"id":"$EdUYYTBGSaLUNHGw9OlsVgtCO8fqfql519qJh0gfs4",

"from": "somePaginationTokenStringA",

"dir": "f" | "b",

}

toger5 · 2021-12-29T21:01:05Z

proposals/2762-widget-event-receiving.md

+the widget requested 25, it will only ever get 5 maximum back. When `limit` is not present it is
+assumed that the widget wants as many events as the client will give it. When negative, the client
+can reject the request with an error.
+


Here it could be discussed, if a new capability should be introduced. Something like: m.completeTimeline.

Suggested change

If the client approves the `m.completeTimeline` capability, it guarantees to give access to all events until

the time where the capability was approved. It is still possible to return less events in the response to a `read_events` action then the limit but it has to

return at least one event, if there is one available after the `from` pagination token and before the capability

approval. If this capability got approved the client is also responsible to query more events (this is NOT the case if if the widget does not have this capability) from the server

if the widget requests them and it is expected that non-gappy event lists are sent.

toger5 · 2021-12-30T13:35:44Z

proposals/2762-widget-event-receiving.md

+      }
+    ]
+  }
+}


Suggested change

}

}

// Response read_events (room events)

{

"api": "fromWidget",

"widgetId": "20200827_WidgetExample",

"requestid": "generated-id-1234",

"action": "read_events",

"data": {

"msgtype": "m.text",

"type": "m.room.message",

"limit": 50,

"id":"$EdUYYTBGSaLUNHGw9OlsVgtCO8fqfql519qJh0gfs4",

"from": "somePaginationTokenStringA",

"dir": "f" | "b"

}

"response": {

"start": "somePaginationTokenStringA",

"end": "somePaginationTokenStringC",

"events": [

{

"type": "m.room.message",

"sender": "@alice:example.org",

"event_id": "$example",

"room_id": "!room:example.org",

"origin_server_ts": 1574383781154,

"content": {

"msgtype": "m.text",

"body": "Hello"

},

"unsigned": {

"age": 12345

}

}

]

}

}

toger5 · 2021-12-30T14:41:31Z

proposals/2762-widget-event-receiving.md

+The `events` array is simply the array of events requested. When no matching events are found, this
+array must be defined but can be empty.


With the addition of pagination tokes it would make the action more explicit if a dir parameter is introduced. (see: #2762 (comment))

Suggested change

The `events` array is simply the array of events requested. When no matching events are found, this

array must be defined but can be empty.

The `events` array is simply the array of events requested. When no matching events are found, this

array must be defined but can be empty. The order of this array is given by the `dir` parameter of the request.

The widget is informed if the end of the available events is reached by not providing an end token.

toger5 · 2021-12-30T14:57:57Z

proposals/2762-widget-event-receiving.md

+To complement the send/receive event capabilities, a single capability is introduced to access the timelines
+of other rooms: `m.timeline:<Room ID>`. The `<Room ID>` can either be an actual room ID, or a `*` to denote
+all joined or invited rooms the client is able to see, current and future. The widget can limit its exposure
+by simply requesting highly scoped send/receive capabilities to accompany the timeline capability.
+
+Do note that a widget does not need to request capabilities for all rooms if it only ever interacts with the
+user's currently viewed room. Widgets such as stickerpickers will not need to request timeline capabilities
+because they'll always send events to the user's currently viewed room, and the client will let them do that
+without special room timeline permissions.


This suggestion is not related to the others. Its about having different permissions in different rooms. Here is a list of examples, why I think it might be good to be able to define different capabilities for each room:

I want to check if the user is part of a specific public room or space to provide additional functionality in the widget. In case this widget has lots of capabilities (send/receive) in the current room. The widget would also be able to receiv and send messages in the public room although i just want to check the m.room.member list if the current user is part of that room/space.

A calendar widget that should query calendar events (containing information about meetings/appointments) from all rooms should also be able to ping me with a message in its associated room to notify me about the events. But it should not be able to post messages in all rooms.

Suggested change

To complement the send/receive event capabilities, a single capability is introduced to access the timelines

of other rooms: `m.timeline:<Room ID>`. The `<Room ID>` can either be an actual room ID, or a `*` to denote

all joined or invited rooms the client is able to see, current and future. The widget can limit its exposure

by simply requesting highly scoped send/receive capabilities to accompany the timeline capability.

Do note that a widget does not need to request capabilities for all rooms if it only ever interacts with the

user's currently viewed room. Widgets such as stickerpickers will not need to request timeline capabilities

because they'll always send events to the user's currently viewed room, and the client will let them do that

without special room timeline permissions.

To send/receive events in different rooms, capabilities can be extended with `:<Room ID>` to access the timelines or states of other rooms. The complete format of capabilities the introduced capabilities is:

`<m.send.event | m.receive.event>:<event type>:<Room ID>`

The `<Room ID>` can either be an actual room ID, or a `*` to denote all joined or invited rooms

the client is able to see, current and future. In the case, where `*` is used,

the widget can limit its exposure by requesting highly scoped `<event type>`'s for the send/receive capabilities.

As presented in the previous sections, omitting the `:<Room ID>` part of the capability implies that the widget can ONLY send/receive events in the room the widget is attached to.

Widgets such as sticker-pickers therefore will not need to provide the `:<Room ID>` part in its capabilities,

because they'll always send events to the user's currently viewed room, and the client will let them do that

without specifying a room id.

In cases where the widget needs access to multiple rooms, but not all rooms (`*`) it needs to request multiple capabilities (e.g. `m.receive.event:m.room.message:$roomA` and `m.receive.event:m.room.message:$roomB`). It is NOT possible to provide a list of room id's in a single capability.

toger5 · 2021-12-30T15:54:06Z

proposals/2762-widget-event-receiving.md

+object):
+
+```json
+{


Suggested change

{

// Response read_events (state events)

{

dbkr · 2023-02-13T13:08:48Z

proposals/2762-widget-event-receiving.md

+the [`/redact` endpoint](https://matrix.org/docs/spec/client_server/r0.6.1#put-matrix-client-r0-rooms-roomid-redact-eventid-txnid)
+on behalf of the widget.
+
+## Proposal (receiving events in a widget)


I think this section needs to specify behaviour for encrypted events.

it should be covered somewhere in this proposal: the widget always receives decrypted events, and always sends unencrypted (to be encrypted by the client).

If you mean toDevice messages, those are #3819

turt2live added 2 commits September 2, 2020 13:49

Proposal to refine/define the send/receive event API for widgets

d53447b

Assign number

748a57f

turt2live changed the title ~~MSC0000: Allowing widgets to send/receive events~~ MSC2762: Allowing widgets to send/receive events Sep 2, 2020

turt2live added kind:feature MSC for not-core and not-maintenance stuff proposal A matrix spec change proposal proposal-in-review labels Sep 2, 2020

deepbluev7 reviewed Sep 2, 2020

View reviewed changes

proposals/2762-widget-event-receiving.md Show resolved Hide resolved

turt2live mentioned this pull request Sep 2, 2020

Widgets, the spec #2764

Draft

6 tasks

turt2live commented Nov 2, 2020

View reviewed changes

proposals/2762-widget-event-receiving.md Outdated Show resolved Hide resolved

turt2live added a commit to matrix-org/matrix-widget-api that referenced this pull request Nov 3, 2020

Add an implementation of MSC2762: send/receive events

06a06af

MSC: matrix-org/matrix-spec-proposals#2762

turt2live mentioned this pull request Nov 3, 2020

Add an implementation of MSC2762: send/receive events matrix-org/matrix-widget-api#9

Merged

turt2live mentioned this pull request Nov 3, 2020

Add new widget API actions for changing rooms and sending/receiving events matrix-org/matrix-react-sdk#5385

Merged

turt2live added the widgets anything to do with widgets label Nov 23, 2020

turt2live added 3 commits November 23, 2020 15:52

Apply suggestions from code review

a6eafc1

Fix parsing of state keys/msgtypes

8ad1926

Clarify rules for other event types

3aa58ee

turt2live mentioned this pull request Nov 24, 2020

Support MSC2762 proper once landed matrix-org/matrix-widget-api#22

Open

KB1RD reviewed Jan 26, 2021

View reviewed changes

turt2live mentioned this pull request May 14, 2021

MSC2876: Allowing widgets to read events in a room #2876

Closed

turt2live added 2 commits May 14, 2021 13:55

Merge branch 'master' into travis/msc/widgets-send-receive-events

5050f1e

Absorb MSC2876

9a06744

This was referenced May 14, 2021

Support UI for MSC2762: Widgets reading events from rooms matrix-org/matrix-react-sdk#5960

Merged

Replace read with receive as per MSC2762 matrix-org/matrix-widget-api#37

Merged

skolmer mentioned this pull request Jul 14, 2021

Allow widgets to read all existing events of a certain type element-hq/element-web#17994

Closed

turt2live commented Jul 23, 2021

View reviewed changes

Sorunome reviewed Jul 24, 2021

View reviewed changes

proposals/2762-widget-event-receiving.md Show resolved Hide resolved

Sorunome reviewed Jul 24, 2021

View reviewed changes

proposals/2762-widget-event-receiving.md Show resolved Hide resolved

kevincox approved these changes Jul 26, 2021

View reviewed changes

turt2live mentioned this pull request Aug 16, 2021

Expose the redact endpoint to widgets element-hq/element-web#18573

Closed

turt2live added 3 commits August 24, 2021 14:25

Merge branch 'master' into travis/msc/widgets-send-receive-events

b671246

Add timeline access

8eedf1d

widget

7786f95

turt2live added a commit to matrix-org/matrix-widget-api that referenced this pull request Aug 26, 2021

Add timeline support from MSC2762

baa1b4e

MSC: matrix-org/matrix-spec-proposals#2762 Additions: matrix-org/matrix-spec-proposals@8eedf1d

This was referenced Aug 26, 2021

Add timeline support from MSC2762 matrix-org/matrix-widget-api#41

Merged

Add support for MSC2762's timeline functionality matrix-org/matrix-react-sdk#6684

Merged

Special case redactions for now

853bed2

afranke force-pushed the old_master branch from dca99ee to 70192e8 Compare August 30, 2021 13:00

Remove arbitrary limits

aeadae8

turt2live force-pushed the old_master branch from e895827 to dca99ee Compare August 30, 2021 22:34

turt2live force-pushed the travis/msc/widgets-send-receive-events branch from 12e6603 to aeadae8 Compare August 30, 2021 22:35

toger5 reviewed Oct 29, 2021

View reviewed changes

toger5 reviewed Dec 30, 2021

View reviewed changes

AndrewRyanChama mentioned this pull request Mar 5, 2022

MSC3751: Allowing widgets to read account data #3751

Open

turt2live removed the proposal-in-review label May 5, 2022

This was referenced Sep 7, 2022

Support the widget API features needed for Element Call integration element-hq/element-android#7067

Open

Support the widget API features needed for Element Call integration element-hq/element-ios#6494

Open

yostyle mentioned this pull request Sep 7, 2022

Support the widget API features needed for Element Call integration element-hq/element-android#7066

Open

justinbot mentioned this pull request Oct 26, 2022

Integration managers should use the Widget API element-hq/element-meta#792

Open

AndrewKvalheim mentioned this pull request Nov 1, 2022

update language and link around a/v chat SeaGL/attend.seagl.org#21

Merged

justinbot mentioned this pull request Dec 12, 2022

Add actions for integration manager to send and read certain events matrix-org/matrix-react-sdk#9740

Merged

dbkr reviewed Feb 13, 2023

View reviewed changes

dbkr mentioned this pull request Feb 13, 2023

sendToDevice in StopGapWidgetDriver doesn't handle encrypted events properly element-hq/element-web#24470

Open

AndrewFerr mentioned this pull request Jul 10, 2024

MSC4157: Futures (widget api) #4157

Draft

5 tasks

robintown mentioned this pull request Sep 3, 2024

Always pass recent decryption retry successes to widgets matrix-org/matrix-react-sdk#12890

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MSC2762: Allowing widgets to send/receive events #2762

MSC2762: Allowing widgets to send/receive events #2762

turt2live commented Sep 2, 2020 •

edited

Loading

turt2live commented Nov 3, 2020

KB1RD Jan 26, 2021

turt2live Jan 26, 2021

toger5 Dec 30, 2021 •

edited

Loading

turt2live commented May 14, 2021

turt2live Jul 23, 2021

toger5 Dec 30, 2021 •

edited

Loading

kevincox Jul 26, 2021

turt2live Aug 24, 2021

kevincox Aug 28, 2021

kevincox Jul 26, 2021

turt2live Aug 24, 2021

toger5 Oct 29, 2021

toger5 left a comment

toger5 Dec 29, 2021

toger5 Dec 29, 2021

toger5 Dec 29, 2021

toger5 Dec 29, 2021 •

edited

Loading

toger5 Dec 29, 2021

toger5 Dec 30, 2021

toger5 Dec 30, 2021 •

edited

Loading

toger5 Dec 30, 2021 •

edited

Loading

toger5 Dec 30, 2021

dbkr Feb 13, 2023

turt2live Feb 13, 2023

		The `events` array is simply the array of events requested. When no matching events are found, this
		array must be defined but can be empty.

		`state_key` for `m.room.message` requests.

		The `type` is simply the event type to go searching for.



	The `id` specifies an event id that the widget wants to load. If an `id` is present, all the other fields in `data` are not required anymore, but can still be provided.
	The client will only send an event if an event with the requested id exists, the event conforms to the approved capabilities and to the other optionally provided fields in `data`.

-}
+}
+// Read room events
+{
+  "api": "fromWidget",
+  "widgetId": "20200827_WidgetExample",
+  "requestid": "generated-id-1234",
+  "action": "read_events",
+  "data": {
+    "msgtype": "m.text",
+    "type": "m.room.message",
+    "limit": 50,
+    "id":"$EdUYYTBGSaLUNHGw9OlsVgtCO8fqfql519qJh0gfs4",
+    "from": "somePaginationTokenStringA",
+    "dir": "f" | "b",
+}

+If the client approves the `m.completeTimeline` capability, it guarantees to give access to all events until
+the time where the capability was approved. It is still possible to return less events in the response to a `read_events` action then the limit but it has to
+return at least one event, if there is one available after the `from` pagination token and before the capability
+approval. If this capability got approved the client is also responsible to query more events (this is NOT the case if if the widget does not have this capability) from the server
+if the widget requests them and it is expected that non-gappy event lists are sent.

MSC2762: Allowing widgets to send/receive events #2762

Are you sure you want to change the base?

MSC2762: Allowing widgets to send/receive events #2762

Conversation

turt2live commented Sep 2, 2020 • edited Loading

turt2live commented Nov 3, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

toger5 Dec 30, 2021 • edited Loading

Choose a reason for hiding this comment

turt2live commented May 14, 2021

Choose a reason for hiding this comment

toger5 Dec 30, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

toger5 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

toger5 Dec 29, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

toger5 Dec 30, 2021 • edited Loading

Choose a reason for hiding this comment

toger5 Dec 30, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

turt2live commented Sep 2, 2020 •

edited

Loading

toger5 Dec 30, 2021 •

edited

Loading

toger5 Dec 30, 2021 •

edited

Loading

toger5 Dec 29, 2021 •

edited

Loading

toger5 Dec 30, 2021 •

edited

Loading

toger5 Dec 30, 2021 •

edited

Loading