Merge branch 'develop' of github.com:matrix-org/synapse into hawkowl/…

…cache-config-without-synctl * 'develop' of github.com:matrix-org/synapse: (76 commits) 1.12.4 Revert "Revert "Merge pull request #7315 from matrix-org/babolivier/request_token"" Revert "Merge pull request #7315 from matrix-org/babolivier/request_token" Stop the master relaying USER_SYNC for other workers (#7318) Config option to inhibit 3PID errors on /requestToken Fix replication metrics when using redis (#7325) formatting for the changelog Another go at fixing one-word commands (#7326) 1.12.4rc1 1.12.4rc1 fix changelog name Extend StreamChangeCache to support multiple entities per stream ID (#7303) Extend room admin api with additional attributes (#7225) Add ability to run replication protocol over redis. (#7040) Do not treat display names as globs for push rules. (#7271) Reduce logging verbosity of URL cache cleanup. (#7295) Query missing cross-signing keys on local sig upload (#7289) import urllib.parse when using urllib.parse.quote (#7319) Reduce federation logging on success (#7321) Fix changelog file ...
matrix-org · Apr 24, 2020 · f300c08 · f300c08
2 parents 905c833 + 68384d9
commit f300c08
Show file tree

Hide file tree

Showing 179 changed files with 5,087 additions and 2,534 deletions.
diff --git a/.buildkite/worker-blacklist b/.buildkite/worker-blacklist
@@ -5,8 +5,6 @@ Message history can be paginated
 
 Can re-join room if re-invited
 
-/upgrade creates a new room
-
 The only membership state included in an initial sync is for all the senders in the timeline
 
 Local device key changes get to remote servers

diff --git a/CHANGES.md b/CHANGES.md
@@ -1,10 +1,39 @@
 Next version
 ============
 
-* A new template (`sso_auth_confirm.html`) was added to Synapse. If your Synapse
-  is configured to use SSO and a custom `sso_redirect_confirm_template_dir`
-  configuration then this template will need to be duplicated into that
-  directory.
+* New templates (`sso_auth_confirm.html`, `sso_auth_success.html`, and
+ `sso_account_deactivated.html`) were added to Synapse. If your Synapse is
+ configured to use SSO and a custom  `sso_redirect_confirm_template_dir`
+ configuration then these templates will need to be duplicated into that
+ directory.
+
+* Plugins using the `complete_sso_login` method of `synapse.module_api.ModuleApi`
+  should update to using the async/await version `complete_sso_login_async` which
+  includes additional checks. The non-async version is considered deprecated.
+
+
+Synapse 1.12.4 (2020-04-23)
+===========================
+
+No significant changes.
+
+
+Synapse 1.12.4rc1 (2020-04-22)
+==============================
+
+Features
+--------
+
+- Always send users their own device updates. ([\#7160](https://github.com/matrix-org/synapse/issues/7160))
+- Add support for handling GET requests for `account_data` on a worker. ([\#7311](https://github.com/matrix-org/synapse/issues/7311))
+
+
+Bugfixes
+--------
+
+- Fix a bug that prevented cross-signing with users on worker-mode synapses. ([\#7255](https://github.com/matrix-org/synapse/issues/7255))
+- Do not treat display names as globs in push rules. ([\#7271](https://github.com/matrix-org/synapse/issues/7271))
+- Fix a bug with cross-signing devices belonging to remote users who did not share a room with any user on the local homeserver. ([\#7289](https://github.com/matrix-org/synapse/issues/7289))
 
 Synapse 1.12.3 (2020-04-03)
 ===========================
@@ -15,14 +44,10 @@ correctly fix the issue with building the Debian packages. ([\#7212](https://git
 Synapse 1.12.2 (2020-04-02)
 ===========================
 
-This release works around [an
-issue](https://github.com/matrix-org/synapse/issues/7208) with building the
-debian packages.
+This release works around [an issue](https://github.com/matrix-org/synapse/issues/7208) with building the debian packages.
 
 No other significant changes since 1.12.1.
 
->>>>>>> master
-
 Synapse 1.12.1 (2020-04-02)
 ===========================
 
@@ -42,12 +67,19 @@ Bugfixes
 Synapse 1.12.0 (2020-03-23)
 ===========================
 
-No significant changes since 1.12.0rc1.
-
 Debian packages and Docker images are rebuilt using the latest versions of
 dependency libraries, including Twisted 20.3.0. **Please see security advisory
 below**.
 
+Potential slow database update during upgrade
+---------------------------------------------
+
+Synapse 1.12.0 includes a database update which is run as part of the upgrade,
+and which may take some time (several hours in the case of a large
+server). Synapse will not respond to HTTP requests while this update is taking
+place. For imformation on seeing if you are affected, and workaround if you
+are, see the [upgrade notes](UPGRADE.rst#upgrading-to-v1120).
+
 Security advisory
 -----------------
 

diff --git a/UPGRADE.rst b/UPGRADE.rst
@@ -75,6 +75,71 @@ for example:
      wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
      dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
 
+Upgrading to v1.12.0
+====================
+
+This version includes a database update which is run as part of the upgrade,
+and which may take some time (several hours in the case of a large
+server). Synapse will not respond to HTTP requests while this update is taking
+place.
+
+This is only likely to be a problem in the case of a server which is
+participating in many rooms.
+
+0. As with all upgrades, it is recommended that you have a recent backup of
+   your database which can be used for recovery in the event of any problems.
+
+1. As an initial check to see if you will be affected, you can try running the
+   following query from the `psql` or `sqlite3` console. It is safe to run it
+   while Synapse is still running.
+
+   .. code:: sql
+
+      SELECT MAX(q.v) FROM (
+        SELECT (
+          SELECT ej.json AS v
+          FROM state_events se INNER JOIN event_json ej USING (event_id)
+          WHERE se.room_id=rooms.room_id AND se.type='m.room.create' AND se.state_key=''
+          LIMIT 1
+        ) FROM rooms WHERE rooms.room_version IS NULL
+      ) q;
+
+   This query will take about the same amount of time as the upgrade process: ie,
+   if it takes 5 minutes, then it is likely that Synapse will be unresponsive for
+   5 minutes during the upgrade.
+
+   If you consider an outage of this duration to be acceptable, no further
+   action is necessary and you can simply start Synapse 1.12.0.
+
+   If you would prefer to reduce the downtime, continue with the steps below.
+
+2. The easiest workaround for this issue is to manually
+   create a new index before upgrading. On PostgreSQL, his can be done as follows:
+
+   .. code:: sql
+
+      CREATE INDEX CONCURRENTLY tmp_upgrade_1_12_0_index
+      ON state_events(room_id) WHERE type = 'm.room.create';
+
+   The above query may take some time, but is also safe to run while Synapse is
+   running.
+
+   We assume that no SQLite users have databases large enough to be
+   affected. If you *are* affected, you can run a similar query, omitting the
+   ``CONCURRENTLY`` keyword. Note however that this operation may in itself cause
+   Synapse to stop running for some time. Synapse admins are reminded that
+   `SQLite is not recommended for use outside a test
+   environment <https://github.com/matrix-org/synapse/blob/master/README.rst#using-postgresql>`_.
+
+3. Once the index has been created, the ``SELECT`` query in step 1 above should
+   complete quickly. It is therefore safe to upgrade to Synapse 1.12.0.
+
+4. Once Synapse 1.12.0 has successfully started and is responding to HTTP
+   requests, the temporary index can be removed:
+
+   .. code:: sql
+
+      DROP INDEX tmp_upgrade_1_12_0_index;
 
 Upgrading to v1.10.0
 ====================

diff --git a/changelog.d/6899.bugfix b/changelog.d/6899.bugfix
@@ -0,0 +1 @@
+Improve error responses when accessing remote public room lists.
diff --git a/changelog.d/7040.feature b/changelog.d/7040.feature
@@ -0,0 +1 @@
+Add support for running replication over Redis when using workers.
diff --git a/changelog.d/7160.feature b/changelog.d/7160.feature
diff --git a/changelog.d/7185.misc b/changelog.d/7185.misc
@@ -0,0 +1 @@
+Move client command handling out of TCP protocol.
diff --git a/changelog.d/7186.feature b/changelog.d/7186.feature
@@ -0,0 +1 @@
+Support SSO in the user interactive authentication workflow.
diff --git a/changelog.d/7187.misc b/changelog.d/7187.misc
@@ -0,0 +1 @@
+Move server command handling out of TCP protocol.
diff --git a/changelog.d/7192.misc b/changelog.d/7192.misc
@@ -0,0 +1 @@
+Remove sent outbound device list pokes from the database.
diff --git a/changelog.d/7193.misc b/changelog.d/7193.misc
@@ -0,0 +1 @@
+Add a background database update job to clear out duplicate `device_lists_outbound_pokes`.
diff --git a/changelog.d/7199.bugfix b/changelog.d/7199.bugfix
@@ -0,0 +1 @@
+Fix a bug that could cause a user to be invited to a server notices (aka System Alerts) room without any notice being sent.
diff --git a/changelog.d/7207.misc b/changelog.d/7207.misc
@@ -0,0 +1 @@
+Remove some extraneous debugging log lines.
diff --git a/changelog.d/7213.misc b/changelog.d/7213.misc
@@ -0,0 +1 @@
+Add explicit Python build tooling as dependencies for the snapcraft build.
diff --git a/changelog.d/7219.misc b/changelog.d/7219.misc
@@ -0,0 +1 @@
+Add typing information to federation server code.
diff --git a/changelog.d/7225.misc b/changelog.d/7225.misc
@@ -0,0 +1 @@
+Extend room admin api (`GET /_synapse/admin/v1/rooms`) with additional attributes.
diff --git a/changelog.d/7226.misc b/changelog.d/7226.misc
@@ -0,0 +1 @@
+Move catchup of replication streams logic to worker.
diff --git a/changelog.d/7228.misc b/changelog.d/7228.misc
@@ -0,0 +1 @@
+Unblacklist '/upgrade creates a new room' sytest for workers.
diff --git a/changelog.d/7230.feature b/changelog.d/7230.feature
@@ -0,0 +1 @@
+Require admin privileges to enable room encryption by default. This does not affect existing rooms.
diff --git a/changelog.d/7233.misc b/changelog.d/7233.misc
@@ -0,0 +1 @@
+Remove redundant checks on `daemonize` from synctl.
diff --git a/changelog.d/7234.doc b/changelog.d/7234.doc
@@ -0,0 +1 @@
+Update the contributed documentation on managing synapse workers with systemd, and bring it into the core distribution.
diff --git a/changelog.d/7235.feature b/changelog.d/7235.feature
@@ -0,0 +1 @@
+Improve the support for SSO authentication on the login fallback page.
diff --git a/changelog.d/7236.misc b/changelog.d/7236.misc
@@ -0,0 +1 @@
+Upgrade jQuery to v3.4.1 on fallback login/registration pages.
diff --git a/changelog.d/7237.misc b/changelog.d/7237.misc
@@ -0,0 +1 @@
+Change log line that told user to implement onLogin/onRegister fallback js functions to a warning, instead of an info, so it's more visible.
diff --git a/changelog.d/7238.doc b/changelog.d/7238.doc
@@ -0,0 +1 @@
+Add documentation to the `password_providers` config option. Add known password provider implementations to docs.
diff --git a/changelog.d/7239.misc b/changelog.d/7239.misc
@@ -0,0 +1 @@
+Move catchup of replication streams logic to worker.
diff --git a/changelog.d/7240.bugfix b/changelog.d/7240.bugfix
@@ -0,0 +1 @@
+Do not allow a deactivated user to login via SSO.
diff --git a/changelog.d/7241.misc b/changelog.d/7241.misc
@@ -0,0 +1 @@
+Convert some of synapse.rest.media to async/await.
diff --git a/changelog.d/7243.misc b/changelog.d/7243.misc
@@ -0,0 +1 @@
+Correct the parameters of a test fixture. Contributed by Isaiah Singletary.
diff --git a/changelog.d/7248.doc b/changelog.d/7248.doc
@@ -0,0 +1 @@
+Add documentation to the `password_providers` config option. Add known password provider implementations to docs.
diff --git a/changelog.d/7249.bugfix b/changelog.d/7249.bugfix
@@ -0,0 +1 @@
+Fix --help command-line argument.
diff --git a/changelog.d/7251.doc b/changelog.d/7251.doc
@@ -0,0 +1 @@
+Modify suggested nginx reverse proxy configuration to match Synapse's default file upload size. Contributed by @ProCycleDev.
diff --git a/changelog.d/7259.bugfix b/changelog.d/7259.bugfix
@@ -0,0 +1 @@
+ Do not allow a deactivated user to login via SSO.
diff --git a/changelog.d/7260.bugfix b/changelog.d/7260.bugfix
@@ -0,0 +1 @@
+Fix room publish permissions not being checked on room creation.
diff --git a/changelog.d/7261.misc b/changelog.d/7261.misc
@@ -0,0 +1 @@
+Convert auth handler to async/await.
diff --git a/changelog.d/7265.feature b/changelog.d/7265.feature
@@ -0,0 +1 @@
+Add a config option for specifying the value of the Accept-Language HTTP header when generating URL previews.
diff --git a/changelog.d/7268.bugfix b/changelog.d/7268.bugfix
@@ -0,0 +1 @@
+Reject unknown session IDs during user interactive authentication instead of silently creating a new session.
diff --git a/changelog.d/7272.doc b/changelog.d/7272.doc
@@ -0,0 +1 @@
+Documentation of media_storage_providers options updated to avoid misunderstandings. Contributed by Tristan Lins.
diff --git a/changelog.d/7274.bugfix b/changelog.d/7274.bugfix
@@ -0,0 +1 @@
+Fix a sql query introduced in Synapse 1.12.0 which could cause large amounts of logging to the postgres slow-query log.
diff --git a/changelog.d/7279.feature b/changelog.d/7279.feature
@@ -0,0 +1 @@
+ Support SSO in the user interactive authentication workflow.
diff --git a/changelog.d/7286.misc b/changelog.d/7286.misc
@@ -0,0 +1 @@
+Move catchup of replication streams logic to worker.
diff --git a/changelog.d/7290.misc b/changelog.d/7290.misc
@@ -0,0 +1 @@
+Move catchup of replication streams logic to worker.
diff --git a/changelog.d/7291.misc b/changelog.d/7291.misc
@@ -0,0 +1 @@
+Improve typing annotations in `synapse.replication.tcp.streams.Stream`.
diff --git a/changelog.d/7295.misc b/changelog.d/7295.misc
@@ -0,0 +1 @@
+Reduce log verbosity of url cache cleanup tasks.
diff --git a/changelog.d/7300.misc b/changelog.d/7300.misc
@@ -0,0 +1 @@
+Fix sample SAML Service Provider configuration. Contributed by @frcl.
diff --git a/changelog.d/7303.misc b/changelog.d/7303.misc
@@ -0,0 +1 @@
+Fix StreamChangeCache to work with multiple entities changing on the same stream id.
diff --git a/changelog.d/7315.feature b/changelog.d/7315.feature
@@ -0,0 +1 @@
+Allow `/requestToken` endpoints to hide the existence (or lack thereof) of 3PID associations on the homeserver.
diff --git a/changelog.d/7318.misc b/changelog.d/7318.misc
@@ -0,0 +1 @@
+Move catchup of replication streams logic to worker.
diff --git a/changelog.d/7319.misc b/changelog.d/7319.misc
@@ -0,0 +1 @@
+Fix an incorrect import in IdentityHandler.
diff --git a/changelog.d/7321.misc b/changelog.d/7321.misc
@@ -0,0 +1 @@
+Reduce logging verbosity for successful federation requests.
diff --git a/changelog.d/7325.feature b/changelog.d/7325.feature
@@ -0,0 +1 @@
+Add support for running replication over Redis when using workers.
diff --git a/changelog.d/7326.misc b/changelog.d/7326.misc
@@ -0,0 +1 @@
+Move catchup of replication streams logic to worker.