Merge tag 'v1.45.0' into rav/sw1v-hotfixes

Synapse 1.45.0 (2021-10-19)
===========================

No functional changes since Synapse 1.45.0rc2.

Known Issues
------------

- A suspected [performance regression](#11049) which was first reported after the release of 1.44.0 remains unresolved.

  We have not been able to identify a probable cause. Affected users report that setting up a federation sender worker appears to alleviate symptoms of the regression.

Improved Documentation
----------------------

- Reword changelog to clarify concerns about a suspected performance regression in 1.44.0. ([\#11117](#11117))

Synapse 1.45.0rc2 (2021-10-14)
==============================

This release candidate [fixes](#11053) a user directory [bug](#11025) present in 1.45.0rc1.

Known Issues
------------

- A suspected [performance regression](#11049) which was first reported after the release of 1.44.0 remains unresolved.

  We have not been able to identify a probable cause. Affected users report that setting up a federation sender worker appears to alleviate symptoms of the regression.

Bugfixes
--------

- Fix a long-standing bug when using multiple event persister workers where events were not correctly sent down `/sync` due to a race. ([\#11045](#11045))
- Fix a bug introduced in Synapse 1.45.0rc1 where the user directory would stop updating if it processed an event from a
  user not in the `users` table. ([\#11053](#11053))
- Fix a bug introduced in Synapse 1.44.0 when logging errors during oEmbed processing. ([\#11061](#11061))

Internal Changes
----------------

- Add an 'approximate difference' method to `StateFilter`. ([\#10825](#10825))
- Fix inconsistent behavior of `get_last_client_by_ip` when reporting data that has not been stored in the database yet. ([\#10970](#10970))
- Fix a bug introduced in Synapse 1.21.0 that causes opentracing and Prometheus metrics for replication requests to be measured incorrectly. ([\#10996](#10996))
- Ensure that cache config tests do not share state. ([\#11036](#11036))

Synapse 1.45.0rc1 (2021-10-12)
==============================

**Note:** Media storage providers module that read from Synapse's configuration need changes as of this version, see the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1450) for more information.

Known Issues
------------

- We are investigating [a performance issue](#11049) which was reported after the release of 1.44.0.
- We are aware of [a bug](#11025) with the user directory when using application services. A second release candidate is expected which will resolve this.

Features
--------

- Add [MSC3069](matrix-org/matrix-spec-proposals#3069) support to `/account/whoami`. ([\#9655](#9655))
- Support autodiscovery of oEmbed previews. ([\#10822](#10822))
- Add a `user_may_send_3pid_invite` spam checker callback for modules to allow or deny 3PID invites. ([\#10894](#10894))
- Add a spam checker callback to allow or deny room joins. ([\#10910](#10910))
- Include an `update_synapse_database` script in the distribution. Contributed by @Fizzadar at Beeper. ([\#10954](#10954))
- Include exception information in JSON logging output. Contributed by @Fizzadar at Beeper. ([\#11028](#11028))

Bugfixes
--------

- Fix a minor bug in the response to `/_matrix/client/r0/voip/turnServer`. Contributed by @lukaslihotzki. ([\#10922](#10922))
- Fix a bug where empty `yyyy-mm-dd/` directories would be left behind in the media store's `url_cache_thumbnails/` directory. ([\#10924](#10924))
- Fix a bug introduced in Synapse v1.40.0 where the signature checks for room version 8 and 9 could be applied to earlier room versions in some situations. ([\#10927](#10927))
- Fix a long-standing bug wherein deactivated users still count towards the monthly active users limit. ([\#10947](#10947))
- Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state. ([\#10956](#10956))
- Fix a long-standing bug where rebuilding the user directory wouldn't exclude support and deactivated users. ([\#10960](#10960))
- Fix [MSC2716](matrix-org/matrix-spec-proposals#2716) `/batch_send` endpoint rejecting subsequent batches with unknown batch ID error in existing room versions from the room creator. ([\#10962](#10962))
- Fix a bug that could leak local users' per-room nicknames and avatars when the user directory is rebuilt. ([\#10981](#10981))
- Fix a long-standing bug where the remainder of a batch of user directory changes would be silently dropped if the server left a room early in the batch. ([\#10982](#10982))
- Correct a bugfix introduced in Synapse v1.44.0 that would catch the wrong error if a connection is lost before a response could be written to it. ([\#10995](#10995))
- Fix a long-standing bug where local users' per-room nicknames/avatars were visible to anyone who could see you in the user directory. ([\#11002](#11002))
- Fix a long-standing bug where a user's per-room nickname/avatar would overwrite their profile in the user directory when a room was made public. ([\#11003](#11003))
- Work around a regression, introduced in Synapse v1.39.0, that caused `SynapseError`s raised by the experimental third-party rules module callback `check_event_allowed` to be ignored. ([\#11042](#11042))
- Fix a bug in [MSC2716](matrix-org/matrix-spec-proposals#2716) insertion events in rooms that could cause cross-talk/conflicts between batches. ([\#10877](#10877))

Improved Documentation
----------------------

- Change wording ("reference homeserver") in Synapse repository documentation. Contributed by @maxkratz. ([\#10971](#10971))
- Fix a dead URL in development documentation (SAML) and change wording from "Riot" to "Element". Contributed by @maxkratz. ([\#10973](#10973))
- Add additional content to the Welcome and Overview page of the documentation. ([\#10990](#10990))
- Update links to MSCs in documentation. Contributed by @dklimpel. ([\#10991](#10991))

Internal Changes
----------------

- Improve type hinting in `synapse.util`. ([\#10888](#10888))
- Add further type hints to `synapse.storage.util`. ([\#10892](#10892))
- Fix type hints to be compatible with an upcoming change to Twisted. ([\#10895](#10895))
- Update utility code to handle C implementations of frozendict. ([\#10902](#10902))
- Drop old functionality which maintained database compatibility with Synapse versions before v1.31. ([\#10903](#10903))
- Clean-up configuration helper classes for the `ServerConfig` class. ([\#10915](#10915))
- Use direct references to config flags. ([\#10916](#10916), [\#10959](#10959), [\#10985](#10985))
- Clean up some of the federation event authentication code for clarity. ([\#10926](#10926), [\#10940](#10940), [\#10986](#10986), [\#10987](#10987), [\#10988](#10988), [\#11010](#11010), [\#11011](#11011))
- Refactor various parts of the codebase to use `RoomVersion` objects instead of room version identifier strings. ([\#10934](#10934))
- Refactor user directory tests in preparation for upcoming changes. ([\#10935](#10935))
- Include the event id in the logcontext when handling PDUs received over federation. ([\#10936](#10936))
- Fix logged errors in unit tests. ([\#10939](#10939))
- Fix a broken test to ensure that consent configuration works during registration. ([\#10945](#10945))
- Add type hints to filtering classes. ([\#10958](#10958))
- Add type-hint to `HomeserverTestcase.setup_test_homeserver`. ([\#10961](#10961))
- Fix the test utility function `create_room_as` so that `is_public=True` will explicitly set the `visibility` parameter of room creation requests to `public`. Contributed by @AndrewFerr. ([\#10963](#10963))
- Make the release script more robust and transparent. ([\#10966](#10966))
- Refactor [MSC2716](matrix-org/matrix-spec-proposals#2716) `/batch_send` mega function into smaller handler functions. ([\#10974](#10974))
- Log stack traces when a missing opentracing span is detected. ([\#10983](#10983))
- Update GHA config to run tests against Python 3.10 and PostgreSQL 14. ([\#10992](#10992))
- Fix a long-standing bug where `ReadWriteLock`s could drop logging contexts on exit. ([\#10993](#10993))
- Add a `CODEOWNERS` file to automatically request reviews from the `@matrix-org/synapse-core` team on new pull requests. ([\#10994](#10994))
- Add further type hints to `synapse.state`. ([\#11004](#11004))
- Remove the deprecated `BaseHandler` object. ([\#11005](#11005))
- Bump mypy version for CI to 0.910, and pull in new type stubs for dependencies. ([\#11006](#11006))
- Fix CI to run the unit tests without optional deps. ([\#11017](#11017))
- Ensure that cache config tests do not share state. ([\#11019](#11019))
- Add additional type hints to `synapse.server_notices`. ([\#11021](#11021))
- Add additional type hints for `synapse.push`. ([\#11023](#11023))
- When installing the optional developer dependencies, also include the dependencies needed for type-checking and unit testing. ([\#11034](#11034))
- Remove unnecessary list comprehension from `synapse_port_db` to satisfy code style requirements. ([\#11043](#11043))

.ci/scripts/test_synapse_port_db.sh

@@ -25,7 +25,7 @@ python -m synapse.app.homeserver --generate-keys -c .ci/sqlite-config.yaml
 echo "--- Prepare test database"
 
 # Make sure the SQLite3 database is using the latest schema and has no pending background update.
-scripts-dev/update_database --database-config .ci/sqlite-config.yaml
+scripts/update_synapse_database --database-config .ci/sqlite-config.yaml --run-background-updates
 
 # Create the PostgreSQL database.
 .ci/scripts/postgres_exec.py "CREATE DATABASE synapse"
@@ -46,7 +46,7 @@ echo "--- Prepare empty SQLite database"
 # we do this by deleting the sqlite db, and then doing the same again.
 rm .ci/test_db.db
 
-scripts-dev/update_database --database-config .ci/sqlite-config.yaml
+scripts/update_synapse_database --database-config .ci/sqlite-config.yaml --run-background-updates
 
 # re-create the PostgreSQL database.
 .ci/scripts/postgres_exec.py \

.github/CODEOWNERS

@@ -0,0 +1,2 @@
+# Automatically request reviews from the synapse-core team when a pull request comes in.
+* @matrix-org/synapse-core

.github/workflows/tests.yml

@@ -76,22 +76,25 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.6", "3.7", "3.8", "3.9"]
+        python-version: ["3.6", "3.7", "3.8", "3.9", "3.10"]
         database: ["sqlite"]
+        toxenv: ["py"]
         include:
           # Newest Python without optional deps
-          - python-version: "3.9"
-            toxenv: "py-noextras,combine"
+          - python-version: "3.10"
+            toxenv: "py-noextras"
 
           # Oldest Python with PostgreSQL
           - python-version: "3.6"
             database: "postgres"
             postgres-version: "9.6"
+            toxenv: "py"
 
-          # Newest Python with PostgreSQL
-          - python-version: "3.9"
+          # Newest Python with newest PostgreSQL
+          - python-version: "3.10"
             database: "postgres"
-            postgres-version: "13"
+            postgres-version: "14"
+            toxenv: "py"
 
     steps:
       - uses: actions/checkout@v2
@@ -111,7 +114,7 @@ jobs:
         if: ${{ matrix.postgres-version }}
         timeout-minutes: 2
         run: until pg_isready -h localhost; do sleep 1; done
-      - run: tox -e py,combine
+      - run: tox -e ${{ matrix.toxenv }}
         env:
           TRIAL_FLAGS: "--jobs=2"
           SYNAPSE_POSTGRES: ${{ matrix.database == 'postgres' || '' }}
@@ -169,7 +172,7 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
       - run: pip install tox
-      - run: tox -e py,combine
+      - run: tox -e py
         env:
           TRIAL_FLAGS: "--jobs=2"
       - name: Dump logs
@@ -256,8 +259,8 @@ jobs:
           - python-version: "3.6"
             postgres-version: "9.6"
 
-          - python-version: "3.9"
-            postgres-version: "13"
+          - python-version: "3.10"
+            postgres-version: "14"
 
     services:
       postgres:

.gitignore

@@ -40,6 +40,7 @@ __pycache__/
 /.coverage*
 /.mypy_cache/
 /.tox
+/.tox-pg-container
 /build/
 /coverage.*
 /dist/

README.rst

@@ -55,11 +55,8 @@ solutions. The hope is for Matrix to act as the building blocks for a new
 generation of fully open and interoperable messaging and VoIP apps for the
 internet.
 
-Synapse is a reference "homeserver" implementation of Matrix from the core
-development team at matrix.org, written in Python/Twisted.  It is intended to
-showcase the concept of Matrix and let folks see the spec in the context of a
-codebase and let you run your own homeserver and generally help bootstrap the
-ecosystem.
+Synapse is a Matrix "homeserver" implementation developed by the matrix.org core 
+team, written in Python 3/Twisted.
 
 In Matrix, every user runs one or more Matrix clients, which connect through to
 a Matrix homeserver. The homeserver stores all their personal chat history and
@@ -288,7 +285,7 @@ Quick start
 
 Before setting up a development environment for synapse, make sure you have the
 system dependencies (such as the python header files) installed - see
-`Installing from source <https://matrix-org.github.io/synapse/latest/setup/installation.html#installing-from-source>`_.
+`Platform-specific prerequisites <https://matrix-org.github.io/synapse/latest/setup/installation.html#platform-specific-prerequisites>`_.
 
 To check out a synapse for development, clone the git repo into a working
 directory of your choice::
@@ -301,7 +298,7 @@ to install using pip and a virtualenv::
 
     python3 -m venv ./env
     source ./env/bin/activate
-    pip install -e ".[all,test]"
+    pip install -e ".[all,dev]"
 
 This will run a process of downloading and installing all the needed
 dependencies into a virtual env. If any dependencies fail to install,

debian/changelog

@@ -1,3 +1,49 @@
+matrix-synapse-py3 (1.45.0) stable; urgency=medium
+
+  * New synapse release 1.45.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 19 Oct 2021 11:18:53 +0100
+
+matrix-synapse-py3 (1.45.0~rc2) stable; urgency=medium
+
+  * New synapse release 1.45.0~rc2.
+
+ -- Synapse Packaging team <packages@matrix.org>  Thu, 14 Oct 2021 10:58:24 +0100
+
+matrix-synapse-py3 (1.45.0~rc1) stable; urgency=medium
+
+  [ Nick @ Beeper ]
+  * Include an `update_synapse_database` script in the distribution.
+
+  [ Synapse Packaging team ]
+  * New synapse release 1.45.0~rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 12 Oct 2021 10:46:27 +0100
+
+matrix-synapse-py3 (1.44.0) stable; urgency=medium
+
+  * New synapse release 1.44.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 05 Oct 2021 13:43:57 +0100
+
+matrix-synapse-py3 (1.44.0~rc3) stable; urgency=medium
+
+  * New synapse release 1.44.0~rc3.
+
+ -- Synapse Packaging team <packages@matrix.org>  Mon, 04 Oct 2021 14:57:22 +0100
+
+matrix-synapse-py3 (1.44.0~rc2) stable; urgency=medium
+
+  * New synapse release 1.44.0~rc2.
+
+ -- Synapse Packaging team <packages@matrix.org>  Thu, 30 Sep 2021 12:39:10 +0100
+
+matrix-synapse-py3 (1.44.0~rc1) stable; urgency=medium
+
+  * New synapse release 1.44.0~rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 28 Sep 2021 13:41:28 +0100
+
 matrix-synapse-py3 (1.43.0) stable; urgency=medium
 
   * New synapse release 1.43.0.

debian/matrix-synapse-py3.links

@@ -3,3 +3,4 @@ opt/venvs/matrix-synapse/bin/register_new_matrix_user usr/bin/register_new_matri
 opt/venvs/matrix-synapse/bin/synapse_port_db usr/bin/synapse_port_db
 opt/venvs/matrix-synapse/bin/synapse_review_recent_signups usr/bin/synapse_review_recent_signups
 opt/venvs/matrix-synapse/bin/synctl usr/bin/synctl
+opt/venvs/matrix-synapse/bin/update_synapse_database usr/bin/update_synapse_database

docker/Dockerfile-dhvirtualenv

@@ -47,8 +47,9 @@ RUN apt-get update -qq -o Acquire::Languages=none \
     && cd /dh-virtualenv \
     && env DEBIAN_FRONTEND=noninteractive mk-build-deps -ri -t "apt-get -y --no-install-recommends"
 
-# build it
-RUN cd /dh-virtualenv && dpkg-buildpackage -us -uc -b
+# Build it. Note that building the docs doesn't work due to differences in
+# Sphinx APIs across versions/distros.
+RUN cd /dh-virtualenv && DEB_BUILD_OPTIONS=nodoc dpkg-buildpackage -us -uc -b
 
 ###
 ### Stage 1

docker/Dockerfile-pgtests

@@ -1,12 +1,30 @@
 # Use the Sytest image that comes with a lot of the build dependencies
 # pre-installed
-FROM matrixdotorg/sytest:latest
+FROM matrixdotorg/sytest:bionic
 
 # The Sytest image doesn't come with python, so install that
 RUN apt-get update && apt-get -qq install -y python3 python3-dev python3-pip
 
 # We need tox to run the tests in run_pg_tests.sh
 RUN python3 -m pip install tox
 
-ADD run_pg_tests.sh /pg_tests.sh
-ENTRYPOINT /pg_tests.sh
+# Initialise the db
+RUN su -c '/usr/lib/postgresql/10/bin/initdb -D /var/lib/postgresql/data -E "UTF-8" --lc-collate="C.UTF-8" --lc-ctype="C.UTF-8" --username=postgres' postgres
+
+# Add a user with our UID and GID so that files get created on the host owned
+# by us, not root.
+ARG UID
+ARG GID
+RUN groupadd --gid $GID user
+RUN useradd --uid $UID --gid $GID --groups sudo --no-create-home user
+
+# Ensure we can start postgres by sudo-ing as the postgres user.
+RUN apt-get update && apt-get -qq install -y sudo
+RUN echo "user ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
+
+ADD run_pg_tests.sh /run_pg_tests.sh
+# Use the "exec form" of ENTRYPOINT (https://docs.docker.com/engine/reference/builder/#entrypoint)
+# so that we can `docker run` this container and pass arguments to pg_tests.sh
+ENTRYPOINT ["/run_pg_tests.sh"]
+
+USER user

docker/run_pg_tests.sh

@@ -10,11 +10,10 @@ set -e
 # Set PGUSER so Synapse's tests know what user to connect to the database with
 export PGUSER=postgres
 
-# Initialise & start the database
-su -c '/usr/lib/postgresql/9.6/bin/initdb -D /var/lib/postgresql/data -E "UTF-8" --lc-collate="en_US.UTF-8" --lc-ctype="en_US.UTF-8" --username=postgres' postgres
-su -c '/usr/lib/postgresql/9.6/bin/pg_ctl -w -D /var/lib/postgresql/data start' postgres
+# Start the database
+sudo -u postgres /usr/lib/postgresql/10/bin/pg_ctl -w -D /var/lib/postgresql/data start
 
 # Run the tests
 cd /src
 export TRIAL_FLAGS="-j 4"
-tox --workdir=/tmp -e py35-postgres
+tox --workdir=./.tox-pg-container -e py36-postgres "$@"

docs/MSC1711_certificates_FAQ.md

@@ -3,7 +3,7 @@
 ## Historical Note
 This document was originally written to guide server admins through the upgrade
 path towards Synapse 1.0. Specifically,
-[MSC1711](https://github.com/matrix-org/matrix-doc/blob/master/proposals/1711-x509-for-federation.md)
+[MSC1711](https://github.com/matrix-org/matrix-doc/blob/main/proposals/1711-x509-for-federation.md)
 required that all servers present valid TLS certificates on their federation
 API. Admins were encouraged to achieve compliance from version 0.99.0 (released
 in February 2019) ahead of version 1.0 (released June 2019) enforcing the
@@ -282,7 +282,7 @@ coffin of the Perspectives project (which was already pretty dead). So, the
 Spec Core Team decided that a better approach would be to mandate valid TLS
 certificates for federation alongside the rest of the Web. More details can be
 found in
-[MSC1711](https://github.com/matrix-org/matrix-doc/blob/master/proposals/1711-x509-for-federation.md#background-the-failure-of-the-perspectives-approach).
+[MSC1711](https://github.com/matrix-org/matrix-doc/blob/main/proposals/1711-x509-for-federation.md#background-the-failure-of-the-perspectives-approach).
 
 This results in a breaking change, which is disruptive, but absolutely critical
 for the security model. However, the existence of Let's Encrypt as a trivial

docs/README.md

@@ -6,9 +6,9 @@ Please update any links to point to the new website instead.
 ## About
 
 This directory currently holds a series of markdown files documenting how to install, use
-and develop Synapse, the reference Matrix homeserver. The documentation is readable directly
-from this repository, but it is recommended to instead browse through the
-[website](https://matrix-org.github.io/synapse) for easier discoverability.
+and develop Synapse. The documentation is readable directly from this repository, but it is 
+recommended to instead browse through the [website](https://matrix-org.github.io/synapse) for 
+easier discoverability.
 
 ## Adding to the documentation