[1/2] Allow homeservers to send registration emails | Sending the ema…

…il (#5835)

~~Fixes #5833 Moved out to ~~#5863
Part of fixing #5751

Decouples the activity of sending registration emails and binding them to an identity server.

This PR simply sends the registration email, but clicking it does not approve the user for registration. That will come in PR #2.

Some of this makes use of existing stuff for sending password reset emails from Synapse. Some work was done to make that stuff even more generic.

Upgrade notes:
* There is a new top-level config option, `account_threepid_delegate` which defines the address of an identity server that you would like to send registration/password reset emails on your behalf.

  The option `email.trust_identity_server_for_password_resets` has been replaced by this. If you set `email.trust_identity_server_for_password_resets` in your config to `true`, please remove it and configure `account_threepid_delegate` instead. The [sample config](https://github.com/matrix-org/synapse/blob/anoa/reg_email_sending_email/docs/sample_config.yaml) has information on how to configure it.

Note: This PR does not allow homeservers to send emails when simply adding an email to your account. That will come after this and will be blocked on a new MSC.

Part [2/2] will be successfully completing the registration step when `/submit_token` is hit with the correct details, and clearing out the `password_servlet` flag stuff, which is no longer needed. Will be a much smaller PR than this one.

~~Requires #5863 has been merged into the base branch.
~~Requires #5876 has been merged into the base branch.

UPGRADE.rst

@@ -49,6 +49,55 @@ returned by the Client-Server API:
     # configured on port 443.
     curl -kv https://<host.name>/_matrix/client/versions 2>&1 | grep "Server:"
 
+Upgrading to v1.4.0
+===================
+
+Config options
+--------------
+
+**Note: Registration by email address or phone number will not work in this release unless
+some config options are changed from their defaults.**
+
+This is due to Synapse v1.4.0 now defaulting to sending registration and password reset tokens
+itself. This is for security reasons as well as putting less reliance on identity servers.
+However, currently Synapse only supports sending emails, and does not have support for
+phone-based password reset or account registration. If Synapse is configured to handle these on
+its own, phone-based password resets and registration will be disabled. For Synapse to send
+emails, the ``email`` block of the config must be filled out. If not, then password resets and
+registration via email will be disabled entirely.
+
+This release also deprecates the ``email.trust_identity_server_for_password_resets`` option
+and replaces it with ``account_threepid_delegate``. This option defines whether the homeserver
+should delegate an external server (typically an `identity server
+<https://matrix.org/docs/spec/identity_service/r0.2.1>`_) to handle sending password reset
+or registration messages via email or SMS.
+
+If ``email.trust_identity_server_for_password_resets`` was changed from its default to
+``true``, and ``account_threepid_delegate`` is not set to an identity server domain, then the
+server handling password resets and registration via third-party addresses will be set to the
+first entry in the Synapse config's ``trusted_third_party_id_servers`` entry. If no domains are
+configured, Synapse will throw an error on startup.
+
+If ``email.trust_identity_server_for_password_resets`` is not set to ``true`` and
+``account_threepid_delegate`` is not set to a domain, then Synapse will attempt to send
+password reset and registration messages itself.
+
+Email templates
+---------------
+
+If you have configured a custom template directory with the ``email.template_dir`` option, be
+aware that there are new templates regarding registration. ``registration.html`` and
+``registration.txt`` have been added and contain the text that is sent to a client upon
+registering via email address.
+
+``registration_success.html`` and ``registration_failure.html`` are templates containing HTML
+that will be shown to the user when they click the link in their registration email (if a
+redirect URL is not configured), either showing them a success or failure page.
+
+Synapse will expect these files to exist inside the configured template directory. To view the
+default templates, see `synapse/res/templates
+<https://github.com/matrix-org/synapse/tree/master/synapse/res/templates>`_.
+
 Upgrading to v1.2.0
 ===================
 
@@ -132,6 +181,19 @@ server for password resets, set ``trust_identity_server_for_password_resets`` to
 See the `sample configuration file <docs/sample_config.yaml>`_
 for more details on these settings.
 
+New email templates
+---------------
+Some new templates have been added to the default template directory for the purpose of the
+homeserver sending its own password reset emails. If you have configured a custom
+``template_dir`` in your Synapse config, these files will need to be added.
+
+``password_reset.html`` and ``password_reset.txt`` are HTML and plain text templates
+respectively that contain the contents of what will be emailed to the user upon attempting to
+reset their password via email. ``password_reset_success.html`` and
+``password_reset_failure.html`` are HTML files that the content of which (assuming no redirect
+URL is set) will be shown to the user after they attempt to click the link in the email sent
+to them.
+
 Upgrading to v0.99.0
 ====================
 

changelog.d/5835.feature

@@ -0,0 +1 @@
+Add the ability to send registration emails from the homeserver rather than delegating to an identity server.

docs/sample_config.yaml

@@ -1217,11 +1217,22 @@ password_config:
 #   #password_reset_template_html: password_reset.html
 #   #password_reset_template_text: password_reset.txt
 #
+#   # Templates for registration emails sent by the homeserver
+#   #
+#   #registration_template_html: registration.html
+#   #registration_template_text: registration.txt
+#
 #   # Templates for password reset success and failure pages that a user
 #   # will see after attempting to reset their password
 #   #
 #   #password_reset_template_success_html: password_reset_success.html
 #   #password_reset_template_failure_html: password_reset_failure.html
+#
+#   # Templates for registration success and failure pages that a user
+#   # will see after attempting to register using an email or phone
+#   #
+#   #registration_template_success_html: registration_success.html
+#   #registration_template_failure_html: registration_failure.html
 
 
 #password_providers:

synapse/config/emailconfig.py

@@ -75,7 +75,7 @@ def read_config(self, config, **kwargs):
             "renew_at"
         )
 
-        self.email_threepid_behaviour = (
+        self.threepid_behaviour = (
             # Have Synapse handle the email sending if account_threepid_delegate
             # is not defined
             ThreepidBehaviour.REMOTE
@@ -87,9 +87,14 @@ def read_config(self, config, **kwargs):
         # if they have this set and tell them to use the updated option, while using a default
         # identity server in the process.
         self.using_identity_server_from_trusted_list = False
-        if config.get("trust_identity_server_for_password_resets", False) is True:
+        if (
+            not self.account_threepid_delegate
+            and config.get("trust_identity_server_for_password_resets", False) is True
+        ):
             # Use the first entry in self.trusted_third_party_id_servers instead
             if self.trusted_third_party_id_servers:
+                # XXX: It's a little confusing that account_threepid_delegate is modifed
+                # both in RegistrationConfig and here. We should factor this bit out
                 self.account_threepid_delegate = self.trusted_third_party_id_servers[0]
                 self.using_identity_server_from_trusted_list = True
             else:
@@ -98,16 +103,13 @@ def read_config(self, config, **kwargs):
                     '"trusted_third_party_id_servers" but it is empty.'
                 )
 
-        self.local_threepid_emails_disabled_due_to_config = False
-        if (
-            self.email_threepid_behaviour == ThreepidBehaviour.LOCAL
-            and email_config == {}
-        ):
+        self.local_threepid_handling_disabled_due_to_email_config = False
+        if self.threepid_behaviour == ThreepidBehaviour.LOCAL and email_config == {}:
             # We cannot warn the user this has happened here
             # Instead do so when a user attempts to reset their password
-            self.local_threepid_emails_disabled_due_to_config = True
+            self.local_threepid_handling_disabled_due_to_email_config = True
 
-            self.email_threepid_behaviour = ThreepidBehaviour.OFF
+            self.threepid_behaviour = ThreepidBehaviour.OFF
 
         # Get lifetime of a validation token in milliseconds
         self.email_validation_token_lifetime = self.parse_duration(
@@ -117,7 +119,7 @@ def read_config(self, config, **kwargs):
         if (
             self.email_enable_notifs
             or account_validity_renewal_enabled
-            or self.email_threepid_behaviour == ThreepidBehaviour.LOCAL
+            or self.threepid_behaviour == ThreepidBehaviour.LOCAL
         ):
             # make sure we can import the required deps
             import jinja2
@@ -127,7 +129,7 @@ def read_config(self, config, **kwargs):
             jinja2
             bleach
 
-        if self.email_threepid_behaviour == ThreepidBehaviour.LOCAL:
+        if self.threepid_behaviour == ThreepidBehaviour.LOCAL:
             required = ["smtp_host", "smtp_port", "notif_from"]
 
             missing = []
@@ -146,28 +148,45 @@ def read_config(self, config, **kwargs):
                     % (", ".join(missing),)
                 )
 
-            # Templates for password reset emails
+            # These email templates have placeholders in them, and thus must be
+            # parsed using a templating engine during a request
             self.email_password_reset_template_html = email_config.get(
                 "password_reset_template_html", "password_reset.html"
             )
             self.email_password_reset_template_text = email_config.get(
                 "password_reset_template_text", "password_reset.txt"
             )
+            self.email_registration_template_html = email_config.get(
+                "registration_template_html", "registration.html"
+            )
+            self.email_registration_template_text = email_config.get(
+                "registration_template_text", "registration.txt"
+            )
             self.email_password_reset_template_failure_html = email_config.get(
                 "password_reset_template_failure_html", "password_reset_failure.html"
             )
-            # This template does not support any replaceable variables, so we will
-            # read it from the disk once during setup
+            self.email_registration_template_failure_html = email_config.get(
+                "registration_template_failure_html", "registration_failure.html"
+            )
+
+            # These templates do not support any placeholder variables, so we
+            # will read them from disk once during setup
             email_password_reset_template_success_html = email_config.get(
                 "password_reset_template_success_html", "password_reset_success.html"
             )
+            email_registration_template_success_html = email_config.get(
+                "registration_template_success_html", "registration_success.html"
+            )
 
             # Check templates exist
             for f in [
                 self.email_password_reset_template_html,
                 self.email_password_reset_template_text,
+                self.email_registration_template_html,
+                self.email_registration_template_text,
                 self.email_password_reset_template_failure_html,
                 email_password_reset_template_success_html,
+                email_registration_template_success_html,
             ]:
                 p = os.path.join(self.email_template_dir, f)
                 if not os.path.isfile(p):
@@ -177,9 +196,15 @@ def read_config(self, config, **kwargs):
             filepath = os.path.join(
                 self.email_template_dir, email_password_reset_template_success_html
             )
-            self.email_password_reset_template_success_html_content = self.read_file(
+            self.email_password_reset_template_success_html = self.read_file(
                 filepath, "email.password_reset_template_success_html"
             )
+            filepath = os.path.join(
+                self.email_template_dir, email_registration_template_success_html
+            )
+            self.email_registration_template_success_html_content = self.read_file(
+                filepath, "email.registration_template_success_html"
+            )
 
         if self.email_enable_notifs:
             required = [
@@ -291,11 +316,22 @@ def generate_config_section(self, config_dir_path, server_name, **kwargs):
         #   #password_reset_template_html: password_reset.html
         #   #password_reset_template_text: password_reset.txt
         #
+        #   # Templates for registration emails sent by the homeserver
+        #   #
+        #   #registration_template_html: registration.html
+        #   #registration_template_text: registration.txt
+        #
         #   # Templates for password reset success and failure pages that a user
         #   # will see after attempting to reset their password
         #   #
         #   #password_reset_template_success_html: password_reset_success.html
         #   #password_reset_template_failure_html: password_reset_failure.html
+        #
+        #   # Templates for registration success and failure pages that a user
+        #   # will see after attempting to register using an email or phone
+        #   #
+        #   #registration_template_success_html: registration_success.html
+        #   #registration_template_failure_html: registration_failure.html
         """
 
 

synapse/handlers/auth.py

@@ -461,10 +461,10 @@ def _check_threepid(self, medium, authdict, password_servlet=False, **kwargs):
         logger.info("Getting validated threepid. threepidcreds: %r", (threepid_creds,))
         if (
             not password_servlet
-            or self.hs.config.email_threepid_behaviour == ThreepidBehaviour.REMOTE
+            or self.hs.config.threepid_behaviour == ThreepidBehaviour.REMOTE
         ):
             threepid = yield identity_handler.threepid_from_creds(threepid_creds)
-        elif self.hs.config.email_threepid_behaviour == ThreepidBehaviour.LOCAL:
+        elif self.hs.config.threepid_behaviour == ThreepidBehaviour.LOCAL:
             row = yield self.store.get_threepid_validation_session(
                 medium,
                 threepid_creds["client_secret"],

synapse/handlers/identity.py

@@ -24,6 +24,7 @@
 from twisted.internet import defer
 
 from synapse.api.errors import CodeMessageException, HttpResponseException, SynapseError
+from synapse.util.stringutils import random_string
 
 from ._base import BaseHandler
 
@@ -196,6 +197,84 @@ def try_unbind_threepid_with_id_server(self, mxid, threepid, id_server):
 
         return changed
 
+    @defer.inlineCallbacks
+    def send_threepid_validation(
+        self,
+        email_address,
+        client_secret,
+        send_attempt,
+        send_email_func,
+        next_link=None,
+    ):
+        """Send a threepid validation email for password reset or
+        registration purposes
+
+        Args:
+            email_address (str): The user's email address
+            client_secret (str): The provided client secret
+            send_attempt (int): Which send attempt this is
+            send_email_func (func): A function that takes an email address, token,
+                                    client_secret and session_id, sends an email
+                                    and returns a Deferred.
+            next_link (str|None): The URL to redirect the user to after validation
+
+        Returns:
+            The new session_id upon success
+
+        Raises:
+            SynapseError is an error occurred when sending the email
+        """
+        # Check that this email/client_secret/send_attempt combo is new or
+        # greater than what we've seen previously
+        session = yield self.store.get_threepid_validation_session(
+            "email", client_secret, address=email_address, validated=False
+        )
+
+        # Check to see if a session already exists and that it is not yet
+        # marked as validated
+        if session and session.get("validated_at") is None:
+            session_id = session["session_id"]
+            last_send_attempt = session["last_send_attempt"]
+
+            # Check that the send_attempt is higher than previous attempts
+            if send_attempt <= last_send_attempt:
+                # If not, just return a success without sending an email
+                return session_id
+        else:
+            # An non-validated session does not exist yet.
+            # Generate a session id
+            session_id = random_string(16)
+
+        # Generate a new validation token
+        token = random_string(32)
+
+        # Send the mail with the link containing the token, client_secret
+        # and session_id
+        try:
+            yield send_email_func(email_address, token, client_secret, session_id)
+        except Exception:
+            logger.exception(
+                "Error sending threepid validation email to %s", email_address
+            )
+            raise SynapseError(500, "An error was encountered when sending the email")
+
+        token_expires = (
+            self.hs.clock.time_msec() + self.hs.config.email_validation_token_lifetime
+        )
+
+        yield self.store.start_or_continue_validation_session(
+            "email",
+            email_address,
+            session_id,
+            client_secret,
+            send_attempt,
+            next_link,
+            token,
+            token_expires,
+        )
+
+        return session_id
+
     @defer.inlineCallbacks
     def requestEmailToken(
         self, id_server, email, client_secret, send_attempt, next_link=None

synapse/handlers/register.py

@@ -414,18 +414,6 @@ def register_email(self, threepidCreds):
             if not check_3pid_allowed(self.hs, threepid["medium"], threepid["address"]):
                 raise RegistrationError(403, "Third party identifier is not allowed")
 
-    @defer.inlineCallbacks
-    def bind_emails(self, user_id, threepidCreds):
-        """Links emails with a user ID and informs an identity server.
-
-        Used only by c/s api v1
-        """
-
-        # Now we have a matrix ID, bind it to the threepids we were given
-        for c in threepidCreds:
-            # XXX: This should be a deferred list, shouldn't it?
-            yield self.identity_handler.bind_threepid(c, user_id)
-
     def check_user_id_not_appservice_exclusive(self, user_id, allowed_appservice=None):
         # don't allow people to register the server notices mxid
         if self._server_notices_mxid is not None: