UTF-8 encoded mails are not sent #30

viktordick · 2020-07-14T07:01:23Z

According to the documentation in smptlib.sendmail,

    msg may be a string containing characters in the ASCII range, or a byte
    string.  A string is encoded to bytes using the ascii codec, and lone
    \\r and \\n characters are converted to \\r\\n characters.

Since switching to Zope 4, we are unable to send mails that have a UTF-8 encoded body - Even if we transform it to bytes ourselves, Products.MailHost converts it back to strings before sending it to zope.sendmail and smptlib fails to encode the passed message using ASCII during the second phase of the two phase commit. Our quick fix was to change zope.sendmail to encode the message using utf-8 before passing it to smtplib.sendmail.

A really clean way would probably be to inspect the headers (which should be ASCII) and check for a Content-Type and Content-Transfer-Encoding. Maybe it is even possible to have a multipart message with different encodings (not sure). But since UTF-8 is such a popular choice, we used that for now.

I am not sure if the fix should happen in zope.sendmail or somewhere further up like Products.MailHost. If it should be zope.sendmail, I could try to supply a PR. Otherwise I would need to be pointed to the right direction.

The text was updated successfully, but these errors were encountered:

d-maurer · 2020-07-14T08:20:33Z

Viktor Dick wrote at 2020-7-14 00:01 -0700:

According to the documentation in `smptlib.sendmail`, msg may be a string containing characters in the ASCII range, or a byte string. A string is encoded to bytes using the ascii codec, and lone \\r and \\n characters are converted to \\r\\n characters.

I read this as: the message must be ASCII only. This is not surprising as it is a major requirement for low level mail messages. If the orignal (high level) message contains non ASCII, it must be encoded to ASCII (typically via "quoted-printable" or "base64"). Python's `email` package contains logic to do this.

viktordick · 2020-07-14T08:49:07Z

Thanks for the reply.

I actually read this as: Either ensure that you handled the encoding yourself (passing bytes) or only use ASCII. stmplib will not try to guess the encoding.

I am pretty sure that current mail clients assume that every MTA on the way supports https://tools.ietf.org/html/rfc6152. At least if I send a message with Thunderbird with default settings, it uses an 8bit-encoding (In fact, utf-8).

More to the point, passing UTF-8 encoded messages to smtplib.send yields the expected result (assuming the correct headers are set and only the body uses UTF-8, not the headers).

d-maurer · 2020-07-14T09:26:31Z

Viktor Dick wrote at 2020-7-14 01:49 -0700:

... More to the point, passing UTF-8 encoded messages to `smtplib.send` yields the expected result (assuming the correct headers are set and only the body uses UTF-8, not the headers).

Rereading your original post I have the impression that the problem cause is `Products.MailHost`: because messages are at lower level byte sequences, `MailHost` should not convert a byte sequence into `str` - or at least not guessing an encoding. I would not put encoding guessing into `zope.sendmail` or sniff encoding information from the message headers. Encoding handling is implemented elsewhere (e.g. the `email` package); there is no need to implement is another time.

viktordick · 2020-07-14T09:39:36Z

I agree. I posted the issue here because I could not exactly pinpoint the location in Products.MailHost where the issue would have to be fixed, while it was a two-liner if I place the fix in zope.sendmail. But the correct place is Products.MailHost.

Is it possible to move the issue there? Otherwise I will close this and re-open another issue there (and take another look at what needs to be fixed there).

viktordick · 2020-07-15T17:15:49Z

@jugmac00 can you give me a hint how exactly you are using Products.MailHost in Zope4, in particular with non-ASCII characters? My minimal working example for creating a failure is something like a PythonScript containing

mail = '''\
To: test@test.de
From: <no-reply@test.de>
Subject: Test
Mime-Version: 1.0
Content-Type: text/html; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

<!DOCTYPE HTML>
<html>
<head>

<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
Testü
</body>
</html>'''

context.mailhost.send(messageText=mail)

What exactly am I missing?

d-maurer · 2020-07-15T18:26:36Z

Viktor Dick wrote at 2020-7-15 17:16 +0000:

@jugmac00 can you give me a hint how exactly you are using `Products.MailHost` in Zope4, in particular with non-ASCII characters? My minimal working example for creating a failure is something like a PythonScript containing mail = '''\ To: ***@***.*** From: ***@***.***> Subject: Test Mime-Version: 1.0 Content-Type: text/html; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit <!DOCTYPE HTML> <html> <head> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> </head> <body> Testü </body> </html>''' context.mailhost.send(messageText=mail) What exactly am I missing?

I like to look at things in an abstract way: Logically, a message is a sequence of bytes and especially its body is a sequence of bytes. The `Content-Type: text; charset=UTF-8` tells us that the body uses the `UTF-8` encoding. Your message is not a sequence of bytes - but a string representing a message. Especially, the body is not a sequence of bytes. The conversion into a corresponding byte sequence is not trivial. With `Content-Type: text; charset=UTF-8`, one could argue that the body should be encoded with "UTF-8"; but, what about a binary "content-type": how should a string body in this case be converted into a sequence of bytes? I think Python's `email` package has a bug here. ```

>> from email import message_from_string, message_from_bytes >> mail = '''\

... To: test@test.de ... From: <no-reply@test.de> ... Subject: Test ... Mime-Version: 1.0 ... Content-Type: text/html; charset=UTF-8; format=flowed ... Content-Transfer-Encoding: 8bit ... ... <!DOCTYPE HTML> ... <html> ... <head> ... ... <meta http-equiv="content-type" content="text/html; charset=UTF-8"> ... </head> ... <body> ... Testü ... </body> ... </html>'''

>> mb = message_from_bytes(mail.encode("utf-8")) >> ms = message_from_string(mail)

b'To: test@test.de\nFrom: <no-reply@test.de>\nSubject: Test\nMime-Version: 1.0\nContent-Type: text/html; charset=UTF-8; format=flowed\nContent-Transfer-Encoding: 8bit\n\n<!DOCTYPE HTML>\n<html>\n<head>\n\n<meta http-equiv="content-type" content="text/html; charset=UTF-8">\n</head>\n<body>\nTest\xc3\xbc\n</body>\n</html>' ``` Here `mb` and `ms` should represent the same message. But: ```

>> mb.as_bytes()

b'To: test@test.de\nFrom: <no-reply@test.de>\nSubject: Test\nMime-Version: 1.0\nContent-Type: text/html; charset=UTF-8; format=flowed\nContent-Transfer-Encoding: 8bit\n\n<!DOCTYPE HTML>\n<html>\n<head>\n\n<meta http-equiv="content-type" content="text/html; charset=UTF-8">\n</head>\n<body>\nTest\xc3\xbc\n</body>\n</html>'

>> ms.as_bytes()

Traceback (most recent call last): ... File "/usr/local/lib/python3.9/email/generator.py", line 155, in _write_lines self.write(line) File "/usr/local/lib/python3.9/email/generator.py", line 406, in write self._fp.write(s.encode('ascii', 'surrogateescape')) UnicodeEncodeError: 'ascii' codec can't encode character '\xfc' in position 4: ordinal not in range(128) ```

d-maurer · 2020-07-15T19:25:50Z

Dieter Maurer wrote at 2020-7-15 20:26 +0200:

Viktor Dick wrote at 2020-7-15 17:16 +0000: ... I think Python's `email` package has a bug here.

I filed bug report "https://bugs.python.org/issue41307". Unfortunately, this bug prevents the use of `as_bytes` (as I proposed in an earlier comment) for the messages you are interested in (unless we use a `Message` class where this bug is fixed).

d-maurer · 2020-07-15T21:12:40Z

Dieter Maurer wrote at 2020-7-15 21:25 +0200:

Dieter Maurer wrote at 2020-7-15 20:26 +0200: >Viktor Dick wrote at 2020-7-15 17:16 +0000: > ... >I think Python's `email` package has a bug here. I filed bug report "https://bugs.python.org/issue41307". Unfortunately, this bug prevents the use of `as_bytes` (as I proposed in an earlier comment) for the messages you are interested in (unless we use a `Message` class where this bug is fixed).

"https://bugs.python.org/issue41307#msg373724" contains a workaround for your kind of messages. It applies "charset" and targets "8bit" transfer encodings. For other transfer encodings, this likely needs to be applied in addition to "charset". As a side note: the Python 3 `email` package uses "surrogateescape" to convert between a sequence of bytes and "str". To be specific: byte sequence "bs" is represented as "str" by `bs.decode("ascii", "surrogateescape")`; `.decode("ascii", "surrogateescape")` gives back the original `bs`.

d-maurer · 2020-07-16T14:44:08Z

@viktordick Could you check whether #32 solves your issue?

jugmac00 · 2020-07-16T14:49:54Z

@viktordick When I inherited my Zope app 2015 I just continued to work with the idioms my predecessor left in the app.

So, it is something similar like this...

        outer = MIMEMultipart()
        outer['Subject'] = SUBJECT
        outer['To'] = TO
        outer['From'] = FROM
        outer.preamble = 'This is a MIME encoded multipart message.\n'
        outer.epilogue = ''
        msg = MIMEText(text, _subtype='html', _charset='utf-8')
        outer.attach(msg)
        ...
        self.MailHost.send(outer.as_string(), mto=TO, mfrom=FROM)

…esponding messages could not be sent anyway due to #30

viktordick · 2020-07-18T08:38:51Z

Thanks very much for your efforts so far!
Sorry for not answering sooner, I have not found time yet to understand and test the suggestions. I hope I find the time today or tomorrow.

viktordick · 2020-07-18T09:11:08Z

    msg = MIMEText(text, _subtype='html', _charset='utf-8')
    outer.attach(msg)
    ...
    self.MailHost.send(outer.as_string(), mto=TO, mfrom=FROM)

@jugmac00 I tested this and it results in a base64 encoded message. Which of course also gets the job done, it just makes inspection more difficult and increases the message size. I guess my problem with this solution is purely aesthetic (we have 2020 and I do not think that there is a MTA out there that is unable to use a Content-Transfer-Encoding of 8bit).

d-maurer · 2020-07-18T09:26:37Z

Viktor Dick wrote at 2020-7-18 02:11 -0700:

...

we have 2020 and I do not think that there is a MTA out there that is unable to use a Content-Transfer-Encoding of 8bit. Python's `email` package is not yet fully "Content-Transfer-Encoding: 8bit" ready.

This is based on the discussion in `https://github.com/zopefoundation/Products.MailHost/issues/30` and `https://github.com/zopefoundation/Products.MailHost/pull/32`. In order to allow messages with 8bit-encoding to be sent using smtplib, they have to be prepared as bytes. Since any preparation and header parsing and manipulation is done in `Products.MailHost`, `zope.sendmail` only needs to store messages internally as bytes and allow them to be passed as bytes.

jugmac00 · 2020-07-20T19:42:40Z

@d-maurer and @viktordick - could you add a test somewhere, which makes sure the way I create emails ( see #30 (comment) ) will still work with your pull requests?

Don't get me wrong - I trust you both you know what you do... it is just... if something gets fixed which worked for 15 years I am always a bit .. anxious :-)

d-maurer · 2020-07-20T21:00:05Z

Jürgen Gmach wrote at 2020-7-20 12:42 -0700:

@d-maurer and @viktordick - could you add a test somewhere, which makes sure the way I create emails ( see #30 (comment) ) will still work with your pull requests? Don't get me wrong - I trust you both you know what you do... it is just... if something gets fixed which worked for 15 years I am always a bit .. anxious :-)

As I wrote to Viktor, I do not mind if others contribute to "my" PR. I invite you to add the test. Note that is difficult to have a black box test; it would involve really sending the message - which all available tests avoid. That's why Victor's problem was not detected by a test. As a consequence, your test may pass and nevertheless in real life something goes wrong. I therefore propose an alternative: include the versions from the relevant branches into your environment and make a functional test there (involving real sending). Report back if you should observe any problems.

jugmac00 · 2020-07-21T08:39:00Z

As I wrote to Viktor, I do not mind if others contribute to "my" PR. I invite you to add the test.

@d-maurer

Is this test ok? Then I push it to your pr / branch (except I will put the import statements at the top).

    def testSendMultipartMessageObject(self):
        from email.mime.multipart import MIMEMultipart
        from email.mime.text import MIMEText
        TO = 'Name, Nick <recipient@example.com>'
        FROM = 'sender@example.com'

        outer = MIMEMultipart()
        outer['Subject'] = 'This is the subject'
        outer['To'] = TO
        outer['From'] = FROM
        outer.preamble = 'This is a MIME encoded multipart message.\n'
        outer.epilogue = ''
        body = 'This is the message body.'
        msg = MIMEText(body, _subtype='text', _charset='utf-8')
        outer.attach(msg)
        outer.set_boundary('===============8011890429167670482==')

        mailhost = self._makeOne('MailHost')
        mailhost.send(
            outer.as_string(),
            mto=TO,
            mfrom=FROM,
        )
        # date_pattern matches e.g. `Tue, 21 Jul 2020 08:50:04 +0200`
        date_pattern = rb'[a-zA-Z]{2,3}, [0-9]{1,2} [a-zA-Z]{3} [0-9]{4} [0-9]{2}:[0-9]{2}:[0-9]{2} [+-]{1}[0-9]{4}'  # noqa: E501
        pattern = rb"""Content-Type: multipart/mixed; """\
                  rb"""boundary="===============8011890429167670482=="\n"""\
                  rb"""MIME-Version: 1.0\n"""\
                  rb"""Subject: This is the subject\n"""\
                  rb"""To: Name, Nick <recipient@example.com>\n"""\
                  rb"""From: sender@example.com\n"""\
                  rb"""Date: %s\n\n"""\
                  rb"""This is a MIME encoded multipart message.\n\n"""\
                  rb"""--===============8011890429167670482==\n"""\
                  rb"""Content-Type: text/text; charset="utf-8"\n"""\
                  rb"""MIME-Version: 1.0\n"""\
                  rb"""Content-Transfer-Encoding: quoted-printable\n\n"""\
                  rb"""This is the message body.\n"""\
                  rb"""--===============8011890429167670482==--\n""" % date_pattern  # noqa: E501
        self.assertRegex(mailhost.sent, pattern)

d-maurer · 2020-07-21T10:02:45Z

Jürgen Gmach wrote at 2020-7-21 01:39 -0700:

... @d-maurer Is this test ok? Then I push it to your pr / branch (except I will put the import statements at the top).

I likely would try to avoid the "noqa: E501" - but that is not a serious objection (I am often very unhappy with the `flake8` enforced policies -- especially those regarding continuation lines which are either too much indented or where the indentation is the same of that for a nested level). Go ahead with your push.

jugmac00 · 2020-07-21T10:39:10Z

@d-maurer Thanks your reply.

I pushed the code to your branch, after I updated my code slightly to also pass the Python 2 tests.

The two noqa: E501 were necessary as those lines were slightly longer than allowed.

While possible, breaking those lines into multiple lines seemed very unnatural.

This is based on the discussion in `https://github.com/zopefoundation/Products.MailHost/issues/30` and `https://github.com/zopefoundation/Products.MailHost/pull/32`. In order to allow messages with 8bit-encoding to be sent using smtplib, they have to be prepared as bytes. Since any preparation and header parsing and manipulation is done in `Products.MailHost`, `zope.sendmail` only needs to store messages internally as bytes and allow them to be passed as bytes. Co-authored-by: dieter <dieter@handshake.de>

I saw test failures earlier when I tried it in combination with other new packages. Now let's try it on its own. I expect failures due to the changes here: zopefoundation/Products.MailHost#30

mauritsvanrees · 2020-09-16T08:03:32Z

Possible regression on Python 3, at least in Plone tests: #33

I saw test failures earlier when I tried it in combination with other new packages. Now let's try it on its own. I expect failures due to the changes here: zopefoundation/Products.MailHost#30

dataflake transferred this issue from zopefoundation/zope.sendmail Jul 14, 2020

viktordick mentioned this issue Jul 14, 2020

WIP: Produce messages as bytes if 8bit encoding is set #31

Closed

d-maurer added a commit that referenced this issue Jul 16, 2020

try to fix #30

c74bb0b

d-maurer mentioned this issue Jul 16, 2020

Workaround python issue41307#30 #32

Merged

d-maurer added a commit that referenced this issue Jul 17, 2020

skip content-transfer-encoding: 8bit tests for Python 3.5; the corr…

895596b

…esponding messages could not be sent anyway due to #30

dataflake closed this as completed in #32 Jul 31, 2020

This was referenced Sep 15, 2020

Update Products.MailHost to 4.10. plone/buildout.coredev#679

Closed

Plone 5.2 on Python 3 tests fail with MailHost 4.10 #33

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UTF-8 encoded mails are not sent #30

UTF-8 encoded mails are not sent #30

viktordick commented Jul 14, 2020

d-maurer commented Jul 14, 2020 via email

viktordick commented Jul 14, 2020

d-maurer commented Jul 14, 2020 via email

viktordick commented Jul 14, 2020

viktordick commented Jul 15, 2020

d-maurer commented Jul 15, 2020 via email

d-maurer commented Jul 15, 2020 via email

d-maurer commented Jul 15, 2020 via email

d-maurer commented Jul 16, 2020

jugmac00 commented Jul 16, 2020

viktordick commented Jul 18, 2020

viktordick commented Jul 18, 2020

d-maurer commented Jul 18, 2020 via email

jugmac00 commented Jul 20, 2020

d-maurer commented Jul 20, 2020 via email

jugmac00 commented Jul 21, 2020

d-maurer commented Jul 21, 2020 via email

jugmac00 commented Jul 21, 2020

mauritsvanrees commented Sep 16, 2020

UTF-8 encoded mails are not sent #30

UTF-8 encoded mails are not sent #30

Comments

viktordick commented Jul 14, 2020

d-maurer commented Jul 14, 2020 via email

viktordick commented Jul 14, 2020

d-maurer commented Jul 14, 2020 via email

viktordick commented Jul 14, 2020

viktordick commented Jul 15, 2020

d-maurer commented Jul 15, 2020 via email

d-maurer commented Jul 15, 2020 via email

d-maurer commented Jul 15, 2020 via email

d-maurer commented Jul 16, 2020

jugmac00 commented Jul 16, 2020

viktordick commented Jul 18, 2020

viktordick commented Jul 18, 2020

d-maurer commented Jul 18, 2020 via email

jugmac00 commented Jul 20, 2020

d-maurer commented Jul 20, 2020 via email

jugmac00 commented Jul 21, 2020

d-maurer commented Jul 21, 2020 via email

jugmac00 commented Jul 21, 2020

mauritsvanrees commented Sep 16, 2020