Plone 5.2 on Python 3 tests fail with MailHost 4.10

[mauritsvanrees]

Plone 5.2.2 uses Products.MailHost 4.9. When I change the core development buildout to use 4.10, as in this PR, there are 26 failures.

• The tests on Python 2.7 are fine
• The tests on Python 3.6 fail
• The tests on Python 3.7 fail

Errors are like this:

initial_value must be str or None, not bytes

  File "/srv/python3.7/lib/python3.7/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/srv/python3.7/lib/python3.7/unittest/case.py", line 628, in run
    testMethod()
  File "/home/jenkins/workspace/pull-request-5.2-3.7/src/Products.CMFPlone/Products/CMFPlone/tests/testRegistrationTool.py", line 212, in testMailPassword
    msg = message_from_string(mails.messages[0])
  File "/srv/python3.7/lib/python3.7/email/__init__.py", line 38, in message_from_string
    return Parser(*args, **kws).parsestr(s)
  File "/srv/python3.7/lib/python3.7/email/parser.py", line 67, in parsestr
    return self.parse(StringIO(text), headersonly=headersonly)

I am not sure yet if this is only in tests, or for real. We are certainly mocking the MailHost object in tests, so maybe it needs to be changed.

But does this seem like an error in how Products.MailHost handles this? Certainly this is caused by the changes in #30.

[jugmac00]

My app's test suite also failed after updating to MailHost 4.10. I updated the tests, deployed it to staging, and experienced no errors while sending/receiving emails. This may be a (weak) indication the changes were probably ok, but the tests need an update.

[dataflake]

I concur with @jugmac00 . I would take a look at your tests first.

[d-maurer]

Maurits van Rees wrote at 2020-9-16 01:03 -0700:

Plone 5.2.2 uses Products.MailHost 4.9. When I change the core development buildout to use 4.10, as in [this PR](plone/buildout.coredev#679), there are 26 failures. - The tests on Python 2.7 are [fine](https://jenkins.plone.org/job/pull-request-5.2/1592/) - The tests on Python 3.6 [fail](https://jenkins.plone.org/job/pull-request-5.2-3.6/1556/) - The tests on Python 3.7 [fail](https://jenkins.plone.org/job/pull-request-5.2-3.7/1521/#showFailuresLink) Errors are like [this](https://jenkins.plone.org/job/pull-request-5.2-3.7/1521/testReport/junit/Products.CMFPlone.tests.testRegistrationTool/TestRegistrationTool/testMailPassword/):

I will look into this -- likely tomorrow.

[d-maurer]

Dieter Maurer wrote at 2020-9-16 19:18 +0200:

Maurits van Rees wrote at 2020-9-16 01:03 -0700: >Plone 5.2.2 uses Products.MailHost 4.9. When I change the core development buildout to use 4.10, as in [this PR](plone/buildout.coredev#679), there are 26 failures. > >- The tests on Python 2.7 are [fine](https://jenkins.plone.org/job/pull-request-5.2/1592/) >- The tests on Python 3.6 [fail](https://jenkins.plone.org/job/pull-request-5.2-3.6/1556/) >- The tests on Python 3.7 [fail](https://jenkins.plone.org/job/pull-request-5.2-3.7/1521/#showFailuresLink) > >Errors are like [this](https://jenkins.plone.org/job/pull-request-5.2-3.7/1521/testReport/junit/Products.CMFPlone.tests.testRegistrationTool/TestRegistrationTool/testMailPassword/): I will look into this -- likely tomorrow.

You need to change the tests. Explanation following. At the protocol level, email messages are a sequence of bytes. Python's `sendmail` enforces this by requiring that the message to be delivered is either a seqeunce of bytes or contains only ASCII characters. Formerly, `Products.Mailhost` and `zope.sendmail` did not address this requirement. Usually, email messages use an ASCII encoding and then are ASCII only. However, email messages may use `Content-Transfer-Encoding: 8bit` (this means no encoding of the message body) and for those messages, `Products.Mailhost + zope.sendmail` could fail to satisfy the `sendmail` requirement. The current versions of `Products.MailHost` and `zope.sendmail` properly handle this case. For "8bit" messages which are not already a sequence of bytes, someone must convert them. With the new versions, both `Products.MailHost` as well `zope.sendmail` can do this (using different heuristics). In particular, both packages switched their internal message representation from `str` to `bytes`. This means: a message presented as a sequence of bytes stays as it is (formerly, it was converted to `str`); a message presented as `str` is converted to a sequence of bytes early on. As a consequence of this change, tests likely need to be adjusted. Email message tests usually do not actually send a message but use some kind of `sendmail` mockup to intercept the message to be sent and check it. That mockup will no receive a sequence of bytes (formerly an `str`) and must be able to cope with this.

[mauritsvanrees]

I have made the fixes on the Plone side. If I recall correctly, there were indeed only changes needed in tests, not in production code.