Port synapse_port_db to async/await

[babolivier]

Also raise an exception if there are pending background updates so we return with a non-0 code in the CI if something wrong happened when running the background updates.

Fixes #6719

[tadzik]

Running this locally still fails for me. As is:

Preparing sqlite3...
2020-01-16 21:56:47,381 - synapse_port_db - 601 - ERROR - 
Traceback (most recent call last):
  File "/home/synapse/synapse/env/bin/synapse_port_db", line 501, in run
    allow_outdated_version=True,
  File "/home/synapse/synapse/env/bin/synapse_port_db", line 471, in build_db_store
    db_conn, allow_outdated_version=allow_outdated_version
TypeError: check_database() got an unexpected keyword argument 'allow_outdated_version'
Traceback (most recent call last):
  File "/home/synapse/synapse/env/bin/synapse_port_db", line 501, in run
    allow_outdated_version=True,
  File "/home/synapse/synapse/env/bin/synapse_port_db", line 471, in build_db_store
    db_conn, allow_outdated_version=allow_outdated_version
TypeError: check_database() got an unexpected keyword argument 'allow_outdated_version'

Removing that parameter pushes it further, but it still dies:

Preparing sqlite3...
2020-01-16 21:57:45,737 - synapse.storage.prepare_database - 304 - INFO - Upgrading schema to v56
Preparing psycopg2...
2020-01-16 21:57:45,771 - synapse_port_db - 601 - ERROR - 
Traceback (most recent call last):
  File "/home/synapse/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: []

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/synapse/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: []

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/synapse/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: True

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/synapse/synapse/env/bin/synapse_port_db", line 516, in run
    self.hs_config.get_single_database()
  File "/home/synapse/synapse/env/bin/synapse_port_db", line 471, in build_db_store
    db_conn
  File "/home/synapse/synapse/env/lib/python3.7/site-packages/synapse/storage/engines/postgres.py", line 36, in check_database
    txn.execute("SHOW SERVER_ENCODING")
AttributeError: 'psycopg2.extensions.connection' object has no attribute 'execute'
Traceback (most recent call last):
  File "/home/synapse/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: []

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/synapse/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: []

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/synapse/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: True

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/synapse/synapse/env/bin/synapse_port_db", line 516, in run
    self.hs_config.get_single_database()
  File "/home/synapse/synapse/env/bin/synapse_port_db", line 471, in build_db_store
    db_conn
  File "/home/synapse/synapse/env/lib/python3.7/site-packages/synapse/storage/engines/postgres.py", line 36, in check_database
    txn.execute("SHOW SERVER_ENCODING")
AttributeError: 'psycopg2.extensions.connection' object has no attribute 'execute'

[babolivier]

@tadzik I looked into these logs and the corresponding code, it looks like it's caused by the changes in #6675 which haven't been released yet, hence why the patched script is working fine with a Synapse on the develop branch.

Given my local tests and the CI (which is now fixed, and passing), I'm pretty confident that the script now does the right thing.

So to try this patch, instead of blindly applying it onto your local version of synapse_port_db, there are two options:

• Setup a development environment for Synapse and checkout the babolivier/port_db_bg_update_fix branch
• Wait for the next release which will ship this fix (which should happen next week, or on the following one I think)

[tadzik]

Alright, having checked your branch (I'm at 75d60c2), I'm still getting errors (though different ones now :)). They originate from synapse/storage/data_stores/main/schema/delta/57/local_current_membership.py, which I've modified as follows for debugging's sake:

--- a/synapse/storage/data_stores/main/schema/delta/57/local_current_membership.py
+++ b/synapse/storage/data_stores/main/schema/delta/57/local_current_membership.py
@@ -68,6 +68,8 @@ def run_upgrade(cur, database_engine, config, *args, **kwargs):
                 INNER JOIN room_memberships AS r USING (event_id)
                 WHERE type = 'm.room.member' and state_key like '%' || ?
         """
+    print(f"sql is '{sql}'")
+    print(f"config.server_name is '{config.server_name}'")
     cur.execute(sql, (config.server_name,))
 
     cur.execute(

The crash looks like so:

2020-01-17 13:25:39,547 - synapse.storage.prepare_database - 373 - INFO - Running script 57/local_current_membership.py
sql is '
            INSERT INTO local_current_membership (room_id, user_id, event_id, membership)
                SELECT c.room_id, state_key AS user_id, event_id, r.membership
                FROM current_state_events AS c
                INNER JOIN room_memberships AS r USING (event_id)
                WHERE type = 'm.room.member' and state_key like '%' || ?
        '
config.server_name is 'tadzik.net'
(...)
Traceback (most recent call last):
  File "/home/synapse/synapse/synapse.git/scripts/synapse_port_db", line 516, in run
    self.hs_config.get_single_database()
  File "/home/synapse/synapse/synapse.git/scripts/synapse_port_db", line 473, in build_db_store
    prepare_database(db_conn, engine, config=self.hs_config)
  File "/home/synapse/synapse/synapse.git/synapse/storage/prepare_database.py", line 85, in prepare_database
    data_stores=data_stores,
  File "/home/synapse/synapse/synapse.git/synapse/storage/prepare_database.py", line 376, in _upgrade_existing_database
    module.run_upgrade(cur, database_engine, config=config)
  File "/home/synapse/synapse/synapse.git/synapse/storage/data_stores/main/schema/delta/57/local_current_membership.py", line 73, in run_upgrade
    cur.execute(sql, (config.server_name,))
ValueError: unsupported format character ''' (0x27) at index 349

Some further debugging revealed that the offending character is the leading ' in state_key like '%'. I don't see anything wrong with it myself, but perhaps you do :)

[babolivier]

Looks like that was #6727, fixed in #6728. I've merged the latest state of develop into my branch to pull the fix in, so feel free to git pull and try again.

[babolivier]

@tadzik just told me it worked and he's now live on PostgreSQL, so it looks like everything works correctly now 🎉