DB upgrade from 8 -> preview fails

[Andrew-Roth]

Issue Summary

Running in EKS, pulling latest preview and using LDAP auth. Getting the following error, during DB upgrade

[2024-05-08 02:18:27,791][PID:145][INFO][alembic.runtime.migration] Running upgrade e5c7a4e2df4d -> d7d747033183, encrypt alert destinations
Traceback (most recent call last):
  File "./manage.py", line 9, in <module>
    manager()
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flask/cli.py", line 357, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flask_migrate/cli.py", line 134, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/usr/local/lib/python3.8/site-packages/flask_migrate/__init__.py", line 95, in wrapped
    f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flask_migrate/__init__.py", line 280, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/usr/local/lib/python3.8/site-packages/alembic/command.py", line 403, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.8/site-packages/alembic/script/base.py", line 583, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "migrations/env.py", line 93, in <module>
    run_migrations_online()
  File "migrations/env.py", line 85, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.8/site-packages/alembic/runtime/environment.py", line 948, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.8/site-packages/alembic/runtime/migration.py", line 627, in run_migrations
    step.migration_fn(**kw)
  File "/app/migrations/versions/d7d747033183_encrypt_alert_destinations.py", line 60, in upgrade
    .values(encrypted_options=dest.options)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/type_api.py", line 1278, in process
    return process_value(impl_processor(value), dialect)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/sqltypes.py", line 946, in process
    value = bytes(value)
TypeError: string argument without an encoding

[Andrew-Roth]

If anyone runs into this, I was able to work around by updating to the stable release (10.1.0.b50633), manually upgrading the db, then switching to the preview release and manually upgrading the db again

[justinclift]

@Andrew-Roth Yeah, the published recommendation is to upgrade one major version at a time (eg from 8 -> 9, then upgrade again from 9 -> 10, etc) rather than doing the large jump directly from 8/9 to the preview image.

[Andrew-Roth]

@Andrew-Roth Yeah, the published recommendation is to upgrade one major version at a time (eg from 8 -> 9, then upgrade again from 9 -> 10, etc) rather than doing the large jump directly from 8/9 to the preview image.

All the official guidance I've seen suggests going from v8 straight to v10, never using v9. The key for me here was going to 10.1 first rather than straight to preview.

"For best results you should upgrade Redash by one semantic version at a time. To move from V6 to V10, for example, you should upgrade V6 to V7 to V8 to V10."
https://redash.io/help/open-source/admin-guide/how-to-upgrade

"From V8 or earlier

Follow the complete steps outlined in the V10.0 release but use this Docker Tag in step 3: redash/redash:10.1.0.b50633
Read the Impact segment at this link. If your installation is affected, follow the instructions under the Patches heading to secure the secret fields in your database."
https://github.com/getredash/redash/releases

[justinclift]

V8 to V10.

Interesting. I've never noticed that version skip before. 😄

[zachliu]

This error is due to a potential 🐛 in the alembic db migration code.
See my comment #6682 (comment).
I reverted the d7d747033183 part of PR #6682 and was able to use manage db upgrade cli to finish the db upgrade

[justinclift]

k, this sounds like something we'd better look into. @eradman @wlach Any interest in taking a look? 😄

[eradman]

This should be solved, but #6836 needs to be resolved first so that we can run upgrade/downgrade.