Migrate the application to Python 3 #4251
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
No backward compatibility with Python 2.7 is kept. This commit mostly contains changes made with 2to3 and manual tweaking when necessary.
Since it is not possible to change redash/base:debian to Python 3 without breaking future relases, its Dockerfile is temporarly copied here.
Some of the older versions were not compatible with Python 3.
In Python 3, HMAC only works with bytes so the strings and the float used in the sign function need to be encoded. Hopefully this is still backward compatible with already generated signatures.
The latter is not available in Python 3. See https://bugs.python.org/issue17866
These functions shadow the builtin list function which is problematic since 2to3 adds a fair amount of calls to the builtin list when it finds dict.keys() and dict.values(). Only the Python function is renamed, from the perspective of the CLI nothing changes.
`message` is not available anymore, instead use the string representation of the exception.
psycopg2 returns `buffer` objects in Python 2.7 and `memoryview` in Python 3. See #3156
Exception.message is not available in Python 3 anymore, except for some exceptions defined by third-party libraries.
The buffer for the file should be made of bytes and the actual content written to it strings. Note: I do not know why the diff is so large as it's only a two lines change. Probably a white space or file encoding issue.
The UnicodeWriter is not used anymore. In Python 3, the interface provided by the CSV module only deals with strings, in and out. The encoding of the output is left to the user, in our case it is given to Flask via `make_response`.
* Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow) * Make sure Redis connection uses decoded_responses option * Remove unused imports. * Use Redis' decode_responses option * Remove cases of explicit Redis decoding * Rename helper function and make sure it doesn't apply twice. * Don't add decode_responses to Celery Redis connection URL
The exception message is always a string in Python 3, so no need to try to decode things.
SimpleJSON assumes the bytes it receives contain text data, so it tries to UTF-8 encode them. It is sometimes not true, for instance the SQLite datasource returns bytes for BLOB types, which typically do not contain text but truly binary data. This commit disables SimpleJSON auto encoding of bytes to str and instead uses the same method as for memoryviews: generating a hex representation of the data.
1 task
jezdez
reviewed
Oct 17, 2019
Comment on lines
-84
to
-89
| except Exception as e: | ||
| # handle unicode error message | ||
| err_class = sys.exc_info()[1].__class__ | ||
| err_args = [arg.decode('utf-8') for arg in sys.exc_info()[1].args] | ||
| unicode_err = err_class(*err_args) | ||
| reraise(unicode_err, None, sys.exc_info()[2]) |
There was a problem hiding this comment.
Previously this was catching all errors, even if they weren't UnicodeErrors, so I would suggest to keep this.
There was a problem hiding this comment.
I think this was not catching UnicodeError but rather catching everything, decoding content and reraising. I still think that this is not needed on Python 3.
There was a problem hiding this comment.
I agree w/ @NicolasLM , this seems unnecessary anymore (it was supposed to handle error message with Unicode).
- Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects. - Also fixed some naming issues with one character variables in list comprehensions. - Fix Flask warning.
NicolasLM
force-pushed
the
python-3-rq
branch
from
fc8a480
to
2f2a280
Compare
|
I upgraded most dependencies as requested in #4181, but the bump of Werkzeug from 0.11 to 0.16 breaks tests related to redirecting to the next url after login. I've spent some time debugging the problem, but I didn't find the reason. Any idea? |
* Removed setting SERVER_NAME in tests setup to avoid a warning. * Change get_next_path to not return empty string in case of a domain only value. * Fix redirect tests: Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path. * Remove explicit dependency for Werkzeug
arikfr
reviewed
Oct 23, 2019
Comment on lines
-84
to
-89
| except Exception as e: | ||
| # handle unicode error message | ||
| err_class = sys.exc_info()[1].__class__ | ||
| err_args = [arg.decode('utf-8') for arg in sys.exc_info()[1].args] | ||
| unicode_err = err_class(*err_args) | ||
| reraise(unicode_err, None, sys.exc_info()[2]) |
There was a problem hiding this comment.
I agree w/ @NicolasLM , this seems unnecessary anymore (it was supposed to handle error message with Unicode).
`python-geoip-geolite2` is not compatible with Python 3, instead use `maxminddb-geolite2` which is very similar as it includes the geolite2 database in the package .
* show current worker job (alongside with minor cosmetic column tweaks) * avoid loading entire job data for queued jobs * track general RQ queues (default, periodic and schemas) * get all active RQ queues * call get_celery_queues in another place * merge dicts the Python 3 way * extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions
|
@arikfr !!!! 🐍🚀 |
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this? (check all applicable)
Description
Migrate the application to Python 3.
Related Tickets & Documents
#4181