Set up and teardown for individual tests

[hackebrot]

We need a way to run tests in isolation that they don't leak into other test items. The order in which tests are being executed must not impact the test results.

• pytest setup to create clean database tables etc.
• pytest teardown to drop database tables etc.

[hackebrot]

Check out redash_client!

[b4handjr]

For our setup, should there be data added to the database? Or just the creation of the tables and such.

[hackebrot]

Only what we need to log in a user.

[b4handjr]

Only what we need to log in a user.

So kind of like what we have now? I just want to see if I need to clean up what we have now, or do something different.

[hackebrot]

Since we don't have access to the Factory that is being used in the redash unit tests, we need to create test data manually. For the login test we need an organization and a user, see https://github.com/mozilla/redash-ui-tests/blob/master/tests/conftest.py#L58

[b4handjr]

Why can't we create test data before we run the tests? Currently on addons-server we create the database and fill it with data, and then run the tests. It helps with using the tests in an environment outside of the one we can control, say a dev or stage instance.

[hackebrot]

We can do that, sure. For some tests we will have to clean the data that we created during the test execution. I'm not entirely sure I understand what you're asking. Are you suggesting to use make rather than pytest fixtures for the setup?

[b4handjr]

I think that is what I am saying. I did some research on how to use the python postgres client, and it would be more work than I think it is worth. We are really at a loss here not being able to use upstream redash features. If we can somehow call the commands within the redash container, that would help a bunch. But I don't think we want to rewrite every sql command that we need by hand,

[b4handjr]

Per chat:

Initial Setup: Create DB, maybe setup root user.
Per-Test: Ability to add/remove data from database as needed.

[b4handjr]

Per IRC

I suggest to have commands in Makefile/CI to
1. Create the database
2. Create the database tables
3. Create root user

And a pytest fixture to 
1. create all users from variables.json

And tests that
1. log in a user who was created in the pytest fixture

[b4handjr]

I reached out to Jannis and he pointed me here. I've got this working locally.

[b4handjr]

For teardown, looks like we can only 'disable' a user, not delete a user. This could be a problem if we are changing user profile data via the UI in one test, but not another.

Found this issue.

Looks like there is a way we can do it via the DB via postgres commands. Is that something we want to do?

[jezdez]

@jrbenny35 There is a users CLI subcommand for the manage.py script, that can be used to delete a user: ./manage.py users delete foo@bar.com

Could you use that instead?

[b4handjr]

@jezdez We could but we would like to use the api as that would work with a local or hosted instance.

[jezdez]

@jrbenny35 Right, since the API doesn't support this out of the box I'm afraid there is only one option left to add a new endpoint ourselves using the fairly new Redash extensions mechanism. It's the way we're going to move our Redash customizations into a separate package called redash-stmo as well.

It's a bit more involved but here are the steps to do that:

1. Create a custom Python package, e.g. redash_ui_tests, that follows the same API as redash-stmo
2. To that package add a Redash extension that registers a new custom UserDeleteResource class that does the deletion like you want with the passed Flask app instance similar to how the built-in UserDisableResource is registered under the endpoint /api/users/<user_id>/delete.
3. Release the package to PyPI
4. Install it in the Docker image by adding it to the Pipfile
5. Call to the custom user delete API endpoint in your teardown code

[b4handjr]

@jezdez that is a great solution I will have to look into! Thanks for that

[jezdez]

@jrbenny35 The great thing about it is that you can extend it with other API endpoints or code changes if needed and keep it in sync with this repo. I guess that package can easily just live in this repo next to the tests to simplify coordination. Just put it into src/redash_ui_tests and add the setup.py etc files.

[hackebrot]

Hi @jrbenny35 @jezdez! 👋

Can we create a story for this over at getredash/redash or mozilla/redash, as this seems like a missing capability of Redash rather than testing specific requirement?

Keep in mind that we can't install Redash extensions into a hosted Redash instance. So if we want to be able to run the UI tests against staging or a release candidate we need to work with the official API.

For the two login tests that we have, we don't need to be able to delete users, so we can go ahead without adding this at this point.

[b4handjr]

I agree. It seems like the delete issue has been open for over 2 years. Makes sense if you are able to create you should be able to delete.

[jezdez]

@hackebrot There is already getredash/redash#799 and a related issue was fixed recently in getredash/redash#1600. Not sure if upstream will add an API endpoint to delete users since right now the frontend client is the only consumer of that API. Since we're working on a new consumer with the ui tests I would keep it contained here until there is an indication of interest to implement it in the main repo.

The long term goal to support any Redash instance shouldn't be a blocker to implement our user deletion endpoint now to get this off the ground. For our instance of Redash we already install redash-stmo in stage and prod and it's perfectly fine to assume that we can install redash-ui-tests there as well.

[hackebrot]

The goal to support both Redash running inside of Docker container or on a hosted environment is not a blocker to implement such an endpoint. However I feel like this belongs in redash-stmo rather than the UI testing harness.

I'd rather move forward on getting the login tests up and running, than spending time to create a new PyPI package with an endpoint that is not strictly required at this point.

[jezdez]

@hackebrot Makes sense, let's table it until we really need it.

[b4handjr]

@hackebrot so for now just get the user creation working? I kinda hate half-implementing stuff but I can understand as there isn't a solution as of right now.

[jezdez]

@hackebrot @jrbenny35 Let's open a ticket in https://github.com/mozilla/redash-stmo and link to this tread, ok?

[hackebrot]

Don't worry about that, @jrbenny35! Since we're not testing the creation of users via the GUI, there's not need for us to remove the users at this point. Our goal is to get the login tests working! 👍

[b4handjr]

Solved the first one in #26

[hackebrot]

Closing this now. Let's open new issues for specific test cases!