Skip some serverless tests after roles and spaces changes in QA #190599
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pheyos
added
release_note:skip
alisonelizabeth
approved these changes
Aug 15, 2024
💚 Build Succeeded
Metrics [docs]
HistoryTo update your PR or re-run it, just comment with: cc @pheyos |
|
Corresponding MKI run on PR code level was successful ✔️ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR skips some serverless tests after changes on roles and spaces have been applied to the QA environment.
Details
After role management and multiple spaces have been enabled in the QA environment, a number of tests started to fail when running against MKI. Since we run the same set of tests against QA and Staging, we can't just fix the tests to work with the changes in QA because then they would fail when running against Staging.
After carefully checking the assertions in the tests (mainly checking that certain things are NOT available, while we gradually make them available again env by env), we decided in a small group discussion it's the best way forward to temporarily skip these tests and revisit them once the change has been applied to all environments. Some of the tests might be deleted then, others might be adjusted to the new behavior.
To be clear: this is not ideal and this outcome is a trade-off for this specific situation and should not be used as future guidance to handle this kind of problem. We have some ideas of how to deal with changes that are applied gradually to the environments, but it takes more time to consolidate these. In the interest of moving forward quickly, the test skip was considered the best solution for now.