Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests: add default_root_dir=tmpdir #2392

Merged
merged 6 commits into from
Jul 28, 2020
Merged

Conversation

Borda
Copy link
Member

@Borda Borda commented Jun 27, 2020

What does this PR do?

some test were influenced by previous runs...

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

@Borda Borda added bug Something isn't working ci Continuous Integration labels Jun 27, 2020
@Borda Borda added this to the 0.8.x milestone Jun 27, 2020
@mergify mergify bot requested a review from a team June 27, 2020 19:49
@Borda
Copy link
Member Author

Borda commented Jun 27, 2020

Not sure why MLFlow fails in CI while locally pass

        file_path = os.path.join(root, file_name)
        if not exists(file_path):
>           raise MissingConfigException("Yaml file '%s' does not exist." % file_path)
E           mlflow.exceptions.MissingConfigException: Yaml file '/tmp/pytest-of-runner/pytest-0/test_loggers_fit_test_MLFlowLo0/default/a7c4907de41943a7b958534289a872a2/meta.yaml' does not exist.

@awaelchli
Copy link
Member

same here #2391 what's going on? I've been trying to debug this for literally almost the whole day.
I think it is a mismatch between the logger dir (weights_save_path) and the checkpoint dir .

@Borda
Copy link
Member Author

Borda commented Jun 28, 2020

@awaelchli so yo say that weights_save_path and default_root_dir does not like each other?

@awaelchli
Copy link
Member

yes, the bug is somewhere burried in the many if else statements that sets these two variables. There seems to be a mismatch betwen these two paths.
Basically some loggers (e.g. mlflow) save a config file like meta.yaml in the log dir and this seems to be in the wrong folder.
Then when you try to resume from checkpoint (e.g. in trainer.test) it will not find that file.

@Borda
Copy link
Member Author

Borda commented Jun 28, 2020

yes, the bug is somewhere burried in the many if else statements that sets these two variables. There seems to be a mismatch betwen these two paths.

at first I thought just to skip it and use the passing test maybe it is a chance to find the bug and fix it...

@mergify
Copy link
Contributor

mergify bot commented Jun 29, 2020

This pull request is now in conflict... :(

@Borda Borda changed the title tests: add default_root_dir=tmpdir [blocked by #2502] tests: add default_root_dir=tmpdir Jul 4, 2020
@mergify
Copy link
Contributor

mergify bot commented Jul 7, 2020

This pull request is now in conflict... :(

@awaelchli
Copy link
Member

@Borda #2502 got merged, this is now unblocked

@Borda Borda changed the title [blocked by #2502] tests: add default_root_dir=tmpdir tests: add default_root_dir=tmpdir Jul 11, 2020
@codecov
Copy link

codecov bot commented Jul 12, 2020

Codecov Report

Merging #2392 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #2392   +/-   ##
======================================
  Coverage      91%     91%           
======================================
  Files          82      82           
  Lines        6784    6784           
======================================
  Hits         6151    6151           
  Misses        633     633           

@mergify mergify bot requested a review from a team July 12, 2020 20:25
@Borda
Copy link
Member Author

Borda commented Jul 12, 2020

@awaelchli thx for fixing missed changes after rebase :]

@Borda Borda added the ready PRs ready to be merged label Jul 12, 2020
@mergify mergify bot requested a review from a team July 12, 2020 23:17
Copy link
Contributor

@jeremyjordan jeremyjordan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍
excited for this PR, those lingering test artifacts are definitely annoying

@mergify mergify bot requested a review from a team July 13, 2020 01:38
@Borda Borda requested a review from yukw777 July 28, 2020 08:20
@williamFalcon williamFalcon merged commit 590e7fb into master Jul 28, 2020
@Borda Borda deleted the tests/default_root_dir branch July 28, 2020 14:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working ci Continuous Integration ready PRs ready to be merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants