[WB-4478] Feature/tb sync #1852
Conversation
| @@ -484,6 +491,8 @@ def sync( | |||
| clean_old_hours=24, | |||
| clean_force=None, | |||
| ): | |||
| # TODO: rather unfortunate, needed to avoid creating a `wandb` directory | |||
| os.environ["WANDB_DIR"] = TMPDIR.name | |||
There was a problem hiding this comment.
@raubitsj this one really sucks. I guess we can address it once we get rid of wandb.old and _get_cling_api... Not sure if it was worth it to dig deeper.
There was a problem hiding this comment.
Other people took a pass at not creating wandb dirs, but i think it wasnt done cleanly so it keeps coming back
| self._step, len(dropped_keys) | ||
| ) | ||
| ) | ||
| print("\t" + ("\n\t".join(dropped_keys))) |
There was a problem hiding this comment.
This was the error that Graphcore was seeing, atleast now we tell the user exactly what keys we're dropping if a single "row" is more than 4 megabytes.
Codecov Report
@@ Coverage Diff @@
## master #1852 +/- ##
==========================================
+ Coverage 73.81% 74.90% +1.08%
==========================================
Files 233 233
Lines 28554 28754 +200
==========================================
+ Hits 21078 21538 +460
+ Misses 7476 7216 -260
Continue to review full report at Codecov.
|
https://wandb.atlassian.net/browse/WB-4478
https://wandb.atlassian.net/browse/CLI-708
Description
Syncs tensorboard event files with
wandb syncand adds tests.Testing
How was this PR tested?