Tensorboard logger check if lightning_logs directory exists (Lightnin…

…g-AI#1377)

* tensorboard logger version if root_dir not exist

* update changelog

* resolve comments

Co-authored-by: Alexander Reshytko <areshytko@Alexanders-MacBook-Pro.local>
Co-authored-by: William Falcon <waf2107@columbia.edu>

CHANGELOG.md

@@ -83,6 +83,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Fixed running `on_validation_end` only on main process in DDP ([#1125](https://github.com/PyTorchLightning/pytorch-lightning/pull/1125))
 - Fixes `use_amp` issue ([#1145](https://github.com/PyTorchLightning/pytorch-lightning/pull/1145))
 - Fixes using deprecated `use_amp` attribute ([#1145](https://github.com/PyTorchLightning/pytorch-lightning/pull/1145))
+- Fixed Tensorboard logger error: lightning_logs directory not exists in multi-node DDP on nodes with rank != 0 ([#1375](https://github.com/PyTorchLightning/pytorch-lightning/issues/1375)).
 - Fixed `Unimplemented backend XLA` error on TPU ([#1387](https://github.com/PyTorchLightning/pytorch-lightning/pull/1387))
 
 ## [0.7.1] - 2020-03-07

pytorch_lightning/loggers/tensorboard.py

@@ -9,6 +9,7 @@
 from torch.utils.tensorboard import SummaryWriter
 
 from pytorch_lightning.loggers.base import LightningLoggerBase, rank_zero_only
+from pytorch_lightning import _logger as log
 
 
 class TensorBoardLogger(LightningLoggerBase):
@@ -163,6 +164,11 @@ def version(self) -> int:
 
     def _get_next_version(self):
         root_dir = os.path.join(self.save_dir, self.name)
+
+        if not os.path.isdir(root_dir):
+            log.warning('Missing logger folder: %s', root_dir)
+            return 0
+
         existing_versions = []
         for d in os.listdir(root_dir):
             if os.path.isdir(os.path.join(root_dir, d)) and d.startswith("version_"):