Skip to content

Commit

Permalink
BF: guard download_generator to not propagate errors
Browse files Browse the repository at this point in the history
This way instead of dumping all tracebacks at the end, some times for benign
FileExistsError -- they will be displayed neatly in pyout and also output
in detail in the log.

Closes #1007
  • Loading branch information
yarikoptic committed May 11, 2022
1 parent 231f7f0 commit debdf1f
Showing 1 changed file with 15 additions and 2 deletions.
17 changes: 15 additions & 2 deletions dandi/download.py
Original file line number Diff line number Diff line change
Expand Up @@ -313,10 +313,23 @@ def download_generator(
lock=lock,
)

def _guarded_generator(path, generator):
try:
for resp in generator:
yield dict(resp, path=path)
except Exception as exc:
lgr.error("Caught while downloading %s", path, exc_info=exc)
yield {
"path": path,
"status": "error",
"message": str(exc.__class__.__name__),
}

gen = _guarded_generator(path, _download_generator)
if yield_generator_for_fields:
yield {"path": path, yield_generator_for_fields: _download_generator}
yield {"path": path, yield_generator_for_fields: gen}
else:
for resp in _download_generator:
for resp in gen:
yield dict(resp, path=path)


Expand Down

0 comments on commit debdf1f

Please sign in to comment.