feat: add cached dataset wrapper #148

wangraying · 2021-07-28T09:48:34Z

No description provided.

bagua/torch_api/contrib/cached_dataset.py

todo · 2021-08-04T10:06:31Z

cleanup automatically

bagua/bagua/torch_api/contrib/cache_loader.py

Lines 53 to 58 in 6c6ab3d

    
                   # TODO: cleanup automatically 
        
                   self.store.shutdown() 
        
           class BatchFetcher: 
        
               def __init__(self, store, batch_reads, batch_writes):

This comment was generated by todo based on a `TODO` comment in `6c6ab3d` in #148. cc @BaguaSys.

bagua/torch_api/contrib/utils/redis_store.py

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

bagua/torch_api/contrib/cache_dataset.py

bagua/torch_api/contrib/utils/redis_store.py

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

bagua/torch_api/contrib/cache_dataset.py

bagua/torch_api/contrib/utils/redis_store.py

bagua/torch_api/contrib/utils/store.py

NOBLES5E

see comments

bagua/torch_api/contrib/utils/redis_store.py

bagua/torch_api/contrib/cached_dataset.py

bagua/torch_api/contrib/utils/store.py

NOBLES5E

see comments

wangraying added 2 commits July 28, 2021 16:37

init commit

62325c6

add

2c640ba

pr-triage bot added the PR: unreviewed label Jul 28, 2021

update

9dda8cc

github-actions bot reviewed Jul 28, 2021

View reviewed changes

bagua/torch_api/contrib/cached_dataset.py Outdated Show resolved Hide resolved

wangraying added 2 commits July 28, 2021 23:35

update redis config

d41667e

.

5d130b7

wangraying added PR: draft and removed PR: unreviewed labels Jul 29, 2021

fix

b6f36d2

pr-triage bot added PR: unreviewed and removed PR: draft labels Jul 29, 2021

batch writes

ec998cc

wangraying added PR: draft and removed PR: unreviewed labels Aug 2, 2021

add cluster

6c6ab3d

pr-triage bot added PR: unreviewed and removed PR: draft labels Aug 4, 2021

github-actions bot reviewed Aug 4, 2021

View reviewed changes

bagua/torch_api/contrib/utils/redis_store.py Outdated Show resolved Hide resolved

Update bagua/torch_api/contrib/utils/redis_store.py

241b21e

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

wangraying added PR: draft and removed PR: unreviewed labels Aug 6, 2021

add

14d6d17

pr-triage bot added PR: unreviewed and removed PR: draft labels Aug 11, 2021

github-actions bot reviewed Aug 11, 2021

View reviewed changes

wangraying and others added 2 commits August 11, 2021 23:26

Update bagua/torch_api/contrib/cache_dataset.py

4b7a7d6

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update bagua/torch_api/contrib/cache_dataset.py

0bb1d15

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

github-actions bot reviewed Aug 11, 2021

View reviewed changes

bagua/torch_api/contrib/cache_dataset.py Outdated Show resolved Hide resolved

bagua/torch_api/contrib/cache_dataset.py Outdated Show resolved Hide resolved

Update redis_store.py

5a67148