Distributed training

[ebgoldstein]

Make the distributed training branch conform to the config file..

config will need:

• "DIST_TRAIN" : {true | false} optional (default is false) to turn on mirror strat
• "GPUS" : {dict} optional .. to set the GPU numbers to use (optional, for people who have non sequential gpus, or different size, etc.
• Documentation in readme

[ebgoldstein]

i played around with this for a bit today....there are some tricky things for me in this implementation.. primarily the need for a conditional with statement. it for sure can be done: e.g.,... but it will take a bit of skill growth for me to get it implemented..

My opinion right now is that this should be shelved until the active development on the train is over, and then i can try to do this again, and introduce the newer configs that will be needed...

so this can either be migrated to a discussion or closed for the time being.

[dbuscombe-usgs]

Agree this can now wait until train has incorporated better reporting of metrics, and more widespread testing of model training with the new "nd makedatasets" npz files

[dbuscombe-usgs]

I feel like much of this is now implemented, except DIST_TRAIN

[ebgoldstein]

i totally agree.. and the string for GPU nums eliminates the need for a DIST_TRAIN config... I will lock this conversation... we could also delete it if we want..