This repository has been archived by the owner on Mar 19, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 334
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Update numpy and pytorch seeding for dataloader and multiple processe…
…s per machine. (#299) Summary: Pull Request resolved: #299 *Current state:* Currently we set different seeds per-nodes, but the same seed among all training processes on a node. However, each of our Dataloader process seeds are all different each epoch, but non-deterministic. *Proposed State:* Different random number seeds for each dist_rank, and different, deterministic, for each Dataloader process seeds per epoch. https://fb.quip.com/hVIcAahpVLo2 *Effects:* Fixes randomization for a few losses, hooks, and trunks. Fixes randomization when using the Fork multiprocessing option for transformations. Fixes collapse of seeds to 0 when config seed set to 0. There are 3 changes summarized as: 1. Use dist_rank instead of node_id for seeding training processes. 2. Create worker_init_fn to manually set numpy seed for dataloader workers. 3. Add knowledge of training process seed to sampler. Reviewed By: prigoyal, QuentinDuval Differential Revision: D27784137 fbshipit-source-id: 55008e1a10d88637f7b8ff44c45fd4658706a465
- Loading branch information
1 parent
8e0476b
commit e775a26
Showing
6 changed files
with
47 additions
and
19 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters