Updated the examples folder readme file #7208
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
duncantech
reviewed
Jun 10, 2024
examples/README.md
Outdated
|
|
||
| ```shell | ||
| pip install --no-deps --pre torchvision -i https://download.pytorch.org/whl/nightly/cu118 | ||
| ``` | ||
|
|
||
| ## Run the example | ||
| You can run all models directly. Only environment you want to set is `PJRT_DEVICE`. | ||
| You can run all models directly. The only environment you want to set is `PJRT_DEVICE`. |
There was a problem hiding this comment.
Add "variable" after environment for specificity.
There was a problem hiding this comment.
Done! Updated in the latest commit.
JackCaoG
reviewed
Jun 13, 2024
examples/README.md
Outdated
| - `train_resnet_flash_attention_fsdp_v2.py`: Combines flash attention with FSDP, showcasing the integration of custom kernels with FSDP for scalable and efficient model training. | ||
|
|
||
| - fsdp: A trainer implementation to run a decoder-only model using FSDP (Fully Sharded Data Parallelism). | ||
| - `train_decoder_only_fsdp_v2.py`: Employs FSDP for training the decoder-only model, demonstrating parallel training of large transformer models on TPUs. |
There was a problem hiding this comment.
Employs FSDP -> Employs FSDPv2(FSDP algorithm implemented with PyTorch/XLA GSPMD)
JackCaoG
reviewed
Jun 13, 2024
examples/README.md
Outdated
| - flash_attention: A trainer implementation to run a decoder-only model using Flash Attention. | ||
|
|
||
| - `train_decoder_only_flash_attention.py`: Incorporates flash attention, an efficient attention mechanism, utilizing custom kernels for accelerated training. | ||
| - `train_resnet_flash_attention_fsdp_v2.py`: Combines flash attention with FSDP, showcasing the integration of custom kernels with FSDP for scalable and efficient model training. |
There was a problem hiding this comment.
with FSDP -> with FSDPv2
There was a problem hiding this comment.
Done in latest commit
JackCaoG
approved these changes
Jun 14, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses the update of the readme file on the examples folder as requested here in this issue.
cc: @duncantech @JackCaoG
The following describes the Git difference for the changed files:
Changes:
Run the example
-You can run all models directly. Only environment you want to set is
PJRT_DEVICE.+You can run all models directly. The only environment you want to set is
PJRT_DEVICE.+## Examples and Description
+-
train_resnet_base.py: A minimal example of training ResNet50. This is the baseline example for comparing performance with other trainingstrategies.
+-
train_decoder_only_base.py: A minimal example of training a decoder-only model. This serves as a baseline for comparison with other training strategies.
+-
train_resnet_amp.py: Shows how to use Automatic Mixed Precision (AMP) with PyTorch/XLA to improve performance. This example demonstratesthe benefits of AMP for reducing memory usage and accelerating training.
+
+- data_parallel: A trainer implementation to run ResNet50 on multiple devices using data-parallel.
+
train_resnet_ddp.py: Shows how to use PyTorch's DDP implementation for distributed training on TPUs. This example showcases how to integrate PyTorch's DDP with PyTorch/XLA for distributed training.
train_resnet_spmd_data_parallel.py: Leverages SPMD (Single Program Multiple Data) for distributed training. It shards the batch dimension across multiple devices and demonstrates how to achieve higher performance than DDP for specific workloads.
train_resnet_xla_ddp.py: Shows how to use PyTorch/XLA's built-in DDP implementation for distributed training on TPUs. It demonstratesthe benefits of distributed training and the simplicity of using PyTorch/XLA's DDP.
+- debug: A trainer implementation to run ResNet50 with debug mode.
+
train_resnet_profile.py: Captures performance insights with PyTorch/XLA's profiler to identify bottlenecks. Helps diagnose and optimizemodel performance.
train_resnet_benchmark.py: Provides a simple way to benchmark PyTorch/XLA, measuring device execution and tracing time for overall efficiency analysis.
+- flash_attention: A trainer implementation to run a decoder-only model using Flash Attention.
+
train_decoder_only_flash_attention.py: Incorporates flash attention, an efficient attention mechanism, utilizing custom kernels for accelerated training.
train_resnet_flash_attention_fsdp_v2.py: Combines flash attention with FSDP, showcasing the integration of custom kernels with FSDP forscalable and efficient model training.
+- fsdp: A trainer implementation to run a decoder-only model using FSDP (Fully Sharded Data Parallelism).
train_decoder_only_fsdp_v2.py: Employs FSDP for training the decoder-only model, demonstrating parallel training of large transformer models on TPUs.
train_resnet_fsdp_auto_wrap.py: Demonstrates FSDP (Fully Sharded Data Parallel) for model training, automatically wrapping model partsbased on size or type criteria.
(END)