Releases: mosaicml/composer
v0.12.0
π Composer v0.12.0
Composer v0.12.0 is released! Install via pip
:
pip install mosaicml==0.12.0
New Features
-
πͺ΅ Logging and ObjectStore Enhancements
There are multiple improvements to our logging and object store support in this release.
-
Image visualization using our
CometMLLogger
(#1710)We've added support for using our
ImageVisualizer
callback with CometML to log images and segmentation masks to CometML.from composer.trainer import Trainer trainer = Trainer(..., callbacks=[ImageVisualizer()], loggers=[CometMLLogger()] )
-
Added direct support for Oracle Cloud Infrastructure (OCI) as an
ObjectStore
(#1774) and support for Google Cloud Storage (GCS) via URI (#1833)To use, you can simply set your
save_folder
orload_path
to a URI beginning withoci://
orgs://
, to save and load with OCI and GCS respectively.from composer.trainer import Trainer # Checkpoint saving to Google Cloud Storage. trainer = Trainer( model=model, save_folder="gs://my-bucket/{run_name}/checkpoints", run_name='my-run', save_interval="1ep", save_filename="ep{epoch}.pt", save_num_checkpoints_to_keep=0, # delete all checkpoints locally ... ) trainer.fit()
-
Added basic support for logging with MLFlow (#1795)
We've added basic support for using MLFlow to log experiment metrics.
from composer.loggers import MLFlowLogger from composer.trainer import Trainer mlflow_logger = MLFlowLogger(experiment_name=mlflow_exp_name, run_name=mlflow_run_name, tracking_uri=mlflow_uri) trainer = Trainer(..., loggers=[mlflow_logger])
-
Simplified console and progress bar logging (#1694)
To turn off the progress bar, set
progress_bar=False
. To turn on logging directly to the console, setlog_to_console=True
. To control the frequency of logging to console, setconsole_log_interval
(e.g. to1ep
or1ba
). -
Our
get_file
utility now supports URIs directly (s3://
,oci://
, andgs://
) for downloading files.
-
-
πββοΈ Support for Mid-Epoch Resumption with the latest release of Streaming
We've added support in Composer for the latest release of our Streaming library. This includes awesome new features like instant mid epoch resumption and deterministic shuffling, regardless of the number of nodes. See the Streaming release notes for more!
-
π¨ New algorithm -
GyroDropout
!Thanks to @jelite for adding a new algorithm,
GyroDropout
to Composer! Please see the method card for more details. -
π€ HuggingFace + Composer improvements
We've added a new utility to load a π€ HuggingFace model and tokenizer out of a Composer checkpoint (#1754), making the pretraining -> finetuning workflow even easier in Composer. Check out the docs for more details, and our example notebook for a full tutorial (#1775)!
-
π GradMonitor -> OptimizerMonitor
Renames our
GradMonitor
callback toOptimizerMonitor
, and adds the ability to track optimizer specific metrics. Check out the docs for more details, and add to your code just like any other callback!from composer.callbacks import OptimizerMonitor from composer.trainer import Trainer trainer = Trainer( ..., callbacks=[OptimizerMonitor(log_optimizer_metrics=log_optimizer_metrics)] )
-
π³ New PyTorch and CUDA versions
We've expanded our library of Docker images with support for PyTorch 1.13 + CUDA 11.7:
mosaicml/pytorch:1.13.0_cu117-python3.10-ubuntu20.04
mosaicml/pytorch:1.13.0_cpu-python3.10-ubuntu20.04
The
mosaicml/pytorch:latest
,mosaicml/pytorch:cpu_latest
andmosaicml/composer:0.12.0
tags are now built from PyTorch 1.13 based images. Please see our DockerHub repository for additional details.
API changes
-
Replace
grad_accum
withdevice_train_microbatch_size
(#1749, #1776)We're deprecating the
grad_accum
Trainer argument in favor of the more intuitivedevice_train_microbatch_size
. Instead of thinking about how to divide your specified minibatch into microbatches, simply specify the size of your microbatch. For example, let's say you want to split your minibatch of 2048 into two microbatches of 1024:from composer import Trainer trainer = Trainer( ..., device_train_microbatch_size=1024, )
If you want Composer to tune the microbatch for you automatically, enable automatic microbatching as follows:
from composer import Trainer trainer = Trainer( ..., device_train_microbatch_size='auto', )
The
grad_accum
argument is still supported but will be deprecated in the next Composer release. -
Renamed precisions (#1761)
We've renamed precision attributes for clarity. The following values have been removed:
['amp', 'fp16', bf16']
.We have added the following values, prefixed with 'amp' to clarify when an Automatic Mixed Precision type is being used:
['amp_fp16', 'amp_bf16']
.The
fp32
precision value remains unchanged.
Deprecations
- Removed support for YAHP (#1512)
- Removed COCO and SSD datasets (#1717)
- Fully removed Streaming v1 support, please see the mosaicml/streaming project for our next-gen streaming datasets (#1787)
- Deprecated
FusedLayerNorm
algorithm (#1789) - Fully removed
grad_clip_norm
training argument, please use theGradientClipping
algorithm instead (#1768) - Removed
data_fit
,data_epoch
, anddata_batch
fromLogger
(#1826)
Bug Fixes
- Fix FSDP checkpoint strategy (#1734)
- Fix gradient clipping with FSDP (#1740)
- Adds more supported FSDP config flags (
sync_module_states
,forward_prefecth
,limit_all_gathers
) (#1794) - Allow
FULL
precision with FSDP (#1796) - Fix
eval_microbatch
modification onEVAL_BEFORE_FORWARD
event (#1739) - Fix algorithm API backwards compatibility in checkpoints (#1741)
- Fixes a bad
None
check preventing settingdevice_id
to0
(#1767) - Unregister engine to make cleaning up memory easier (#1769)
- Fix issue if
metric_names
is not a list (#1798) - Match implementation for list and tensor batch splitting (#1804)
- Fixes infinite eval issue (#1815)
What's Changed
- Update installation constraints for streaming by @karan6181 in #1661
- Update decoupled_weight_decay.md by @jacobfulano in #1672
- Notebooks part 2 by @dakinggg in #1659
- Add trainer arg for engine passes by @mvpatel2000 in #1673
- Autoload algorithms by @mvpatel2000 in #1658
- Faster metrics calculations + Fix warnings added by the new version of torchmetrics by @dskhudia in #1674
- Update coolname requirement from <2,>=1.1.0 to >=1.1.0,<3 by @dependabot in #1666
- Bump ipykernel...
v0.11.1
π Composer v0.11.1
Composer v0.11.1 is released! Install via pip
:
pip install --upgrade mosaicml==0.11.1
Bug Fixes
- Fixes for Notebooks (#1659)
- Documentation updates and fixes (#1685, #1696, #1702, #1709)
- Addressed warnings and speed improvements for Torchmetrics (#1674)
- Fixes to Gated Linear Units method (#1575, #1689)
- Set
NCCL_ASYNC_ERROR_HANDLING
ENV variable in Composer launcher to enable distributed timeout (#1695) - Fix epoch count when
eval
is called beforefit
(#1697) - Constrain PyTorch package versions to avoid unintended upgrades (#1688)
- Fix Optimizer state sharding issue with FSDP (#1732)
- Rase
ValueError
with if evaluation dataloader of infinite length is specified
Full Changelog: v0.11.0...v0.11.1
v0.11.0
π Composer v0.11.0
Composer v0.11.0 is released! Install via pip
:
pip install --upgrade mosaicml==0.11.0
New Features
-
π§° FSDP Beta Support
Composer now supports PyTorch FSDP! PyTorch FSDP is a strategy for distributed training, similar to PyTorch DDP, that distributes work using data-parallelism only. On top of this, FSDP uses model, gradient, and optimizer sharding to dramatically reduce device memory requirements, and enables users to easily scale and train large models.
Here's how easy it is to use FSDP with Composer:
import torch.nn as nn from composer import Trainer class Block (nn.Module): ... # Your custom model class Model(nn.Module): def __init__(self, n_layers): super().__init__() self.blocks = nn.ModuleList([ Block(...) for _ in range(n_layers) ]), self.head = nn.Linear(...) def forward(self, inputs): ... # FSDP Wrap Function def fsdp_wrap_fn(self, module): return isinstance(module, Block) # Activation Checkpointing Function def activation_checkpointing_fn(self, module): return isinstance(module, Block) # ComposerModel wrapper, used by the Trainer # to compute loss, metrics, etc. class MyComposerModel(ComposerModel): def __init__(self, n_layers): super().__init__() self.model = Model(n_layers) ... def forward(self, batch): ... def eval_forward(self, batch, outputs=None): ... def loss(self, outputs, batch): ... # Pass your ComposerModel and fsdp_config into the Trainer composer_model = MyComposerModel(n_layers=3) fsdp_config = { 'sharding_strategy': 'FULL_SHARD', 'min_params': 1e8, 'cpu_offload': False, # Not supported yet 'mixed_precision': 'DEFAULT', 'backward_prefetch': 'BACKWARD_POST', 'activation_checkpointing': False, 'activation_cpu_offload': False, 'verbose': True } trainer = Trainer( model=composer_model, fsdp_config=fsdp_config, ... ) trainer.fit()
For more information, please see our FSDP docs.
-
π° Streaming v0.1
We've spun off Streaming datasets into it's own repository! Streaming datasets is a high-performance drop-in for TorchΒ
IterableDataset
, enabling users to stream training data from cloud based object stores. Streaming is shipping with built-in support for popular open source datasets (ADE20K, C4, COCO, Enwiki, ImageNet, etc.)To get started, install the Streaming PyPi package:
pip install mosaicml-streaming
You can use the streaming Dataset class with the PyTorch native DataLoader class as follows:
import torch from streaming import Dataset dataloader = torch.utils.data.DataLoader(dataset=Dataset(remote='s3://...'))
For more information, please check out the Streaming docs.
-
βπ Simplified Checkpointing Interface
With this release weβve greatly simplified configuration of loading and saving checkpoints in Composer.
To save checkpoints to S3, all you need to do is:
- Specify with
save_folder
your full URI to your save directory destination (e.g.'s3://my-bucket/{run_name}/checkpoints'
) - Optionally, set
save_filename
to the pattern you want for your checkpoint file names
from composer.trainer import Trainer # Checkpoint saving to S3. trainer = Trainer( model=model, save_folder="s3://my-bucket/{run_name}/checkpoints", run_name='my-run', save_interval="1ep", save_filename="ep{epoch}.pt", save_num_checkpoints_to_keep=0, # delete all checkpoints locally ... ) trainer.fit()
Likewise, to load checkpoints from S3, all you have to do is:
- Set
load_path
to the full URI to your desired checkpoint file (e.g.'s3://my-bucket/my-run/checkpoints/epoch13.pt'
)
from composer.trainer import Trainer # Checkpoint loading from S3. new_trainer = Trainer( model=model, train_dataloader=train_dataloader, max_duration="10ep", load_path="s3://my-bucket/my-run/checkpoints/ep13.pt", ) new_trainer.fit()
For more information, please see our Checkpointing guide.
- Specify with
-
π³ Improved Distributed Experience
Weβve made it easier to write your own custom distributed entry points by exposing our distributed API. You can now leverage all of our helpful distributed functions and contexts.
For example, let's say we want to need to download a dataset in a distributed training application. To avoid race conditions where different ranks try to write the dataset to the same place, we need to ensure that only rank 0 downloads the dataset first:
import datetime from composer.trainer.devices import DeviceGPU from composer.utils import dist dist.initialize(DeviceGPU(), datetime.timedelta(seconds=30)) # Initialize distributed module if dist.get_local_rank() == 0: # Download dataset on rank zero dataset = download_my_dataset() dist.barrier() # All ranks wait until dataset is downloaded # Create and train your model!
For more information, please check out our Distributed API docs.
Bug Fixes
- fix loss and eval_forward for HF models (#1597)
- add more robust casting to int for fsdp min_params (#1608)
- Deepspeed Docs Typo (#1605)
- Fix mmdet typo (#1618)
- Blurpool idempotent (#1625)
- When model is not on
meta
device, initialization should occur on compute device not CPU (#1623) - Auto resumption (#1615)
- Adjust speed monitor (#1645)
- Hot fix console logging (#1643)
- Lazy Logging + pretty print dict for hparams (#1653)
- Fix many failing notebook tests (#1646)
What's Changed
- Bump coverage[toml] from 6.4.4 to 6.5.0 by @dependabot in #1583
- Bump furo from 2022.9.15 to 2022.9.29 by @dependabot in #1584
- Add English Wikipedia 2020-01-01 dataset by @knighton in #1572
- Add pull request template by @dakinggg in #1588
- Bump ipykernel from 6.15.3 to 6.16.0 by @dependabot in #1587
- Update importlib-metadata requirement from <5,>=4.11.0 to >=5.0,<6 by @dependabot in #1585
- Bump sphinx-argparse from 0.3.1 to 0.3.2 by @dependabot in #1586
- Add step explicitly to ImageVisualizer logging calls by @dakinggg in #1591
- Image viz test by @dakinggg in #1592
- Remove unused fixture by @mvpatel2000 in #1594
- Fixes RandAugment API by @mvpatel2000 in #1596
- fix loss and eval_forward for HF models by @dskhudia in #1597
- Remove tensorflow-io from setup.py by @eracah in #1577
- Fixes enwiki for the newly processed wiki dataset by @dskhudia in #1600
- Change install to all by @mvpatel2000 in #1599
- Remove log level and should_log_artifact by @dakinggg in #1603
- Add more robust casting to int for fsdp min_params by @dblalock in #1608
- Deepspeed Docs Typo by @mvpatel2000 in #1605
- Object store logger refactor by @dakinggg in #1601
- Bump gitpython from 3.1.27 to 3.1.28 by @dependabot in #1609
- Bump tabulate from 0.8.10 to 0.9.0 by @dependabot in #1610
- Log the number of GPUs and nodes Composer running on. by @eracah in #1604
- Update MLPerfCallback for v2.1 by @hanlint in #1607
- Remove object store cls by @dakinggg in #1606
- Add LAMB Optimizer by @hanlint in #1613
- Mmdet adapter by @A-Jacobson in #1545
- Fix mmdet typo by @Landanjs in #1618
- update torchmetrics requirement by @hanlint in #1620
- Add distributed sampler error by @mvpatel2000 in #1598
- Landan/deeplabv3 ade20k example by @Landanjs in #1593
- Upgrade CodeQL Action to version 2 by @karan6181 in #1628
- Blurpool idempotent by @mvpatel2000 in #1625
- Defaulting streaming dataset version to 2 by @karan6181 in #1616
...
v0.10.1
π Composer v0.10.1
Composer v0.10.1 is released! Install via pip
:
pip install --upgrade mosaicml==0.10.1
New Features
-
π· Weight Standardization
Weight Standardization reparametrizes convolutional weights such that the fan-in dimensions have zero mean and unit standard deviation. This could slightly improve performance at the expensive of 5% lower throughput. This has been used in several papers to train with smaller batch sizes, with normalization layers besides batch norm, and for transfer learning.
Using Weight Standardization with the Composer Trainer:
import composer # Apply Weight Standardization (when training is initialized) weight_std = composer.algorithms.WeightStandardization() # Train with Weight Standardization trainer = composer.trainer.Trainer( ... algorithms=[weight_std] ) trainer.fit()
Using Weight Standardization with the Composer functional interface:
import composer from torchvision.models import resnet50 my_model = resnet50() # Apply weight standardization to model my_model = composer.functional.weight_standardization(my_model)
Please see the Weight Standardization Method Card for more details.
Bug Fixes
- Fix for checkpoints not being saved automatically at the end of a run (#1552)
- Fix Onnx export for Composer HuggingFaceModels (#1557)
- Fix for MIoU metric producing NaN's (#1558)
- CometML logger documentation updates and fixes (#1567, #1570, #1571)
- WandB image visualizer fix (#1591)
What's Changed
- Update evaluate_periodically() when eval interval is of type Duration by @karan6181 in #1523
- Quality of life updates to EMA by @coryMosaicML in #1524
- Add ADE20K and COCO v2 dataset behind a version flag by @karan6181 in #1528
- Pinned setuptools version to fix distutils version error by @karan6181 in #1536
- Less strict name formatting by @hanlint in #1535
- Defaulting streaming dataset version to 1 and add a deprecation warning by @karan6181 in #1532
- Changing 'stable' to 'latest' in notebooks in examples by @bcui19 in #1534
- Bump furo from 2022.6.21 to 2022.9.15 by @dependabot in #1540
- Bump fasteners from 0.17.3 to 0.18 by @dependabot in #1538
- Add Pandoc to Docker images, bump version to 2.19.2 by @bandish-shah in #1550
- Removed streaming version 2 from yaml since version 1 is default by @karan6181 in #1551
- Bump ipykernel from 6.15.2 to 6.15.3 by @dependabot in #1548
- Bump yamllint from 1.27.1 to 1.28.0 by @dependabot in #1546
- Bump traitlets from 5.3.0 to 5.4.0 by @dependabot in #1539
- Object Store Logger Race Condition + EMA Fix by @mvpatel2000 in #1552
- Adding in erroring for when using GradMonitor and DeepSpeed by @bcui19 in #1555
- Bump pypandoc from 1.8.1 to 1.9 by @dependabot in #1559
- Update context to raise errror by @mvpatel2000 in #1561
- Fix MIoU metric when
self.total_union==0
by @abhi-mosaic in #1558 - Move dataloader
initialize_object
to factory methods by @hanlint in #1510 - Weight Standardization method by @Landanjs in #1562
- Update comet links to include query params and point to main site by @dakinggg in #1567
- remove dead line in alibi by @mvpatel2000 in #1568
- GLU Fixes by @mvpatel2000 in #1564
- Add FSDP strategy by @abhi-mosaic in #1553
- Comet example by @dakinggg in #1570
- Add missing _enabled flag, post_close, and clean up comet ml tests by @dakinggg in #1571
- Consistent Method Card Style by @growlix in #1407
- add missing return in context by @mvpatel2000 in #1574
- Remove eval batch split by @mvpatel2000 in #1576
- Fix Onnx Export for Composer HuggingFaceModels by @nik-mosaic in #1557
- Revert checkpoint rename by @hanlint in #1579
New Contributors
Full Changelog: v0.10.0...v0.10.1
v0.10.0
π Composer v0.10.0
Composer v0.10.0 is out! This latest release adds support for CometML Experiment tracking, automatic selection of evaluation batch size, API enhancements for Evaluation/Logging/Metrics and a preview of our new streaming datasets repository!
pip install --upgrade mosaicml==0.10.0
New Features
-
βοΈ Comet Experiment Tracking (#1490)
We've added support for the popular Comet experiment tracker! To enable, simply create the logger and pass it to the
Trainer
object at initialization:from composer import Trainer from composer.loggers import CometMLLogger cometml_logger = CometMLLogger() trainer = Trainer( ... loggers=[cometml_logger], )
Please see our Logging and CometMLLogger docs pages for details on usage.
-
πͺ Automatic Evaluation Batch Size Selection (#1417)
Composer now supports
eval_batch_size='auto'
, which will choose the right evaluation batch size to avoid CUDA OOMs! Now, in conjunction withgrad_accum='auto'
, you can run the same code on any hardware with no changes necessary. This makes it easy to add evaluation to a training script without having to pick and choose the right batch sizes to avoid CUDA OOMs. -
π― Evaluation API Changes (#1479)
The Evaluation API has been updated to be consistent with the Trainer API. If the
eval_dataloader
was provided to the Trainer during initialization,eval
can be invoked without needing to provide anything additional:trainer = Trainer( eval_dataloader=... ) trainer.eval()
Alternatively, the
eval_dataloader
can be passed directly to theeval()
method:trainer = Trainer( ... ) trainer.eval( eval_dataloader=... )
The
eval_dataloader
can be a pytorch dataloader, or for multiple metrics, a list ofEvaluator
objects. -
πͺ΅ Simplified Logging (#1416)
We've significantly simplified our internal logging interface:
- Removed the use of
LogLevel
throughout the logging, which was a mostly unused feature. Filtering logs are the responsibility of the logger. - For better compatibility with external logging interfaces such as CometML or Weights & Biases, loggers now support the following methods:
log_metrics
,log_hyperparameters
, andlog_artifacts
. Previous calls todata_fit, data_epeoch, ..
have been removed.
- Removed the use of
-
π― validate --> eval_forward (#1411 , #1419)
Previously,
ComposerModel
implemented thevalidate(batch: Any) -> Tuple[Any, Any]
method which returns an(input, target)
tuple, and the Trainer handles updating the metrics. Inv0.10
, we return the metrics updating control to the user.Now, models instead implement
def eval_forward(batch: Any)
which returns the outputs of evaluation, and alsodef update_metric(batch, outputs, metric)
which updates the metric.An example implementation for classification can be found in our
ComposerClassifer
base class:def update_metric(self, batch: Any, outputs: Any, metric: Metric) -> None: _, targets = batch metric.update(outputs, targets) def eval_forward(self, batch: Any, outputs: Optional[Any] = None) -> Any: return outputs if outputs is not None else self.forward(batch)
-
π΅οΈββοΈ Evaluator changes
The
Evaluator
class now stores evaluation metric names instead of metric instances. For example:glue_mrpc_task = Evaluator( label='glue_mrpc', dataloader=mrpc_dataloader, metric_names=['BinaryF1Score', 'Accuracy'] )
These metric names are matched against the metrics returned by the
ComposerModel
. The metric instances are now stored as deep copies in theState
class asstate.train_metrics
orstate.eval_metrics
. -
π§ Streaming Datasets Repository Preview
We're in the process of splitting out streaming datasets into it's own repository! Streaming datasets is a high-performance drop-in replacement for Torch
IterableDataset
objects and enables you to stream your training data from cloud based object stores. For an early preview, please checkout the Streaming repo. -
β YAHP deprecation
We are deprecating support for yahp, our hyperparameter configuration tool. Support for this will be removed in the following minor version release of Composer. We recommend users migrate to OmegaConf, or Hydra as tools.
Bug Fixes
- Documentation fixes (#1408, #1422, #1425, #1413, #1432, #1403, #1426, #1396, #1446, #1466, #1443)
- Upgrade WandB version (#1440)
- fix import (#1442)
- fix wrong extra deps group (#1449)
- wandb bug fix (#1488)
- Reset train metrics every batch (#1496)
- fix auto grad accum (#1515)
- Fix compression file remote download exception handling (#1526)
- Add Pandoc to Docker images, bump version to 2.19.2 (#1550)
What's Changed
- current metrics docs by @A-Jacobson in #1402
- merge nlp+hf notebooks by @A-Jacobson in #1406
- Add break epoch exception by @mvpatel2000 in #1415
- Upgrade to torch 1.12.1 by @abhi-mosaic in #1409
- Metrics refactor pt1 by @ishanashastri in #1411
- Use state algos by @mvpatel2000 in #1412
- Add default ignore index by @moinnadeem in #1421
- Update default hparams for ResNet model card by @abhi-mosaic in #1423
- update colout link in custom speedup notebook by @A-Jacobson in #1408
- Clean up prose in key files by @dblalock in #1422
- Relax codeowners by @bandish-shah in #1424
- Fix typo by @Landanjs in #1425
- Fix pre-commit checks failing on fresh checkout of dev by @dblalock in #1414
- Have docs use preferred import paths, not longest import paths by @dblalock in #1413
- Fix missing indent by @Landanjs in #1432
- eval_batch_size=auto by @mvpatel2000 in #1417
- Simplify helper for conflicting files by @hanlint in #1427
- add install from dev instructions by @A-Jacobson in #1403
- Style/tone consistency update for tutorial notebooks by @alextrott16 in #1426
- Dynamic quantization + minor improvements in inference APIs by @dskhudia in #1433
- Upgrade WandB version by @moinnadeem in #1440
- Log multiple losses by @Landanjs in #1375
- Fix attribute by @mvpatel2000 in #1442
- Expand evaluation doc by @alextrott16 in #1396
- Metrics Refactor Part 2 by @ishanashastri in #1419
- Create dependabot.yml by @mvpatel2000 in #1448
- Methods overview fix by @growlix in #1446
- Bump custom-inherit from 2.3.2 to 2.4.0 by @dependabot in #1451
- Bump junitparser from 2.4.3 to 2.8.0 by @dependabot in #1453
- Update moto[s3] requirement from <3.2,>=3.1.12 to >=4.0.1,<5 by @dependabot in #1450
- Update monai requirement from <0.9,>=0.8.0 to >=0.9.0,<0.10 by @dependabot in #1452
- Update torch-optimizer requirement from <0.2,>=0.1.0 to >=0.3.0,<0.4 by @dependabot in #1454
- Bump cryptography from 37.0.2 to 37.0.4 by @dependabot in #1457
- Bump sphinxext-opengraph from 0.6.1 to 0.6.3 by @dependabot in #1458
- Bump coverage[toml] from 6.3.2 to 6.4.4 by @dependabot in #1460
- Bump nbsphinx from 0.8.8 to 0.8.9 by @dependabot in #1459
- Fix incorrect deps group in
streaming
requirement by @hanlint in #1449 - Logger Destination Refactor by @eracah in #1416
- Bump sphinx-markdown-tables from 0.0.15 to 0.0.17 by @dependabot in #1463
- Bump traitlets from 5.1.1 to 5.3.0 by @dependabot in #1462
- Bump vit-pytorch from 0.27 to 0.35.8 by @dependabot in #1465
- Bump furo from 2022.3.4 to 2022.6.21 by @dependabot in #1467
- Bump ipykernel from 6.9.2 to 6.15.1 by @dependabot in #1470
- Bump pytest from 7.1.0 to 7.1.2 by @dependabot in #1469
- Bump sph...
v0.9.0
π Composer v0.9.0
Excited to share the release of Composer v0.9.0, which comes with an Inference Export API, beta support for Apple Silicon and TPU training, as well as expanded usability of NLP-related speed-up methods. This release includes 175 commits from 34 contributors, including 10 new contributors π !
pip install --upgrade mosaicml==0.9.0
Alternatively, install Composer with Conda:
conda install -c mosaicml mosaicml=0.9.0
New Features
-
π¦ Export for inference APIs
Train with Composer and deploy anywhere! We have added a dedicated export API as well as an export training callback to allow you to export Composer-trained models for inference, supporting popular formats such as torchscript and ONNX.
For example, hereβs how to export a model in torchscript format:
from composer.utils import export_for_inference # Invoking export with a trained model export_for_inference(model=model, save_format='torchscript', save_path=model_save_path)
Hereβs an example of using the training callback, which automatically exports the model at the end of training to ONNX format:
from composer.callbacks import ExportForInferenceCallback # Initializing Trainer with the export callback callback = ExportForInferenceCallback(save_format='onnx', save_path=model_save_path) trainer = Trainer(model=model, callbacks=callback, train_dataloader=dataloader, max_duration='10ep') # Model will be exported at the end of training trainer.fit()
Please see our Exporting for Inference notebook for more information.
-
π ALiBi support for BERT training
You can now use ALiBi (Attention with Linear Biases; Press et al., 2021) when training BERT models with Composer, delivering faster training and higher accuracy by leveraging shorter sequence lengths.
ALiBi improves the quality of BERT pre-training, especially when pre-training uses shorter sequence lengths than the downstream (fine-tuning) task. This allows models with ALiBi to reach higher downstream accuracy with less pre-training time.
Example of using ALiBi as an algorithm with the Composer Trainer:
# Create an instance of a BERT masked language model model = composer.models.create_bert_mlm() # Apply ALiBi (when training is initialized) alibi = composer.algorithms.alibi(max_sequence_length=1024) # Train with ALiBi trainer = composer.trainer.Trainer( model=model, train_dataloader=train_dataloader, algorithms=[alibi] ) trainer.fit()
Example using the Composer Functional API:
import composer.functional as cf # Create an instance of a BERT masked language model model = composer.models.create_bert_mlm() # Apply ALiBi and expand the model's maximum sequence length to 1024 cf.apply_alibi(model=model, max_sequence_length=1024)
AliBi can also now be extended to work with custom models by registering your attention and embedding layers. Please see our ALiBi method card for more information.
-
π§ Entry point for GLUE tasks pre-training and fine-tuning
You can now easily pre-train and fine-tune NLP models across all GLUE (General Language Understanding Evaluation) tasks through one simple entry point! The entry point handles model saving and loading, spawns GLUE tasks in parallel across all available GPUs, and delivers a highly efficient evaluation of model performance.
Example of launching the entrypoint:
# This runs pre-training followed by fine-tuning. # --training_scheme can take either pretrain, finetune, or all depending on the task! python run_glue_trainer.py -f glue_example.yaml --training_scheme all
Please see our GLUE entrypoint notebook for more information.
-
π€ TPU support (in beta)
You can now use Composer to train your models on TPUs! Support is now available in Beta, and currently only supports single-core TPU training. Try it out, explore optimizations, and share your feedback and feature requests with us so we can make it better for you and for the community.
To use TPUs with Composer, simply specify a
tpu
device:# Set device to `tpu` trainer = composer.trainer.Trainer( model=model, train_dataloader=train_dataloader, max_duration=train_epochs, device='tpu') # Run fit trainer.fit()
Please see our Training with TPUs notebook for more information.
-
π Apple Silicon support (beta)
Leverage Apple Silicon chips to train your models with Composer by providing the
device='mps'
argument:trainer = Trainer( ..., device='mps' )
We use the latest PyTorch MPS backend to execute the training. This requires torch version β₯1.12, and Max OSX 12.3+.
For more information on training with Apple M chips, see the PyTorch 1.12 blog and our API Reference for Composer specific details.
-
π§ Contrib repository
Got a new method idea, or published a paper and want those methods to be easily accessible? Weβve created the
mcontrib
repository, with a lightweight process to contribute new algorithms. Weβre happy to work directly with you to benchmark these methods and eventually βpromoteβ them to Composer for use by end customers.Please checkout the README for details on how to contribute a new algorithm. For more details on how to write speed-up methods, see our notebook on custom speed-up methods.
Additional API Changes
-
π’ Passes Module
The order in which algorithms are run matters significantly during composition. With this release we refactored algorithm passes into their own
passes
module. Users can now register custom passes (for custom algorithms) with the Engine. Please see #1377 for more information. -
ποΈ Default Checkpoint Extension
The CheckpointSaver now defaults to using the
*.pt
extension for checkpoint fienames. Please see #1370 for more information. -
ποΈ Models Refactor
Most vision models (ResNet, MNIST, ViT, EfficientNet) have been refactored from classes to a factory function. For example
ComposerResNet
->composer_resnet
.# before from composer.models import ComposerResNet model = ComposerResNet(..) from composer.models import composer_resnet # after model = composer_resnet(..)
The same refactor has been done for NLP as well, e.g.
BERTModel
->create_bert_mlm
andcreate_bert_classification
. -
β Misc API Changes
BreakEpochException
has been removed.state.is_model_deepspeed
has been moved tocomposer.utils.is_model_deepspeed
.- Helper function
monitored_barrier
has been added tocomposer
distributed.
Bug Fixes
- Add informative error for infer batch size issues (#1401)
- Fix ImagenetDatasetHparams bug (#1392), resolves #1111
- Fix hparams error condition checking (#1394)
- Fix AMP resumption with grad scaler (#1376)
- Auto Grad Accum Cache Clearing (#1380), fixes issue reported in #1331
- Fix default precision (#1369)
- Fix the profiler on multi-node training (#1358), resolves #1270
- Retry SFTP on Size Mismatch (#1300)
- Fix scheduler edge cases (#1350), resolves #1077
- Fix a race condition in the object store logger (#1328)
- Fix WandB load from checkpoint (#1326)
- Fix Notebook Progress Bars (#1313)
Commits
What's Changed
- Fix DeepSpeed typo in docstring by @abhi-mosaic in #1188
- Move grad_accum logging to every step by @coryMosaicML in #1187
- Update STYLE_GUIDE with details on Documentation by @bandish-shah in #1183
- ProgressBar Units by @hanlint in #1190
- Added Xavier Normal initializer by @vladd-i in #1196
- Updated cost figure by @nqn in #1180
- Remove algorithm yamls by @hanlint in #1193
- Fix the Composer Launch Script for the C...
v0.8.2
π Composer v0.8.2
Composer v0.8.2 is released! Install via pip
:
pip install --upgrade mosaicml==0.8.2
Alternatively, install Composer with Conda:
conda install -c mosaicml mosaicml=0.8.2
π Bug Fixes
-
Fixed Notebook Progress Bars in Colab
Fixes a bug introduced by #1264 which causes Composer running in Colab notebooks to error out with:
UnsupportedOperation: fileno.
Changelog
v0.8.1
π Composer v0.8.1
Composer v0.8.1 is released! Install via pip
:
pip install --upgrade mosaicml==0.8.1
Alternatively, install Composer with Conda:
conda install -c mosaicml mosaicml=0.8.1
π New Features
-
πΌοΈ Image Visualizer
The
ImageVisualizer
callback periodically logs the training and validation images when using the WandB logger. This is great for validating your dataloader pipeline, especially if extensive data augmentations are used. Also, when training on a semantic segmentation task, the callback can log the target segmentation mask and the predicted segmentation mask by setting the argumentmode='segmentation'
. See PR #1266 for more details. Here is an example of using theImageVisualizer
callback:from composer import Trainer from composer.callbacks import ImageVisualizer # Callback to log 8 training images after every 100 batches image_visualizer = ImageVisualizer() # Construct trainer trainer = Trainer( ..., callbacks=image_visualizer ) # Train! trainer.fit()
Here is an example visualization from the training set of ADE20k:
-
πΆ TensorBoard Logging
You can now log metrics and losses from your Composer training runs with Tensorboard! See #1250 and #1283 for more details. All you have to do is create a
TensorboardLogger
object and add it
to the list of loggers in yourTrainer
object like so:from composer import Trainer from composer.loggers import TensorboardLogger tb_logger = TensorboardLogger(log_dir="./my_tensorboard_logs") trainer = Trainer( ... # Add your Tensorboard Logger to the trainer here. loggers=[tb_logger], ) trainer.fit()
For more information, see this tutorial.
-
π Multiple Losses
Adds support for multiple losses. If a model returns a tuple of losses, they are summed before the
loss.backward()
call. See #1240 for more details. -
ποΈ Stream Datasets from HTTP URIs
You can now specify a HTTP URI for a Streaming Dataset remote. See #1258 for more detials. For example:
from composer.datasets.streaming import StreamingDataset from torch.utils.data import DataLoader # Construct the Dataset dataset = StreamingDataset( ..., remote="https://example.com/dataset/", ) # Construct the DataLoader train_dl = DataLoader(dataset) # Construct the Trainer trainer = Trainer( ..., train_dataloader=train_dl, ) # Train! trainer.fit()
For more information on streaming datasets, see this tutorial.
-
ποΈ GPU Devices default to TF32 Matmuls
Beginning with PyTorch 1.12, the default behavior for computing FP32 matrix multiplies on NVIDIA Ampere devices was switched from TF32 to FP32. See PyTorch documentation here.
Since Composer is designed specifically for ML training with a focus on efficiency, we choose to preserve the old default of using TF32 on Ampere devices. This leads to significantly higher throughput when training in single precision, without impact training convergence. See PR #1275 for implementation details.
-
π Set the Device ID for GPU Devices
Specify the device ID within a DeviceGPU to train on when instantiating a Trainer object instead of using the local ID! For example,
from composer.trainer.devices.device_gpu import DeviceGPU # Specify to use GPU 3 to train device = DeviceGPU(device_id=3) # Construct the Trainer trainer = Trainer( ..., device = device ) # Train! trainer.fit()
-
BERT and C4 Updates
We make some minor adjustments to our
bert-base-uncased.yaml
training config. In particular, we make the global train and eval batch sizes a power of 2. This maintains divisibility when using many GPUs in multi-node training. We also adjust themax_duration
so that it converts cleanly to 70,000 batches.We also upgrade our StreamingDataset C4 conversion script (
scripts/mds/c4.py
) to use a multi-threaded reader. On a 64-core machine we are able to convert the 770GB train split to.mds
format in ~1.5hr. -
π Set a
prefix
when using aS3ObjectStore
When using
S3ObjectStore
for applications like checkpointing, it can be useful to provide path prefixes, mimickingfolder/subfolder
directories like on a local filesystem. Whenprefix
is provided, any objects uploaded withS3ObjectStore
will be stored atf's3://{self.bucket}/{self.prefix}{object_name}'
. -
βοΈ Scale the Warmup Period of Composer Schedulers
Added a new flag
scale_warmup
to schedulers that will scale the warmup period when a scale schedule ratio is applied. Default isFalse
to mirror default behavior. See #1268 for more detials. -
π§ Stochastic Depth on Residual Blocks
Residual blocks are detected automatically and replaced with stochastic versions. See #1253 for more details.
π Bug Fixes
-
Fixed Progress Bars
Fixed a bug where the the Progress Bars jumped around and did not stream properly when tailing the terminal over the network. Fixed in #1264, #1287, and #1289.
-
Fixed S3ObjectStore in Multithreaded Environments
Fixed a bug where the
boto3
crashed when creating the default session in multiple threads simultaniously (see boto/boto3#1592). Fixed in #1260. -
Retry on
ChannelException
errors in theSFTPObjectStore
Catch
ChannelException
SFTP transient error and retry. Fixed in #1245. -
Treating S3 Permission Denied Errors as Not Found Errors
We update our handling of
botocore
403 ClientErrors to interpret them asFileNotFoundErrors
. We do this because of a situation that occurs when a user has no S3 credentials configured, and tries to read from a bucket with public files. For privacy, Amazon S3 raises 403 (Permission Denied) instead of 404 (Not Found) errors. As such, PR #1249 treats 403 ClientErrors as FileNotFoundErrors. -
Fixed Parsing of
grad_accum
in theTrainerHparams
Fixes an error where the command line override
--grad_accum
lead to incorrect parsing. Fixed in #1256. -
Fixed Example YAML Files
Our recipe configurations (YAML) are updated to the latest version, and a test was added to enforce correctness moving forward. Fixed in #1235 and #1257.
Changelog
v0.8.0
π Composer v0.8.0
Composer v0.8.0 is released! Install via pip
:
pip install --upgrade mosaicml==0.8.0
Alternatively, install Composer with Conda:
conda install -c mosaicml mosaicml=0.8.0
New Features
-
π€ HuggingFace ComposerModel
Train your HuggingFace models with Composer! We introduced a
HuggingFaceModel
that converts your existing π€ Transformers models into a ComposerModel.For example:
import transformers from composer.models import HuggingFaceModel # Define the model hf_model = transformers.AutoModelForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2) # Convert it into a ComposerModel model = HuggingFaceModel(hf_model) # Construct the trainer trainer = Trainer( ..., model, ) # Train! trainer.fit()
For more information, see the example on fine-tuning a pretrained BERT with Composer.
-
π« Fused Layer Norm
Fused LayerNorm replaces implementations of
torch.nn.LayerNorm
with aapex.normalization.fused_layer_norm
. The fused kernel provides increased GPU utilization.For example:
from composer.trainer import Trainer from composer.algorithms import FusedLayerNorm # Initialize the algorithm alg = FusedLayerNorm() # Construct the trainer trainer = Trainer( algorithms=alg, ) # Train! trainer.fit()
See the method card for more information.
-
πΎ Ignore Checkpoint Parameters
If you have a checkpoint and don't want to restore some elements of the chceckpoint to the state, we added a
load_ignore_keys
parameter. Any specified (nested) keys will be ignored. Glob syntax is supported!For example, to restore a checkpoint without the seed:
from composer import Trainer trainer = Trainer( ..., load_path="path/to/my/checkpoint.pt", load_ignore_keys=["state/rank_zero_seed", "rng"], )
See the Trainer API Reference for more information.
-
πͺ£ Object Stores
Composer v0.8.0 introduces an abstract Object Store API to support multiple object store drivers, such as boto3 (for Amazon S3) and Paramiko (for SFTP), in addition to the existing libcloud implementation.
For example, if you are training on AWS where credentials are available in the environment, here's how to to save checkpoints to a S3 object store via Boto3.
from composer import Trainer from composer.loggers import ObjectStoreLogger from composer.utils.object_store import S3ObjectStore logger = ObjectStoreLogger( object_store_cls=S3ObjectStore, object_store_kwargs={ # These arguments will be passed into the S3ObjectStore -- e.g.: # object_store = S3ObjectStore(**object_store_kwargs) # Refer to the S3ObjectStore class for documentation 'bucket': 'my-bucket', }, ) trainer = Trainer( ..., loggers=logger, ) # Train! trainer.fit()
See the Object Store API Reference for more information.
-
πͺ¨ Artifact Metadata
Composer automatically logs the epoch, batch, sample, and token counts as metadata when storing artifacts in Weights & Biases. See the API Reference for more information.
API Changes
-
βοΈ Gradient Clipping is now an Algorithm
To clean up the Trainer, we moved gradient clipping into an Algorithm. The
grad_clip_norm
argument in the Trainer is deprecated and will be removed in a future version of Composer. Instead, use the Gradient Clipping algorithm:For example:
from composer.algorithms import GradientClipping from composer.trainer import Trainer # Configure gradient clipping gradient_clipping = GradientClipping() # Configure the trainer trainer = Trainer( ..., algorithms=gradient_clipping, ) # Train! trainer.fit()
See the method card for more information.
-
ποΈ Removed
batch_num_samples
andbatch_num_tokens
from the state.State properties
batch_num_samples
andbatch_num_tokens
have been removed.
Instead, useState.timestamp
for token and sample tracking. -
π§βπ€βπ§ DDP Sync Strategy
We changed the default DDP Sync Strategy to
MULTI_AUTO_SYNC
, asFORCED_SYNC
doesn't work with all algorithms. -
π Moved the
run_name
into theState
The
run_name
has been added to the State object, so it is persisted with checkpoints. It has been removed from the Logger.
Bug Fixes
- In the Object Store Logger, added in retries for credential validation, and validating credentials only on global rank zero. (#1144)
- Fixed a bug in the speed monitor where it returned negative wall clock times. (#1123)
- Fixed how block-wise Stochastic Depth could freeze the trainer. (#1087)
- Fixed a bug in the [MLPerfCallback] where sample counts were incorrect on per-sharded datasets. (#1156)
Changelog
v0.7.1
π Composer v0.7.1
Composer v0.7.1 is released! Install via pip
:
pip install --upgrade mosaicml==0.7.1
Alternatively, install Composer with Conda:
conda install -c mosaicml mosaicml=0.7.1
Bug Fixes
- Upgraded
wandb>=0.12.17
, to fix incompatibility with protobuf >= 4 (wandb/wandb#3709)