Skip to content

Commit

Permalink
Transformers doc fixes (#1578)
Browse files Browse the repository at this point in the history
  • Loading branch information
KSGulin committed May 26, 2023
1 parent 3dd3af9 commit 990fa3a
Show file tree
Hide file tree
Showing 15 changed files with 20 additions and 22 deletions.
14 changes: 6 additions & 8 deletions integrations/huggingface-transformers/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,9 @@ Once trained, SparseML enables you to export models to the ONNX format, such tha
Install with `pip`:

```bash
pip install sparseml[torch]
pip install sparseml[transformers]
```

**Note**: Transformers will not immediately install with this command. Instead, a sparsification-compatible version of Transformers will install on the first invocation of the Transformers code in SparseML.

## **Tutorials**

- [Sparse Transfer Learning with the Python API](tutorials/sparse-transfer-learning-bert-python.md) [**RECOMMENDED**]
Expand All @@ -50,13 +48,13 @@ pip install sparseml[torch]
### **Use Case Examples - Python**

- [Sparse Transfer with GLUE Datasets (SST2) for sentiment analysis](tutorials/sentiment-analysis/docs-sentiment-analysis-python-sst2.ipynb)
- [Sparse Transfer with Custom Datasets (RottenTomatoes) and Custom Teacher from HF Hub for sentiment analysis](tutorials/sentiment-analysis/docs-sentiment-analysis-python-custom-teacher-rottentomatoes)
- [Sparse Transfer with Custom Datasets (RottenTomatoes) and Custom Teacher from HF Hub for sentiment analysis](tutorials/sentiment-analysis/docs-sentiment-analysis-python-custom-teacher-rottentomatoes.ipynb)
- [Sparse Transfer with GLUE Datasets (QQP) for multi-input text classification](tutorials/text-classification/docs-text-classification-python-qqp.ipynb)
- [Sparse Transfer with Custom Datasets (SICK) for multi-input text classification](tutorials/text-classification/docs-text-classification-python-sick.ipynb)
- [Sparse Transfer with Custom Datasets (TweetEval) and Custom Teacher for single input text classificaiton](tutorials/text-classification/docs-text-classification-python-custom-teacher-tweeteval.ipynb)
- [Sparse Transfer with Custom Datasets (GoEmotions) for multi-label text classification](tutorials/text-classification/docs-text-classification-python-multi-label-go_emotions.ipynb)
- [Sparse Transfer with Conll2003 for named entity recognition](tutorials/token-classification/docs-token-classification-python-conll2003.ipynb)
- [Sparse Transfer with Custom Datasets (WNUT) and Custom Teacher for named entity recognition](tutorials/token-classification/docs-token-classification-custom-teacher-wnut.ipynb)
- [Sparse Transfer with Custom Datasets (WNUT) and Custom Teacher for named entity recognition](tutorials/token-classification/docs-token-classification-python-custom-teacher-wnut.ipynb)
- Sparse Transfer with SQuAD (example coming soon!)
- Sparse Transfer with Squadshifts Amazon (example coming soon!)

Expand All @@ -66,7 +64,7 @@ pip install sparseml[torch]

SparseZoo is an open-source repository of pre-sparsified models, including BERT-base, BERT-large, RoBERTa-base, RoBERTa-large, and DistillBERT. With SparseML, you can fine-tune these pre-sparsified checkpoints onto custom datasets (while maintaining sparsity) via sparse transfer learning. This makes training inference-optimized sparse models almost identical to your typical training workflows!

[Check out the available models](https://sparsezoo.neuralmagic.com/?repo=huggingface&page=1)
[Check out the available models](https://sparsezoo.neuralmagic.com/?repos=huggingface)

### **Recipes**

Expand Down Expand Up @@ -140,15 +138,15 @@ Currently supported tasks include:

Sparse Transfer is very similiar to the typical transfer learing process used to train NLP models, where we fine-tune a checkpoint pretrained on a large upstream dataset using masked language modeling onto a smaller downstream dataset. With Sparse Transfer Learning, however, we simply start the fine-tuning process from a pre-sparsified checkpoint and maintain sparsity while the training process occurs.

Here, we will fine-tune a [90% pruned version of BERT](https://sparsezoo.neuralmagic.com/models/nlp%2Fmasked_language_modeling%2Fobert-base%2Fpytorch%2Fhuggingface%2Fwikipedia_bookcorpus%2Fpruned90-none) from the SparseZoo onto SST2.
Here, we will fine-tune a [90% pruned version of BERT](https://sparsezoo.neuralmagic.com/models/obert-base-wikipedia_bookcorpus-pruned90?comparison=obert-base-wikipedia_bookcorpus-base) from the SparseZoo onto SST2.

### **Kick off Training**

We will use SparseML's `sparseml.transformers.text_classification` training script.

To run sparse transfer learning, we first need to create/select a sparsification recipe. For sparse transfer, we need a recipe that instructs SparseML to maintain sparsity during training and to quantize the model.

For the SST2 dataset, there is a [transfer learning recipe available in SparseZoo](https://sparsezoo.neuralmagic.com/models/nlp%2Fsentiment_analysis%2Fobert-base%2Fpytorch%2Fhuggingface%2Fsst2%2Fpruned90_quant-none), identified by the following SparseZoo stub:
For the SST2 dataset, there is a [transfer learning recipe available in SparseZoo](https://sparsezoo.neuralmagic.com/models/obert-base-sst2_wikipedia_bookcorpus-pruned90_quantized?comparison=obert-base-sst2_wikipedia_bookcorpus-base&tab=0), identified by the following SparseZoo stub:
```
zoo:nlp/sentiment_analysis/obert-base/pytorch/huggingface/sst2/pruned90_quant-none
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ In this tutorial, you will learn how to:
Install SparseML via `pip`:

```bash
pip install sparseml[torch]
pip install sparseml[transformers]
```

## **Sparse Transfer Learning onto SQuAD**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
},
"outputs": [],
"source": [
"!pip install sparseml[torch]"
"!pip install sparseml[transformers]"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
},
"outputs": [],
"source": [
"!pip install sparseml[torch]"
"!pip install sparseml[transformers]"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ In this tutorial, you will learn how to:
Install SparseML via `pip`:

```bash
pip install sparseml[torch]
pip install sparseml[transformers]
```

## Sparse Transfer Learning onto SST2 (GLUE Task)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ SparseZoo contains pre-sparsified checkpoints of common NLP models like BERT and
Install via `pip`:

```
pip install sparseml[torch]
pip install sparseml[transformers]
```

## **Sparse Transfer Learning onto SST2**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ SparseZoo contains pre-sparsified checkpoints of common NLP models like BERT and
Install via `pip`:

```
pip install sparseml[torch]
pip install sparseml[transformers]
```

## **Example: Sparse Transfer Learning onto SST2**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
},
"outputs": [],
"source": [
"!pip install sparseml[torch]"
"!pip install sparseml[transformers]"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@
},
"outputs": [],
"source": [
"!pip install sparseml[torch]"
"!pip install sparseml[transformers]"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
},
"outputs": [],
"source": [
"!pip install sparseml[torch]"
"!pip install sparseml[transformers]"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
},
"outputs": [],
"source": [
"!pip install sparseml[torch]"
"!pip install sparseml[transformers]"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ In this tutorial, you will learn how to:
Install SparseML via `pip`:

```bash
pip install sparseml[torch]
pip install sparseml[transformers]
```

## SparseML CLI
Expand Down

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
},
"outputs": [],
"source": [
"!pip install sparseml[torch]"
"!pip install sparseml[transformers]"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ In this tutorial, you will learn how to:
Install SparseML via `pip`:

```bash
pip install sparseml[torch]
pip install sparseml[transformers]
```

## Sparse Transfer Learning onto Conll2003
Expand Down

0 comments on commit 990fa3a

Please sign in to comment.