Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transformers doc fixes #1578

Merged
merged 3 commits into from
May 25, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 6 additions & 8 deletions integrations/huggingface-transformers/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,9 @@ Once trained, SparseML enables you to export models to the ONNX format, such tha
Install with `pip`:

```bash
pip install sparseml[torch]
pip install sparseml[transformers]
```

**Note**: Transformers will not immediately install with this command. Instead, a sparsification-compatible version of Transformers will install on the first invocation of the Transformers code in SparseML.

## **Tutorials**

- [Sparse Transfer Learning with the Python API](tutorials/sparse-transfer-learning-bert-python.md) [**RECOMMENDED**]
Expand All @@ -50,13 +48,13 @@ pip install sparseml[torch]
### **Use Case Examples - Python**

- [Sparse Transfer with GLUE Datasets (SST2) for sentiment analysis](tutorials/sentiment-analysis/docs-sentiment-analysis-python-sst2.ipynb)
- [Sparse Transfer with Custom Datasets (RottenTomatoes) and Custom Teacher from HF Hub for sentiment analysis](tutorials/sentiment-analysis/docs-sentiment-analysis-python-custom-teacher-rottentomatoes)
- [Sparse Transfer with Custom Datasets (RottenTomatoes) and Custom Teacher from HF Hub for sentiment analysis](tutorials/sentiment-analysis/docs-sentiment-analysis-python-custom-teacher-rottentomatoes.ipynb)
- [Sparse Transfer with GLUE Datasets (QQP) for multi-input text classification](tutorials/text-classification/docs-text-classification-python-qqp.ipynb)
- [Sparse Transfer with Custom Datasets (SICK) for multi-input text classification](tutorials/text-classification/docs-text-classification-python-sick.ipynb)
- [Sparse Transfer with Custom Datasets (TweetEval) and Custom Teacher for single input text classificaiton](tutorials/text-classification/docs-text-classification-python-custom-teacher-tweeteval.ipynb)
- [Sparse Transfer with Custom Datasets (GoEmotions) for multi-label text classification](tutorials/text-classification/docs-text-classification-python-multi-label-go_emotions.ipynb)
- [Sparse Transfer with Conll2003 for named entity recognition](tutorials/token-classification/docs-token-classification-python-conll2003.ipynb)
- [Sparse Transfer with Custom Datasets (WNUT) and Custom Teacher for named entity recognition](tutorials/token-classification/docs-token-classification-custom-teacher-wnut.ipynb)
- [Sparse Transfer with Custom Datasets (WNUT) and Custom Teacher for named entity recognition](tutorials/token-classification/docs-token-classification-python-custom-teacher-wnut.ipynb)
- Sparse Transfer with SQuAD (example coming soon!)
- Sparse Transfer with Squadshifts Amazon (example coming soon!)

Expand All @@ -66,7 +64,7 @@ pip install sparseml[torch]

SparseZoo is an open-source repository of pre-sparsified models, including BERT-base, BERT-large, RoBERTa-base, RoBERTa-large, and DistillBERT. With SparseML, you can fine-tune these pre-sparsified checkpoints onto custom datasets (while maintaining sparsity) via sparse transfer learning. This makes training inference-optimized sparse models almost identical to your typical training workflows!

[Check out the available models](https://sparsezoo.neuralmagic.com/?repo=huggingface&page=1)
[Check out the available models](https://sparsezoo.neuralmagic.com/?repos=huggingface)

### **Recipes**

Expand Down Expand Up @@ -140,15 +138,15 @@ Currently supported tasks include:

Sparse Transfer is very similiar to the typical transfer learing process used to train NLP models, where we fine-tune a checkpoint pretrained on a large upstream dataset using masked language modeling onto a smaller downstream dataset. With Sparse Transfer Learning, however, we simply start the fine-tuning process from a pre-sparsified checkpoint and maintain sparsity while the training process occurs.

Here, we will fine-tune a [90% pruned version of BERT](https://sparsezoo.neuralmagic.com/models/nlp%2Fmasked_language_modeling%2Fobert-base%2Fpytorch%2Fhuggingface%2Fwikipedia_bookcorpus%2Fpruned90-none) from the SparseZoo onto SST2.
Here, we will fine-tune a [90% pruned version of BERT](https://sparsezoo.neuralmagic.com/models/obert-base-wikipedia_bookcorpus-pruned90?comparison=obert-base-wikipedia_bookcorpus-base) from the SparseZoo onto SST2.

### **Kick off Training**

We will use SparseML's `sparseml.transformers.text_classification` training script.

To run sparse transfer learning, we first need to create/select a sparsification recipe. For sparse transfer, we need a recipe that instructs SparseML to maintain sparsity during training and to quantize the model.

For the SST2 dataset, there is a [transfer learning recipe available in SparseZoo](https://sparsezoo.neuralmagic.com/models/nlp%2Fsentiment_analysis%2Fobert-base%2Fpytorch%2Fhuggingface%2Fsst2%2Fpruned90_quant-none), identified by the following SparseZoo stub:
For the SST2 dataset, there is a [transfer learning recipe available in SparseZoo](https://sparsezoo.neuralmagic.com/models/obert-base-sst2_wikipedia_bookcorpus-pruned90_quantized?comparison=obert-base-sst2_wikipedia_bookcorpus-base&tab=0), identified by the following SparseZoo stub:
```
zoo:nlp/sentiment_analysis/obert-base/pytorch/huggingface/sst2/pruned90_quant-none
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ In this tutorial, you will learn how to:
Install SparseML via `pip`:

```bash
pip install sparseml[torch]
pip install sparseml[transformers]
```

## **Sparse Transfer Learning onto SQuAD**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
},
"outputs": [],
"source": [
"!pip install sparseml[torch]"
"!pip install sparseml[transformers]"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
},
"outputs": [],
"source": [
"!pip install sparseml[torch]"
"!pip install sparseml[transformers]"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ In this tutorial, you will learn how to:
Install SparseML via `pip`:

```bash
pip install sparseml[torch]
pip install sparseml[transformers]
```

## Sparse Transfer Learning onto SST2 (GLUE Task)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ SparseZoo contains pre-sparsified checkpoints of common NLP models like BERT and
Install via `pip`:

```
pip install sparseml[torch]
pip install sparseml[transformers]
```

## **Sparse Transfer Learning onto SST2**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ SparseZoo contains pre-sparsified checkpoints of common NLP models like BERT and
Install via `pip`:

```
pip install sparseml[torch]
pip install sparseml[transformers]
```

## **Example: Sparse Transfer Learning onto SST2**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
},
"outputs": [],
"source": [
"!pip install sparseml[torch]"
"!pip install sparseml[transformers]"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@
},
"outputs": [],
"source": [
"!pip install sparseml[torch]"
"!pip install sparseml[transformers]"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
},
"outputs": [],
"source": [
"!pip install sparseml[torch]"
"!pip install sparseml[transformers]"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
},
"outputs": [],
"source": [
"!pip install sparseml[torch]"
"!pip install sparseml[transformers]"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ In this tutorial, you will learn how to:
Install SparseML via `pip`:

```bash
pip install sparseml[torch]
pip install sparseml[transformers]
```

## SparseML CLI
Expand Down

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
},
"outputs": [],
"source": [
"!pip install sparseml[torch]"
"!pip install sparseml[transformers]"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ In this tutorial, you will learn how to:
Install SparseML via `pip`:

```bash
pip install sparseml[torch]
pip install sparseml[transformers]
```

## Sparse Transfer Learning onto Conll2003
Expand Down
Loading