[breaking|pipelines|tokenizers] Adding slow-fast tokenizers equivalence tests pipelines - Removing sentencepiece as a required dependency

[thomwolf]

What does this PR do?

Breaking: Auto-tokenizers and pipelines:

• switch to use_fast=True by default (Fast tokenizers by default)
  => The main expected breaking change is the handling of overflowing tokens which is different between slow and fast tokenizers.
• removing sentencepiece from the required dependencies (in some special case this may require you to install sentencepiece in addition to the normal install).

Pipelines:

• Add slow/fast tokenizers equivalence tests in pipelines
• upgrade QA/NER processing pipeline to handle fast tokenizers
• remove test_pipelines_dialog.py which was a duplicated test file

Tokenizers:

• Update and add new alignement method in BatchEncoding

Dependencies:

• upgrade to tokenizers==0.9.4 to allow QA processing with fast tokenizers
• remove sentencepiece from the required dependencies

Misc:

• Fix bug in RobertaFast and test for XLM-Prophetnet and RAG

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to the it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

[thomwolf]

On hold for now until we update the input/output of the alignement methods in tokenizers to handle pairs of input sentences following internal discussion with @n1t0 and @Narsil.

[thomwolf]

Remaining error on the CI (NER pipeline not working for slow tokenizers) should be solved by #8364

Edit: ok solved now that #8364 is merged

[stas00]

examples/seq2seq/finetune.py is failing after this PR:

CUDA_VISIBLE_DEVICES=0 pytest -sv examples/seq2seq/test_seq2seq_examples.py::TestTheRest::test_finetune_0_patrickvonplaten_t5_tiny_random

Traceback (most recent call last):
  File "finetune.py", line 442, in <module>
    main(args)
  File "finetune.py", line 409, in main
    trainer: pl.Trainer = generic_train(
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/lightning_base.py", line 398, in generic_train
    trainer.fit(model)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 444, in fit
    results = self.accelerator_backend.train()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 63, in train
    results = self.train_or_test()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 74, in train_or_test
    results = self.trainer.train()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 493, in train
    self.train_loop.run_training_epoch()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 554, in run_training_epoch
    for batch_idx, (batch, is_last_batch) in train_dataloader:
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pytorch_lightning/profiler/profilers.py", line 80, in profile_iterable
    value = next(iterator)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py", line 46, in _with_is_last
    last = next(it)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 519, in __next__
    data = self._next_data()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1169, in _next_data
    return self._process_data(data)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1195, in _process_data
    data.reraise()
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/_utils.py", line 428, in reraise
    raise self.exc_type(msg)
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/seq2seq/utils.py", line 251, in collate_fn
    batch_encoding: Dict[str, torch.Tensor] = self.tokenizer.prepare_seq2seq_batch(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/models/bart/tokenization_bart_fast.py", line 127, in prepare_seq2seq_batch
    model_inputs: BatchEncoding = self(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 2319, in __call__
    return self.batch_encode_plus(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 2504, in batch_encode_plus
    return self._batch_encode_plus(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/models/gpt2/tokenization_gpt2_fast.py", line 167, in _batch_encode_plus
    return super()._batch_encode_plus(*args, **kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/tokenization_utils_fast.py", line 433, in _batch_encode_plus
    return BatchEncoding(sanitized_tokens, sanitized_encodings, tensor_type=return_tensors)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 242, in __init__
    n_sequences = encoding[0].n_sequences
AttributeError: 'tokenizers.Encoding' object has no attribute 'n_sequences'

Exception ignored in: <function tqdm.__del__ at 0x7f60b55d7b80>
Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tqdm/std.py", line 1128, in __del__
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tqdm/std.py", line 1341, in close
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tqdm/std.py", line 1520, in display
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tqdm/std.py", line 1131, in __repr__
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/tqdm/std.py", line 1481, in format_dict
TypeError: cannot unpack non-iterable NoneType object

[LysandreJik]

Hi @stas00, it seems you don't have the latest tokenizer version!

[stas00]

Bummer! Thank you for identifying that, @LysandreJik - I confirm updating tokenizers fixes it.

Since this happened more than once now, would it be possible to add and maintain run-time checks as we have done here:

transformers/examples/lightning_base.py

Lines 38 to 47 in dd52804

	def require_min_ver(pkg, min_ver):
	got_ver = pkg_resources.get_distribution(pkg).version
	if packaging.version.parse(got_ver) < packaging.version.parse(min_ver):
	logger.warning(
	f"{pkg}>={min_ver} is required for a normal functioning of this module, but found {pkg}=={got_ver}. "
	"Try: pip install -r examples/requirements.txt"
	)
	require_min_ver("pytorch_lightning", "1.0.4")

How a developer is to know that they need to update a dependency of a project otherwise? git pull doesn't trigger pip install -e ".[dev]"

Except the code above should emit an error on failed check. It should be an error in lightning_base.py too, but I couldn't convince @sshleifer to make it so. Warnings are plentiful and this approach of using a warning just doesn't serve its purpose, IMHO.

[sgugger]

I like the function (though we can expand its name and replace ver by version inside to make it more readable ;-) ) and I think we could have a dynamic check at init. I don't see any problem with throwing an error if the minimum version of tokenizers isn't installed, but maybe @LysandreJik or @patrickvonplaten have a different opinion.

[stas00]

Oh, any name works, it was just code first, then I thought that it'll eventually end up being a reusable function, so the half-baked function emerged.

I'm glad you agree that it should assert if the minimal requirements aren't met.

Technically we should then add packaging and pkg_resources to the project dependencies, but since they are prerequisites for setuptools - every user should already have them.

And we will need 2 versions of errors:

• Try: pip install -r examples/requirements.txt for examples,
• Try: pip install transformers -U" for the core.

[patrickvonplaten]

Agree! I actually also already ran into the same issue @stas00 -> I'm definitely in favor of such a function! Should we run the function at import transformers already ?

[LysandreJik]

So did I, and I thought everything was broken until @sgugger showed me the way. Your function would definitely help in that regard @stas00 :)

[stas00]

Should we run the function at import transformers already ?

Two ways I can see:

1. right where the corresponding import is done - so it's easier to see the specific requirement - but it could be scattered - but it could be more difficult to maintain. Ideally, python would have import module 1.2.3, like some other languages have, but it doesn't at the moment.
2. all in one place, at the very top of __init__.py, all requirements next to each other so it's easy to maintain. This check relies on packaging tools - i.e. derives it from the site-packages dir, so it shouldn't even load the module. i.e. we don't try to look for xxx.__version__ here, since not all packages have it.

I'd say (2) is the easier way.

Last night I was dreaming of a trigger feature, where if git sees setup.py modified it'd alert someone to update __init__.py requirements - but it was a dream.

[sgugger]

I think it can be in any file imported by the __init__ (as long as the function is executed), so we could also have this in file_utils.py. Though the __init__ is fine by me too if you think it's better.

[stas00]

Sure, then let's add a dedicated module then? It'd be the most simple/intuitive then

$ cat version_requirements.py
require("tokenizers", "1.2.3")
...

and __init__.py:

import .version_requirements

[stas00]

hear, hear! if we have such a file then we can even use it to feed setup.py! so we have a single place where we edit all the minimal version requirements and don't need to touch setup.py and potentially forget to sync requirements.

In which case I'd use a dict and feed it to require_version (or whatever we end up calling it).

Clearly setup has a lot of optional things, so perhaps then we load this file at the end of init and only check versions for the things that got loaded?

or we just test only the package names that we know we need to check, but use that dict for setup.py's needs.

Let me know if these ideas are an overkill.

[stas00]

Here is a quick prototype to what I'm thinking:

$ cat src/transformers/version_requirements.py
min_vers = dict(
    tokenizers: "==0.9.4",
    tqdm: ">=4.27",
    jaxlib: "==0.1.55",
)
run_time_keys = "tokenizers tqdm".split()
for k in run_time_keys:
    require_min_ver(k, min_vers[k])

$ cat setup.py
from version_requirements import min_vers
# of course we won't hardcode each entry - this is a just to demonstrate
extras["flax"] = [f"jaxlib{min_vers{'jax_lib']}", ...

so you can see the dictionary has all the versions, but we actively check only the versions that are non-optional.

[sgugger]

One downside of this is that it would move dependencies out of the setup.py (which is where people would expect to see them). Do you think there is a way to structure this so the one place we look at minimum version is the setup? It would be less surprising I think.

[stas00]

I agree.

We could have all the version requirements defined in setup.py and when it's run it'd update src/transformers/version_requirements.py instead. Then we would actually want 2 files under transformers - one that setup.py will maintain - it will be just a dict dump - so that it could overwrite the file completely and another for the selective run-time checks that would refer to the first file generated by setup, since we will only check a handful of these many dependencies at run time.

$ cat setup.py
min_vers = dict(
    tokenizers: "==0.9.4",
    tqdm: ">=4.27",
    jaxlib: "==0.1.55",
)

# add code to dump min_vers dict into `src/transformers/version_requirements.py`

# of course we won't hardcode each entry - this is a just to demonstrate
extras["flax"] = [f"jaxlib{min_vers{'jax_lib']}", ...


$ cat src/transformers/version_requirements.py
# AUTOGENERATED - MODIFY setup.py INSTEAD! #
min_vers = dict(
    tokenizers: "==0.9.4",
    tqdm: ">=4.27",
    jaxlib: "==0.1.55",
)

$ cat src/transformers/version_run_time_check.py
from .version_requirements import min_vers

# define which module versions we always want to check (only a few)
run_time_keys = "tokenizers tqdm".split()
for k in run_time_keys:
    require_min_ver(k, min_vers[k])

$ cat src/transformers/__init__.py
import .version_run_time_check

This is of course all just a visual prototype.

[sgugger]

If you want to tackle this, please go ahead with something along these lines. We can refine more on an actual PR.

[stas00]

OK, I will make a partial sub-set of modules and when you like how it looks expand it to all modules.

[Guitaricet]

Is there any way to workaround the version check?
I want to use some features from tokenizers 0.10, but Transformers raise VersionConflict.

Surely, this can cause some very non-obvious bugs, but at least I'll be able to work with my code before the new version of Transformers is released.

[stas00]

I think this is a very reasonable need, @Guitaricet. But it's probably best to discuss it in a dedicated issue. Could you please file a feature request and let's see what others would think?

I'd say an env var to override the checks should do the trick. Should be easy to add if the others agree with having it.