Clear cache upon version upgrade

[lapp0]

Fixes #561

Smoke test:

[nix-shell:~/p/outlines]$ git branch --show-current
clear-cache-upon-version-upgrade


[nix-shell:~/p/outlines]$ python3 -c "from outlines.fsm.fsm import RegexFSM; from outlines.models.transformers import TransformerTokenizer; RegexFSM('a', TransformerTokenizer('gpt2'))" && echo "success"
success


[nix-shell:~/p/outlines]$ git checkout benl/fsm-enhancements
[nix-shell:~/p/outlines]$ pip install . -q
[nix-shell:~/p/outlines]$ python3 -c "from outlines.fsm.fsm import RegexFSM; from outlines.models.transformers import TransformerTokenizer; RegexFSM('a', TransformerTokenizer('gpt2'))" && echo "success"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/andrew/p/outlines/outlines/fsm/fsm.py", line 120, in __init__
    self.states_to_token_maps, self.empty_token_ids = create_states_mapping(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: too many values to unpack (expected 2)


[nix-shell:~/p/outlines]$ git merge clear-cache-upon-version-upgrade
Merge made by the 'ort' strategy.

[nix-shell:~/p/outlines]$ pip install . -q
[nix-shell:~/p/outlines]$ python3 -c "from outlines.fsm.fsm import RegexFSM; from outlines.models.transformers import TransformerTokenizer; RegexFSM('a', TransformerTokenizer('gpt2'))" && echo "success"
success

[lapp0]

Didn't have the pytest-mocker pytest-mock dependency in pyproject.toml, I expect pytest to pass now.

[rlouf]

Thanks! Would you mind rebasing your branch on main to get rid of the merge commit?

[lapp0]

Errors are related to transformers. I'm thinking it's because I changed pyproject.toml which resulted in upgrading a few packages

tests/generate/test_integration_transfomers.py:20: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
outlines/models/transformers.py:220: in transformers
    model = AutoModelForCausalLM.from_pretrained(model_name, **model_kwargs)
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py:566: in from_pretrained
    return model_class.from_pretrained(
/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/transformers/modeling_utils.py:3527: in from_pretrained
    state_dict = load_state_dict(resolved_archive_file)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

checkpoint_file = '/home/runner/.cache/huggingface/hub/models--hf-internal-testing--tiny-random-GPTJForCausalLM/snapshots/1fff390baa45cb187903ebdd269c975bb9ed7386/pytorch_model.bin'

    def load_state_dict(checkpoint_file: Union[str, os.PathLike]):
        """
        Reads a PyTorch checkpoint file, returning properly formatted errors if they arise.
        """
        if checkpoint_file.endswith(".safetensors") and is_safetensors_available():
            # Check format of the archive
            with safe_open(checkpoint_file, framework="pt") as f:
                metadata = f.metadata()
            if metadata.get("format") not in ["pt", "tf", "flax"]:
                raise OSError(
                    f"The safetensors archive passed at {checkpoint_file} does not contain the valid metadata. Make sure "
                    "you save your model with the `save_pretrained` method."
                )
            return safe_load_file(checkpoint_file)
        try:
            if (
                is_deepspeed_zero3_enabled() and torch.distributed.is_initialized() and torch.distributed.get_rank() > 0
            ) or (is_fsdp_enabled() and not is_local_dist_rank_0()):
                map_location = "meta"
            else:
                map_location = "cpu"
            extra_args = {}
            # mmap can only be used with files serialized with zipfile-based format.
            if (
                isinstance(checkpoint_file, str)
                and map_location != "meta"
                and version.parse(torch.__version__) >= version.parse("2.1.0")
                and is_zipfile(checkpoint_file)
            ):
                extra_args = {"mmap": True}
            return torch.load(
                checkpoint_file,
                map_location=map_location,
                weights_only=is_torch_greater_or_equal_than_1_13,
                **extra_args,
            )
        except Exception as e:
            try:
                with open(checkpoint_file) as f:
                    if f.read(7) == "version":
                        raise OSError(
                            "You seem to have cloned a repository without having git-lfs installed. Please install "
                            "git-lfs and run `git lfs install` followed by `git lfs pull` in the folder "
                            "you cloned."
                        )
                    else:
                        raise ValueError(
                            f"Unable to locate the file {checkpoint_file} which is necessary to load this pretrained "
                            "model. Make sure you have saved the model properly."
                        ) from e
            except (UnicodeDecodeError, ValueError):
>               raise OSError(
                    f"Unable to load weights from pytorch checkpoint file for '{checkpoint_file}' "
                    f"at '{checkpoint_file}'. "
                    "If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True."
                )
E               OSError: Unable to load weights from pytorch checkpoint file for '/home/runner/.cache/huggingface/hub/models--hf-internal-testing--tiny-random-GPTJForCausalLM/snapshots/1fff390baa45cb187903ebdd269c975bb9ed7386/pytorch_model.bin' at '/home/runner/.cache/huggingface/hub/models--hf-internal-testing--tiny-random-GPTJForCausalLM/snapshots/1fff390baa45cb187903ebdd269c975bb9ed7386/pytorch_model.bin'. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.

Investigating...

[lapp0]

Works for me on transformers=4.36.2, but errors on 4.37.0 (released 11 hours ago).

I'll pin transformers to 4.36.2 in this PR as well.

[rlouf]

I get the same error locally, we should open an issue for that.

[lapp0]

Done
#568

[rlouf]

Thank you so much for fixing this!