Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support loading HF repos with hf: stubs #1260

Merged
merged 1 commit into from
Sep 20, 2023

Conversation

mgoin
Copy link
Member

@mgoin mgoin commented Sep 20, 2023

Adds the capability to download models straight from HF repos if the stub is prepended with hf:.
If you want to use this model https://huggingface.co/mgoin/TinyStories-1M-deepsparse, then passing in model_path="hf:mgoin/TinyStories-1M-deepsparse" will suffice

import deepsparse
pipe = deepsparse.Pipeline.create(task="text-generation", model_path="hf:mgoin/TinyStories-1M-deepsparse")
pipe(sequences="hello")

Output:

>>> import deepsparse
>>> pipe = deepsparse.Pipeline.create(task="text-generation", model_path="hf:mgoin/TinyStories-1M-deepsparse")
2023-09-20 14:10:19 deepsparse.transformers WARNING  The neuralmagic fork of transformers may not be installed. It can be installed via `pip install nm_transformers`
2023-09-20 14:10:20.061685: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/mgoin/code/nyann/external/onnx-runtime/build/Linux/RelWithDebInfo:/home/mgoin/code/nyann/external/onnx-runtime/build/Linux/RelWithDebInfo:::
2023-09-20 14:10:20.061710: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Downloading (…)50ea1/.gitattributes: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.52k/1.52k [00:00<00:00, 93.5kB/s]
Downloading (…)e1050ea1/config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.02k/1.02k [00:00<00:00, 70.5kB/s]
Downloading (…)neration_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 119/119 [00:00<00:00, 7.86kB/s]
Downloading (…)cial_tokens_map.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 438/438 [00:00<00:00, 37.4kB/s]
Downloading (…)okenizer_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 727/727 [00:00<00:00, 64.9kB/s]
Downloading (…)9e1050ea1/merges.txt: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 456k/456k [00:00<00:00, 2.87MB/s]
Downloading (…)9e1050ea1/vocab.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 798k/798k [00:00<00:00, 3.25MB/s]
Downloading (…)50ea1/tokenizer.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.11M/2.11M [00:00<00:00, 3.65MB/s]
Downloading model.onnx: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28.0M/28.0M [00:05<00:00, 5.47MB/s]
Fetching 9 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:05<00:00,  1.56it/s]
Using pad_token, but it is not set yet.█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28.0M/28.0M [00:05<00:00, 5.65MB/s]
2023-09-20 14:10:27 deepsparse.transformers.pipelines.text_generation INFO     Compiling an auxiliary engine to process a prompt with a larger processing length. This improves performance, but may result in additional memory consumption.
2023-09-20 14:10:27 deepsparse.utils.onnx INFO     Overwriting in-place the input shapes of the transformer model at /home/mgoin/.cache/huggingface/hub/models--mgoin--TinyStories-1M-deepsparse/snapshots/62256599e36d99faefb58461e0030e19e1050ea1/model.onnx
DeepSparse, Copyright 2021-present / Neuralmagic, Inc. version: 1.6.0 COMMUNITY | (4e69fdec) (optimized) (system=avx512, binary=avx512)
2023-09-20 14:10:29 deepsparse.utils.onnx INFO     Overwriting in-place the input shapes of the transformer model at /home/mgoin/.cache/huggingface/hub/models--mgoin--TinyStories-1M-deepsparse/snapshots/62256599e36d99faefb58461e0030e19e1050ea1/model.onnx
>>> pipe(sequences="hello").generations[0].text
'.\n\nThe little girl was so excited that she couldn\'t wait to tell her mommy. She asked her mommy if she could go on the beach. Her mommy said yes, but she was too busy to go.\n\nThe little girl was so excited to go on the beach. She ran to the beach and saw a big wave. She was so excited to see the wave. She ran to the shore and said, "Look, Mommy! I\'m going to go!"\n\nHer mommy smiled and said, "Yes, you can go on the beach!"\n\nThe little girl was so happy. She ran to the beach and played with her friends. She was so happy to have her friends and they all played together.\n'

@mgoin mgoin merged commit d22bfce into feature/damian/deployment_folder_only Sep 20, 2023
@mgoin mgoin deleted the hf-download branch September 20, 2023 19:47
mgoin added a commit that referenced this pull request Sep 29, 2023
* initial commit

* initial commit

* ready for review

* Apply suggestions from code review

* fix (failing tests)

* fix test

* Support loading HF repos with `hf:` stubs (#1260)

---------

Co-authored-by: Michael Goin <michael@neuralmagic.com>
Co-authored-by: George <george@neuralmagic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants