Tokenizers: Updated huggingface_models.py to support Safetensors models as well as pytorch

[dameikle]

This updates huggingface_models.py to support converting Safetensors models as well as pytorch when running the Model Zoo importer

[codecov-commenter]

Codecov Report

Attention: 1365 lines in your changes are missing coverage. Please review.

Comparison is base (bb5073f) 72.08% compared to head (f0164bd) 72.33%.
Report is 929 commits behind head on master.

Files	Patch %	Lines
...va/ai/djl/modality/nlp/generate/TextGenerator.java	2.81%	276 Missing ⚠️
.../java/ai/djl/modality/nlp/generate/SeqBatcher.java	0.75%	132 Missing ⚠️
...ity/nlp/generate/ContrastiveSeqBatchScheduler.java	2.97%	98 Missing ⚠️
...i/djl/modality/nlp/generate/SeqBatchScheduler.java	9.83%	55 Missing ⚠️
.../java/ai/djl/modality/cv/BufferedImageFactory.java	40.96%	47 Missing and 2 partials ⚠️
...a/ai/djl/modality/nlp/generate/StepGeneration.java	2.04%	48 Missing ⚠️
api/src/main/java/ai/djl/ndarray/NDArray.java	43.42%	39 Missing and 4 partials ⚠️
...n/java/ai/djl/modality/cv/output/CategoryMask.java	22.00%	39 Missing ⚠️
...i/src/main/java/ai/djl/ndarray/NDArrayAdapter.java	71.21%	31 Missing and 7 partials ⚠️
.../cv/translator/SemanticSegmentationTranslator.java	37.50%	35 Missing ⚠️
... and 76 more

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #2880      +/-   ##
============================================
+ Coverage     72.08%   72.33%   +0.24%     
- Complexity     5126     7170    +2044     
============================================
  Files           473      708     +235     
  Lines         21970    31928    +9958     
  Branches       2351     3317     +966     
============================================
+ Hits          15838    23095    +7257     
- Misses         4925     7256    +2331     
- Partials       1207     1577     +370     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.