[tokenizer] Allows import non-english model

[frankfliu]

Description

Brief description of what this PR is about

• If this change is a backward incompatible change, why must this change be made?
• Interesting edge cases to note here

[codecov-commenter]

Codecov Report

Attention: 1375 lines in your changes are missing coverage. Please review.

Comparison is base (bb5073f) 72.08% compared to head (4e9ca98) 72.22%.
Report is 888 commits behind head on master.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #2797      +/-   ##
============================================
+ Coverage     72.08%   72.22%   +0.13%     
- Complexity     5126     7112    +1986     
============================================
  Files           473      702     +229     
  Lines         21970    31680    +9710     
  Branches       2351     3284     +933     
============================================
+ Hits          15838    22881    +7043     
- Misses         4925     7236    +2311     
- Partials       1207     1563     +356     

Files	Coverage Δ
...ava/ai/djl/inference/streaming/StreamingBlock.java	100.00% <100.00%> (ø)
api/src/main/java/ai/djl/metric/Dimension.java	100.00% <100.00%> (ø)
api/src/main/java/ai/djl/metric/Unit.java	100.00% <100.00%> (ø)
api/src/main/java/ai/djl/modality/audio/Audio.java	100.00% <100.00%> (ø)
api/src/main/java/ai/djl/modality/cv/Image.java	69.23% <ø> (-4.11%)	⬇️
...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java	76.00% <ø> (ø)
...ava/ai/djl/modality/cv/output/DetectedObjects.java	96.29% <100.00%> (+1.29%)	⬆️
...rc/main/java/ai/djl/modality/cv/output/Joints.java	71.42% <100.00%> (ø)
.../main/java/ai/djl/modality/cv/output/Landmark.java	100.00% <ø> (ø)
...i/djl/modality/cv/transform/RandomResizedCrop.java	94.11% <100.00%> (+5.22%)	⬆️
... and 226 more

... and 372 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.