-
Notifications
You must be signed in to change notification settings - Fork 885
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
What are the biggest performance bottlenecks? #1390
Comments
The constituency parser is probably the slowest, so if you don't need conparses, you could consider dropping it. The tokenizer is a surprising amount of CPU work for building all the token objects. We have in fact talked about pushing some of that into C++ or Rust |
import time
import stanza
text = "The United Nations is a diplomatic and political international organization whose stated purposes are to maintain international peace and security, develop friendly relations among nations, achieve international cooperation, and serve as a centre for harmonizing the actions of nations. It is the world's largest international organization. The UN is headquartered in New York City (in the United States, but with certain extraterritorial privileges), and the UN has other offices in Geneva, Nairobi, Vienna, and The Hague, where the International Court of Justice is headquartered at the Peace Palace."
processors = [
'tokenize',
'tokenize,pos',
'tokenize,pos,constituency',
'tokenize,mwt',
'tokenize,mwt,pos',
'tokenize,mwt,pos,lemma',
'tokenize,mwt,pos,lemma,depparse'
]
res = {}
for proc in processors:
nlp = stanza.Pipeline(lang='en', processors=proc)
start = time.time()
doc = nlp(text)
end = time.time()
res[proc] = end-start You are right, the time doubles. The community could certainly participate in putting them to C++, if you could start listing specific functions that would |
It's not the cuda usage that's the problem. It's the object creation in the tokenizer and the code that determines whether or not a transition is legal in the parser. |
Could you point to a specific file or function? |
Sure, I think the tokenizer is mostly slower than it could be because of stanza/stanza/models/tokenization/utils.py Line 463 in 6e442a6
|
It is a bit difficult to get into a system from outside. A very helpful step would be to
When I look at the code, I sometimes see, that it has a defaultvalue of "None" and one would have to print out any argument type at runtime to infer the necessary type. Ps.: The annotation does not have any effect in python, it is just for readeability and when you want to move to a static language. |
You're not wrong, but, it's also not the limitation stopping us or outside folks from making a faster tokenizer. I'm pretty sure the right answer is to make all of the random little python objects in a compiled language instead. |
Yes, but for the translation to a statically typed language, they would be necessary and would be a great help. When we translate It is probably easier for somebody that has a general overview over the package to write the annotations. After a deeper analysis of the code, I think that the first steps towards a C++ implementation could be:
|
The package appears to consist of many different parts along the processing pipeline, for which the information is always encoded and then again decoded and as I saw, they use even different vocabs. |
The models are all optimized for each individual annotation task. If you found a way to efficiently answer each task with one network, that would be a very powerful research result |
@AngledLuffa We saw that if you run the models directly on onnx runtime or TensorRT that you get a 2X-4X improvement in the inference runtime - which is critical for any production deployment. If you can allow users of Stanza to not be dependent on torch and enable exporting of the models to other formats (for example ONNX) then we can use existing frameworks to optimize inference time. We use 2 NER models from Stanza using the Stanza pipeline, and it is nearly impossible to export the models and remove the dependency on torch. This is what HuggingFace transformers pipeline enable us to do, and we would want Stanza to support this as well. |
Hey,
I noticed that for very large amount of text data, the algorithm takes a long time to finish.
We can probably not simplify the pytorch models (or can we?), but maybe the authors could write a
list of the most time consuming operations that could be improved.
This would help to support your efforts with stanza.
One could certainly merge some docs here, but I would have to know what the perfect text length would be (for a 8GB, 16 GB, 32 GB ... GPU)
The text was updated successfully, but these errors were encountered: