You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@RemyLau I was building some of the cross species networks and I noticed that as the threads were increased (I have got up to 120 cpus going), generating the walks became very fast, however, the training time of the gensim model slows down a considerable amount. Using too many threads in gensim seems to be a known problem as shared in this post.. I think there could be two ways to help fix this
The easy fix would be too add separate arguments for the number of workers in random walk generation and the number of workers used by gensim and let the user find what is best
The other could be to use the corpus_file argument as described in the post above. Maybe it would be fast with large parallel processing to generate the corpus_file, and if users have some scratch system like MSU could easily be saved there temporarily
Use corpus file or not (also report the corpus file size)
Other hyperparameter choices (Optional)
Embedding dimensions
Number of walks
Window size
p & q (should not have any effects)
Testing 1-3 can answer the following two questions:
Do gensim scale better / run faster on AMD's or INTEL's chip.
Does the corpus file approach provide noticeable speedup compared to not using it.
If indeed the answer to 2 is using the corpus file approach provides significant speedup, I'll proceed to make a CLI option for that. Some potential things to keep in mind.
Need to be careful about cleaning up cache (corpus files).
Normal exit: shutil.rmtree to remove the temporary cache directory
Either save the cache to /tmp or make a dedicated dir for pecanpy_cache. The former can take advantage of the system to help clean up any missed files but is hard to keep track of. The latter will not be automatically cleaned up but can be easily located.
How should gensim word2vec be called using the corpus file? Should it be a separate process from the main PecanPy process?
@RemyLau I was building some of the cross species networks and I noticed that as the threads were increased (I have got up to 120 cpus going), generating the walks became very fast, however, the training time of the gensim model slows down a considerable amount. Using too many threads in gensim seems to be a known problem as shared in this post.. I think there could be two ways to help fix this
corpus_file
argument as described in the post above. Maybe it would be fast with large parallel processing to generate the corpus_file, and if users have some scratch system like MSU could easily be saved there temporarilyOriginally posted by @ChristopherMancuso in #19 (comment)
The text was updated successfully, but these errors were encountered: