-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parallel downloads #825
Comments
This comment has been minimized.
This comment has been minimized.
Yes we do care. However it's largely a problem of there being bigger issues to tackle first. Sorry that nobody responded to your ticket though! |
If we're looking at parallelizing just the download part, is it much more On Mon, Jan 20, 2014 at 7:23 PM, Donald Stufft notifications@github.comwrote:
|
Parallelizing disk writes can actually make downloading slower on HDDs since the needle skips from one place to another. You'll have to store the entire downloaded content in memory and extract it from there in order to actually gain something. It's not that easy. As far as I know this is not the case with SSDs. Also, depending on your network's throughput downloading in parallel might actually slow down the installation process. I'm not sure if this is a good idea. Downloading and installing in parallel might be a good idea though as long as you download into memory so there's only one concurrent disk write at the time. Again see the note about SSDs. |
I disagree. Most downloads are far slower than a disk can write, even if writing many files. As an optional --feature, I think it is a trival choice -- people who know they are downloading over a LAN and onto a slow disk would chose not to --enable such feature. People like myself downloading several-hundred megabyte packages over broadband would gain much to chose to use such --feature. |
+1 to parallel download (and install if possible). |
+1, I frequently work on projects that have hundreds of dependencies and are built from scratch repeatedly throughout the average workday. This stuff crawls even with a pypi mirror on the LAN so having this would be great! http://stackoverflow.com/questions/11021130/parallel-pip-install Options presented are not great, but it does show there is (and has been) interest in a solution to the issue |
+1, this would be huge. Willing to take a crack at implementing. |
If anyone is curious why this should be a core feature and not something an external tool provides I can summarize the problem. Using something like "xargs --max-args=1 --max-procs=4 sudo pip install < requires.txt" in general can actually increase the total downloads quite a bit. This happens when for example "django-foo" and "django-bar" both require Django, but specify different constraints for which version is acceptable. I think pip itself has to compute the unified requirement set (probably including requirements-of-requirements) to avoid redundant downloads. |
Do knowledgeable people know if parallel install is an option? When I tried implementing a build system that executed pip in parallel, I hit problems specifically related to installing modules that had native extensions, but didn't dig further into it. If people know what might be wrong there, I'd love to know. |
+1 to parallelization. Not just for downloading but also for things like "Collecting ... File was already downloaded" with |
note that enabling complete "parallelism" amounts to a relatively complete re-factoring of many internals also there is ux issues, while downloading/unpacking is simple, wheel generation and setup.py invocation are full of error potentials/conditions (however they are needed until everybody uses wheels whenever possible and they are needed right in the middle of execution) |
so if the UX complication is sidestepped with only parallel downloads/unpacking, doesn't it make sense for that to be the first goal? full parallelism does seem like a huge undertaking.. i'm no expert but as far as that implementation here's two ideas that came to mind a) in the set of all the requirements and the requirements of requirements, subsets must be discovered wherein individual requirements must be installed sequentially. all such subsets can then be installed in parallel at least. as far as ux, errors here can be shown whenever they occur because any error encountered is truly an error. b) in the set of all the requirements and the requirements of requirements, a parallel worker pops something out and tries installing it. if the req installs successfully, great. if it fails, we assume another requirement must be installed first and put it back in. as far as ux, errors here must be suppressed unless they persist until the end of this procedure, because any error encountered might be something we can recover from later in the process. |
the ux issue cannot be side-stepped if any of the downloaded packages is a sdist the requirement graph is nonlinear and changes whenever more about the dependencies and constrains of a just freshly downloaded package gets known as soon as a sdist is downloaded, the build/egg-info/wheel process of it has to be triggered |
I just took a stab at it. It seems that the progress bar/spinner is an issue since it registers a signal handler in order to clean up downloads. |
Here's what I think... To be able to have parallel downloads, pip would need to determine which packages are to be installed and where they need to be fetched from. As of today, it is not possible since pip cannot determine the dependencies of a package without downloading it. That said, this is definitely something that'll be pretty awesome to have. :) |
I understand that you won't know follow-up dependencies before downloading a package. But looking at a requirements.txt, I would assume you could start a parallel download of all of those packages and then expand the list of things to download as you discover more dependencies. The only edge case I can think of right now would be different version requirements on the same dependency. But that problem exists with regular downloading as well I would assume. |
I didn't mean to post that message yet. Oops. Anyway, the point I was making was that the way pip currently handles dependencies, there's a race condition where 2 packages depend on a common third with compatible but different version specifiers. Whichever package is downloaded first, its specifier would be used and a bad thing - you have behaviour that changes because of how the network behaved. The only right way to do this is then to have dependencies metadata on pypi and only then can we determine the packages beforehand and proceed to parallel download/installation. Or somehow managing this during downloading? Or I'm missing something about this issue. |
@fruechel Yes. Except with the serial downloads, the version of the common dependency is deterministic. |
Understood. You're right, it would make the process non-deterministic, based on random network behaviour. So yeah until that issue is resolved, implementing this would introduce problematic behaviour that you wouldn't want to build in. |
This problem would seem to be limited to parallel installation. If we're talking strictly about parallel downloads followed by standard, serial installs.. you might experience an extra, useless download.. but I'm not clear on why it should introduce anything nondeterministic. I see a few potentially different scenarios for this improvement, and maybe it's useful to avoid conflating them:
Of course having several of these things would be awesome, but any could be an improvement. In this thread people have raised at least 3 separate blockers from what I can tell:
I'm less clear on which blockers affect which scenarios |
Hmmm this is where class dicts with an entry of the downloaded dependencies names would work so then it can bite the issue about extra downloads. And then the install system can look at that dict that is only populated at run time to install all the packages in that dict. And the dict itself on every entry could have a class that stores the information needed for the install mechanism in pip. I think this can theoretically be used for parallel installs. |
What is probably the most feasible is to download all requirements in parallel until reaching a source distribution, and then download in serial. Something like:
|
I've labelled this issue as an "deferred till PR". This label is essentially for indicating that further discussion related to this issue should be deferred until someone comes around to make a PR. This does not mean that the said PR would be accepted - |
That said, I think it's pretty much clear that this is would be a welcome improvement. :) |
Let's stick to the topic of adding parallel downloads to pip here. If there are questions about the broader picture or how pip does vendoring etc, please file a new issue / start a thread on the Discourse forum etc. I've gone ahead and hidden all comments unrelated to the topic at hand. |
The issue I created on This is particularly true:
@pradyunsg due you consider this a separate concern? If so I'll create a new issue. I guess in an ideal world pip would utilise asyncio for download, and In summary the question for me from this thread are:
|
IMHO precompiled wheels are better solution. Compiling C++ stuff is a job of CMake and ninja, not pip. |
That's a completely different issue, I entirely agree that pre-compiled binaries would be great - but they're not at all easy apparently, see pypa/manylinux#37 and pypa/packaging-problems#69. My suggestion is a work around that makes use of multiple cores to speed up installing/compiling many packages by ~10x. Maybe I wasn't clear, I'm not suggesting a single package is installed using multiple threads or processes, but rather that packages can be compiled in parallel, so if I do (on alpine) |
@samuelcolvin I know it's "just" a workaround, but couldn't you do something like
Using GNU Parallel to run multiple commands in parallel - excuse me if I got the syntax wrong, I'm not a Unix user. Basically, build wheels in parallel, and then pip install them. If you have multiple machines that have the same architecture/are binary compatible, you can re-use the |
Interesting idea, that might help a lot, the main problem is that multiple packages might have the same dependencies which would then be installed multiple times, even worse they might have conflicting dependencies which pip wouldn't know about in this case. |
Not if you |
Good point, sorry. For me that reduced the install time in the above example from 134s to 54s. |
Yep. Even if we're going to solve them together, it's useful to have separation of concerns and discussion.
I don't know. Someone has to do the work of figuring out what the scope of things is and actually spend time implementing this stuff. Since no one has stepped up to do it yet, and the volunteered time from the maintainers isn't going to be all pulled into this issue, I don't think that we'll have the answers soon either. |
I have a PR which works and shortens installation time. |
@thedrow please feel free to file a PR and we can have more concrete discussions on it. :) |
@pradyunsg I did a while ago... |
I am beginner to open-source please help what technologies should I learn so that I can do this project? |
Hi @jsar3004, I'm working on this with pip's maintainers and contributors this summer. Turns out, it is not a trivial task as it touches many corner that I never thought that it would. FYI the current work is implementing GH-7819 and if you want to help out, feel free to join the discussion. If you are really beginning to contributing free software development, it's more than just technical skills and understanding that you should possess. For medium-large-sized project like pip, a decent amount of time is actually spent in discussion of all level (from idea to nitpicking)—trust me I'm still learning on how to do it properly. Personally I suggest starting from the list of issues awaiting PRs and see what you're also interested in. Increasing the scope a bit, I believe others project under the hood of PyPA also welcome people looking to contribute. |
You may wanna look at https://github.com/KOLANICH/fetchers.py/blob/master/fetchers/tools/download.py#L16 |
@jsar3004 Hi! I suggest you get started by looking at other tasks, such as good first issues in pip and [other PyPA tooling](org:pypa label:"good first issue"). This is a significantly large task, with multiple moving parts -- it would not be possible to work on this without an initial ramp-up, gaining a better understanding of how pip works and what pip's development workflows are. |
Hey, I've created a PR #12923 to do this by extending the Please ask questions/comments in #12921 regarding metadata resolves, and #12923 regarding the parallel download strategy (I just used threads instead of any fancy async stuff). I haven't figured out the progress bar yet (although I'm going to try to do so now), so please feel free to leave comments in #12923 regarding how to support a unified progress bar for parallel downloads. |
How about having pip download all the packages in parallel instead of waiting for each one to finish before downloading the next?
After that is implemented, how about having pip start installing one package while it's downloading the next ones?
The text was updated successfully, but these errors were encountered: