Load .whl packages and dependencies dynamically

[Tyldi]

Hello,

I am quite new on python environment and on pyinstaller, but I really need your help.

Context

I have a python app which loads plugins at runtime. Indeed, we provide new features for our users in the form of plugins, and we want them to be able to install and use those that interest them. We don't want to build a new executable file every new plugin.

We bundled our app in a one folder, because it seems easier to add plugins. We have a folder for Linux and Windows
pyinstaller --clean --paths=lib/python3.8/site-packages src/main.py

My plugins have sometimes external dependencies (lib, etc.), so foreach plugin here is my setup.py (with requirements, values from requirements.txt file) :

setup(
    name='package_name',
    version='0.0.1',
    include_package_data=True,
    python_requires='>=3.7',
    install_requires=requirements,
    extras_require={'all': requirements}
)

My issue

I know how to import installed modules dynamically to my code:
module = __import__(module_name, globals(), locals(), [name], 0)
This part works.

But I have difficulties to install completely and dynamically those plugins within my code.

What I tried

Let's imagine I want to import my test.whl plugin, which has a dependency with 'test2' package
1. Install the plugin with pip:
subprocess.run(["pip", "install", "--upgrade", package, "-t", my_one_folder_location])
=> Cons : You need to have python and "pip" (not "pip3" etc) on the user computer. Not the best solution

I also tried to use the python installed in the pyinstaller folder but did not succeed.

2. Import every plugins in a specific directory:

PyInstaller understands the “egg” distribution format often used for Python packages. If your script imports a module from an “egg”, PyInstaller adds the egg and its dependencies to the set of needed files.

I tried to put my .wheel packages in a folder "egg". Then my program detect the plugin, but it doesn't import my lib dependencies
__main__:Unexpected error: no module named 'test2'

I'm going around in circles and can't find any solutions: What is the best way to import my external .whl packages and all it's dependencies in a pyinstaller solution?

[bwoodsend]

Don't do this. It won't work.

[Tyldi]

Thank you for you reply, but can you please elaborate ? This problem is a quite common one, and I am quite sure it can works with a pyinstaller setup.
For example the first solution with the "pip" works, but I don't find it very reliable, and it needs a python installation on the host machine.

[rokm]

You are trying to use PyInstaller for something outside of its design specs - which is bundling python script with python packages that are available at the build/freezing time. So while you can try to force it to work the way you want, you are on your own in doing so and there's no guarantee that the process will be particularly pleasant.

Your first solution with pip is not a practical one - if you expect python and pip to be available on the target system, what do you need PyInstaller for in the first place? Just have the user set up a virtual environment and run your code from there. Also, using pip as a sort of package manager for curating a location where your PyInstaller-frozen application will pick up extra packages will work only as long as the target environment's major python version matches the one that was used to freeze the application (well, unless only pure-python packages are used).

Second, nothing* prevents you from copying extra package directories to top-level application directory; or adding wheels to eggs subdirectory; or adding extra code to some external location and adding that to sys.path (or alternatively, scanning an external location for wheels and adding them to sys.path). But that means that you need to copy over both your plugins' code and its dependencies. And I don't think there's an off-the-shelf automatic process you can use for that - which seems to be the core of your question.

[*] There's a caveat, though: if there is a package partially collected into PyInstaller's PYZ archive, the missing modules from that package won't be found in external location even if you provide an additional full copy of a package there (see #7214). Similarly, if your plugin or its dependencies depend on some part of stdlib that was not collected during original build, those modules will be missing as well.

Basically, with the "I want to have plugins and external code available in my frozen application" route, you are bypassing PyInstaller's dependency analysis and bundling, so you have to handle everything on your own, so I second the "Don't do this" advice; but maybe with added "unless you know what you are doing".

[Tyldi]

Hello,

Your first solution with pip is not a practical one - if you expect python and pip to be available on the target system, what do you need PyInstaller for in the first place? Just have the user set up a virtual environment and run your code from there. Also, using pip as a sort of package manager for curating a location where your PyInstaller-frozen application will pick up extra packages will work only as long as the target environment's major python version matches the one that was used to freeze the application (well, unless only pure-python packages are used).

Indeed, the pip solution is something that "works" but that I do not consider as a solution, especially because of the problems that you have rightly pointed out. I mentioned this technique in case I missed something in the documentation that could have made a "pip" work inside the Pyinstaller app, with no link with the Host Machine environment. From what you say this does not seem to be the case and/or a relevant choice.

Basically, with the "I want to have plugins and external code available in my frozen application" route, you are bypassing PyInstaller's dependency analysis and bundling, so you have to handle everything on your own, so I second the "Don't do this" advice; but maybe with added "unless you know what you are doing".

Thank you for the clarifications, I'll try to find a solution in another way.

[VitalyName]

Hi @Tyldi
Are you found any solution?