Trying to use pyinstaller inside an abstracted venv (Failed to import `module`)

by Dvd Znfin Arch Linux, Python, System Administrationon Posted on

Story of someone trying to use pyinstaller on Arch Linux

Note: If you only care about the issue with pipx and don’t need a pretext, you may skip to here

Trying to use pyinstaller on Arch Linux

Let’s say you are an Arch Linux user and want to install a certain python module via pip

If you try you will get the following error:

Ah yes, PEP 668.
TL;DR: Using multiple package managers can be bad, so just use the OS package manager.

Problem is that, not all Python packages are in the official repos!
Take for example pyinstaller
Despite it being quite a popular package and the goto solution for bundling a python application, it’s not listed in the official repos.

No worries, the externally-managed error provided some, hopefully, helpful information.
It states that if the Python packages is not in the official repos we can use venv to create a separate virtual environment for Python and …………. that seems to much work for simply trying to install a package.
And what if I would need multiple venvs for non-Arch-Python-Packages?
Would I need to create and manage 10 separate dir’s each with its own Python env and installed modules?

No, Thank you!

There’s also some talk about pipx but I’ll leave that aside from now and try my luck with the Arch wiki article for Python

Already we can find conflicting information, at least there’s some warnings about that.

Anyway, since we already tried the Official repos maybe we can try our luck with the AUR

Nice, we can now start the process of manually installing an AUR package, as it is officially recommended.

The typical process looks like this:

g Clone Clone Check PKGBUILD \n(for sanity and security) Check PKGBUILD (for sanity and security) Clone->Check PKGBUILD \n(for sanity and security) makepkg -sirc \n(Use whatever combination \nof flags needed) makepkg -sirc (Use whatever combination of flags needed) Check PKGBUILD \n(for sanity and security)->makepkg -sirc \n(Use whatever combination \nof flags needed)

And if we try:

We get more problems

AUR Packages depending on other AUR Packages

Because of the way the AUR works and because AUR Packages can depend on other AUR Packages, you must install all AUR dependencies for a given AUR Package before you can install that AUR Package. This process must be done manually unless you use an AUR Helper, which is both not officially supported and not recommended, because you should verify all user submitted PKGBUILDs before installing.

If you don’t use an AUR Helper, this will create a big package management problem and will require a lot of manual work(Checking each PKGBUILD individually, Making sure they are up to date, etc…)

Usually for normal software installed from the AUR this isn’t that big of a problem.
But since Python modules can be so inter-mingled and can have very complex dependency graphs, you can quickly get in a situation where you have manually manage a lot of Python Modules just for a single package you actually wanted.

So going back to the main topic of trying to install pyinstaller, I won’t use the AUR package because I don’t want to manually manage more than one package for that single package and because I don’t want to use an AUR Helper either.

This leaves only one more option…

Venv and Pipx

As I said here, individually managing 10 different python venvs is not for me. So this leaves us with the other suggestion of using pipx in order to abstract those virtual environments.

At first glance pipx, seems like a nice tool, that allows you to abstract venvs and just focus on the packages you want.

So let’s try it!

Let’s try to install pyinstaller via pipx

And now let’s try to run pyinstaller on an actual project to see if it works

Looks like pyinstaller couldn’t include the requests module, because it didn’t find it.

This is a direct consequence of running pyinstaller from that abstracted venv, even tho python-requests may be installed normally on the system from the Arch official repos that doesn’t matter, because of the venv.

My main issue is with the added layer of abstraction added by pipx.
When you work with venv directly, you know exactly the mess your getting into. When you work with pipx, all of that context gets abstracted away the moment you enable ensurepath.
This way, someone could easily forget that their even running a program in a venv and get very confused as to why their installed modules don’t get recognized by this other module.

Trying to use pip for pipx managed venv

After all pipx is really just a wrapper for python, pip and venv so there’s no real reason why you couldn’t use the pip it uses directly, this of course strips away at the layers of abstraction.

Using pip in this manner is only a consequence of the “isolation” that pipx provides and the fact that you can’t install any other additional modules besides the ones that are strictly necessary for the package you initially installed. This model would be great if your simply trying to install a cli app from pip, but when that package needs other modules this model simply doesn’t work.

Anyway this is the path I found for the pipx python interpreter:

$HOME/.local/pipx/shared/bin/python

And here is the path for pip:

$HOME/.local/pipx/shared/bin/pip

Note that there is also another python interpreter specific for a venv (Example with pyinstaller):
But there is no pip in that folder

$HOME/.local/pipx/venvs/pyinstaller/bin/python

Not we can finally install any additional modules for all pipx managed venvs
Following my case of trying to install Requests and Prompt Toolkit for pyinstaller:

Now we can retry to build my app and see if pyinstaller works:

Pyinstaller bundled all the modules correctly and my app now actually works!

Conclusion

Trying to avoid dependency hell can sometimes be a pain innit of itself, going out of your way in order to unabstract abstracted virtual environments created by a silly little wrapper for yet another package manager you don’t care about

But since we’re talking about this “pipx” thing, why not try to see it’s full purpose and full set of features?

This has to be a joke …

Moral of the story

RTFM!

Sources/Further reading: