Runtime requirements may be necessary at build time

[zenbot]

Apologies first off if this has been discussed before — I've only had a quick scan through old issues to see if this has already been mentioned.

flit's behaviour of importing the package being built to get its description and version means that any runtime requirements in the top level of the package must be available for import in the build environment. This can be painful when the runtime requirements aren't trivially installable.

My first thought on this was, "these should be just made optionally configurable in flit.ini", but this goes against flit's (excellent!) there-should-be-one-way-to-do-it ethos.

My second thought was "we could just use ast to pull the docstring and version out", but that assumes that __version__ is defined in the __init__.py or module, and not imported from elsewhere.

Do you have any thoughts on this? Would you consider allowing optional configuration of these things in flit.ini, or ast-walking as a fallback if the package fails to be imported? Or do you consider the package being importable at build time a hard requirement?

[zenbot]

One advantage of allowing optional configuration of the description and version in flit.ini that I just thought of: it enables the building of PEP 420-style native namespace packages.

[Carreau]

Thanks for open this on your though on how to improve flit and the kind words !

This seem similar to one of the earlier issue I opened.

I think there are many point to take into consideration when deciding what features to have in flit, and it's always a right balance, and as you pointed out one of flit quality is the strong opinions that @takluyver baked into it.

One things I am curious about is in which case will you built a package when you cannot import it ? One thing I can think of is in an automated building process which is only responsible for building.
Now one of the question I have is should you build a package if you can't test it or at least import it before building ? It feel to me like in most case the release process should be :

• test the pacakge
• if test pass: package.

Now there is already a use-case where this breakdown : flit can build a wheel for Python2, but that does not mean you can test on Python 2.

The other two side are the complexity of implementation, it's cost; vs the number of case where it is actually useful.

In this case, without amore explicit example, I have the feeling that the cost-benefit ratio would not be that great for the use case; in particular because I feel like the env that build the wheel should be the one testing it.

On the other hand I'm not sure there are many packages that actually have dynamic generation of version number; so I'm unsure how many packages would be broken by such an implementation.

As for optional feature, I think that's against the spirit of flit try to move in the opposite direction of setup.py (where everything is a choice). I think it's too early in flit life to decide to add an options; but keeping track of what options are requested for user and reconsider once we have more data make sens.

These are just my thoughts. I'll let @takluyver comment and decide.

[zenbot]

Thanks for your reply!

One things I am curious about is in which case will you built a package when you cannot import it ? One thing I can think of is in an automated building process which is only responsible for building.

Accounting for our automated build system while trying to migrate one of our packages to flit is what led me to create this issue. Our buildbot builds a few dozen packages with different and conflicting requirements; to get flit going we'd need package-specific build environments, & would need to roll our own PEP 518 support to define those environments.

Now one of the question I have is should you build a package if you can't test it or at least import it before building ?

This question assumes a workflow with codependent test and build steps. We can't merge to our master branch if tests are failing — by the time we cut a release and build distribution artefacts tests have long-since passed.

As for optional feature, I think that's against the spirit of flit try to move in the opposite direction of setup.py (where everything is a choice).

For sure.

Do you have any thoughts on how flit could support building native namespace packages, given the requirement that the description and version come from the package itself?

[Carreau]

Thanks for the description. I did not consider this use case, and I think it make some sens (at least to me); I can also see a case where you'd like to build the wheel, and actually test it instead of the repository.

Do you have any thoughts on how flit could support building native namespace packages, given the requirement that the description and version come from the package itself?

Not particularly, I think that might be a case where you want flit to be easy to use as an API and have another package with a different opinionated idea of a workflow to strive. With pep517 you could decide that the description and version number be in pyproject.toml, and that building wheel/sdist would actually embed them in __init__.py instead of pulling them out.

I'll let @takluyver comment on this as well.

[takluyver]

Thanks for the issue, and thanks @Carreau for responding while I was busy. :-)

Yeah, it is tricky. IIRC, when you do flit install, flit will take care to install the dependencies (via pip) before it imports the package to get the version number. I have also thought of getting it by scanning the ast, but that breaks down if the version number isn't a string literal (for Jupyter packages we construct the string from a tuple, for instance), or, as you noted, if you import it from another module.

Maybe one way would be to try getting the version and docstring from the ast first, and fall back to importing if that doesn't work. I'll turn that over in my head for a while to see if I spot an issue with it.

I haven't thought much about namespace packages. I've generally found them to be an annoying complication for a purely aesthetic benefit. They're not used widely or consistently enough to really function as namespaces, so it ends up as just a way to include a dot in your package name, as far as I can see.

[zenbot]

Thanks for the response, @takluyver.

I've submitted a POC PR that includes the ast-first-then-fallback behaviour you've mentioned. Please feel free to bin it if you find the implementation offensive or discover you hate the idea generally — I'm just desperately keen to get us away from setuptools, and this is one of our blockers.