Questions about build customization and compiler specification

[msarahan]

Separated out from #1835

Note that in conda-build 3.0 (coming soon), we'll be deprecating the compiler activation logic in conda-build itself, and moving towards activation scripts in packages. With these packages, you'll be able to just list this package as a run or test time dependency. I'll leave this issue open as something that will be more easily fixed by that transition.
@gabm

gabm commented a day ago
wow, that comes a bit as a surprise... Is there some discussion going on on that topic? I subscribed the mailing list but there has been nothing in that direction?!

Is there some more information on the new technique? Documentation? anything?
@msarahan
Member
msarahan commented a day ago
conda/conda-docs#414
#1585
https://www.youtube.com/watch?v=WzAgRUa4iuA

@gabm commented 7 hours ago
after reading your documents: ufff... thats pretty cool stuff... 😳 🏁 🎉

So do I understand correctly:

you list a jinja variable {{ cxx_compiler }} as a build dependency, which will be replaced by several compiler packages which are listed in a special conda_build_config.yaml in the mentioned locations
this compiler package pins an implicit run dependency of the corresponding runtime package to the built package
when I install the built package, the proper runtime library version gets installed as well
Cool so far, that replaces track_features right?

Now we want to implement this stuff for our own services. The compiler package under windows will be somewhat special, cause it only calls the vcvarsxx.bat, but in general the compiler will be made available through activation script or proper package. I think thats fine.

Questions that arise:

Should the runtime library contain the actual library? Guess so, right? Or is it just a placeholder to group project with of the same compiler?
How do we specify, which runtime library we want to install on our systems? Currently we do that by tracking the corresponding feature. So How would I distinguish vc 9 and vc 14 on client side?

[msarahan]

Thanks. We're very excited about it.

will be replaced by several compiler packages which are listed in a special conda_build_config.yaml

Technically, it's potentially several compiler packages - one per entry in conda_build_config.yaml.

Regarding your questions:

• Runtime packages should contain the actual runtime packages, or at the very least, some script that verifies that they are available on the destination system at install time.
• pinning downstream packages is key to specifying runtimes. Basically, for the compiler package, you would add some additional metadata to meta.yaml. There's a WIP proof of concept with GCC at https://github.com/conda/conda-recipes/blob/master/cross-compilers/gcc/meta.yaml#L26-L27

Windows compilers can do the same thing. They could, for example, add a downstream runtime dependency to the vc package (not to be confused with the vc version) - which would take care of also activating the feature. I think features and this new stuff will coexist for a while, but this new stuff will also allow people to specify debug variants of packages, as well as different optimization levels, static/dynamic linked variants, etc.

[gabm]

thanks a lot for these answers... if I were an end user (without conda-build) and I'd have two variants of the same package: - libpng built with vc9 and libpng built with vc14.. then what happens of I conda install libpng ?

According to your description I expect, that if there is no vc package installed at all I'll get an error, and if I have a vc package installed, I get the corresponding libpng variant?!

And another thing: which of this stuff is already implemented and expected to be stable? We got quite some packages already and this change means we'll have to touch all of them. I'd prefer to to that not more than once... But since our tests broke in 2.1.6, we need a solution for that :/

[msarahan]

if I were an end user (without conda-build) and I'd have two variants of the same package: - libpng built with vc9 and libpng built with vc14.. then what happens of I conda install libpng ?

This is still an active topic of development. It's what we call "Core Dependency Trees" (CDTs). I think that conda-build now supports the notion fairly well, but there is not really support for the install-time equivalent of conda_build_config.yaml in conda. What will happen is that there will be some kind of default file created when you install conda/miniconda, as well as a program to re-run detection. This is going to pick up things like CPU optimization support. Users will be able to modify it to change it or add things like enabling debugging versions.

More specifically, for your vc example, this is simpler. It would probably have a default value being the newest vc, so that you don't get an error. However, I'd expect more often that people install Python, which then activates a particular vc feature, which should take care of the rest.

Everything in the docs is implemented on the master branch. There will be a preview release very soon, but I don't expect it to be completely stable until it's seen a couple of weeks of testing. As a solution to your tests, I recommend something based on selectors instead.

1. create a recipe that runs the compiler activate scripts in an activate.d script
2. add that package to your meta.yaml in the requirements/run or test/requires section, as appropriate, with a [win] selector.

If you'd rather not modify recipes twice, I recommend just pinning conda-build to the last version that worked for you, until the new stuff is ready for you.

[msarahan]

a couple of weeks of testing

That was overly optimistic. =(

rc2 is on the conda-canary anaconda.org channel. Release will really, truly be out soon.

[gabm]

sounds great, we'll test it!

I have a question though: has there been some progress on the conda integration of CDTs?

[msarahan]

@kalefranz can you comment on any example of install-time usage of package pinning specified in .condarc or elsewhere? I know you've said it's done, but I don't know which version(s) of conda it is available in, nor what the syntax is.

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically locked because it has not had recent activity after being closed.

Please open a new issue if needed.

Thanks!