decouple package repository from spack core

[0xaf1f]

It would be great to be able to use the most up to date package repository without having to use the development version of the package manager itself. This was offhandedly talked about around #1325 (comment), but I'm filing it so it's not forgotten.

[adamjstewart]

We've talked about this before, and the conclusion we always reach is that Spack's core is in turmoil. Since we're still in beta mode, features are constantly being added and removed to Spack that affect how package.py files work. Right now, if we add/remove/modify how a directive works, we don't have to worry because we can simultaneously modify any package that needs this behavior. Once we decouple the package repository, this becomes a greater concern.

We've also talked about various ways to handle this. We could add logic to a package.py file that tells you which version of Spack is required for this version of the package.py file.

In general, I think most Spack developers agree that this is the right long term goal. It's just a matter of whether Spack is stable enough for us to move in this direction.

If you are a developer for or know any developers for other package managers with split package repositories (like Homebrew) I would love to here how they handled it and what negative consequences they encountered.

[samfux84]

@0xaf1f : For this reason, I have a second spack installation which is not used to install any package. I just update it all the time to get the newest versions of all packages.

Whenever I think that the newer package version brings some benefit, then I go to the productive spack installation, copy away the current package (to keep the original), replace it with the newest version of the package.

If this works -> keep the new package version (but don't delete the old one, such that you can always go back)
If there is a problem -> go back to the original package version of the productive spack installation

So far I did not run into bigger problems with the mix&match approach.

[0xaf1f]

Thanks. It's really helpful to know someone did this. I was also considering to use a second spack download this way, but then configure it as a secondary package repository.

[0xaf1f]

@adamjstewart,

If you are a developer for or know any developers for other package managers with split package repositories (like Homebrew) I would love to here how they handled it and what negative consequences they encountered.

Maybe ask on their mailing lists or something. I looked through the conda changelog and found this description of a new package feature and how it's handled by older versions of the package manager:

https://github.com/conda/conda/blob/90a0d121dab54fb57bd6e359dd3e673086b7c30a/CHANGELOG.md#new-feature-highlights-1

Constrained optional dependencies are supported starting with conda-build 3.0 (via conda/conda-build#2001. A new run_constrained keyword, which takes a list of package specs similar to the run keyword, is recognized under the requirements section of meta.yaml. For backward compatibility with versions of conda older than 4.4, a requirement may be listed in both the run and the run_constrained section. In that case older versions of conda will see the package as a hard dependency, while conda 4.4 will understand that the package is meant to be optional.

[0xaf1f]

Ok, I've already run into an issue using develop's repo with the latest release. I see

==> Error: name 'filter_compiler_wrappers' is not defined

when doing various spack subcommands.

$ grep -r 'filter_compiler_wrappers' /opt/spack/var/spack/repos/rolling/*
/opt/spack/var/spack/repos/rolling/packages/mvapich2/package.py:    filter_compiler_wrappers(
/opt/spack/var/spack/repos/rolling/packages/openmpi/package.py:    filter_compiler_wrappers('openmpi/*-wrapper-data*', relative_root='share')
Binary file /opt/spack/var/spack/repos/rolling/packages/openmpi/package.pyc matches
/opt/spack/var/spack/repos/rolling/packages/neuron/package.py:    filter_compiler_wrappers('*/bin/nrniv_makefile')
/opt/spack/var/spack/repos/rolling/packages/r/package.py:    filter_compiler_wrappers(
/opt/spack/var/spack/repos/rolling/packages/mpich/package.py:    filter_compiler_wrappers(
/opt/spack/var/spack/repos/rolling/packages/intel-parallel-studio/package.py:    def filter_compiler_wrappers(self):
/opt/spack/var/spack/repos/rolling/packages/hdf5/package.py:    filter_compiler_wrappers('h5cc', 'h5c++', 'h5fc', relative_root='bin')

so I guess this is one of those backwards-incompatible changes to the packaging format.

[adamjstewart]

Yup, that's one of the recent additions.

[alalazo]

Closing this discussion as stale. Feel free to reopen if you think there's something to add. For what is worth I think #7424 (comment) still depicts the current situation very well.

[ChristianTackeGSI]

Please don't close the issue just because it's currently stale.
Possibly give it a label like "much-later"?

What has changed?
Well, it's now nearly two years since the comment and spack is quite mature by now. So we should hopefully slowly reconsider this project.

About other package managers / utils:
dpkg is a package itself, so the build dependencies of some Debian packages have a Build-Depends: dpkg (>= 1.13) in them. Or debhelper (a toolset) is also references like this Build-Depends: debhelper (>= 9).

So we could start by letting spack itself provide a (simulated/virtual) spack package, so that package.py could just contain depends('spack@0.13:', type='build')?