Project Idea: Better haskell-specific tooling for working with nix

[mpickering]

Link to rendered proposal.

https://github.com/mpickering/summer-of-haskell/blob/034bd1b9759e17d2d57ccdf8d0e9bf1c5663ed13/content/ideas/cabal-nix-workflow.md

[mpickering]

Possible mentor: @ElvishJerricco

[ElvishJerricco]

@Ericson2314 and I have plenty of ideas for this.

[Ericson2314]

CC @kmicklas

[kmicklas]

Doesn't callCabal2nix already solve most of the pain here? I agree though that deeper integration between cabal and nix would be nice. In particular per module nix builds is the really missing piece (so a full nix-build is as fast/incremental as cabal new-build inside the nix shell created with callCabal2nix.)

[ElvishJerricco]

@kmicklas I don't think callCabal2nix is enough. There are a few major problems remaining with Nix's Haskell infrastructure.

• Multi-package building. In reflex-platform, we have some kind gross stuff for this. But ideally, cabal2nix would have support for cabal.project files, such that it would provide both a shell for incrementally building the project locally, and an individual derivation for each package (so they can be overlayed into haskellPackages).
• haskellDrv.env is annoying, and isn't strictly necessary.
• Even basic Haskell projects require an unfortunate amount of Nix know-how. It's not bad, but it's certainly enough to deter newcomers at the first sign of danger. Simplifying is really important. Something like reflex-platform's project, but for arbitrary Haskell work (hopefully even reflex work) would be great.
• Updating haskell package sets independently of the system package set is somewhat important for industrial use, but currently requires manually running something like stackage2nix and maintaining a whole infrastructure around that. We've run into this at work.
• Incremental building with cross compilation. John just got his Haskell cross work merged into master, but in my little experiments, I was unable to use cabal in a nix-shell to do these builds incrementally.
• DOCUMENTATION is the absolute worst part about using Nix with Haskell right now, for most people. It's in a really really bad state.

I'm sure there's more that's not off the top of my head.

[gbaz]

@Gabriel439 @intractable and @ixmatus may have thoughts on this...

[Gabriella439]

In theory you could directly translate a stack.yaml file to a Nix project build using import-from-derivation. There's also some prior art that is similar: https://github.com/jyp/styx

[Ericson2314]

I'm still interested in what I layed out in http://lists.science.uu.nl/pipermail/nix-dev/2016-September/021765.html and haskell/cabal#3882 . The big difference between this and other strategies is that it gives most build planning responsibility back to cabal-install. Version solving is back in, hard-pinning versions in Nix is out. Derivations should be component-wise everywhere Cabal/cabal-install is, with per-module grandularity for both to eventially arrive simultaneously down the road.

@kmicklas and I have resumed some preparatory work on Cabal, and hopefully there'd be a summer-sized piece we can break off from the rest.

[Gabriella439]

@Ericson2314: Wouldn't your proposal be subsumed by recursive Nix? cc: @taktoa

The idea behind using Recursive Nix is that you replace the ghc that cabal uses with a fake ghc script that calls one nix-build per object/interface file using the real ghc to build each file. Then you get integration with the /nix/store transparently without any changes to cabal and it works with other Haskell build tools, too (like stack)

[Ericson2314]

Also, with @Hadrian and dreams like haskell/cabal#4174, us Nix+Haskell people should be in better contact with @ndmitchell and @snowleopard. Extracting Nix derivations from Shake build plans would be a phenomenal way to better leverage our resources, but to the best of my knowledge is impossible with Shake's current design.

CC @taktoa

[Gabriella439]

@Ericson2314: The point of recursive Nix is that you don't have to integrate with Shake at all. Once you provide the fake GHC wrapping Nix all Haskell tools that build on top of GHC automatically integrate with Nix. Instead of writing on Nix integration per package manager you write one Nix integration per compiler.

[Ericson2314]

@Gabriel439 that does work. But I've never really been excited about recursive Nix for, admittedly, basically aesthetic reasons. A single IFD step vs pervasive recursive nix is a lot less dynamism.

More practically, it seems like quite a bit less work but changing Nix itself is, erm, politically quite difficult vs changing cabal, so it might not be.

[Gabriella439]

@Ericson2314: The reason I cc:ed @taktoa is that he tried exactly what you suggested for his summer internship at our company: using import-from-derivation to turn a .cabal file into an incremental Haskell build. Remy is highly proficient in both Haskell and Nix and worked on this full time for several months and still ran into several issues that require significant changes to cabal to implement this correctly. This is why I don't think it's an appropriate Summer of Haskell project. He can describe the specific issues in more detail.

[Ericson2314]

Yeah @taktoa and I have talked a lot about it. Basically my plan differs from that summer in that I have lamer goal poasts: per-component builds for now, until the Cabal overhaul that everyone wants happens. I guess I'm more interested in cutting down nix-specific configuration and cabal-install duplication than better caching as my first priority.

[taktoa]

I want to point out a slight inaccuracy in what @Gabriel439 just said: doing incremental builds with recursive Nix will not automatically give us per-object file incrementalism (only per-component incrementalism), since cabal calls ghc --make for every component. Instead, the advantage is mainly that we get access to the right flags and we get injected into the right environment when that component needs to be built, so all we have to do to get full incrementalism is write a drop-in replacement for ghc --make that calls nix-build on a set of n derivations if n = |transitiveClosure(modules-specified-on-command-line)|.

[ElvishJerricco]

AFICT, John's proposal doesn't have much to do with what @Gabriel439 and @taktoa are describing. John's talking about porting the package-level (component-level?) build plan of new-build, not the module-level plan of ghc --make.

John's plan of bringing back the solver in Nix does not resonate with me. I'm personally much more a fan of snapshot curation than solving. It's particularly relevant in Nix since we can provide binary caches for snapshots. Requiring each project to have its own freeze file means that each project needs its own binary cache (granted, most companies do this themselves anyway). reflex-platform, for instance, really relies on a snapshot. Having users come to the table with their own solved plan would be very likely to break Reflex.

As for incremental Nix, I don't think we need to go for full generality just to get something worthwhile for Haskell. At the risk of suggesting something that more knowledgeable people have already discussed: Why can't we just add a command to Cabal that subs ghc --make with ghc -M? Take the generated files, convert them to Nix, and IFD them to build the whole component. This would give us incremental builds without having to invent anything too fancy or time consuming.

One concern I have for recursive Nix: nix-build -E 'with import <nixpkgs> {}; runCommand "foo" {} "mkdir $out"' takes 0.35s to build on my machine, and 0.2s to run when cached. This is an extremely simple derivation. 0.2s is still better than most Haskell modules (not all), but I worry that that time will end up higher with a real derivation that does a real build. With callCabal2nix, we see times upward of 1s for each of the IFDs involved. That's certainly worse than a reasonable percentage of Haskell modules. If this gets too bad, then it will be no better from a dev experience perspective than an ordinary build.

Showstopper for recursive Nix for me: Doing it properly means having to properly interpret GHC's entire command line interface in order to intercept it. This is pretty hard, and likely to work differently between GHC versions. After having countless issues with the cc-wrapper in nixpkgs, I tend to prefer avoiding sophisticated wrappers. Something using IFD would be far more attractive to me (and I'm not convinced this isn't the best option anyway, even ignoring this particular issue).

@Gabriel439 FYI, stackage2nix recently created an IFD solution for converting stack projects at nix-build time. Not sure how pure it actually is...

---

Regardless. I really don't think the focus this year should be on radical reimaginings of how Haskell works at the fundamental level in Nix, unless we can get incremental Nix+Haskell with relative ease. Otherwise, I think the focus absolutely needs to be on things that prevent people from adopting Nix, not on things that already-nix-users would enjoy. Roughly these fall into these categories:

• Ergonomics: Just writing the basic configuration file for a Haskell project should be as simple and comfortable as a stack.yaml file, and it should come complete with as many development features. reflex-platform's project is making headway here, but it's still proving to be problematic for beginners.
• Documentation: Using Nix for Haskell development is horribly underdocumented. There is some exceptional content in the form of stuff like blog posts that would serve as a big benefit to this, but beginners don't know to look for them. This info needs to be centralized and expanded on in something like the nixpkgs manual.
• Reliability: One of the most common things I've seen turn away newcomers has been encountering multiple failing builds in their first foray with Nix. Stupid stuff like packages with no modules failing in the Haddock phase. We need to make it so that the common path is the happy path. Currently, we're not quite there.

Getting people in the door should be priority #1. With the right knowhow, Nix is already powerful enough; so we need to focus on making that power accessible to new people.

[Gabriella439]

@ElvishJerricco: Regarding documentation on using Nix for Haskell development, have you seen: https://github.com/Gabriel439/haskell-nix

Also, to clarify: I'm not recommending that the Summer of Haskell project should be to implement or use recursive Nix (quite the opposite)

I also agree that I prefer the snapshot model (i.e. how both Stackage and nixpkgs already work) over cabal's resolver. The problem with cabal's resolver is that while it is more flexible it does not provide good guidance on how to fix dependency resolution failures when they happen. Now, fixing that would be an interesting Summer of Haskell project, but it's probably out of scope for this particular discussion.

[taktoa]

Interestingly, @Ericson2314 seems to be interested in the only part of this project that is made easier with Recursive Nix, so I suspect this will come down to just overhauling Cabal. Here are some things that people could work on that seem independent of the IFD / Recursive Nix divide:

• Splitting Template Haskell out of GHC and into a separate binary that can emit a serialized representation of the modified AST, and modifying GHC so that it can accept a serialized AST. I mentioned this idea in my recent rant on pain points in Haskell:

  The fact that Template Haskell is built into GHC as opposed to being a separate preprocessing step executed via another binary (distributed with GHC) is a complete travesty. It really does not make any sense to couple the GHC that is used to evaluate Template Haskell to the GHC that used to compile the library or executable; this has caused massive problems with cross-compilation and with GHCJS. There is a workaround for this issue called iserv, but I don't think it is philosophically the right way to deal with it. Admittedly, there is a fairly significant problem with separating TH out this way, which is that it requires some way of pretty-printing the generated Haskell, but I think that problem should be dealt with by creating a versioned binary representation for the GHC Haskell AST, and then modifying the GHC frontend so that it can accept this binary format rather than a Haskell source file. Then the TH execution utility could just output this binary format rather than Haskell source, sidestepping the issue of pretty-printing a TH-modified AST (though admittedly a separate tool for pretty-printing these binary AST files would also be pretty useful). Going back to my point about splitting GHC up into smaller tools, this binary AST would also mean that we could make a ghc-parse executable that converts a (potentially TH-containing) Haskell source file into a binary AST file, and this would be extremely useful for editor tooling, though as usual the devil is in the details of versioning, stability, and ease of parsing this binary AST format. I would probably advocate shipping a tool along with GHC that, in addition to (slowly) pretty-printing the binary AST as Haskell source, can convert the AST to a few non-binary formats like s-expressions and JSON.

  What I didn't mention in that rant is that this would also be very useful for parallelizing and incrementalizing Haskell builds for Nix, as long as you add one other feature: some way of statically analyzing the effects of the TH used in some codebase. I'm thinking that this would look like a pragma that allows you to list the effects that the TH in a given module is allowed to use, plus a binary that, given a list of Haskell files, will return a DAG representing the order in which the Template Haskell for those files must be evaluated (if the pragma isn't specified, then we'd assume that all the effects are used, so the DAG would end up looking like the pointer graph of a linked list). Certainly this will allow for safe parallel evaluation of TH, which is definitely a goal, though I think if you generate the DAG in just the right way it will also have benefits for incrementalism (since, if a file uses TH but never reads or writes to the filesystem during TH execution, it is safe to evaluate the TH exactly once).

  Ideally, the TH effect-listing pragma would have a fine level of granularity, so that you can say "I only ever write to this path" or "I only ever read from this path". This is what can give us incrementalism rather than just parallelism.

• Incrementalizing phases of Cabal execution. For example, there's no reason why we couldn't (with IFD) have a Nix derivation for cabal configure and a separate Nix derivation for cabal build and a third derivation for cabal haddock. In particular, parallelizing the execution of tests and Haddock generation during Haskell builds under Nix could yield a fairly significant performance improvement, as would parallelizing the building of unrelated components (though the latter is significantly more difficult as it requires knowing the component graph).

• I think there is probably interesting work to be done on improving the speed, parallelism, and incrementalism with which Haddock is generated from Haskell source code. I haven't profiled Haddock generation, but if a significant amount of time is spent just parsing the Haskell source code, there would be great benefits from splitting Haddock into two different binaries (just as I was mentioning with GHC earlier): one that parses a single Haskell file into a binary format, and one that takes a directory tree of files that would otherwise be .hs files, but are instead binary-serialized ASTs, and combines them to generate all the HTML and such.

• Along the lines of the pragma I suggested earlier for TH, there could be a pragma for the effects used during test execution (in the Main.hs of the test). This could be used to infer when it is safe to run tests in parallel without sandboxing. I'm not sure how hard it would be to verify that the pragma is being used in a safe way (certainly there are some conservative static analyses we could do, or perhaps main could have a type other than IO (), but it's unclear to me how difficult that would be to implement).

• Overhauling Cabal so that it emits a static description of the build, which is necessary for the IFD version of incremental Haskell Nix builds, would be probably be good even in the case where we use Recursive Nix.

[taktoa]

Oh, somehow I didn't notice that there was actually a proposal to read here. I wrote all that without having read anything other than these comments. I'll read the proposal now and comment again with my feelings on it.

[taktoa]

My responses to the contents of this proposal:

• I use cabal-install inside a nix-shell. It works decently well.... I haven't heard many people say that they use stack with Nix. The support that cabal-install has added in 2.0 for Nix was definitely a big improvement to my development experience.
• Certainly I agree that there are aspects of using Cabal with Nix that could be streamlined. I think a pretty good solution would be to effectively bake this kind of Makefile often used with Cabal + Nix into Cabal (or write a library that makes it easy to use that workflow with a custom Setup.hs). We should be very careful to support workflows where the output of cabal2nix could go into arbitrary parts of the project directory tree, though, since otherwise this approach will not scale to big projects.
• Easy support for pinning nixpkgs commits seems like a great idea.
• Perhaps it would be good to also have baked-in support for using a best-practices Nix workflow with cabal init / stack new. There's no reason that couldn't just be an external script or whatever but in practice people often won't go out of their way to use something like that. We do need to be careful that we get the template right the first time though, since people will probably not regenerate their projects from a new template.

[ElvishJerricco]

@ElvishJerricco: Regarding documentation on using Nix for Haskell development, have you seen: https://github.com/Gabriel439/haskell-nix

Yes, and a few other like it. These are the sorts of resources that need to be put somewhere that a beginner is likely to actually find, specifically the nixpkgs manual.

@Gabriel439 @Ericson2314 @taktoa Can we all agree that the discussion @Ericson2314 and @taktoa are having is about an issue that is out of scope for the SoC? If so, it should probably move to a different thread.

[taktoa]

@ElvishJerricco
I don't have any particular problem with moving to another thread, but I don't see my suggestions above as being necessarily out of scope for the SoC, though admittedly most of them involve hacking on GHC/Haddock, and I'm not very familiar with either of those codebases, so I can't say for sure whether any of those tasks (or parts of them) are feasible to complete in a summer.

[taktoa]

@ElvishJerricco

As for incremental Nix, I don't think we need to go for full generality just to get something worthwhile for Haskell. At the risk of suggesting something that more knowledgeable people have already discussed: Why can't we just add a command to Cabal that subs ghc --make with ghc -M? Take the generated files, convert them to Nix, and IFD them to build the whole component. This would give us incremental builds without having to invent anything too fancy or time consuming.

I didn't notice this comment before, but this is basically what I tried to implement last summer. There are a bunch of problems with this approach:

• ghc -M does not handle Template Haskell (hence my comments above on making TH a preprocessing step rather than part of ghc execution). The only conservative approximation of ghc -M that supports TH will completely evaporate all incrementalism when there is even a little bit of TH in the codebase. A lot of codebases use TH, so I don't think you can just ignore it. Moreover, TH is often responsible for some of the longest Haskell build times, and making TH a preprocessing step will also be a big improvement for Haskell cross-compilation.
• You can't just do ghc -M, since there is also the component-level build graph; IIRC there are situations with components where actual compilation has to happen before ghc -M can be run on the next component, or else ghc -M will fail. This means you can't just run Cabal with a PATH where ghc is actually ghc -M, since the dependency computation and compilation steps are interleaved.
• Sure, nix-build is slow, but IFD is also pretty damn slow! I am of the opinion that we should just build these things, and then optimize Nix based on their usage patterns; I find it likely that there is a lot of room for optimization in the Nix codebase.

Showstopper for recursive Nix for me: Doing it properly means having to properly interpret GHC's entire command line interface in order to intercept it. This is pretty hard, and likely to work differently between GHC versions. After having countless issues with the cc-wrapper in nixpkgs, I tend to prefer avoiding sophisticated wrappers. Something using IFD would be far more attractive to me (and I'm not convinced this isn't the best option anyway, even ignoring this particular issue).

The wrapper script we write that pretends to be GHC can be written in any language, including Haskell, so we could just reuse GHC's command line parser. The algorithm here is basically:

1. Parse command line flags using GHC's parser.
2. Use the code in GHC that implements ghc -M to compute the module dependency graph.
3. Compute a new command line for each node in the graph that is based on the module dependency graph and the output of the GHC CLI parser.
4. Emit a Nix file (or drv file, for speed) that encodes this build plan.
5. Execute nix-build on that file.
6. Symlink the files in ./result/* into the appropriate places based on the semantics of ghc --make.

At the end of the day, Recursive Nix means that we can inject ourselves into the build process in a way that is completely agnostic of the build system, and only concerns itself with the semantics of the compiler. IFD-style approaches are highly coupled to the build system. In Haskell we pretty much only have one build system, but for other languages there are many, so Recursive Nix really is just more scalable solution.

[kmicklas]

I agree with @taktoa 's arguments for recursive Nix. Another reason I am somewhat against IFD as a solution to these problems is that it entrenches the privilege of Nix-the-configuration-language to be the one sole way to control Nix-the-generic-computation-memoization-system. Recursive Nix provides a much more coherent layering story where any tool can hook into the caching system. Someday theoretically even the Nix language evaluator could be implemented this way so the evaluation of nixpkgs could be cached. But issues with the design of Nix are probably outside the scope of Summer of Haskell...

[ElvishJerricco]

it entrenches the privilege of Nix-the-configuration-language to be the one sole way to control Nix-the-generic-computation-memoization-system

I don't see how this is true. An alternate evaluation language would be just as capable of this as Nix.

Anyway, I really don't think we should continue the discussion about recursive Nix here, if we don't think we're going to be proposing it as SoC project.

[mpickering]

This thread quickly spiralled out of control whilst I was away over the weekend! It's great that there is lots
of discussion about using Haskell with nix but I feel the need to bring this proposal back on track.

Recursive nix is definitely off the table and I deliberately left incremental rebuilding from this proposal
as it is a well-known rabbit hole.

It seems that @Ericson2314 has previously considered this question in a lot of detail but I don't understand the concrete proposal from the original email thread. There is more detail in the ticket however it hasn't been updated since 2016 so I'm unsure what the status of the proposal is. It really needs a single cohesive summary so that people can understand it better.

I think @ElvishJerricco summed up the situation well at the bottom of his earlier comment.
Focusing on things which newcomers find difficult is more appropriate as it will have greater impact
and also the contributor is likely to be a newcomer themselves.

Thanks to @taktoa for his thoughtful comments on the proposal. He is right that cabal-install works "decently well" with nix but the point of the proposal is to make it work "very well". There are definitely ergonomic improvements which could be made (another example, the src of a package should be just the files specified in a cabal file rather than ./.).

What is important now is that we can specify a contained, well-motivated proposal rather than bigger design philosophy questions (which can gladly continue in other forums). Can anyone suggest a suitably sized well-specified chunk that they think would be appropriate? Will's earlier comment is a good start I think.

[ElvishJerricco]

Here's one concrete proposal:

Cabal2nix for cabal.project files

I want to be able to use cabal2nix on projects that use cabal.project files to declare multi-package projects. MVP would accept the following cabal.project:

-- cabal.project
packages:
  ./foo
  ./bar

-- foo/foo.cabal
name:                foo
version:             0.1.0.0
license:             BSD3
license-file:        LICENSE
author:              Will Fancher
maintainer:          elvishjerricco@gmail.com
build-type:          Simple
extra-source-files:  ChangeLog.md
cabal-version:       >=1.10

library
  exposed-modules:     Foo
  build-depends:       base >=4.9 && <4.10
                     , baz
  hs-source-dirs:      src
  default-language:    Haskell2010

-- bar/bar.cabal
name:                bar
version:             0.1.0.0
license:             BSD3
license-file:        LICENSE
author:              Will Fancher
maintainer:          elvishjerricco@gmail.com
build-type:          Simple
extra-source-files:  ChangeLog.md
cabal-version:       >=1.10

library
  exposed-modules:     Bar
  build-depends:       base >=4.9 && <4.10
                     , raz
                     , foo
  hs-source-dirs:      src
  default-language:    Haskell2010

And produce the following Nix file:

self: super: {
  foo = self.callPackage({ mkDerivation, base, baz, stdenv }:
    mkDerivation {
      pname = "foo";
      version = "0.1.0.0";
      src = ./foo;
      libraryHaskellDepends = [ base baz ];
      license = stdenv.lib.licenses.bsd3;
    });

  bar = self.callPackage({ mkDerivation, base, foo, raz, stdenv }:
    mkDerivation {
      pname = "bar";
      version = "0.1.0.0";
      src = ./bar;
      libraryHaskellDepends = [ base foo raz ];
      license = stdenv.lib.licenses.bsd3;
    });

  my-project-shell = # or some other name
    self.callPackage ({ stdenv }: stdenv.mkDerivation {
      name = "my-project";
      nativeBuildInputs = [(self.ghcWithPackage (p: with p; [baz raz]))];
    })
}

This is sort of incomplete (we need more in the my-project-shell derivation). But the point is to produce an overlay function that can be applied to haskellPackages, such that it defines all the local packages individually, and provides a my-project-shell derivation that you can use nix-shell on to get a shell for developing those packages incrementally.

Bonus points for creating a callCabal2nix style function for this, using "import-from-derivation" so that my default.nix can simply look like:

# default.nix
# callCabal2nixProject returns a modified `haskellPackages`, with the above overlay applied.
(import <nixpkgs> {}).haskellPackages.callCabal2nixProject ./cabal.project {}

# shell.nix
(import ./.).my-project-shell

[gwils]

"should load enter an" doesn't look quite right.

[alpmestan]

I very much second @ElvishJerricco's proposal. I have in fact been discussing an idea like this on and off in the past few months with @hvr and @bgamari. It is the biggest missing bit of infra for me.

I'm quite often not really happy with the package sets that we have in nixpkgs and would more generally welcome some more tooling to easily create them "from scratch" or from existing ones. It seems very natural to just lean on cabal-install's solver to figure out everything we need to generate a nix expression that will faithfully reproduce the versions and what not that cabal-install would use if it were to build the packages itself.

Handling dead-simple scenarios should not require a lot of work, especially given that we have things like cabal-plan to help us. The devil will probably be in the details (like being able to interpret everything we can possibly see in a cabal.project or .cabal file in Nix terms). It would be nice of course to have more user-friendly Nix functions or CLI tools to generate our own package sets for all sorts of situations (that is, more user-friendly than the ones we have in nixpkgs at the moment), but I'm unfortunately not able to provide a good specification for what I would like to have, so in the meantime @ElvishJerricco's proposal would already go a long way towards making my life easier :)

Regarding cabal --nix and stack --nix: I have used both in the past too, they work OK but it seems to me like there isn't much more to be done except polish things up a little bit and write more documentation and examples so that people who're just getting started with Nix can easily build with their projects with {cabal, stack} --nix, using nix only to provision non-haskell packages. On the other hand, @ElvishJerricco's proposal does require some serious work and is IMO a more fruitful avenue to explore, and should be doable in a summer given the number of people interested in this and knowledgeable enough to help if/when needed.

This would also allow us to build more things on top of it: generate a haskellPackages-like package set from .cabal/cabal.project/etc files, maybe provide helpers for exposing those as overlays to nixpkgs, and surely more on top of that.

[mpickering]

Thanks @ElvishJerricco, I think that's definitely a good place where a student could get started. There is plenty of food for thought in this thread now but it isn't our responsibility to completely write a student's proposal for them!

I'm also curious about what additional documentation you think is necessary for the haskell nix ecosystem? I have been trying to think of things that need writing about but nothing seems to jump out at me. Gabriel's haskell-nix github repo is quite good imo. There's also another similar project by @shajra (https://github.com/shajra/example-nix).

[ElvishJerricco]

I think @bgamari's work is a bit stronger than I was hoping for. It looks like it actually looks at your cabal build plan and produces an entire package set from that? @alpmestan is this something more like what you would have hoped for? I think it's valuable to have this as an option, but I think it's important that our tools are built around the default of using nixpkgs's Haskell package set.

@mpickering Those github guides are indeed great, but they're in very much the wrong place. I usually have users come to me asking for help saying they've looked at my documentation on reflex-platform, sometimes the nixpkgs manual, and rarely anything else. Preferably, everything that is general to nixpkgs needs to be in the manual. And preferably, more of the reflex-platform project infrastructure can be upstreamed and put in said manual. These third party resources are just not discoverable and are more likely to fall out of date with time.

Additionally, you'll find almost no documentation on the functions in haskell.lib or haskellPackages unless you go and physically look at their definitions; a journey most newcomers are unwilling to brave. There's also not much documentation on when to use what; i.e. even if there were clear docs on callHackage, I'd be surprised if a beginner knew to look for it. So there needs to be some documentation outlining the happy paths, and how to do whatever you would normally do with stack.yaml in Nix.

EDIT: Spelling

[alpmestan]

@alpmestan is this something more like what you would have hoped for? I think it's valuable to have this as an option, but I think it's important that our tools are built around the default of using nixpkgs's Haskell package set.

Yes, I'm absolutely fine with defaulting to what nixpkgs comes with, really all I'm suggesting is to make it easy to use other package sets, which naturally prompts the question of how we can make it easy/trivial to create other ones. We already have overriding/overlay mechanisms in nixpkgs to tweak existing ones, but there is no simple way to "mass-import" a bunch of specific package versions in an attr set and call that a package set, with the same capabilities as haskellPackages and its siblings, in nixpkgs. This is what I'm missing the most. That would be a good first milestone and is what Ben's patches do more or less IIRC, but I don't think it is rock solid just yet.

If you have packages a and b in your cabal.project, it might happen that a and b need different versions of some package c. cabal can deal with that just fine but it would be a lot more annoying in nix land, as most of the time we just instantiate some dependency with whatever is in scope in the package set with the very precise name of the argument for that dependency in the derivation. It's trickier here as we'd have to explicitly wire, say, c-0.1 to a and c-0.2 to b, in the nix expressions generated by the program.

I haven't looked in great detail at Ben's code but I don't think it handles that yet. It would also be nice to be able to retrieve and apply the flags specified in cabal.project files (or more generally the flags that cabal would configure the packages with if it were to build the project(s), as well as other things that one may find in cabal.project files (but not everything I suppose, as we can't accept to work with any compiler like a local ghc build, at least not without a bit of Nix code).

Anyway, sorry for hijacking the discussion with my specific use case. I do think the "mass-import" thing would be nice, with the possible bonus of handling flags and all sorts of reasonable cabal.project options and what not faithfully. This looks like a rather well-defined project, Ben's patches are a good starting point and I don't think a summer is either way too long or way too short to build the "mass import" bit in a way that supports multiple versions of a same package (or even same version but configured differently, e.g with or without some flag?) and ideally even picks a suitable compiler version when there are constraints on it (and defaults to the latest major release or something), and then "decorate" the list of packages so that it behaves like haskellPackages and in particular has a callPackage attribute etc (there's a "low-level" make-package-set function somewhere IIRC which would be the building block for this). At that point, it would be usable and we could think of a few cherries to add on top.

Not sure that kind of project is an easy sell to the broader haskell community though.

[Ericson2314]

@ElvishJerricco

I think @bgamari's work is a bit stronger than I was hoping for...

So generic-builder.nix will always be playing catch-up with Cabal, and this is really bad for new users. It's not that the missing features are necessarily crucial, but the fact that there is an impedence mismatch that makes it inhospitable. This is why I want to move towards leveraging cabal and cabal-install more: less nix code means fewer pitfalls relative to Cabal.

As far as package sets go, we can easily replicate them with cabal constraints in project files. Nixpkgs can contain big fat ones of those which it will cache, and projects can copy it into their cabal.project.local. This is great because it's a completely nix-agnostic solution.

[ElvishJerricco]

If you have packages a and b in your cabal.project, it might happen that a and b need different versions of some package c.

@alpmestan That doesn't sound like a thing cabal.project is supposed to be used for...

Anyway, we do have makePackageSet, which is about as close to what you're asking for as I can think of.

[alpmestan]

@alpmestan That doesn't sound like a thing cabal.project is supposed to be used for...

Hm? I'm just describing the simple case with packages: a/ b/ in a cabal.project, where it just happens that b needs an old c but a needs a recent one. I'm pretty confident that cabal-install can deal with this.

Anyway, we do have makePackageSet, which is about as close to what you're asking for as I can think of.

Right, this is what I alluded to in a comment above. What we're missing is some infra to populate package sets using this by directly getting the versions and what not from cabal.

[mpickering]

@Ericson2314 I really want to understand what your vision for your projects are but I'm having a hard time. Please can you clarify with lots of examples so that I can understand how you are planning to proceed.

You say that "this is why I want to move towards leveraging cabal and cabal-install more" but isn't that what @alpmestan is also suggesting? He is suggesting taking the output from cabal, whatever plan it decides, to set up the environment. This is already using Cabal more as currently what Cabal thinks the plan should be isn't really taken into account, the information is just parsed from the cabal file. This seems to also be exactly in line with the first paragraph on haskell/cabal#3882.

Perhaps it would be useful to everyone if you could briefly describe what you imagine the precise responsibilities of cabal to be and the precise responsibilities of nix. I think that would go a long way to clarifying the situation.

[Ericson2314]

@mpickering yeah sorry, was responding to @ElvishJerricco; edited to make clear. @alpmestan and I are advocating for the same sort of thing, as far as I can tell.

I want cabal to solve for all versions and everything, and come up with a completely static dependency graph which Nix then builds. Ideally that graph would eventually be per-module granularity, but I think offloading the planning to Cabal is more important than making the graph fine-grained.

I need to do more investigating of cabal-install's source, but I recall it does already strive to be as static as possible. So exporting that into a course pkg- and component-granular (need to be conservative with custom setups) build graph for Nix shouldn't be too hard.

[ElvishJerricco]

@alpmestan Ah I see. Regardless, I do not think new-build is intended to provide different versions of a dependency to different packages in the cabal.project file. When I try the following:

-- cabal.project
packages:
  foo/
  bar/

-- foo/foo.cabal
name:                foo
version:             0.1.0.0
license:             BSD3
license-file:        LICENSE
author:              Will Fancher
maintainer:          elvishjerricco@gmail.com
build-type:          Simple
extra-source-files:  ChangeLog.md
cabal-version:       >=1.10

library
  build-depends:       base >=4.9 && <4.10
                     , mtl == 2.2.1
  hs-source-dirs:      src
  default-language:    Haskell2010

-- bar/bar.cabal
name:                bar
version:             0.1.0.0
license:             BSD3
license-file:        LICENSE
author:              Will Fancher
maintainer:          elvishjerricco@gmail.com
build-type:          Simple
extra-source-files:  ChangeLog.md
cabal-version:       >=1.10

library
  build-depends:       base >=4.9 && <4.10
                     , mtl == 2.2.0.1
  hs-source-dirs:      src
  default-language:    Haskell2010

I get:

Resolving dependencies...
cabal: Could not resolve dependencies:
trying: base-4.9.1.0/installed-4.9... (dependency of bar-0.1.0.0)
next goal: mtl (dependency of bar-0.1.0.0)
rejecting: mtl-2.2.1 (conflict: bar => mtl==2.2.0.1)
trying: mtl-2.2.0.1
next goal: transformers (dependency of mtl-2.2.0.1)
rejecting: transformers-0.5.2.0/installed-0.5..., transformers-0.5.5.0,
transformers-0.5.4.0, transformers-0.5.2.0, transformers-0.5.1.0,
transformers-0.5.0.1, transformers-0.5.0.0 (conflict: mtl =>
transformers==0.4.*)
rejecting: transformers-0.4.3.0, transformers-0.4.2.0 (conflict:
base==4.9.1.0/installed-4.9..., transformers => base>=2 && <4.9)
rejecting: transformers-0.4.1.0 (conflict: base==4.9.1.0/installed-4.9...,
transformers => base>=2 && <4.8 || >=1.0 && <2)
rejecting: transformers-0.3.0.0, transformers-0.2.2.1, transformers-0.2.1.0,
transformers-0.2.0.0, transformers-0.1.4.0, transformers-0.1.3.0,
transformers-0.1.1.0, transformers-0.1.0.1, transformers-0.0.1.0,
transformers-0.0.0.0, transformers-0.5.3.1, transformers-0.5.3.0,
transformers-0.5.0.2 (conflict: mtl => transformers==0.4.*)
rejecting: transformers-0.4.0.0 (conflict: base==4.9.1.0/installed-4.9...,
transformers => base>=2 && <4.8 || >=1.0 && <2)
rejecting: transformers-0.2.2.0, transformers-0.1.0.0 (conflict: mtl =>
transformers==0.4.*)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: mtl, base, transformers, bar

It wants to use one solved plan for the entire project, and I think this is sane.

@Ericson2314, requiring users to copy a cabal config out of nixpkgs, and somehow keep that in sync as nixpkgs gets updated, in order to keep the cache working, is exactly the sort of thing that causes astonishment for newcomers. And I personally would be extremely frustrated to have such a workflow by default. I agree that using a cabal solved plan should be possible, but the default should definitely be as it is today: A curated snapshot that requires zero hassle, and minimal astonishment to get to work with. Point being, a cabal config -> nix program that yields something usable with makePackageSet would be nice, but it's a fundamentally different project than my proposed idea above, and I'm not convinced it aids in making Nix accessible to newcomers in any way.

[Ericson2314]

@ElvishJerricco Don't take my "copying" method as something permanent. People already want a way to import constraints for LTSes, so I'm sure something could be put together.

They way curated packages work today is atrocious. We rely on each ./Setup configure preferring the packages already installed, but from the Cabal perspective this is a terrible impurity / source of non-determinism: planning should not be biased towards what's already installed. Indeed that bias was the source of much "Cabal hell'. Cabal and cabal-install are and will continue migrating away from this, leaving us on an increasingly stale and moribund code path.

So, no problem if you want "use the cached package set" to be the default, but we really ought to accomplish it in the way I describe. We could wrap cabal (or have the IFD derivation) pass the nixpkgs default constraints by default, for example. Really, my priority here is not fighting new-build over joining the version-solving vs curated package sets brawl, here. (Though yes, full disclosure for everyone else, I do prefer the former as I've told you in person.)

[gbaz]

The plan.json generated by new-build gives a solution that is not biased towards any installed packages. Perhaps some low-hanging fruit would be to teach the nix tools to extract buid plans from that instead?

[Ericson2314]

@gbaz Yeah plan.json is a good start. But it's just sort of an ad-hoc sample of different properties of the install plan, more as a demo and for debugging than anything else. I'm not sure whether its better to continue dumping more stuff in it, or start over. For the record, there was a similar issue where the Rust Cargo people were wondering whether the "lockfile" (like new-build freeze file) was sufficient, but no it wasn't Cargo never attempts to do everything from the lockfile alone.

What we need is cabal-install to dump something which it can then ingest as its sole input and do everything else with, to prove that the information therein is indeed sufficient. At this point, I'm pretty convinced that such a proof is the only way to keep the language package managers honest :).

[gbaz]

My understanding is that plan.json doesn't contain everything, but it contains "everything you need to get everything". I.e. it can point you to what cabal files you want as well, and the union of the cabal files and the plan.json information is sufficient?

Or is there something I'm missing...

[hvr]

@Ericson2314 IMO (I may be biased, because I'm in part to blame for the way plan.json looks today) it's a bit unfair to call plan.json "more as a demo and for debugging than anything else", as tooling like http://matrix.hackage.haskell.org/ are built on top of plan.json to get access to everything it needs to uniquely determine the properties of the install-plan. In your opinion, what specifically is missing from plan.json to describe the install-plan configuration (and also note that there's s distinct cabal build-info feature planned which is deliberately separate from plan.json)?

[Ericson2314]

Ok I'm just out of date I think. Glad to be wrong :).

[ElvishJerricco]

So here's a brief attempt to form @Ericson2314's thoughts into a more concrete proposal, potentially for the SoC:

Building custom Nix package sets from plan.json

I want to be able to take a plan.json file from cabal new-build, and produce a Nix package set like the one in hackage-packages.nix, for use with haskell.lib.makePackageSet. This would allow users to produce custom package sets for their project from the latest Hackage DB with relative ease. This may require changes to cabal, in case plan.json isn't enough, or the tooling can't produce a good enough plan.json easily enough.

@Ericson2314 does that sound like an accurate and actionable proposal for the SoC? We can worry about whether or not it makes sense to make it the default in nixpkgs after the summer.

[Ericson2314]

@ElvishJerricco maybe that's a good first step, but I want to stop using generic-builder.nix and all that.

[ElvishJerricco]

@Ericson2314 Could you briefly summarize how/why a SoC student would work on getting rid of generic-builder?

[Ericson2314]

generic-builder.nix is duplicating cabal-install and forever playing catch up in terms of feature completeness. We should just blindly do what cabal-install does.

[Gabriella439]

@Ericson2314: I don't follow. generic-builder.nix uses ./Setup.hs which in turn uses the Cabal library, so what is cabal-install providing that the Cabal library is not providing?

[hvr]

(@Ericson2314 maybe off-topic, but is there anything interesting mentioned around haskell/cabal#4646 (comment) that could be relevant to this discussion?)

[Ericson2314]

@Gabriel439 So concretely I don't think we should be ./Setup configureing at all (except for custom setup pkgs where even cabal-install does).

But more important to me in the prinicple of deduplicating stuff. cabal-install has a bunch of "nix-style" stuff. We, to the extent that is possible, should literally be sending the same plan to either real nix, or the nix-style implementation. The more we do the same thing, the more Nix can be a drop-in replacement.

I'll need to get back in the cabal-install codebase to have more specifics; its been a year until a few days ago.

[mpickering]

@jaspervdj What is needed in order to merge this idea?

[jaspervdj]

@mpickering Nothing from your side -- I got a bit behind on the voluminous discussion on this ticket. I'll review it this week to make sure it's somewhat agreeable to all parties involved and then get it merged.

[jaspervdj]

Reformatted/tweaked and merged in #49