toShell*: turn nix data-structures into its shell equivalents

[kamadorueda]

Motivation for this change

Bash's data-structures are quite limited, mapping from Nix data structures into Bash ones can be sometimes tricky. By adding this built-in we enhance our scripts capabilities in a very clean way

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 21.11 Release Notes (or backporting 21.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[andersk]

That eval mechanism is unnecessarily complicated. Why not simplify to this?

toShellMap = attrs:
  "(${
    concatStringsSep " " (mapAttrsToList
      (key: val: "[${escapeShellArg key}]=${escapeShellArg val}") attrs)
  })";

Usage:

''
  declare -A myMap=${toShellMap myMap}
''

[kamadorueda]

That eval mechanism is unnecessarily complicated. Why not simplify to this?

toShellMap = attrs:
  "(${
    concatStringsSep " " (mapAttrsToList
      (key: val: "[${escapeShellArg key}]=${escapeShellArg val}") attrs)
  })";

Usage:

''
  declare -A myMap=${toShellMap myMap}
''

---

I find your proposal very interesting,
changes were applied,
thanks for your review

I also added two flavors of toShellMap and toShellArray
(called toShellMapFile and toShellArrayFile)

Let me explain why this is a good idea

There are two widely used ways of calling mkDerivation:

• external builder:

  mkDerivation {
    builder = ./builder.sh;
  }

• in-place builder:

  mkDerivation {
    builder = builtins.toFile "builder" ''
      source $stdenv/setup
      ...
    '';
  }

If we keep only toShellMap
some users would be forced to write
boilerplate like this:

nixpkgs.stdenv.mkDerivation {
  name = "example";
  builder = ./builder.sh;

  # Unnecessary wrapper, can be abstracted with `toShellMapFile`
  myMapSetup = builtins.toFile "setup" ''
    declare -A myMap=${toShellMap testData}
  '';
}

source $stdenv/setup
source $myMapSetup  

for key in "${!myMap[@]}"; do  
  val=''${myMap[$key]}
  echo $key: $val
done

Now, with toShellMapFile, users could write:

nixpkgs.stdenv.mkDerivation {
  name = "example";
  builder = ./builder.sh;

  # Minimal code
  myMapSetup = toShellMapFile "myMap" testData;
}

source $stdenv/setup
source $myMapSetup 

for key in "${!myMap[@]}"; do
  val="${myMap[$key]}"
  echo $key: $val
done

and

nixpkgs.stdenv.mkDerivation {
  name = "example";

  # Minimal code, also
  builder = builtins.toFile "builder" ''
    declare -A myMap=${toShellMap testData}
  '';
}

covering common uses of mkDerivation with minimal boilerplate ✨

[andersk]

Even in the external builder case, instead of myMapSetup = toShellMapFile "myMap" testData and source $myMapSetup, one could just as easily write myMapCode = toShellMap testData and eval "declare -A myMap=$myMapCode", which is much clearer that it’s creating an associative array. I don’t see the advantage of wrapping this in a file.

[kamadorueda]

one could just as easily write myMapCode = toShellMap testData
I don’t see the advantage of wrapping this in a file.

env vars length, CLI arguments count, and others, have hard limits

example

let
  testData = genAttrs (map toString (range 1 10000)) (name: "value");
in
nixpkgs.stdenv.mkDerivation {
  name = "example";
  builder = ./builder.sh;
  someVarSetup = toShellMap testData;
}

this is why files (example: passAsFile) come into play, they allow people to overcome those limits by encapsulading lots of data into small identifiers

now, this do happen in real life, for example:

• Using toBashMap at scale fluidattacks/makes#746

toShellMapFile is not bound to these limits

anyway, if people prefer and their maps are small, they can use toShellMap :)

I don’t see the advantage of wrapping this in a file.

For big maps or people who do not like using eval, toShellMapFile is the way to go

[andersk]

For big maps, passAsFile already exists and can be used with toShellMap. For people who object to using eval, they should equally object to generating a file and using source, since that’s just a less efficient reimplementation of eval. Neither seems like a good reason to double the surface area of this library interface, to me. But I’m happy to hear what other reviewers think.

[kamadorueda]

@andersk The PR was updated to be exactly as you propose

Thanks for the feedback

[roberth]

Isn't structuredAttrs supposed this problem, well, structurally? (Excuse the bad pun)
Perhaps that could be a simpler solution to your original problem?

[kamadorueda]

Isn't structuredAttrs supposed this problem, well, structurally? (Excuse the bad pun)
Perhaps that could be a simpler solution to your original problem?

@roberth I don't think structuredAttrs is a solution, because it does not exist right now

• nix stable

  

• nix unstable

  

• nixpkgs unstable

  

• Still a draft after 2 years: stdenv: enable __structuredAttrs #72074

On the other hand, this PR gets the job done, today, with docs: