lib/generators: add RFC42-style libconfig format

[ckiee]

Description of changes

Hey again. I've split out the libconfig serializer from #167388 and took the time to acknowledge a few more edge-cases, as well as adding a test and auto-validator.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/1647

[infinisil]

I'm wondering, wouldn't it be better to write these generators in a generic programming language, converting from the builtins.toJSON output to the desired format? While this wouldn't work for lib.generators since it needs to return a Nix string, it should work for pkgs.formats, which is mostly what's used for configuration generation nowadays. The benefits would be faster evaluation, ability to reuse existing tooling, better testability.

[ckiee]

@infinisil: I think that is the way forward generally but libconfig doesn't really have much of an ecosystem or any CLI tools as far as I can tell so I'd like to continue with this PR as-is. I'm already 3.5 layers deep (#165936 -> #167388 -> [this PR] -> taking over as upstream maintainer soon most likely) and would really rather not recurse more.

[infinisil]

It's just that this adds a whole bunch of code that's slow to evaluate, hard to debug, test and maintain. And yes that's half of nixpkgs already but I don't think it's a good idea to make this worse. Others might review and merge this PR, and I won't interfere with that, but personally I'm gonna abstain from this and focus on cleaner code instead.

[roberth]

I agree with infinisil, and the semantics are fragile; see comment.

[roberth]

Please don't assign meaning without context. Such assumptions are likely to lead to bugs.

"12A" -> "12A"
"12B" -> "12B"
...
"12L" -> 12

This is not how Nix works. Strings are strings. If you want a number, you write a number, or call a function that returns one.

Instead, you could represent these special with special wrappers instead mkOctal, etc, but do you really need multiple ways to write the same number?

[ckiee]

but do you really need multiple ways to write the same number?

Spec says there are a few scalar types and @anund previously said:

~ This [couldn't] currently generate valid config for the example. :(

Other then that mk{Octal|…} seems reasonable but annoying, future note: make it return just a simple { _type = …; … } attrset.

I'll get to this Eventually.

[anund]

Playing with https://github.com/PixlOne/logiops/blob/master/logid.example.cfg as an example.

devices = [
  {
    #<snip> out direct translations
    buttons = [
      {
        cid: (toLibConfig.mkOctal "0xc3");
        action =
          {
            type: "Gestures";
            gestures: [
              {
                direction: "Up";
                mode: "OnRelease";
                action =
                {
                  type: "Keypress";
                  keys: (toLibConfig.mkArray ["KEY_UP"]);
                };
              }
            ];
          };
      }
    ];
  }
];

#nesting would also need to work
(toLibConfig.mkArray [(toLibConfig.mkOctal "0x3")]);

Is that the sort of wrappers you have in mind @roberth?

[roberth]

If NixOS/nix#7578 is implemented, the hex and octal numbers won't need any sort of escaping or wrapping. That is, unless the program parsing the config makes decisions based on the syntax used, which would be decimal regardless of what base the module user used to configured the number.

I don't think we have many examples of wrappers yet, but in the module system we generally use a representation that's an attrset with a _type attribute. For example:

  mkOverride = priority: content:
    { _type = "override";
      inherit priority content;
    };

I don't think we have any examples of formats using this approach. (I kind of expected at least something RFC42-like to use this, so sorry for my brevity earlier)
We could add a general mkUnescape to lib, but otherwise I don't know exactly where to put it. Perhaps we could have (formats.foo {}).type and also (formats.foo {}).lib?
The meaning of this mkUnescape is that it would take a string and concatenate it directly into the config file without any processing (except perhaps re-indenting, for some formats). This is in contrast to regular strings, which should be escaped to become strings in the target config language. This means that the string is reinterpreted according to the target configuration language syntax, and this would allow a user to specify any syntax, assuming it's going to be valid in the end. This includes such things as hex: lib.mkUnescape "0xc3" would do the job.

cc @infinisil

[roberth]

That's not to say that a formats.libconfig.mkOctal wouldn't be nice to have.

[anund]

I also notice I've mixed up octal and hex in my example(s).

[anund]

A quick survey: octal is 0o[0-7]+ in python/haskell, 0[0-7]+ in java/c/golang/libconfig. This might be a format that requires a wrapper. (or a pattern of read as octal write as decimal?)

[roberth]

These should be escaped and/or validated, but that would be better to do in a proper program as infinisil suggests.

[anund]

The type system for libconfig includes types not expressible directly in nix (or json). Lists and Arrays can't be unambiguously selected given a nix list. (a List is close to a nix list, an Array must contain only scalars of the same type). It's unclear if there are consumers of libconfig files that would need to care about a Group vs Array vs List. Expressible numbers include 0x123, -1, +1.0, or 5L. Octal and positive/negative I don't think are expressible. Long values might be? Include directives are also awkward to encode.

For logiops in particular the usually documented config style includes references to octal number values. Ideally experimenting with packages in Nixos doesn't also come with experimenting with translating config examples.

With that context I have trouble interpreting the intention of RFC 42. Something like toLibConfig would require documenting the above mismatch everywhere it's used and generally adding .file inputs as an escape hatch. That sort of approach generates documentation issues. Is there a good way to handle this in Nixpkgs? Examples where nix config and file inputs are mixed perhaps?

[ckiee]

The type system for libconfig includes types not expressible directly in nix (or json). Lists and Arrays can't be unambiguously selected given a nix list. (a List is close to a nix list, an Array must contain only scalars of the same type). It's unclear if there are consumers of libconfig files that would need to care about a Group vs Array vs List.

In this PR I poked your patch a bit more to get the behavior closer to what we want. (though I don't remember the specifics very confidently ATM) and perfection is the enemy of good. (no toLibconfig currently!)

Expressible numbers include 0x123, -1, +1.0, or 5L. Octal and positive/negative I don't think are expressible. Long values might be?

deferring on @roberth's thread →

Include directives are also awkward to encode.

Yes and although I'm not sure if these should block this PR I think "@include" = keys as (also similarly discussed in the previous thread (@#167388)) are at least partially no go due to the awkward merging config value merging: Two includes? Maybe. "foo.list" and "foo.array"? Definitely not.

For logiops in particular the usually documented config style includes references to octal number values. Ideally experimenting with packages in [NixOS] doesn't also come with experimenting with translating config examples.

This is kind of a wider "found problem" with RFC 42 and I have ran into it before but it's usually been resolved quickly by opening the docs and looking at the example or defaults. (see @#167388)

With that context I have trouble interpreting the intention of RFC 42. Something like toLibConfig would require documenting the above mismatch everywhere it's used and generally adding .file inputs as an escape hatch. That sort of approach generates documentation issues. Is there a good way to handle this in Nixpkgs? Examples where nix config and file inputs are mixed perhaps?

RFC42 is already widely deployed (search has 51 merged results) and with all of the things I've tried so far this hasn't been that much of a problem? libconfig isn't too complicated a format considering #192285 akkoma: init at 3.5.0 was recently merged using this elixrConf voodoo.

I'm pretty sure neither of us are calibrated just right, but after all of the nixpkgs NixOS modules I've consumed I have yet to touch environment.etc in my own config, which is always there along with mkForce.

[anund]

You are right, this work is in the land of compromise. I'm no expert in what's being done in the scope of nix options. Consider the above a mixture of documentation on the mismatch between nix and libconfig and an open ended request for a pattern, any pattern, to use in cases where nix types are insufficient for the target config schema. (I favour a .file option with my limited experience)

One of the least enjoyable loops in nix happens when grabbing a new package with a nix options example of {}. This typically involves needing to; learn a new config format, learn a new translation scheme, and iterate through a cycle of which step is wrong? the nix -> the generated config -> the program + config. That's a lot of the motivation for my original requests for examples and attempts to limit the need to mentally map things like 'the source docs use octal but decimal will also work'.

[ckiee]

Proceeded by #246115.