Automatically generate LUA_PATH

[teto]

Issue description

I've been working towards better lua support in nixos (automatic export of LUA_PATH/LUA_CPATH so that it doesn't have to be done for each derivation as it is now). The start was pretty rough but I have now a better understanding of the building process.
When I do this, I would like LUA_CPATH to be set automatically so that lua can correctly require "cjson" when doing nix-shell ~/nixpkgs2 -p lua52Packages.cjson lua.
So far my buildLuaPackage function has lua as a propagatedBuildInputs so the previous command should install lua. My lua interpreter has a setupHook that runs a script which should populate the LUA_PATH via:
envHooks+=(addToLuaPath).
The problem is that the hook iterates over buildInputs of the lua package and thus don't scan the lua52Packages.cjson folder (since it's not a buildInput of lua). I am not sure how to cleanly solve this.

Having cjson adding itself to lua as a buildInput would mean recompiling lua depending on the derivations which is bad. I wonder if cjson could add itself to the lua passthrough and then use this in the hook ?
I believe python generates some kind of an environment but I am not sure really.

NB: the code is here https://github.com/teto/nixpkgs/tree/luafix but not sure that would help. It's a fucking mess. Most interesting files are mk-lua-package and 5.2.nix and the lua hooks.

[vyp]

Missing a backtick in your text. 😉

[orivej]

The problem is that the hook iterates over buildInputs of the lua package

I think that you need to make it so that every lua package has a propagatedNativeBuildInputs dependency on a package whose setupHook augments envHooks with a shell function that accepts the path to a lua package as its first argument and extends LUA_PATH/LUA_CPATH.

[teto]

@FRidh you have done sthg similar with julia package/python. I am interested in your opinion; I've read https://gist.github.com/FRidh/c3e974e34eac738405af699786b462d5#current-implementation-of-python-on-nix and I am still not sure what's the point in building an environment ? I wonder if the python packages are found via the "pythonpath" variable in buildInputs ? there is a comment in mk-python-package # DEPRECATED: use propagatedBuildInputs

[vcunat]

Again, for reference, very closely related to: #25830

[teto]

thanks for the link, I had seen this but it said it didn't work. I will try a similar approach and maybe if environments are helpful I can add them afterwards. 2017-10-01 21:50 GMT+09:00 Vladimír Čunát <notifications@github.com>:

…

Again, for reference, very closely related to: #25830 <#25830> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#29957 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA2FOvOdbIfXTwXHjBttp1vVT4JfU5-jks5sn4qxgaJpZM4Ppltr> .

[teto]

arf I can't make anything work reliably. An envHook works on the native dependencies (nativeBuildInputs + propagatedNativeBuildInputs) as shown in stdenv/generic/setup.sh

_addToNativeEnv() {
    local pkg="$1"
    if [[ -n "${crossConfig:-}" ]]; then
        local -i depOffset=-1
    else
        local -i depOffset=0
    fi

    # Run the package-specific hooks set by the setup-hook scripts.
    runHook envHook "$pkg"
}

for i in ${nativePkgs+"${nativePkgs[@]}"}; do
    _addToNativeEnv "$i"
done

for i in ${nativePkgs+"${nativePkgs[@]}"}; do
    _addToNativeEnv "$i"
done
    declare -a nativePkgs
    for i in ${nativeBuildInputs:-} ${defaultNativeBuildInputs:-} ${propagatedNativeBuildInputs:-}; do
        findInputs "$i" nativePkgs propagated-native-build-inputs
    done

but it is not run on the package itself; in situations where I run
$ nix-shell -p lua52Packages.cjson
the envHook won't look at cjson directory. Even if cjson depends on i.e. lpeg. then the hook will add lpeg but not cjson.
I notice don't get the mechanism of having a setupHook in the interpreters for the same reason. cjson adds lua as a buildInput but is not a nativeBuildInput of lua so the envHook installed by lua won't find cjson either.

I don't get how but doing
nix-shell -p python36Packages.six ~/nixpkgs2
will pull the python interpreter too in nativePkgs :s

buildInputs='/nix/store/08g2fqyk93s3hyd4d7lhxr7k5lgnnsal-python3.6-six-1.10.0 /nix/store/0lqkw7k9spha405a4v0i90nq1952n5gq-nixpkgs2'
defaultBuildInputs=
defaultNativeBuildInputs='/nix/store/bl1mv4i6ywns4hy0nvbdn93m9i12ryna-patchelf-0.9 /nix/store/dy1fjqfnk1w0q2nix38b21cjqx218val-paxctl-0.9 /nix/store/mjjy30kxz775bhhi6j9phw81qh6dsbrf-
move-docs.sh /nix/store/81ikflgpwzgjk8b5vmvg9gaw9mbkc86k-compress-man-pages.sh /nix/store/vgz4sms56q3dnwrvcpkjq26vs16gr646-strip.sh /nix/store/xfgjfjcq34fwfbfi1idp2nh81kpigch7-patch-
shebangs.sh /nix/store/j1q2bm9qjrhr6q2z819dgzi7bmy3hhjq-audit-tmpdir.sh /nix/store/scf9318hx6pad0wf0qx8a79h635aamfb-multiple-outputs.sh /nix/store/z82dl6ialp166drqihzkz67nkl6w3l16-mo
ve-sbin.sh /nix/store/a92kz10cwkpa91k5239inl3fd61zp5dh-move-lib64.sh /nix/store/15kgcm8hnd99p7plqzx7p4lcr2jni4df-set-source-date-epoch-to-latest.sh '
dontAddDisableDepTrack=1
envHooks=([0]="addPythonPath")
fixupOutputHooks=([0]="if [ -z \"\$dontPatchELF\" ]; then patchELF \"\$prefix\"; fi" [1]="if [ -z \"\$dontGzipMan\" ]; then compressManPages \"\$prefix\"; fi" [2]="_doStrip" [3]="if 
[ -z \"\$dontPatchShebangs\" -a -e \"\$prefix\" ]; then patchShebangs \"\$prefix\"; fi" [4]="if [ -z \"\$noAuditTmpdir\" -a -e \"\$prefix\" ]; then auditTmpdir \"\$prefix\"; fi" [5]=
"_moveSbin" [6]="_moveLib64")
i=/nix/store/15kgcm8hnd99p7plqzx7p4lcr2jni4df-set-source-date-epoch-to-latest.sh
initialPath='/nix/store/d4yib20lqn7krrqmm0wwbwjxjfa78xqp-coreutils-8.28 /nix/store/871adhqrxl37bpi5w9g9g6y0i1n4f9cd-findutils-4.6.0 /nix/store/kkrg56qh1l655bcdr6amyp7k59ndf81z-diffut
ils-3.6 /nix/store/h5c2vfaxhpv7v0p1vlh3h7sdiphsi4rv-gnused-4.4 /nix/store/s6wz7l5h0ya6qciy0xcxwv05l50snfy8-gnugrep-3.1 /nix/store/pyqzyl3yvam1magahsm8xahx5hs1ngh9-gawk-4.1.4 /nix/sto
re/kpnj0h0340wd0i86q0523527ikbz62ll-gnutar-1.29 /nix/store/abk6h8g0hj6dyk75h320mgls3ygshl0w-gzip-1.8 /nix/store/by3lqpgph3c4maz6vb2xhd64vhagfc67-bzip2-1.0.6.0.1-bin /nix/store/8g2x5q
jqyv9a7glmvpzv1wylnbf1cwvw-gnumake-4.2.1 /nix/store/flb9ar1xdd13c606aa4my9miy3iv4vyk-bash-4.4-p12 /nix/store/44dbwrwfy6lj45jy02d0rjbj9930nwf3-patch-2.7.5 /nix/store/zv86mp9qr1kk2sxv1
74fxrmly82wakr0-xz-5.2.3-bin'
name=shell
nativeBuildInputs=
nativePkgs=([0]="/nix/store/08g2fqyk93s3hyd4d7lhxr7k5lgnnsal-python3.6-six-1.10.0" [1]="/nix/store/85b2460h34awjgyczh8mr542x2xq765y-python3-3.6.2" [2]="/nix/store/2jm5gz3dd744qvrx7ml
mwmyyk0l1i56s-python3.6-setuptools-36.4.0" [3]="/nix/store/vr7f6k4746c50paan4zjr5ax8j2rdx2w-nixpkgs2" [4]="/nix/store/bl1mv4i6ywns4hy0nvbdn93m9i12ryna-patchelf-0.9" [5]="/nix/store/d
y1fjqfnk1w0q2nix38b21cjqx218val-paxctl-0.9" [6]="/nix/store/mjjy30kxz775bhhi6j9phw81qh6dsbrf-move-docs.sh" [7]="/nix/store/81ikflgpwzgjk8b5vmvg9gaw9mbkc86k-compress-man-pages.sh" [8]
="/nix/store/vgz4sms56q3dnwrvcpkjq26vs16gr646-strip.sh" [9]="/nix/store/xfgjfjcq34fwfbfi1idp2nh81kpigch7-patch-shebangs.sh" [10]="/nix/store/j1q2bm9qjrhr6q2z819dgzi7bmy3hhjq-audit-tm
pdir.sh" [11]="/nix/store/scf9318hx6pad0wf0qx8a79h635aamfb-multiple-outputs.sh" [12]="/nix/store/z82dl6ialp166drqihzkz67nkl6w3l16-move-sbin.sh" [13]="/nix/store/a92kz10cwkpa91k5239in
l3fd61zp5dh-move-lib64.sh" [14]="/nix/store/15kgcm8hnd99p7plqzx7p4lcr2jni4df-set-source-date-epoch-to-latest.sh")
out=/nix/store/25kb0wv68cb2i2rr34ifb72kzrn7gs3h-shell

When I do nix-shell -p lua52Packages.cjson ~/nixpkgs2
even though buildLuaPackage automatically adds lua as a buildInput, it doesn't get pulled as python does and so doesn't install the lua interpreter envHook either.
There is comment in mk-python-derivation saying sthg related to nix-shell.

  # propagate python/setuptools to active setup-hook in nix-shell
  propagatedBuildInputs = propagatedBuildInputs ++ [ python setuptools ];

Reading through man nix-shell, -p creates an environment so I suppose python does sthg special when nix-shell creates its environment while lua doesn't ?

[orivej]

When I do nix-shell -p lua52Packages.cjson ~/nixpkgs2 even though buildLuaPackage automatically adds lua as a buildInput, it doesn't get pulled as python does

Certainly it doesn't, because nix-shell -p puts an already built package into the environment, and built packages have already used their buildInputs when they were built. You need to put lua into propagatedNativeBuildInputs (propagatedBuildInputs will also do). Have you tried #29957 (comment) ?

[teto]

nix-repl tells me that lua is in lua52Packages.cjson.propagatedNativeBuildInputs yet when I do nix-shell -p lua52Packages.cjson ~/nixpkgs2, "lua" is not in PATH :'(

[teto]

propagatedNativeBuildInputs is empty in the shell while. Doesn't a change in the lua52Packages.cjson derivation trigger a rebuild even if linked via nix-shell -p ? I've pushed my changes, I need to cool down I am going nuts.

[orivej]

@teto May you commit development/lua-modules/cjson to your luafix branch so I could see?

[teto]

true I moved it to a specific file in order to more quickly find it in vim.
Pushed to https://github.com/teto/nixpkgs/tree/luafix
Thanks for the help. I feel like I understand the idea (which I didn't 2 weeks ago) yet I can't complete it and it's very frustrating. All my hopes on you :)

[FRidh]

I suggest not using a hook and instead create a LUA_PATH with Nix. You could add a certain attribute to Lua packages so they can be recognized as Lua packages.

[orivej]

@FRidh How do we make nix-shell -p luaPackages.cjson affect LUA_PATH without hooks? Or do you propose another way to work with lua packages in nix-shell?

[FRidh]

The following is needed:

1. a method for finding packages/dependencies during a build, with nix-build and nix-shell
2. a method for finding packages/dependencies after a build
3. a method for using packages/dependencies with nix-shell -p

For 1) typically a setupHook is used. This one is also used by 3) if the derivation with the setupHook is propagated by the package added with nix-shell -p.

I think its better to create a LUA_PATH in Nix by adding a certain attribute/identifier to Lua packages, and to have a luaWithPackages function, like we have with Python (where it doesn't use a setupHook and Haskell. This luaWithPackages you can use with nix-shell, i.e. nix-shell -p 'luaWithPackages(ps: with ps; [ pkg1 pkg2 ])'

[teto]

I've taken the withPackages approach and it seems to work. I will push a clean PR when I feel more confident. What I still don't understand (and this explains why I missed it) is python's wrapper.nix:
{ buildEnv, pythonPackages }: f: let packages = f pythonPackages; in buildEnv.override { extraLibs = packages; }" (python wrapper.nix);
it is imported as withPackages = import .../wrapper.nix. so withPackages is a function that takes one parameter and returns an enviroment . I don't get the "f pythonPackages". I believe that f is in general a list (python36.withPackages( [list]) but here it is applied to pythonPackages that is a set (not too sure about that) ?
Thanks for all the help. I think I can now manage on my own xD

[FRidh]

@teto a perhaps clearer example is provided in https://github.com/NixOS/nixpkgs/blob/91f7042aa09eb7488267a01a68c46bab05d8515e/pkgs/development/libraries/aspell/aspell-with-dicts.nix

The expression you see is a function that is called with callPackage, as is commonly done in Nixpkgs.
Instead of returning a derivation, it returns an anonymous function that takes one argument, f, which when called returns a derivation. The aspellDicts is a dictionary of aspell dictionary derivations, e.g.

aspellDicts = {
  en = mkDerivation {...};
  nl = mkDerivation {...};
};

The argument f should then be an anonymous function that takes certain attributes from its own argument, e.g. f = ps: [ ps.en ]. If we then do dicts = f aspellDicts, the argument aspellDicts corresponds to ps and so we take the attribute en from aspellDicts.

[teto]

Of course the f was teh whole f = ps: [ ps.en ] while for some reason I had only retained the [ps.en] part :/ Sorry for being dense.
So far, I test with nix-shell -p 'lua52Packages.lua.withPackages(ps: with ps; [ cjson ])' --show-trace -I nixpkgs=$HOME/nixpkgs2 -K and cjson2lua seems correctly wrapped with valid LUA_PATH (wrap.sh generates a valid LUA_PATH).
My last problem is the same as before, aka LUA_PATH doesn't get exported in the shell, unlike PYTHONPATH (the lua binary is not wrapped, as it seems to be the case for python). I think in the python3 case, setup-hook.sh does the export ( addToSearchPathWithCustomDelimiter : PYTHONPATH $1/@sitePackages@) but I wonder how it finds the libraries ? In my case, a similar lua/setup-hook.sh is run and can't find the cjson folder as it's not in the buildInputs.

I believe I could make it work with adding a setupHook in lua/wrapper.nix but it's a passthru so that's not a good solution. I believe I should be able to export LUA_PATH as PYTHONPATH is exported but I am not sure how it does. I wish I could run the seup.sh with "set -x" but that would recompile everything I think :s

[teto]

So I have a script that can generate nix expressions (unessential fields are temporary ones for now such as the license field). Here is an example of the generated code from a subset of rockspecs.

/* auto-generated file -- DO NOT EDIT! */
{ self, stdenv
, fetchurl
, fetchgit
}:
with self;
rec {
ansicolors = buildLuaPackage rec {
src= fetchurl {
url="https://github.com/kikito/ansicolors.lua/archive/v1.0.2.tar.gz";
sha256="0r4xi57njldmar9pn77l0vr5701rpmilrm51spv45lz0q9js8xps"; }
;
version="1.0.2-3";
name="ansicolors";
propagatedBuildInputs=[ lua];
meta={
homepage="https://github.com/kikito/ansicolors.lua";
description="Library for color Manipulation.";
license=stdenv.lib.licenses.mit; }
; }
;
busted = buildLuaPackage rec {
version="2.0.rc9-0";
name="busted";
src= fetchurl {
sha256="0snaxcq60fj86i75b5lq3d60pd05xzpz1w4qslhm228g8rc0kg4b";
url="https://github.com/Olivine-Labs/busted/archive/v2.0.rc9-0.tar.gz"; }
;
meta={
description="Elegant Lua unit testing.";
license=stdenv.lib.licenses.mit;
homepage="http://olivinelabs.com/busted/"; }
;
propagatedBuildInputs=[ lua lua_cliargs luafilesystem dkjson say luassert ansicolors lua-term penlight mediator_lua luasocket]; }
;
dkjson = buildLuaPackage rec {
src= fetchurl {
url="http://dkolf.de/src/dkjson-lua.fsl/tarball/dkjson-2.5.tar.gz?uuid=release_2_5";
sha256="14wanday1l7wj2lnpabbxw8rcsa0zbvcdi1w88rdr5gbsq3xwasm"; }
;
propagatedBuildInputs=[ lua];
meta={
homepage="http://dkolf.de/src/dkjson-lua.fsl/";
license=stdenv.lib.licenses.mit;
description="David Kolf's JSON module for Lua"; }
;
version="2.5-2";
name="dkjson"; }
;
lua-cmsgpack = buildLuaPackage rec {
propagatedBuildInputs=[ lua];
meta={
description="MessagePack C implementation and bindings for Lua 5.1/5.2/5.3";
license=stdenv.lib.licenses.mit;
homepage="http://github.com/antirez/lua-cmsgpack"; }
;
src= fetchfromgit {
url="git://github.com/antirez/lua-cmsgpack.git";
sha256="0j0ahc9rprgl6dqxybaxggjam2r5i2wqqsd6764n0d7fdpj9fqm0";
rev="0.4.0"; }
;
version="0.4.0-0";
name="lua-cmsgpack"; }
;
lua_cliargs = buildLuaPackage rec {
propagatedBuildInputs=[ lua];
src= fetchurl {
sha256="15sh7d0xwpgfsb346nzc0abj81wkrf9s8iyfrfclyr3wmiyg89d9";
url="https://github.com/downloads/amireh/lua_cliargs/lua_cliargs-1.1.tar.gz"; }
;
version="1.1-1";
name="lua_cliargs";
meta={
homepage="https://github.com/amireh/lua_cliargs";
description="A command-line argument parser.";
license=stdenv.lib.licenses.mit; }
; }
;
luassert = buildLuaPackage rec {
meta={
description="Lua Assertions Extension";
homepage="http://olivinelabs.com/busted/";
license=stdenv.lib.licenses.mit; }
;
propagatedBuildInputs=[ lua say];
name="luassert";
src= fetchurl {
sha256="0qgi3xcm3j6mqk4crc1mwwbj74aw5mjjn93hcf8c9bcvh7sf6cp6";
url="https://github.com/Olivine-Labs/luassert/archive/v1.7.9.tar.gz"; }
;
version="1.7.9-0"; }
;
lua-term = buildLuaPackage rec {
meta={
description="Terminal functions for Lua";
homepage="https://github.com/hoelzro/lua-term";
license=stdenv.lib.licenses.mit; }
;
propagatedBuildInputs=[];
name="lua-term";
version="0.7-1";
src= fetchurl {
sha256="0c3zc0cl3a5pbdn056vnlan16g0wimv0p9bq52h7w507f72x18f1";
url="https://github.com/hoelzro/lua-term/archive/0.07.tar.gz"; }
; }
;
luasocket = buildLuaPackage rec {
propagatedBuildInputs=[ lua];
name="luasocket";
src= fetchurl {
url="http://luaforge.net/frs/download.php/2664/luasocket-2.0.2.tar.gz";
sha256="19ichkbc4rxv00ggz8gyf29jibvc2wq9pqjik0ll326rrxswgnag"; }
;
meta={
homepage="http://luaforge.net/projects/luasocket/";
license=stdenv.lib.licenses.mit;
description="Network support for the Lua language"; }
;
version="2.0.2-6"; }
;
ltermbox = buildLuaPackage rec {
src= fetchfromgit {
rev="v0.2";
url="git://github.com/ukasz/termbox.git";
sha256="0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"; }
;
propagatedBuildInputs=[ lua];
name="ltermbox";
version="0.2-1";
meta={
license=stdenv.lib.licenses.mit;
homepage="http://code.google.com/p/termbox";
description="A termbox library package"; }
; }
;
mediator_lua = buildLuaPackage rec {
src= fetchurl {
sha256="0z1nzc0jf6sri8imbcxy8ijn9scqn2jnhfcnn3rfj96izilj7qhg";
url="https://github.com/Olivine-Labs/mediator_lua/archive/v1.1.tar.gz"; }
;
propagatedBuildInputs=[ lua];
meta={
description="Event handling through channels";
homepage="http://olivinelabs.com/mediator_lua/";
license=stdenv.lib.licenses.mit; }
;
name="mediator_lua";
version="1.1-3"; }
;
penlight = buildLuaPackage rec {
version="1.5.4-1";
name="penlight";
propagatedBuildInputs=[ luafilesystem];
meta={
homepage="http://stevedonovan.github.com/Penlight";
description="Lua utility libraries loosely based on the Python standard libraries";
license=stdenv.lib.licenses.mit; }
;
src= fetchurl {
url="http://stevedonovan.github.io/files/penlight-1.5.4.zip";
sha256="138f921p6kdqkmf4pz115phhj0jsqf28g33avws80d2vq2ixqm8q"; }
; }
;
say = buildLuaPackage rec {
propagatedBuildInputs=[ lua];
version="1.3-1";
name="say";
src= fetchurl {
url="https://github.com/Olivine-Labs/say/archive/v1.3-1.tar.gz";
sha256="1jh76mxq9dcmv7kps2spwcc6895jmj2sf04i4y9idaxlicvwvs13"; }
;
meta={
license=stdenv.lib.licenses.mit;
homepage="http://olivinelabs.com/busted/";
description="Lua String Hashing/Indexing Library"; }
; }
;
}

Next step is to be able to install these packages.
luarocks can install in an arbitrary location via the --tree flag so I tried setting the installPhase to
luarocks install --tree $out ${name} but it fails with Warning: Failed searching manifest: Failed creating temporary cache directory /homeless-shelter/.cache/luarocks/https___luarocks.org
An alternative would be to be able to catpure the installation instructions from luarocks and put them directly into the derivations. It seems doable with luarocks architecture but it might produce bigger derivations.
Not sure if I can tell luarocks to ignore dependancies too.

[teto]

I managed to get something working at #33903 . I have managed to support the lua packages needed to run a neovim tests (a dozen) so I am quite happy with that. I dont have the mental strength to improve it further right now (nor the time) so I hope someone can pick it up.

[teto]

work has been merged recently to achieve this and more is going to follow. Thus closing.