Meta issue about lua ecosystem progress

[teto]

Issue description

Meta issue about changes to how lua is handled in nixpkgs (obsoletes #33903).

Possibie improvements

• rename lua5_1 into lua51 etc to do as python packages. It would also improve consistency within lua, we have lua5_1 and lua51Packages
• improve cmake based luarocks packages cmake build system doesnt take variables from LUAROCKS_CONFIG luarocks/luarocks#1160 (waiting for structured attributes) not sure that's needed anymore since luarocks maintainers dont really encourage it
• speed up maintainers/scripts/update-luarocks (cache rocks/rockspec)
• support the different luarocks mirror
• add luarocks to the generated ones (and rename the current one to luarocks-bootstrap ?)
• implement a lua.mkDerivation
• rewrite the interpreters to share more code (python-like)
• unify the getLuaPathList / getLuaPath / setuphook (between buildLuaPackage / buildLuaRocks
• add propagatedBuildInputs to the generated luarocks config external_deps_dirs = {} (https://github.com/luarocks/luarocks/wiki/Paths-and-external-dependencies)
• ~~ export/use version specific variables such as LUA_PATH_5_2/LUAROCKS_CONFIG_5_2 ~~ (wont do but someone feel free to pickup)
• let luarocks check for dependencies via exporting the different rocktrees in temporary config as is kind of done in torch-distro.nix
• remove the luarocks warning about different folder owners
• a better way to tell luarocks where to look for buildInputs https://github.com/luarocks/luarocks/wiki/Platform-agnostic-external-dependencies
  Here is an upstreaming roadmap:
• convert packages to use lua.withPackages /wrapLuaPrograms
  • mudlet/mpv/vis (Feel free to pick commits from https://github.com/teto/nixpkgs/tree/lua_envs and merge them yourself, I have no time to do it at the moment
  • vis vis: leverage lua environments #62762
  • prosody
  • knot-resolver
  • lumail
  • quvi (quvi: drop #269966)
  • neovim
  • luakit
• convert some packages to luarocks-based packaging (Lua: convert some more packages #55747)
• move packages from torch-distro.nix to the generated ones torchPackages, torch-repl: remove #81173
• update documentation doc: update lua documentation #55302
• leverage the rockspec version 3 to run tests for lua packages
• remove the use of with self; in generated-packages.nix . I have a (messy) working prototype at teto@dde3bd2
• deal with strange packages like cqueue that have one rockspec per interpreter c01fe37#commitcomment-32572249
• added luarocks autocompletion
• While investigating how to reconcile vim plugins and lua packages, I found out the luarocks config entry
  lua_modules_path = "" that creates a flat hierarchy and could reduce the number of nested folders in our nix packages. Investigate its usage.

---

[Shados]

Some nice things that would require upstream luarocks changes (from a brief preliminary investigation):

• Make luarocks doc work for luarocks derivations in the current profile
• Make luarocks show work for luarocks derivations in the current profile
• Make luarocks list work for luarocks derivations in the current profile
• Make the luarocks.loader module and the luarocks which command work for luarocks derivations in the current profile?

None of these are super necessary, but all would be potentially useful.

These would require changes to luarocks to make it capable of searching multiple rocks trees for the relevant operations. It already does this for luarocks make/luarocks install if you use --deps-mode=all (and explicitly specify the locations of the rocks trees), but it does not do this for other operations AFAICT.

They would also need changes to add a way to have luarocks find these rocks trees to begin with.

[doronbehar]

I noticed that some packages which use Lua, (e.g AwesomeWM and Neovim) use an old version of Lua (5.2 && 5.1 - respectively). As a beginner Nixos user I'm not sure if it is easily override-able but I think it'd be worth mentioning that nixpkgs should use 5.3 as the default version of Lua for these packages' inputs.

[teto]

why is that ? lua5.2 looks standard; 5.3 qnd 5.4 seem to have breaking changes. As for neovim they explicitly require 5.1

[doronbehar]

As for Neovim, you are correct but as for AwesomeWM, anything higher than 5.1 is fine. They both support LuaJIT as well.

I'm not sure what should be dictated as nixpkgs' policy but my opinion is that Lua 5.3 should be treated as the default as it's considered the latest version upstream. You are right that it has breaking changes and that's why Neovim Neovim explicitly requires 5.1 but this is an upstream issue, not nixpkgs'.

Additionally, I think that in the specific case of Neovim, since LuaJIT is supported as well I would switch to it for this package specifically, it is certainly seems superior then Lua 5.1, it's nice that it's build uses LuaJIT by default. but I think we should just rename the luaPackages name space to refer to lua53Packages. What do you think?

EDIT: The default Neovim build uses LuaJIT - it surprised me since it's inputs are lua and not luajit.

[teto]

I suspect moving to lua53 would break many things. lua versions are different between eachother.

I've added the support in nix for luarocks mirrors so don't be surprised if luarocks-nix generates some strange urls. I've also moved the luarocks-nix repository to nix-community.
I had a qucik look at supporting luarocks test but seems like it deals with installed packages ? not too complex but just enough for me to skip it.
I wanted to take the opportunity to thank @Shados who improved the infra a lot ! I was able to tick many boxes thanks to his work !

[teto]

@grwlf Many packages are marked broken in pkgs/applications/science/machine-learning/torch/torch-distro.nix . What do you think about replacing them with the current lua infrastructure ?
it unifies the interface and may unbreak some modules.
I can guide you if you have any questions.

[lblasc]

@teto how about removing torch-distro completely? Sadly upstream didn't have any activity since 2017.

[stites]

lua-torch is no longer maintained (the codebase was migrated to pytorch) and communities have been marked as non-existent. I think removing torch is valid. Doing so would also close #71888.

See https://github.com/torch/torch7#development-status
and final commit torch/torch7@dde9e56

[sergei-mironov]

@teto Hi. I think it is OK to remove old torch files. I may be listed as a maintainer of some of them, but actually I don't plan to maintain anything because my focus shifted towards TF/pytorch nowdays..

[teto]

ok thanks for the update. I guess we can remove it then. I was hoping someone
could move some of the packages to the current lua infra to salvage some of the
work (like finding the buildInputs). But that might not be worth the effort.
I don't plan to do anything as I have already other issues to address but I can
advise anyone interested in cleaning up.

[stale]

I marked this as stale due to inactivity. → More info

[teto]

@Mic92 @timokau with the rise of a the lua ecosystem with neovim 0.5, I am looking at ways to consolidate dependency management in that ecosystem (with #131499 for instance). The "maintainers/scripts/update-luarocks-packages: shellscript is not up to the task anymore so I was considerng adaptin pkgs/misc/vim-plugins/update.py and pluginupdate.py to generate pkgs/development/lua-modules/generated-packages.nix as well.
Any objection ?

[timokau]

Sorry for the late response. It has been a while since I touched the vim plugin infrastructure and I don't know much about lua. I think you're more up to date on both of these things than me, so I defer to you :)

[teto]

status update:
I've recently merged :

• lua updater: support in-repo rockspecs #134336 that makes it possible to use a rockspec from the downloaded repo
• the lua updater now shares code with the vim updater. It provides a single interface for both so it's easier to learn. It's also much faster than the old bash script Luarocks pyupdater followup #133296
• I've opened an issue about the format of luarocks.csv adopt a new format for pkgs/misc/vim-plugins/vim-plugin-names and maintainers/scripts/luarocks-packages.csv #136248
• added luarocks autocompletion luarocks: add bash/zsh completion #136507

While investigating how to reconcile vim plugins and lua packages, I found out this luarocks config element

-- to create a flat hierarchy
lua_modules_path = ""

that creates a flat hierarchy and could reduce the number of nested folders in our nix packages.

[teto]

big changes in #140801 to remove the use of with self; in generated-packages.nix, like is done in haskell.

EDIT: fixed link

[doronbehar]

big changes in #56398 to remove the use of with self; in generated-packages.nix, like is done in haskell.

@teto #56398 is this issue :).

[teto]

#166162 fixes the issue we had on luv that was blocking luarocks to 3.2. We can now bump it to 3.8.
I had a nice PR that can automatically add neovim plugin dependencies #160484 (need a rebase) which could serve as a nice poc for https://github.com/nvim-lua/nvim-package-specification. To make it mergeable I would need to make it possible to generate flat lua installs.

One problem is that handling of LUA_PATH across nixpkgs is very heterogeneous, and so it makes harder to make some global changes. I think it's heterogeneous partly because lua utilities are broken. For instance the LUA_PATH generation hook removes the defaults from the lua interpreter (";;"). Many modules hardcode the different lua paths instead of using e.g. wrapLuaProgram, luaPathList from pkgs/development/lua-modules/lib.nix . I believe wrapLuaProgram is broken so let's first fix this and add some tests. I would welcome some help as I wont likely be able to do it anytime.

[teto]

I've merged #195869 that adds a testing infra for the interpreter, which I hope will help prevent naive regressions in the future. I've also merged #195732 which should hopefully allow to have the best of both worlds for neovim plugins: automatic dependency management from the rockspec + uptodate plugins via the vim plugin updater.

Next step will be to track every per-package custom nix code for lua and replace it with code from the luaLib so that I can configure flat installs in luaLib and it will affect all of nixpkgs.

[tomodachi94]

Here's a GitHub Search query that can perhaps help with that: https://github.com/search?q=repo%3ANixOS%2Fnixpkgs+lang%3Anix+lua+NOT+wrapLuaPrograms+NOT+withPackages&type=code

There are definitely some false positives, but I'm also seeing some other packages could use some love.

[tomodachi94]

More packages that could use TLC (pure Lua packages, most likely compatible with the neo-builders):

• z-lua
• vis
• openrussian-cli