Add pre-commit for linting/auto-formatting

[stefsmeets]

This PR adds pre-commit as a tool for managing pre-commit hooks and code linting. It also replaces pylint by ruff as a linter. Ruff also sorts imports similar to isort. New checks can be enabled in pyproject.toml.

Running the linters now goes through pre-commit.

# staged files only
pre-commit

# all files
pre-commit run --all-files

You can enable automatic linting and code formatting with pre-commit on each commit:

pre-commit install

This replaces the git hook from before. The advantages are that this only checks changed files, issues are automatically fixed (if possibe/enabled), and it is much faster than prospector/pylint.

If you use the old hook, you will need to run:

git config --unset core.hooksPath

You can also install ruff locally (pip install ruff) and run it on the entire codebase: ruff check dianna --fix.

I also added yapf as a code formatter. At some point we should run yapf on the entire code base so that everyone is up-to-date. This may hurt a bit with open PRs, but the reward is that we no longer have to worry about code formatting / fighting with pylint. I will create a new PR once this one is accepted.

I realized I could not make this PR without fixing some standing issues to pass the linter, so I fixed some whitespace issues, added missing parameters in docstrings, and looked at all the pylint pragma's.

Closes #492
Closes #493
Closes #496

[stefsmeets]

@cwmeijer Would you be willing to review this? Let me know what you think!

[cwmeijer]

Looks great!
I have to say that I don't fully understand what code is responsible for actually changing the code before a commit, but I'm sure you'll explain me soon ;-)
Could you have a look at my comment?

[stefsmeets]

Yeah, good question!

So what happens is that pre-commit checks which files have changed (essentially git diff). Every file that is changed gets passed to some hooks based on the file extension. For example, it does not make sense to pass a yaml or markdown file to a python linter.

Each of the hooks then runs on the changed files. These can then change the code if they wish. This is no different than running each of the hooks individually on the command line. Some simply emit warnings (like pylint, flake8, or ruff check), others also update and format the code (like ruff check --fix, yapf, isort). You will have to stage these again and if they pass without changes then the pre-commit hook is passed.

All these hooks are defined in: https://github.com/dianna-ai/dianna/blob/61fe210edb8c95e69e2b54bc6ea85cc3eb220c8e/.pre-commit-config.yaml

You'll notice there are some that check whether yaml files are valid (check-yaml), or whether the python code actually parses correcty (check-ast). Then there are the heavy lifters yapf and ruff which format and lint the code. I also tentatively added mypy, which we could enable at some point for type checking.

[stefsmeets]

I guess we can enable mypy here as a step towards #85

[egpbos]

Super nice PR! Sorry for the unasked review, but I was just very interested to see how this would look in practice and in particular in DIANNA which I know well :) My only minor complaints are about disabling pylint complaints, as the examples above illustrate ;) I know there's always a trade-off with being realistic in terms of how much time you can spend on polishing. But DIANNA is growing into quite a big thing, so I would argue that in this case the strictest standards should be seriously considered.

[cwmeijer]

Sorry for the delay in approving. LGTM asap :-)

[stefsmeets]

Cool, I will do a full re-format sometime next week.

Also a reminder to run:

git config --unset core.hooksPath

to remove the old hook.