switch lint workflow to super-linter

[eitsupi]

Close #245

Switch to super-linter, which allows large numbers of linters to be used in a single workflow.
For example, this includes actionlint for GitHub Actions and lintr for R.
From my experience creating the GitHub Actions definition files for this repository, I feel that actionlint is really needed.
(I really don't often realize my mistakes until I run it and it fails!)

Because super-linter is maintained and popular by GitHub, there is little attention to pay to maintaining the workflow.

We have been using markdownlint/markdownlint as a linter for Markdown, but super-linter has DavidAnson/markdownlint built in.
Note that the format of the configuration file is different because of this. (.mdlrc v.s. .markdownlint.jsonc or .markdownlint.yaml)
All config files are here. https://github.com/github/super-linter/tree/main/TEMPLATES

Because of the large number of check items added by swiching the workflow, existing files will need to be modified in the future.
Since modifying everything in this PR would be a large amount of work, I edited only the README.md and the NEWS.md first.

[eitsupi]

Inspection of all files locally yielded the following results.

2022-03-10 10:13:58 [INFO]   ----------------------------------------------
2022-03-10 10:13:58 [INFO]   ----------------------------------------------
2022-03-10 10:13:58 [INFO]   The script has completed
2022-03-10 10:13:58 [INFO]   ----------------------------------------------
2022-03-10 10:13:58 [INFO]   ----------------------------------------------
2022-03-10 10:13:58 [ERROR]   ERRORS FOUND in BASH:[30]
2022-03-10 10:13:58 [ERROR]   ERRORS FOUND in DOCKERFILE_HADOLINT:[84]
2022-03-10 10:13:58 [ERROR]   ERRORS FOUND in GITHUB_ACTIONS:[4]
2022-03-10 10:13:58 [ERROR]   ERRORS FOUND in JSCPD:[2]
2022-03-10 10:13:58 [ERROR]   ERRORS FOUND in MARKDOWN:[1]
2022-03-10 10:13:58 [ERROR]   ERRORS FOUND in NATURAL_LANGUAGE:[2]
2022-03-10 10:13:58 [ERROR]   ERRORS FOUND in R:[1]
2022-03-10 10:13:58 [ERROR]   ERRORS FOUND in SHELL_SHFMT:[39]
2022-03-10 10:13:58 [ERROR]   ERRORS FOUND in YAML:[2]
2022-03-10 10:13:58 [FATAL]   Exiting with errors found!

super-linter.log

It seems that there are some errors that should be ignored, so I will add additional configuration files later.
For example, the following is an error where RUN is used continuously in a Dockerfile. https://github.com/hadolint/hadolint/wiki/DL3059
However, this is intentional and cannot be changed.

RUN /rocker_scripts/install_rstudio.sh
RUN /rocker_scripts/install_pandoc.sh

[eitsupi]

Now.

2022-03-10 16:35:56 [INFO]   ----------------------------------------------
2022-03-10 16:35:56 [INFO]   ----------------------------------------------
2022-03-10 16:35:56 [INFO]   The script has completed
2022-03-10 16:35:56 [INFO]   ----------------------------------------------
2022-03-10 16:35:56 [INFO]   ----------------------------------------------
2022-03-10 16:35:57 [ERROR]   ERRORS FOUND in BASH:[30]
2022-03-10 16:35:57 [ERROR]   ERRORS FOUND in DOCKERFILE_HADOLINT:[17]
2022-03-10 16:35:57 [ERROR]   ERRORS FOUND in GITHUB_ACTIONS:[4]
2022-03-10 16:35:57 [ERROR]   ERRORS FOUND in JSCPD:[1]
2022-03-10 16:35:57 [ERROR]   ERRORS FOUND in MARKDOWN:[2]
2022-03-10 16:35:57 [ERROR]   ERRORS FOUND in NATURAL_LANGUAGE:[2]
2022-03-10 16:35:57 [ERROR]   ERRORS FOUND in R:[1]
2022-03-10 16:35:57 [ERROR]   ERRORS FOUND in SHELL_SHFMT:[39]
2022-03-10 16:35:57 [ERROR]   ERRORS FOUND in YAML:[4]
2022-03-10 16:35:57 [FATAL]   Exiting with errors found!

super-linter.log

If this gets merged, I would like to work on eliminating the above error that is occurring in the existing code.

@cboettig What do you think about this linter workflow switching?

[cboettig]

Thanks, yeah this looks promising, though the configuration for Dockerfiles never to have consecutive RUN elements does feel over-zealous to me; sometimes it makes sense to have multiple layers. Overall though better-linted code would be a nice improvement to readability!

[eitsupi]

the configuration for Dockerfiles never to have consecutive RUN elements does feel over-zealous to me

Yes, this rule is the hadolint's "DL3059" and I set the config file to ignore this rule. 1ff0cd9
https://github.com/hadolint/hadolint/wiki/DL3059

I have not been able to review all the errors because the super-linter log file is too long and difficult to read, but the errors that remain now are others.
As far as I could tell, the rules that were causing the other errors were not enough to ignore, but of course we can ignore those rules by editing the config files.

[cboettig]

Nice, the proposed changes here to deploy super-linter and update the documentation at various points all looks good.

As you know I'm not as familiar with linting tools outside of R / python toolset, but are there utilities that could auto-format the markdown and shell scripts? (similar to styler in R or black in python?) Probably wouldn't cover all issues, but might trim down the log of existing issues?

[eitsupi]

As you know I'm not as familiar with linting tools outside of R / python toolset, but are there utilities that could auto-format the markdown and shell scripts? (similar to styler in R or black in python?)

I use VSCode as my main editor, so I use https://github.com/foxundermoon/vs-shell-format for shell scripts and https://github.com/yzhang-gh/vscode-markdown for Markdown.
When used from VSCode, it can be auto-formatted with the same operation as black for Python files or styler for R files. (Of course, that is one of the reasons for VSCode's popularity.)

And, shellcheck supports automatic modification for some rules.
I have installed VSCode extension https://github.com/koalaman/shellcheck to use shellcheck, but this should work from the CLI.

before

after

https://github.com/DavidAnson/vscode-markdownlint, an extension for Markdownlint for VSCode, also seems to have an automatic modification function for some rules.
The same functionality should be available from the CLI.

before

after

Probably wouldn't cover all issues, but might trim down the log of existing issues?

Yes, I believe some of the errors currently occurring in these auto-formats can be fixed automatically.
(If I look at the super-linter's log, I can see that shellcheck suggests some auto-fixes.)