Add Mac OS tests to Github Action

[Pierre-Sassoulas]

Current problem

The tests suite is not run on Mac so the tests are failing locally for mac developpers.

Desired solution

Run Mac tests in Github Actions.

Additional context

See discussion in #4753 (comment)

[cdce8p]

The test suite might not be run, but that doesn't mean the tests are failing.
Just run pytest tests/test_functional.py on main and everything was fine (macOS Catalina)

[DanielNoord]

I reported this as running the test suite on my local machine produced different errors than those reported by the GitHub actions.

I have added the report. Perhaps this is due to something I messed up in my local environment, but I did not fully understand why these errors were only thrown on my local machine.

GLOB sdist-make: /Users/daniel/DocumentenLaptop/Programming/Github/pylint/setup.py
py39 recreate: /Users/daniel/DocumentenLaptop/Programming/Github/pylint/.tox/py39
py39 installdeps: -r/Users/daniel/DocumentenLaptop/Programming/Github/pylint/requirements_test.txt
py39 inst: /Users/daniel/DocumentenLaptop/Programming/Github/pylint/.tox/.tmp/package/3/pylint-2.9.6.dev0.zip
py39 installed: appdirs==1.4.4,astroid==2.6.5,attrs==19.3.0,backports.entry-points-selectable==1.1.0,black==21.7b0,certifi==2021.5.30,cfgv==3.3.0,charset-normalizer==2.0.3,cli-ui==0.10.3,click==8.0.1,colorama==0.4.4,contextlib2==21.6.0,coverage==5.5,coveralls==3.2.0,distlib==0.3.2,docopt==0.6.2,execnet==1.9.0,filelock==3.0.12,flake8==3.9.2,gprof2dot==2021.2.21,identify==2.2.11,idna==3.2,iniconfig==1.1.1,isort==5.9.2,lazy-object-proxy==1.6.0,mccabe==0.6.1,mypy==0.910,mypy-extensions==0.4.3,nodeenv==1.6.0,packaging==21.0,pathspec==0.9.0,platformdirs==2.1.0,pluggy==0.13.1,pre-commit==2.13.0,py==1.10.0,py-cpuinfo==8.0.0,pycodestyle==2.7.0,pyenchant==3.2.1,pyflakes==2.3.1,pylint @ file:///Users/daniel/DocumentenLaptop/Programming/Github/pylint/.tox/.tmp/package/3/pylint-2.9.6.dev0.zip,pyparsing==2.4.7,pytest==6.2.4,pytest-benchmark==3.4.1,pytest-cov==2.12.1,pytest-forked==1.3.0,pytest-profiling==1.7.0,pytest-xdist==2.3.0,PyYAML==5.4.1,regex==2021.7.6,requests==2.26.0,schema==0.7.4,six==1.16.0,tabulate==0.8.9,tbump==6.3.2,toml==0.10.2,tomli==1.1.0,tomlkit==0.7.2,types-pkg-resources==0.1.3,types-toml==0.1.3,typing-extensions==3.10.0.0,Unidecode==1.2.0,urllib3==1.26.6,virtualenv==20.6.0,wrapt==1.12.1
py39 run-test-pre: PYTHONHASHSEED='2685728459'
py39 run-test: commands[0] | pytest --benchmark-disable /Users/daniel/DocumentenLaptop/Programming/Github/pylint/tests/
============================= test session starts ==============================
platform darwin -- Python 3.9.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
cachedir: .tox/py39/.pytest_cache
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /Users/daniel/DocumentenLaptop/Programming/Github/pylint, configfile: setup.cfg, testpaths: tests
plugins: cov-2.12.1, benchmark-3.4.1, xdist-2.3.0, profiling-1.7.0, forked-1.3.0
collected 1681 items / 204 deselected / 1477 selected

tests/test_check_parallel.py ...........................                 [  1%]
tests/test_config.py ....                                                [  2%]
tests/test_epylint.py ..                                                 [  2%]
tests/test_func.py ............                                          [  3%]
tests/test_functional.py F.............................................. [  6%]
........F.........................s....................................s [ 11%]
......................................................FF.....F.......... [ 15%]
.............................................s.s.ss....s................ [ 20%]
........s..........F....s...............................s............... [ 25%]
.F....F.................F...............................s............... [ 30%]
..........................ss.s.s.............F.F.s...................... [ 35%]
.....s....s.s.s................................                          [ 38%]
tests/test_import_graph.py ......                                        [ 39%]
tests/test_numversion.py ...........                                     [ 39%]
tests/test_pragma_parser.py ............                                 [ 40%]
tests/test_pylint_runners.py ....                                        [ 40%]
tests/test_regr.py ..................                                    [ 42%]
tests/test_self.py ..................................................... [ 45%]
....................................................                     [ 49%]
tests/unittest_config.py ......                                          [ 49%]
tests/unittest_pyreverse_diadefs.py .........                            [ 50%]
tests/unittest_pyreverse_inspector.py ........                           [ 50%]
tests/unittest_pyreverse_writer.py ..................                    [ 51%]
tests/unittest_reporters_json.py .                                       [ 52%]
tests/unittest_reporting.py .....                                        [ 52%]
tests/benchmark/test_baseline_benchmarks.py ...........                  [ 53%]
tests/checkers/unittest_base.py .............s..........s                [ 54%]
tests/checkers/unittest_classes.py .........                             [ 55%]
tests/checkers/unittest_deprecated.py ........................           [ 57%]
tests/checkers/unittest_exceptions.py ..                                 [ 57%]
tests/checkers/unittest_format.py ..............                         [ 58%]
tests/checkers/unittest_imports.py ......                                [ 58%]
tests/checkers/unittest_logging.py ........                              [ 59%]
tests/checkers/unittest_misc.py ..........                               [ 59%]
tests/checkers/unittest_python3.py ..................................... [ 62%]
.....................................................                    [ 65%]
tests/checkers/unittest_refactoring.py .                                 [ 65%]
tests/checkers/unittest_similar.py ...............                       [ 66%]
tests/checkers/unittest_spelling.py ssssssssssssssssssssssssssssssssss   [ 69%]
tests/checkers/unittest_stdlib.py ......                                 [ 69%]
tests/checkers/unittest_strings.py ...                                   [ 69%]
tests/checkers/unittest_typecheck.py ....................                [ 71%]
tests/checkers/unittest_utils.py ..................................      [ 73%]
tests/checkers/unittest_variables.py .....................               [ 74%]
tests/extensions/test_bad_builtin.py .                                   [ 75%]
tests/extensions/test_broad_try_clause.py .                              [ 75%]
tests/extensions/test_check_docs.py .................................... [ 77%]
........................................................................ [ 82%]
..............                                                           [ 83%]
tests/extensions/test_check_docs_utils.py ..............                 [ 84%]
tests/extensions/test_check_mccabe.py ..                                 [ 84%]
tests/extensions/test_check_raise_docs.py .............................. [ 86%]
................                                                         [ 87%]
tests/extensions/test_check_return_docs.py ............................. [ 89%]
...........                                                              [ 90%]
tests/extensions/test_check_yields_docs.py ............................  [ 92%]
tests/extensions/test_comparetozero.py .                                 [ 92%]
tests/extensions/test_confusing_elif.py ........                         [ 92%]
tests/extensions/test_docstyle.py .                                      [ 92%]
tests/extensions/test_elseif_used.py .                                   [ 92%]
tests/extensions/test_empty_comment.py .                                 [ 92%]
tests/extensions/test_emptystring.py .                                   [ 93%]
tests/extensions/test_overlapping_exceptions.py .F                       [ 93%]
tests/extensions/test_redefined.py .                                     [ 93%]
tests/functional/r/redundant_unittest_assert.py ss                       [ 93%]
tests/lint/unittest_expand_modules.py ....                               [ 93%]
tests/lint/unittest_lint.py ............................................ [ 96%]
.......                                                                  [ 97%]
tests/message/unittest_message.py .                                      [ 97%]
tests/message/unittest_message_definition.py ......                      [ 97%]
tests/message/unittest_message_definition_store.py .................     [ 98%]
tests/message/unittest_message_id_store.py .......                       [ 99%]
tests/profile/test_profile_against_externals.py s                        [ 99%]
tests/testutils/test_output_line.py .......                              [ 99%]
tests/utils/unittest_ast_walker.py ..                                    [ 99%]
tests/utils/unittest_utils.py ..                                         [100%]

=================================== FAILURES ===================================
___________________ test_functional[import_outside_toplevel] ___________________

self = <pylint.testutils.lint_module_test.LintModuleTest object at 0x10ca69d00>

    def runTest(self):
>       self._runTest()
E       AssertionError: Wrong results for file "import_outside_toplevel":
E       
E       Unexpected in testdata:
E         42: no-name-in-module

.tox/py39/lib/python3.9/site-packages/pylint/testutils/lint_module_test.py:80: AssertionError
______________ test_functional[regression_1326_crash_uninferable] ______________

self = <pylint.testutils.lint_module_test.LintModuleTest object at 0x1118d3f40>

    def runTest(self):
>       self._runTest()
E       AssertionError: Wrong results for file "regression_1326_crash_uninferable":
E       
E       Unexpected in testdata:
E          5: no-member

.tox/py39/lib/python3.9/site-packages/pylint/testutils/lint_module_test.py:80: AssertionError
__________________ test_functional[invalid_exceptions_raised] __________________

self = <pylint.testutils.lint_module_test.LintModuleTest object at 0x113a0b520>

    def runTest(self):
>       self._runTest()
E       AssertionError: Wrong results for file "invalid_exceptions_raised":
E       
E       Unexpected in testdata:
E         22: c-extension-no-member

.tox/py39/lib/python3.9/site-packages/pylint/testutils/lint_module_test.py:80: AssertionError
__________________ test_functional[invalid_exceptions_caught] __________________

self = <pylint.testutils.lint_module_test.LintModuleTest object at 0x114c4e760>

    def runTest(self):
>       self._runTest()
E       AssertionError: Wrong results for file "invalid_exceptions_caught":
E       
E       Unexpected in testdata:
E         19: no-member
E         98: c-extension-no-member

.tox/py39/lib/python3.9/site-packages/pylint/testutils/lint_module_test.py:80: AssertionError
____________________ test_functional[inconsistent_returns] _____________________

self = <pylint.testutils.lint_module_test.LintModuleTest object at 0x112f174c0>

    def runTest(self):
>       self._runTest()
E       AssertionError: Wrong results for file "inconsistent_returns":
E       
E       Unexpected in testdata:
E         10: c-extension-no-member
E         17: c-extension-no-member
E        162: c-extension-no-member
E        167: c-extension-no-member

.tox/py39/lib/python3.9/site-packages/pylint/testutils/lint_module_test.py:80: AssertionError
_____________________ test_functional[socketerror_import] ______________________

self = <pylint.testutils.lint_module_test.LintModuleTest object at 0x118105d00>

    def runTest(self):
>       self._runTest()
E       AssertionError: Wrong results for file "socketerror_import":
E       
E       Unexpected in testdata:
E          3: no-name-in-module

.tox/py39/lib/python3.9/site-packages/pylint/testutils/lint_module_test.py:80: AssertionError
_____________________ test_functional[function_redefined] ______________________

self = <pylint.testutils.lint_module_test.LintModuleTest object at 0x11801d370>

    def runTest(self):
>       self._runTest()
E       AssertionError: Wrong results for file "function_redefined":
E       
E       Unexpected in testdata:
E         85: no-name-in-module

.tox/py39/lib/python3.9/site-packages/pylint/testutils/lint_module_test.py:80: AssertionError
_______________ test_functional[f_string_without_interpolation] ________________

self = <pylint.testutils.lint_module_test.LintModuleTest object at 0x114277580>

    def runTest(self):
>       self._runTest()
E       AssertionError: Wrong results for file "f_string_without_interpolation":
E       
E       Unexpected in testdata:
E          7: c-extension-no-member

.tox/py39/lib/python3.9/site-packages/pylint/testutils/lint_module_test.py:80: AssertionError
________________________ test_functional[member_checks] ________________________

self = <pylint.testutils.lint_module_test.LintModuleTest object at 0x1182b4eb0>

    def runTest(self):
>       self._runTest()
E       AssertionError: Wrong results for file "member_checks":
E       
E       Unexpected in testdata:
E        151: no-member

.tox/py39/lib/python3.9/site-packages/pylint/testutils/lint_module_test.py:80: AssertionError
_____________________ test_functional[builtin_module_test] _____________________

self = <pylint.testutils.lint_module_test.LintModuleTest object at 0x11ac0fbb0>

    def runTest(self):
>       self._runTest()
E       AssertionError: Wrong results for file "builtin_module_test":
E       
E       Unexpected in testdata:
E          4: no-name-in-module

.tox/py39/lib/python3.9/site-packages/pylint/testutils/lint_module_test.py:80: AssertionError
____________________ test_functional[bad_exception_context] ____________________

self = <pylint.testutils.lint_module_test.LintModuleTest object at 0x11aba2bb0>

    def runTest(self):
>       self._runTest()
E       AssertionError: Wrong results for file "bad_exception_context":
E       
E       Unexpected in testdata:
E         19: no-member

.tox/py39/lib/python3.9/site-packages/pylint/testutils/lint_module_test.py:80: AssertionError
_______________________ test_overlapping_exceptions_py33 _______________________

linter = Checker 'master' (responsible for 'F0001', 'F0002', 'F0010', 'I0001', 'I0010', 'I0011', 'I0013', 'I0020', 'I0021', 'I0022', 'E0001', 'E0011', 'E0012', 'E0013')

>   ???
E   AssertionError: assert 'E1101' == 'W0714'
E     - W0714
E     + E1101

/Users/daniel/Documents/Programming/Github/DanielNoord/pylint/tests/extensions/test_overlapping_exceptions.py:84: AssertionError
=========================== short test summary info ============================
FAILED tests/test_functional.py::test_functional[import_outside_toplevel] - A...
FAILED tests/test_functional.py::test_functional[regression_1326_crash_uninferable]
FAILED tests/test_functional.py::test_functional[invalid_exceptions_raised]
FAILED tests/test_functional.py::test_functional[invalid_exceptions_caught]
FAILED tests/test_functional.py::test_functional[inconsistent_returns] - Asse...
FAILED tests/test_functional.py::test_functional[socketerror_import] - Assert...
FAILED tests/test_functional.py::test_functional[function_redefined] - Assert...
FAILED tests/test_functional.py::test_functional[f_string_without_interpolation]
FAILED tests/test_functional.py::test_functional[member_checks] - AssertionEr...
FAILED tests/test_functional.py::test_functional[builtin_module_test] - Asser...
FAILED tests/test_functional.py::test_functional[bad_exception_context] - Ass...
FAILED tests/extensions/test_overlapping_exceptions.py::test_overlapping_exceptions_py33
=== 12 failed, 1406 passed, 59 skipped, 204 deselected in 144.66s (0:02:24) ====
ERROR: InvocationError for command /Users/daniel/DocumentenLaptop/Programming/Github/pylint/.tox/py39/bin/pytest --benchmark-disable tests (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   py39: commands failed

[cdce8p]

Can you run these two commands and check if anything changes?

# The output might help
pylint --version

tox -r -e py39

--
You could also try the manual route:

1. Create a virtual environment and activate it

# Inside the 'pylint' folder
python3.9 -m venv venv-39
source venv-39/bin/activate

2. Install all requirements

python -m pip install -U pip setuptools wheel
pip install -r requirements_test.txt

3. Run pytest

pytest tests/test_functional.py

[DanielNoord]

Manual route still gives me the same failed tests..

Perhaps it has something to do with my local python installation, but I wouldn't know why.
Could it be that I linked python to my python 3.9 installation instead of the standard python 2.7?

[DudeNr33]

I also have this issue on MacOS 11.4 BigSur, and they only occur with Python 3.9.

Python 3.8 works fine.
These errors have in common, that they:

• are reported for modules of the standard library
• the modules in question are no plain Python modules, but written in C.

[cdce8p]

Quite strange. I have no issues with Python 3.7 through 3.10
Used the installer from python.org

[DanielNoord]

Might it be something with Big Sur? In your earlier comment you mentioned you were using Catalina.
Perhaps something changed with the naming of modules in Big Sur?

[DudeNr33]

Another possibility would be the architecture, as I am running an M1 with arm64. At least the Downloads page on python.org has two separate installers (Intel and universal2).
However, the Python 3.8 installer is also already available for M1.

[Pierre-Sassoulas]

Maybe we need to change those options in the pylintrc ?

# Allow loading of arbitrary C extensions. Extensions are imported into the
# active Python interpreter and may run arbitrary code.
unsafe-load-any-extension=no

# A comma-separated list of package or module names from where C extensions may
# be loaded. Extensions are loading into the active Python interpreter and may
# run arbitrary code
extension-pkg-allow-list=

[DudeNr33]

As this currently only happens with some yet to determine specific combinations of Python version / architecture / OS I would say no - I don't think it is pylint's (or astroid's) fault.
Or even if it is, we should try to fix it instead just suppressing it.

Edit: just checked: running file on both executable reports them both as Mach-O 64-bit executable arm64.
Perhaps they are still optimising their Apple Silicon builds (they are marked as "experimental")?

[DudeNr33]

OK, I just dug a little deeper.

The versions installed with pyenv work fine, regardless of the Python version (tested 3.8.10, 3.9.5, 3.9.6).
The Python version I used before I switched to Pylint (Python 3.9.6 installed via homebrew) does have the problem.

I do not know if homebrew installs Python via a .pkg or from source.
At least pyenv definitely builds from source.

[DanielNoord]

This became a real issue when working on #4784.

To test my implementation I copied the code to my local pylint installation. This worked and all messages showed up correctly.
However, when using the testsuite to test the fix some messages did not show up. After some frustrating 30 minutes of trying to find the problem I tried to see if the CI would give me the same problems. Just like with the errors reported above, they disappeared as soon as I ran the tests via Github.

Do we have any idea about how to go about solving this issue? I would gladly help, but have no idea where to start..

[DudeNr33]

How did you install your Python version?
For me using a venv created with pyenv worked fine, which compiles Python from source.
Maybe this could be a workaround until someone finds a permanent solution.

When I first stumbled over this I made a list to check what the culprits are, and it boiled down to two modules:

• socket (socket.error, socket.AF_INET, socket.SOCK_STREAM)
• math (math.log10, math.pi, math.ceil, math.sqrt)

For example, the math.log10 is found in math.pyi (i.e. the stub file) when using command+SHIFT to jump to source.
Maybe those stub files are not installed correctly with some installation methods.
I'll try to figure out where they should live - if I jump from VScode to the .pyi file, it opens one of the fallback stub files provided by the Pylance plugin for VScode.

Edit:
Ok, after creating a new environment with Python 3.9.6 using pyenv and then running pip install -r requirements_test.txt, I can find a math.pyi file in this environment coming from mypy:

(tmp396) 3.9.6 % find . -name "math.pyi" 
./envs/tmp396/lib/python3.9/site-packages/mypy/typeshed/stdlib/math.pyi

... whereas my originally created Python 3.9 environment does not have a math.pyi file - at least not in the correct location?

(tmp396) 3.9.6 % find ~/programming/forks/pylint/venv39 -name "math.pyi"
/Users/andreas/programming/forks/pylint/venv39/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/2and3/math.pyi
/Users/andreas/programming/forks/pylint/venv39/lib/python3.9/site-packages/jedi/third_party/django-stubs/django-stubs/db/models/functions/math.pyi
/Users/andreas/programming/forks/pylint/venv39/lib/python3.9/site-packages/mypy/typeshed/stdlib/2and3/math.pyi

However, upgrading mypy in this virtual environment from V0.812 to V0.910 moved the math.pyi file to

/Users/andreas/programming/forks/pylint/venv39/lib/python3.9/site-packages/mypy/typeshed/stdlib/math.pyi

but pylint still complains.

Hmm, I really hoped that some missing *.pyi file could have been the solution.
I document the result of this analysis anyway in case somebody wants to analyse as well and to save them some time.

Maybe @Pierre-Sassoulas or @cdce8p who have more knowledge about how inference from C implementations work in astroid can give a hint in which direction we could go for analysis?

[DanielNoord]

Currently I'm using HomeBrew.

Going to look into pyenv in the meantime, thanks!

Edit: virtualenv from pyenv seems to work!

[Pierre-Sassoulas]

I don't have a mac so I can't test but could it be that the version of socket is not the same ? I remember that Mac often have outdated lib by default.

[DanielNoord]

I noticed that socket is a wrapper for _socket. However, my homebrew installation does not seem to have a _socket.py file.
@DudeNr33 does your homebrew installation have a _socket.py file?

[DudeNr33]

@DanielNoord my installation has neither (both the "working" installation from pyenv and the "non-working" from homebrew).
But I think that's because _socket is a C module, so you won't find a _socket.py.
Rather I have a ./lib/python3.9/lib-dynload/_socket.cpython-39-darwin.so, which should be the compiled shared object of the C implementation.

[dc5048]

Similar to @DudeNr33, I am getting false positives when importing from math (all members as far as I can tell, noticed it for pi, sin, cos, fmod, and others). I'm running Big Sur 11.4 on an intel architecture, and the problem started occurring after updating python to 3.9.6 yesterday using homebrew.

[ghost]

Same here. Using pyenv's python fixed this problem.

[tgree]

I'm also getting this same issue with math, socket and also struct which hasn't been mentioned before that I see.

[DanielNoord]

It might be interesting to take another look at this now that astroid resolves symlinks a little better.

[DanielNoord]

I just did another test on python3.9 from homebrew, so not pyenv and I think all issues have been solved. At least, for me the test suite passes locally.

I'm still going to open a PR to run one suite on CI though as it wouldn't hurt to do so.