Update build to support Python 3.8

[brahmaneya]

Description of proposed changes

Build config updates:

• Add Python 3.8 build to Travis
• Remove unneeded coverage and documentation checks for >3.6

Package version updates:

• Clamp NumPy to a max of 1.20 which introduces typing and other incompatible changes
• Clamp blis, which requires newer Python versions for latest releases
• Pin Dask and distributed requirements to pull versions before version scheme change (now YYYY.MM)
• Bump scikit-learn version upper bound to allow recent versions (which correctly has a Cython dependency for isolated wheel builds on Python 3.8+)

Fixes #1627

Test plan

• Travis runs pass for 3.6, 3.7, 3.8
• Will manually monitor full trunk runs once this lands

Checklist

Need help on these? Just ask!

• I have read the CONTRIBUTING document.
• I have updated the documentation accordingly.
• I have added tests to cover my changes.
• I have run tox -e complex and/or tox -e spark if appropriate.
• All new and existing tests passed.

[rth]

(which correctly has a Cython dependency)

For information, the issue not a Cython dependency, but that for python 3.8 users are trying to build scikit-learn 0.22 from sources since binary wheels do not exist for that Python version. And there yes, it looks like Cython should have been declared a build dependency but it wasn't.

[rth]

BTW, if you want to auto-close the associate issue the PR description needs to say "Fixes #1620" (there is a formatting issue currently).

[rjurney]

(which correctly has a Cython dependency)

For information, the issue not a Cython dependency, but that for python 3.8 users are trying to build scikit-learn 0.22 from sources since binary wheels do not exist for that Python version. And there yes, it looks like Cython should have been declared a build dependency but it wasn't.

Building Cython doesn't actually fix the problem, however.

[rth]

Building Cython doesn't actually fix the problem, however.

You shouldn't need Cython to install a recent version of scikit-learn . What issue are you experiencing? If I install this branch of snorkel on Python 3.8

conda create -n test-env python=3.8 scipy
conda activate test-env
pip install https://github.com/snorkel-team/snorkel/archive/sklearn_bump.zip

it works fine for me:

  Downloading https://github.com/snorkel-team/snorkel/archive/sklearn_bump.zip
     \ 1.5 MB 3.0 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: numpy<2.0.0,>=1.16.0 in ./miniconda3/envs/scipy-dev/lib/python3.8/site-packages (from snorkel==0.9.6+dev) (1.20.1)
Collecting munkres>=1.0.6
  Downloading munkres-1.1.4-py2.py3-none-any.whl (7.0 kB)
Collecting networkx<2.4,>=2.2
  Downloading networkx-2.3.zip (1.7 MB)
     |████████████████████████████████| 1.7 MB 19.2 MB/s 
Collecting decorator>=4.3.0
  Using cached decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Collecting pandas<2.0.0,>=0.25.0
  Downloading pandas-1.2.2-cp38-cp38-manylinux1_x86_64.whl (9.7 MB)
     |████████████████████████████████| 9.7 MB 76.6 MB/s 
Collecting python-dateutil>=2.7.3
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting pytz>=2017.3
  Downloading pytz-2021.1-py2.py3-none-any.whl (510 kB)
     |████████████████████████████████| 510 kB 100.7 MB/s 
Collecting scikit-learn<0.25.0,>=0.20.2
  Using cached scikit_learn-0.24.1-cp38-cp38-manylinux2010_x86_64.whl (24.9 MB)
Collecting joblib>=0.11
  Downloading joblib-1.0.1-py3-none-any.whl (303 kB)
     |████████████████████████████████| 303 kB 112.5 MB/s 
Collecting scipy<2.0.0,>=1.2.0
  Using cached scipy-1.6.0-cp38-cp38-manylinux1_x86_64.whl (27.2 MB)
Collecting six>=1.5
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting tensorboard<2.0.0,>=1.14.0
  Downloading tensorboard-1.15.0-py3-none-any.whl (3.8 MB)
     |████████████████████████████████| 3.8 MB 92.4 MB/s 
Requirement already satisfied: wheel>=0.26 in ./miniconda3/envs/scipy-dev/lib/python3.8/site-packages (from tensorboard<2.0.0,>=1.14.0->snorkel==0.9.6+dev) (0.36.2)
Requirement already satisfied: setuptools>=41.0.0 in ./miniconda3/envs/scipy-dev/lib/python3.8/site-packages (from tensorboard<2.0.0,>=1.14.0->snorkel==0.9.6+dev) (52.0.0.post20210125)
Collecting absl-py>=0.4
  Downloading absl_py-0.11.0-py3-none-any.whl (127 kB)
     |████████████████████████████████| 127 kB 107.3 MB/s 
Collecting grpcio>=1.6.3
  Downloading grpcio-1.35.0-cp38-cp38-manylinux2014_x86_64.whl (4.1 MB)
     |████████████████████████████████| 4.1 MB 99.6 MB/s 
Collecting markdown>=2.6.8
  Downloading Markdown-3.3.3-py3-none-any.whl (96 kB)
     |████████████████████████████████| 96 kB 6.4 MB/s 
Collecting protobuf>=3.6.0
  Downloading protobuf-3.14.0-cp38-cp38-manylinux1_x86_64.whl (1.0 MB)
     |████████████████████████████████| 1.0 MB 98.6 MB/s 
Collecting threadpoolctl>=2.0.0
  Using cached threadpoolctl-2.1.0-py3-none-any.whl (12 kB)
Collecting torch<2.0.0,>=1.2.0
  Downloading torch-1.7.1-cp38-cp38-manylinux1_x86_64.whl (776.8 MB)
     |████████████████████████████████| 776.8 MB 20 kB/s 
Collecting tqdm<5.0.0,>=4.33.0
  Downloading tqdm-4.56.2-py2.py3-none-any.whl (72 kB)
     |████████████████████████████████| 72 kB 1.3 MB/s 
Collecting werkzeug>=0.11.15
  Using cached Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
Collecting typing-extensions
  Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Building wheels for collected packages: snorkel, networkx
  Building wheel for snorkel (PEP 517) ... done
  Created wheel for snorkel: filename=snorkel-0.9.6+dev-py3-none-any.whl size=145128 sha256=bccc603807dbbee4a4827dafff6fac5d14e85bdb29b73defe73debd45a042268
  Stored in directory: /tmp/pip-ephem-wheel-cache-jjuqwci_/wheels/37/0e/82/c9e52f544e98eed20b9c6dd50df8842828bdafcf6819c658c6
  Building wheel for networkx (setup.py) ... done
  Created wheel for networkx: filename=networkx-2.3-py2.py3-none-any.whl size=1555990 sha256=c69626ad1eab9ba996f64d9b85d0d7c045af9ba2f19f8f385ff6d5b37b52a497
  Stored in directory: /home/rth/.cache/pip/wheels/ff/62/9e/0ed2d25fd4f5761e2d19568cda0c32716556dfa682e65ecf64
Successfully built snorkel networkx
Installing collected packages: six, werkzeug, typing-extensions, threadpoolctl, scipy, pytz, python-dateutil, protobuf, markdown, joblib, grpcio, decorator, absl-py, tqdm, torch, tensorboard, scikit-learn, pandas, networkx, munkres, snorkel
Successfully installed absl-py-0.11.0 decorator-4.4.2 grpcio-1.35.0 joblib-1.0.1 markdown-3.3.3 munkres-1.1.4 networkx-2.3 pandas-1.2.2 protobuf-3.14.0 python-dateutil-2.8.1 pytz-2021.1 scikit-learn-0.24.1 scipy-1.6.0 six-1.15.0 snorkel-0.9.6+dev tensorboard-1.15.0 threadpoolctl-2.1.0 torch-1.7.1 tqdm-4.56.2 typing-extensions-3.7.4.3 werkzeug-1.0.1

[codecov]

Codecov Report

Merging #1628 (af6f152) into master (ed77718) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1628   +/-   ##
=======================================
  Coverage   97.21%   97.21%           
=======================================
  Files          68       68           
  Lines        2151     2151           
  Branches      345      345           
=======================================
  Hits         2091     2091           
  Misses         31       31           
  Partials       29       29           

[henryre]

LGTM! @brahmaneya can you take a look now since I made some changes?

[henryre]

As a note: we turned on weekly cron builds to detect issues from changing dependency on an ongoing basis

[rjurney]

@henryre thanks!