add in reinstall for dask and distributed after feast install #713

jperez999 · 2022-11-01T15:55:31Z

This will fix issues with container integration testing that are caused by downgrade of dask when feast<0.20.0 is installed (installs dask==2022.01.0). That downgrade creates a mismatch between distributed and dask which causes issues in testing.

github-actions · 2022-11-01T15:57:32Z

Documentation preview

https://nvidia-merlin.github.io/Merlin/review/pr-713

nvidia-merlin-bot · 2022-11-01T16:08:57Z

Click to view CI Results

GitHub pull request #713 of commit f3ab75709ed145a5bcf157f1001be66b8ecd1c63, no merge conflicts.
Running as SYSTEM
Setting status of f3ab75709ed145a5bcf157f1001be66b8ecd1c63 to PENDING with url https://10.20.13.93:8080/job/merlin_merlin/545/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/713/*:refs/remotes/origin/pr/713/* # timeout=10
 > git rev-parse f3ab75709ed145a5bcf157f1001be66b8ecd1c63^{commit} # timeout=10
Checking out Revision f3ab75709ed145a5bcf157f1001be66b8ecd1c63 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f f3ab75709ed145a5bcf157f1001be66b8ecd1c63 # timeout=10
Commit message: "add in reinstall for dask and distributed after feast install"
 > git rev-list --no-walk bd56ceecd707dfe26007fc8c1f117b86cd806ac2 # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins7048600700063970656.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.26.5,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.28.5,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.8.2,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.42,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.0,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3526984'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-1hw5faw_
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-1hw5faw_
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit dff7b0274606f25d846edbc851b99f5603b89ae8
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+9.gdff7b02) (0.3.0+12.g78ecddd)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+9.gdff7b02) (1.1.1)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+9.gdff7b02) (2.22.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (7.0.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (0.55.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (1.3.5)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (3.19.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (1.2.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (1.10.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (4.64.1)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (0.4.3)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (0.12.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (2022.5.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (2.4.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (1.20.3)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 67.1 MB/s eta 0:00:00
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+9.gdff7b02) (6.0.1)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+9.gdff7b02-py3-none-any.whl size=86642 sha256=5cd3b5bd6bfc0b9a23f8785bddb9141abc5b6bbc551fb5a25f0ce89758b3aee8
  Stored in directory: /tmp/pip-ephem-wheel-cache-krrhhqiy/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+9.gdff7b02
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/models.git

Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-lgd40brt

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-lgd40brt

Resolved https://github.com/NVIDIA-Merlin/models.git to commit ea40954806b7ffefa3a7afdcd0da4b81331e21ac

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+21.gea409548) (0.3.0+12.g78ecddd)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (21.3)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (7.0.0)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (0.55.1)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (1.3.5)

Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (2022.3.0)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (3.19.5)

Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (2022.3.0)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (1.2.5)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (1.10.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (4.64.1)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (1.2.0)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (0.4.3)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (2.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (5.4.1)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (1.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (0.12.0)

Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (2022.5.0)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (1.7.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (6.1)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (1.0.4)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (8.1.3)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (5.8.0)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (2.0.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (3.1.2)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (2.4.0)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (1.20.3)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (0.38.1)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (65.4.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (2022.2.1)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (1.0.1)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (6.0.2)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (4.1.0)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (2.1.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (4.0.0)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+21.gea409548) (6.0.1)

Building wheels for collected packages: merlin-models

Building wheel for merlin-models (pyproject.toml): started

Building wheel for merlin-models (pyproject.toml): finished with status 'done'

Created wheel for merlin-models: filename=merlin_models-0.9.0+21.gea409548-py3-none-any.whl size=356069 sha256=8fe7d47dc7121f72cdb4ac4bd8bdb8f73623a287aeb8f681ea9f690cf9d83810

Stored in directory: /tmp/pip-ephem-wheel-cache-et7klc74/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5

Successfully built merlin-models

Installing collected packages: merlin-models

Attempting uninstall: merlin-models

Found existing installation: merlin-models 0.7.0+11.g280956aa4

Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu

Can't uninstall 'merlin-models'. No files were found to uninstall.

Successfully installed merlin-models-0.9.0+21.gea409548
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git

Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-__opmue1

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-__opmue1

Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit c0636deadefebcf238935267239406bf7cc6803f

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+1.gc0636dea) (0.3.0+12.g78ecddd)

Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+1.gc0636dea) (1.8.1)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (21.3)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (7.0.0)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (0.55.1)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (1.3.5)

Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (2022.3.0)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (3.19.5)

Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (2022.3.0)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (1.2.5)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (1.10.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (4.64.1)

Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+1.gc0636dea) (1.20.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (1.2.0)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (0.4.3)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (2.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (5.4.1)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (1.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (0.12.0)

Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (2022.5.0)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (1.7.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (6.1)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (1.0.4)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (8.1.3)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (5.8.0)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (2.0.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (3.1.2)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (2.4.0)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (0.38.1)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (65.4.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (2022.2.1)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (1.0.1)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (6.0.2)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (4.1.0)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (2.1.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (4.0.0)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+1.gc0636dea) (6.0.1)

Building wheels for collected packages: nvtabular

Building wheel for nvtabular (pyproject.toml): started

Building wheel for nvtabular (pyproject.toml): finished with status 'done'

Created wheel for nvtabular: filename=nvtabular-1.6.0+1.gc0636dea-cp38-cp38-linux_x86_64.whl size=266739 sha256=8dbf306eae0c3495b4c8a85bfe9ed84213376c59c836cbe52fe419ed20d40d08

Stored in directory: /tmp/pip-ephem-wheel-cache-c_mejcbc/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc

Successfully built nvtabular

Installing collected packages: nvtabular

Attempting uninstall: nvtabular

Found existing installation: nvtabular 1.1.1

Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu

Can't uninstall 'nvtabular'. No files were found to uninstall.

Successfully installed nvtabular-1.6.0+1.gc0636dea
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/core.git

Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-d7om4o30

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-d7om4o30

Resolved https://github.com/NVIDIA-Merlin/core.git to commit eda153c663aa864da66927c7a0a9d4e64c073120

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (21.3)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (7.0.0)

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (2022.5.0)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (0.55.1)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (1.3.5)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (3.19.5)

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (2022.3.0)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (1.2.5)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (1.10.0)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (2022.3.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (4.64.1)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (1.2.0)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (0.4.3)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (5.4.1)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (0.12.0)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.7.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (6.1)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.0.4)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (8.1.3)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (5.8.0)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.0.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (3.1.2)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.4.0)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+4.geda153c) (1.20.3)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+4.geda153c) (0.38.1)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+4.geda153c) (65.4.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+4.geda153c) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+4.geda153c) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+4.geda153c) (2022.2.1)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+4.geda153c) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+4.geda153c) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+4.geda153c) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.0.1)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (6.0.2)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (4.1.0)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.1.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (4.0.0)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (6.0.1)

Building wheels for collected packages: merlin-core

Building wheel for merlin-core (pyproject.toml): started

Building wheel for merlin-core (pyproject.toml): finished with status 'done'

Created wheel for merlin-core: filename=merlin_core-0.8.0+4.geda153c-py3-none-any.whl size=118258 sha256=d5cac776d176223f92146aa613c9d051fa36cdc3b24b271900a722672333868d

Stored in directory: /tmp/pip-ephem-wheel-cache-cgces5w3/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84

Successfully built merlin-core

Installing collected packages: merlin-core

Attempting uninstall: merlin-core

Found existing installation: merlin-core 0.3.0+12.g78ecddd

Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu

Can't uninstall 'merlin-core'. No files were found to uninstall.

Successfully installed merlin-core-0.8.0+4.geda153c
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin

plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0

collected 3 items
tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py F      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py F               [100%]
=================================== FAILURES ===================================

__________________________________ test_func ___________________________________
def test_func():
    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "01-Building-Recommender-Systems-with-Merlin.ipynb",
        execute=False,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["NUM_ROWS"] = "100000"
            os.system("mkdir -p /tmp/examples")
            os.environ["BASE_DIR"] = "/tmp/examples/"
            """
        )


      tb1.execute()


tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:32:

../../../.local/lib/python3.8/site-packages/testbook/client.py:147: in execute

super().execute_cell(cell, index)

../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped

return just_run(coro(*args, **kwargs))

../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run

return loop.run_until_complete(coro)

/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete

return future.result()

../../../.local/lib/python3.8/site-packages/nbclient/client.py:965: in async_execute_cell

await self._check_raise_for_error(cell, cell_index, exec_reply)

self = <testbook.client.TestbookNotebookClient object at 0x7fe02a09bd90>

cell = {'cell_type': 'code', 'execution_count': 22, 'id': 'bfe2aa9e', 'metadata': {'execution': {'iopub.status.busy': '2022-1..., run_eagerly=False, metrics=[tf.keras.metrics.AUC()])\nmodel.fit(train, validation_data=valid, batch_size=16 * 1024)'}

cell_index = 49

exec_reply = {'buffers': [], 'content': {'ename': 'MemoryError', 'engine_info': {'engine_id': -1, 'engine_uuid': '913ec570-aa50-419...e, 'engine': '913ec570-aa50-419c-9e60-6dbf572da561', 'started': '2022-11-01T16:08:25.873824Z', 'status': 'error'}, ...}
async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:


      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)


E E           ------------------
 E E E           ------------------
 E
 E E E E           �[1;32m E E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E E
 E E           �[1;32m E E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E           �[1;32m    841�[0m ):
 E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m    854�[0m )
 E
 E E
 E E

nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
 model.compile(optimizer="adam", run_eagerly=False, metrics=[tf.keras.metrics.AUC()])
 model.fit(train, validation_data=valid, batch_size=16 * 1024)
 �[0;31m---------------------------------------------------------------------------�[0m
 �[0;31mMemoryError�[0m                               Traceback (most recent call last)
 Cell �[0;32mIn [22], line 2�[0m
 1�[0m model�[38;5;241m.�[39mcompile(optimizer�[38;5;241m=�[39m�[38;5;124m"�[39m�[38;5;124madam�[39m�[38;5;124m"�[39m, run_eagerly�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m, metrics�[38;5;241m=�[39m[tf�[38;5;241m.�[39mkeras�[38;5;241m.�[39mmetrics�[38;5;241m.�[39mAUC()])
 �[0;32m----> 2�[0m �[43mmodel�[49m�[38;5;241;43m.�[39;49m�[43mfit�[49m�[43m(�[49m�[43mtrain�[49m�[43m,�[49m�[43m �[49m�[43mvalidation_data�[49m�[38;5;241;43m=�[39;49m�[43mvalid�[49m�[43m,�[49m�[43m �[49m�[43mbatch_size�[49m�[38;5;241;43m=�[39;49m�[38;5;241;43m16�[39;49m�[43m �[49m�[38;5;241;43m�[39;49m�[43m �[49m�[38;5;241;43m1024�[39;49m�[43m)�[49m
 File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py:856�[0m, in �[0;36mBaseModel.fit�[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing, train_metrics_steps, pre, **kwargs)�[0m
 853�[0m     �[38;5;28mself�[39m�[38;5;241m.�[39m_reset_compile_cache()
 854�[0m     �[38;5;28mself�[39m�[38;5;241m.�[39mtrain_pre �[38;5;241m=�[39m pre
 �[0;32m--> 856�[0m out �[38;5;241m=�[39m �[38;5;28;43msuper�[39;49m�[43m(�[49m�[43m)�[49m�[38;5;241;43m.�[39;49m�[43mfit�[49m�[43m(�[49m�[38;5;241;43m�[39;49m�[38;5;241;43m�[39;49m�[43mfit_kwargs�[49m�[43m)�[49m
 858�[0m �[38;5;28;01mif�[39;00m pre:
 859�[0m     �[38;5;28;01mdel�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mtrain_pre
 File �[0;32m~/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:67�[0m, in �[0;36mfilter_traceback..error_handler�[0;34m(args, **kwargs)�[0m
 65�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m �[38;5;28;01mas�[39;00m e:  �[38;5;66;03m# pylint: disable=broad-except�[39;00m
 66�[0m   filtered_tb �[38;5;241m=�[39m process_traceback_frames(e�[38;5;241m.�[39m__traceback_)
 �[0;32m---> 67�[0m   �[38;5;28;01mraise�[39;00m e�[38;5;241m.�[39mwith_traceback(filtered_tb) �[38;5;28;01mfrom�[39;00m �[38;5;28mNone�[39m
 68�[0m �[38;5;28;01mfinally�[39;00m:
 69�[0m   �[38;5;28;01mdel�[39;00m filtered_tb
 File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py:337�[0m, in �[0;36mLoader.__getitem__�[0;34m(self, idx)�[0m
 331�[0m �[38;5;28;01mdef�[39;00m �[38;5;21m__getitem__�[39m(�[38;5;28mself�[39m, idx):
 332�[0m     �[38;5;124;03m"""�[39;00m
 333�[0m �[38;5;124;03m    implemented exclusively for consistency�[39;00m
 334�[0m �[38;5;124;03m    with Keras model.fit. Does not leverage�[39;00m
 335�[0m �[38;5;124;03m    passed idx in any way�[39;00m
 336�[0m �[38;5;124;03m    """�[39;00m
 �[0;32m--> 337�[0m     �[38;5;28;01mreturn�[39;00m �[43mDataLoader�[49m�[38;5;241;43m.�[39;49m�[38;5;21;43m__next__�[39;49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[43m)�[49m
 File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py:356�[0m, in �[0;36mDataLoader.__next__�[0;34m(self)�[0m
 355�[0m �[38;5;28;01mdef�[39;00m �[38;5;21m__next__�[39m(�[38;5;28mself�[39m):
 �[0;32m--> 356�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_get_next_batch�[49m�[43m(�[49m�[43m)�[49m
 File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py:389�[0m, in �[0;36mDataLoader._get_next_batch�[0;34m(self)�[0m
 387�[0m �[38;5;66;03m# get the first chunks�[39;00m
 388�[0m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_batch_itr �[38;5;129;01mis�[39;00m �[38;5;28;01mNone�[39;00m:
 �[0;32m--> 389�[0m     �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_fetch_chunk�[49m�[43m(�[49m�[43m)�[49m
 391�[0m �[38;5;66;03m# try to iterate through existing batches�[39;00m
 392�[0m �[38;5;28;01mtry�[39;00m:
 File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py:368�[0m, in �[0;36mDataLoader._fetch_chunk�[0;34m(self)�[0m
 366�[0m �[38;5;28;01mif�[39;00m �[38;5;28misinstance�[39m(chunks, �[38;5;167;01mException�[39;00m):
 367�[0m     �[38;5;28mself�[39m�[38;5;241m.�[39mstop()
 �[0;32m--> 368�[0m     �[38;5;28;01mraise�[39;00m chunks
 369�[0m �[38;5;28mself�[39m�[38;5;241m.�[39m_batch_itr �[38;5;241m=�[39m �[38;5;28miter�[39m(chunks)
 File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py:156�[0m, in �[0;36mChunkQueue.load_chunks�[0;34m(self, dev)�[0m
 154�[0m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mdataloader�[38;5;241m.�[39mdevice �[38;5;241m!=�[39m �[38;5;124m"�[39m�[38;5;124mcpu�[39m�[38;5;124m"�[39m:
 155�[0m     �[38;5;28;01mwith�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mdataloader�[38;5;241m.�[39m_get_device_ctx(dev):
 �[0;32m--> 156�[0m         �[38;5;28mself�[39m�[38;5;241m.�[39mchunk_logic(itr)
 157�[0m �[38;5;28;01melse�[39;00m:
 158�[0m     �[38;5;28mself�[39m�[38;5;241m.�[39mchunk_logic(itr)
 File �[0;32m/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101�[0m, in �[0;36mannotate.call..inner�[0;34m(args, **kwargs)�[0m
 98�[0m �[38;5;129m@wraps�[39m(func)
 99�[0m �[38;5;28;01mdef�[39;00m �[38;5;21minner�[39m(�[38;5;241m�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs):
 100�[0m     libnvtx_push_range(�[38;5;28mself�[39m�[38;5;241m.�[39mattributes, �[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)
 �[0;32m--> 101�[0m     result �[38;5;241m=�[39m �[43mfunc�[49m�[43m(�[49m�[38;5;241;43m�[39;49m�[43margs�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m�[39;49m�[43mkwargs�[49m�[43m)�[49m
 102�[0m     libnvtx_pop_range(�[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)
 103�[0m     �[38;5;28;01mreturn�[39;00m result
 File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py:135�[0m, in �[0;36mChunkQueue.chunk_logic�[0;34m(self, itr)�[0m
 133�[0m chunks, spill �[38;5;241m=�[39m �[38;5;28mself�[39m�[38;5;241m.�[39mget_batch_div_chunk(chunks, �[38;5;28mself�[39m�[38;5;241m.�[39mdataloader�[38;5;241m.�[39mbatch_size)
 134�[0m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mshuffle:
 �[0;32m--> 135�[0m     chunks �[38;5;241m=�[39m �[43mshuffle_df�[49m�[43m(�[49m�[43mchunks�[49m�[43m)�[49m
 137�[0m �[38;5;28;01mif�[39;00m �[38;5;28mlen�[39m(chunks) �[38;5;241m>�[39m �[38;5;241m0�[39m:
 138�[0m     chunks �[38;5;241m=�[39m �[38;5;28mself�[39m�[38;5;241m.�[39mdataloader�[38;5;241m.�[39mmake_tensors(chunks, �[38;5;28mself�[39m�[38;5;241m.�[39mdataloader�[38;5;241m.�[39m_use_nnz)
 File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py:74�[0m, in �[0;36mshuffle_df�[0;34m(df, size, keep_index)�[0m
 72�[0m �[38;5;28;01melse�[39;00m:
 73�[0m     �[38;5;28;01mif�[39;00m _CUDF_IGNORE_INDEX_SUPPORTED:
 �[0;32m---> 74�[0m         �[38;5;28;01mreturn�[39;00m �[43mdf�[49m�[38;5;241;43m.�[39;49m�[43msample�[49m�[43m(�[49m�[43mn�[49m�[38;5;241;43m=�[39;49m�[43msize�[49m�[43m,�[49m�[43m �[49m�[43mignore_index�[49m�[38;5;241;43m=�[39;49m�[38;5;129;43;01mnot�[39;49;00m�[43m �[49m�[43mkeep_index�[49m�[43m)�[49m
 75�[0m     �[38;5;28;01melse�[39;00m:
 76�[0m         �[38;5;28;01mreturn�[39;00m df�[38;5;241m.�[39msample(n�[38;5;241m=�[39msize, keep_index�[38;5;241m=�[39mkeep_index)
 File �[0;32m/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101�[0m, in �[0;36mannotate.call..inner�[0;34m(args, **kwargs)�[0m
 98�[0m �[38;5;129m@wraps�[39m(func)
 99�[0m �[38;5;28;01mdef�[39;00m �[38;5;21minner�[39m(�[38;5;241m�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs):
 100�[0m     libnvtx_push_range(�[38;5;28mself�[39m�[38;5;241m.�[39mattributes, �[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)
 �[0;32m--> 101�[0m     result �[38;5;241m=�[39m �[43mfunc�[49m�[43m(�[49m�[38;5;241;43m�[39;49m�[43margs�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m�[39;49m�[43mkwargs�[49m�[43m)�[49m
 102�[0m     libnvtx_pop_range(�[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)
 103�[0m     �[38;5;28;01mreturn�[39;00m result
 File �[0;32m/usr/local/lib/python3.8/dist-packages/cudf/core/indexed_frame.py:2123�[0m, in �[0;36mIndexedFrame.sample�[0;34m(self, n, frac, replace, weights, random_state, axis, ignore_index)�[0m
 2120�[0m     weights �[38;5;241m=�[39m weights �[38;5;241m/�[39m weights�[38;5;241m.�[39msum()
 2122�[0m �[38;5;28;01mif�[39;00m axis �[38;5;241m==�[39m �[38;5;241m0�[39m:
 �[0;32m-> 2123�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_sample_axis_0�[49m�[43m(�[49m
 2124�[0m �[43m        �[49m�[43mn�[49m�[43m,�[49m�[43m �[49m�[43mweights�[49m�[43m,�[49m�[43m �[49m�[43mreplace�[49m�[43m,�[49m�[43m �[49m�[43mrandom_state�[49m�[43m,�[49m�[43m �[49m�[43mignore_index�[49m
 2125�[0m �[43m    �[49m�[43m)�[49m
 2126�[0m �[38;5;28;01melse�[39;00m:
 2127�[0m     �[38;5;28;01mif�[39;00m �[38;5;28misinstance�[39m(random_state, cp�[38;5;241m.�[39mrandom�[38;5;241m.�[39mRandomState):
 File �[0;32m/usr/local/lib/python3.8/dist-packages/cudf/core/indexed_frame.py:2153�[0m, in �[0;36mIndexedFrame._sample_axis_0�[0;34m(self, n, weights, replace, random_state, ignore_index)�[0m
 2148�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mNotImplementedError�[39;00m �[38;5;28;01mas�[39;00m e:
 2149�[0m     �[38;5;28;01mraise�[39;00m �[38;5;167;01mNotImplementedError�[39;00m(
 2150�[0m         �[38;5;124m"�[39m�[38;5;124mRandom sampling with cupy does not support these inputs.�[39m�[38;5;124m"�[39m
 2151�[0m     ) �[38;5;28;01mfrom�[39;00m �[38;5;21;01me�[39;00m
 �[0;32m-> 2153�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_gather�[49m�[43m(�[49m
 2154�[0m �[43m    �[49m�[43mcudf�[49m�[38;5;241;43m.�[39;49m�[43mcore�[49m�[38;5;241;43m.�[39;49m�[43mcolumn�[49m�[38;5;241;43m.�[39;49m�[43mas_column�[49m�[43m(�[49m�[43mgather_map_array�[49m�[43m)�[49m�[43m,�[49m
 2155�[0m �[43m    �[49m�[43mkeep_index�[49m�[38;5;241;43m=�[39;49m�[38;5;129;43;01mnot�[39;49;00m�[43m �[49m�[43mignore_index�[49m�[43m,�[49m
 2156�[0m �[43m    �[49m�[43mcheck_bounds�[49m�[38;5;241;43m=�[39;49m�[38;5;28;43;01mFalse�[39;49;00m�[43m,�[49m
 2157�[0m �[43m�[49m�[43m)�[49m
 File �[0;32m/usr/local/lib/python3.8/dist-packages/cudf/core/indexed_frame.py:845�[0m, in �[0;36mIndexedFrame._gather�[0;34m(self, gather_map, keep_index, nullify, check_bounds)�[0m
 839�[0m �[38;5;28;01mif�[39;00m �[38;5;129;01mnot�[39;00m libcudf�[38;5;241m.�[39mcopying�[38;5;241m.�[39m_gather_map_is_valid(
 840�[0m     gather_map, �[38;5;28mlen�[39m(�[38;5;28mself�[39m), check_bounds, nullify
 842�[0m     �[38;5;28;01mraise�[39;00m �[38;5;167;01mIndexError�[39;00m(�[38;5;124m"�[39m�[38;5;124mGather map index is out of bounds.�[39m�[38;5;124m"�[39m)
 844�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_from_columns_like_self(
 �[0;32m--> 845�[0m     �[43mlibcudf�[49m�[38;5;241;43m.�[39;49m�[43mcopying�[49m�[38;5;241;43m.�[39;49m�[43mgather�[49m�[43m(�[49m
 846�[0m �[43m        �[49m�[38;5;28;43mlist�[39;49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_index�[49m�[38;5;241;43m.�[39;49m�[43m_columns�[49m�[43m �[49m�[38;5;241;43m+�[39;49m�[43m �[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_columns�[49m�[43m)�[49m
 847�[0m �[43m        �[49m�[38;5;28;43;01mif�[39;49;00m�[43m �[49m�[43mkeep_index�[49m
 848�[0m �[43m        �[49m�[38;5;28;43;01melse�[39;49;00m�[43m �[49m�[38;5;28;43mlist�[39;49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_columns�[49m�[43m)�[49m�[43m,�[49m
 849�[0m �[43m        �[49m�[43mgather_map�[49m�[43m,�[49m
 850�[0m �[43m        �[49m�[43mnullify�[49m�[38;5;241;43m=�[39;49m�[43mnullify�[49m�[43m,�[49m
 851�[0m �[43m    �[49m�[43m)�[49m,
 852�[0m     �[38;5;28mself�[39m�[38;5;241m.�[39m_column_names,
 853�[0m     �[38;5;28mself�[39m�[38;5;241m.�[39m_index�[38;5;241m.�[39mnames �[38;5;28;01mif�[39;00m keep_index �[38;5;28;01melse�[39;00m �[38;5;28;01mNone�[39;00m,
 File �[0;32mcudf/_lib/copying.pyx:185�[0m, in �[0;36mcudf._lib.copying.gather�[0;34m()�[0m
 �[0;31mMemoryError�[0m: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory
 MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory
../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError

----------------------------- Captured stderr call -----------------------------

2022-11-01 16:08:03.093908: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-01 16:08:07.179884: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-01 16:08:07.180611: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14524 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-11-01 16:08:07.181255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14524 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-11-01 16:08:07.181835: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14524 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

2022-11-01 16:08:07.188156: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 7.95G (8534360064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

2022-11-01 16:08:07.190851: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 7.15G (7680923648 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

__________________________________ test_func ___________________________________
def test_func():
    with testbook(
        REPO_ROOT / "examples" / "scaling-criteo" / "02-ETL-with-NVTabular.ipynb",
        execute=False,
        timeout=180,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["BASE_DIR"] = "/tmp/input/criteo/"
            os.environ["INPUT_DATA_DIR"] = "/tmp/input/criteo/"
            os.environ["OUTPUT_DATA_DIR"] = "/tmp/output/criteo/"
            os.system("mkdir -p /tmp/input/criteo")
            os.system("mkdir -p /tmp/output/criteo")

            from merlin.datasets.synthetic import generate_data

            train, valid = generate_data("criteo", int(100000), set_sizes=(0.7, 0.3))

            train.to_ddf().compute().to_parquet('/tmp/input/criteo/day_0.parquet')
            valid.to_ddf().compute().to_parquet('/tmp/input/criteo/day_1.parquet')
            """
        )


      tb1.execute()


tests/unit/examples/test_scaling_criteo_merlin_models.py:33:

../../../.local/lib/python3.8/site-packages/testbook/client.py:147: in execute

super().execute_cell(cell, index)

../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped

return just_run(coro(*args, **kwargs))

../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run

return loop.run_until_complete(coro)

/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete

return future.result()

../../../.local/lib/python3.8/site-packages/nbclient/client.py:965: in async_execute_cell

await self._check_raise_for_error(cell, cell_index, exec_reply)

self = <testbook.client.TestbookNotebookClient object at 0x7fe1060766a0>

cell = {'cell_type': 'code', 'execution_count': 7, 'metadata': {'jupyter': {'outputs_hidden': False}, 'execution': {'iopub.st...pool_size=(device_pool_size // 256) * 256\n    )\n\n# Create the distributed client\nclient = Client(cluster)\nclient'}

cell_index = 12

exec_reply = {'buffers': [], 'content': {'ename': 'MemoryError', 'engine_info': {'engine_id': -1, 'engine_uuid': '03b34da0-456d-458...e, 'engine': '03b34da0-456d-4587-a5e2-d18a9f9cc72d', 'started': '2022-11-01T16:08:36.748150Z', 'status': 'error'}, ...}
async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:


      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)


E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E           # Dask dashboard

E           dashboard_port = "8787"

E

E           # Deploy a Single-Machine Multi-GPU Cluster

E           protocol = "tcp"  # "tcp" or "ucx"

E           if numba.cuda.is_available():

E               NUM_GPUS = list(range(len(numba.cuda.gpus)))

E           else:

E               NUM_GPUS = []

E           visible_devices = ",".join([str(n) for n in NUM_GPUS])  # Select devices to place workers

E           device_limit_frac = 0.7  # Spill GPU-Worker memory to host at this limit.

E           device_pool_frac = 0.8

E           part_mem_frac = 0.15

E

E           # Use total device size to calculate args.device_limit_frac

E           device_size = device_mem_size(kind="total")

E           device_limit = int(device_limit_frac * device_size)

E           device_pool_size = int(device_pool_frac * device_size)

E           part_size = int(part_mem_frac * device_size)

E

E           # Check if any device memory is already occupied

E           for dev in visible_devices.split(","):

E               fmem = pynvml_mem_size(kind="free", index=int(dev))

E               used = (device_size - fmem) / 1e9

E               if used > 1.0:

E                   warnings.warn(f"BEWARE - {used} GB is already occupied on device {int(dev)}!")

E

E           cluster = None  # (Optional) Specify existing scheduler port

E           if cluster is None:

E               cluster = LocalCUDACluster(

E                   protocol=protocol,

E                   n_workers=len(visible_devices.split(",")),

E                   CUDA_VISIBLE_DEVICES=visible_devices,

E                   device_memory_limit=device_limit,

E                   local_directory=dask_workdir,

E                   dashboard_address=":" + dashboard_port,

E                   rmm_pool_size=(device_pool_size // 256) * 256

E               )

E

E           # Create the distributed client

E           client = Client(cluster)

E           client

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mMemoryError�[0m                               Traceback (most recent call last)

E           Cell �[0;32mIn [7], line 30�[0m

E           �[1;32m     28�[0m cluster �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m  �[38;5;66;03m# (Optional) Specify existing scheduler port�[39;00m

E           �[1;32m     29�[0m �[38;5;28;01mif�[39;00m cluster �[38;5;129;01mis�[39;00m �[38;5;28;01mNone�[39;00m:

E           �[0;32m---> 30�[0m     cluster �[38;5;241m=�[39m �[43mLocalCUDACluster�[49m�[43m(�[49m

E           �[1;32m     31�[0m �[43m        �[49m�[43mprotocol�[49m�[38;5;241;43m=�[39;49m�[43mprotocol�[49m�[43m,�[49m

E           �[1;32m     32�[0m �[43m        �[49m�[43mn_workers�[49m�[38;5;241;43m=�[39;49m�[38;5;28;43mlen�[39;49m�[43m(�[49m�[43mvisible_devices�[49m�[38;5;241;43m.�[39;49m�[43msplit�[49m�[43m(�[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m,�[39;49m�[38;5;124;43m"�[39;49m�[43m)�[49m�[43m)�[49m�[43m,�[49m

E           �[1;32m     33�[0m �[43m        �[49m�[43mCUDA_VISIBLE_DEVICES�[49m�[38;5;241;43m=�[39;49m�[43mvisible_devices�[49m�[43m,�[49m

E           �[1;32m     34�[0m �[43m        �[49m�[43mdevice_memory_limit�[49m�[38;5;241;43m=�[39;49m�[43mdevice_limit�[49m�[43m,�[49m

E           �[1;32m     35�[0m �[43m        �[49m�[43mlocal_directory�[49m�[38;5;241;43m=�[39;49m�[43mdask_workdir�[49m�[43m,�[49m

E           �[1;32m     36�[0m �[43m        �[49m�[43mdashboard_address�[49m�[38;5;241;43m=�[39;49m�[38;5;124;43m"�[39;49m�[38;5;124;43m:�[39;49m�[38;5;124;43m"�[39;49m�[43m �[49m�[38;5;241;43m+�[39;49m�[43m �[49m�[43mdashboard_port�[49m�[43m,�[49m

E           �[1;32m     37�[0m �[43m        �[49m�[43mrmm_pool_size�[49m�[38;5;241;43m=�[39;49m�[43m(�[49m�[43mdevice_pool_size�[49m�[43m �[49m�[38;5;241;43m/�[39;49m�[38;5;241;43m/�[39;49m�[43m �[49m�[38;5;241;43m256�[39;49m�[43m)�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[43m �[49m�[38;5;241;43m256�[39;49m

E           �[1;32m     38�[0m �[43m    �[49m�[43m)�[49m

E           �[1;32m     40�[0m �[38;5;66;03m# Create the distributed client�[39;00m

E           �[1;32m     41�[0m client �[38;5;241m=�[39m Client(cluster)

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/dask_cuda/local_cuda_cluster.py:367�[0m, in �[0;36mLocalCUDACluster.__init__�[0;34m(self, CUDA_VISIBLE_DEVICES, n_workers, threads_per_worker, memory_limit, device_memory_limit, data, local_directory, shared_filesystem, protocol, enable_tcp_over_ucx, enable_infiniband, enable_nvlink, enable_rdmacm, rmm_pool_size, rmm_maximum_pool_size, rmm_managed_memory, rmm_async, rmm_log_directory, rmm_track_allocations, jit_unspill, log_spilling, worker_class, pre_import, **kwargs)�[0m

E           �[1;32m    365�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mcuda_visible_devices �[38;5;241m=�[39m CUDA_VISIBLE_DEVICES

E           �[1;32m    366�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mscale(n_workers)

E           �[0;32m--> 367�[0m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43msync�[49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_correct_state�[49m�[43m)�[49m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:309�[0m, in �[0;36mSyncMethodMixin.sync�[0;34m(self, func, asynchronous, callback_timeout, args, **kwargs)�[0m

E           �[1;32m    307�[0m     �[38;5;28;01mreturn�[39;00m future

E           �[1;32m    308�[0m �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 309�[0m     �[38;5;28;01mreturn�[39;00m �[43msync�[49m�[43m(�[49m

E           �[1;32m    310�[0m �[43m        �[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mloop�[49m�[43m,�[49m�[43m �[49m�[43mfunc�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[43margs�[49m�[43m,�[49m�[43m �[49m�[43mcallback_timeout�[49m�[38;5;241;43m=�[39;49m�[43mcallback_timeout�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m*�[39;49m�[43mkwargs�[49m

E           �[1;32m    311�[0m �[43m    �[49m�[43m)�[49m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:376�[0m, in �[0;36msync�[0;34m(loop, func, callback_timeout, *args, **kwargs)�[0m

E           �[1;32m    374�[0m �[38;5;28;01mif�[39;00m error:

E           �[1;32m    375�[0m     typ, exc, tb �[38;5;241m=�[39m error

E           �[0;32m--> 376�[0m     �[38;5;28;01mraise�[39;00m exc�[38;5;241m.�[39mwith_traceback(tb)

E           �[1;32m    377�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    378�[0m     �[38;5;28;01mreturn�[39;00m result

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:349�[0m, in �[0;36msync..f�[0;34m()�[0m

E           �[1;32m    347�[0m         future �[38;5;241m=�[39m asyncio�[38;5;241m.�[39mwait_for(future, callback_timeout)

E           �[1;32m    348�[0m     future �[38;5;241m=�[39m asyncio�[38;5;241m.�[39mensure_future(future)

E           �[0;32m--> 349�[0m     result �[38;5;241m=�[39m �[38;5;28;01myield�[39;00m future

E           �[1;32m    350�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    351�[0m     error �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762�[0m, in �[0;36mRunner.run�[0;34m(self)�[0m

E           �[1;32m    759�[0m exc_info �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m

E           �[1;32m    761�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 762�[0m     value �[38;5;241m=�[39m �[43mfuture�[49m�[38;5;241;43m.�[39;49m�[43mresult�[49m�[43m(�[49m�[43m)�[49m

E           �[1;32m    763�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    764�[0m     exc_info �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/deploy/spec.py:352�[0m, in �[0;36mSpecCluster._correct_state_internal�[0;34m(self)�[0m

E           �[1;32m    350�[0m     �[38;5;28;01mfor�[39;00m w �[38;5;129;01min�[39;00m workers:

E           �[1;32m    351�[0m         w�[38;5;241m.�[39m_cluster �[38;5;241m=�[39m weakref�[38;5;241m.�[39mref(�[38;5;28mself�[39m)

E           �[0;32m--> 352�[0m         �[38;5;28;01mawait�[39;00m w  �[38;5;66;03m# for tornado gen.coroutine support�[39;00m

E           �[1;32m    353�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mworkers�[38;5;241m.�[39mupdate(�[38;5;28mdict�[39m(�[38;5;28mzip�[39m(to_open, workers)))

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/core.py:299�[0m, in �[0;36mServer.await.._�[0;34m()�[0m

E           �[1;32m    293�[0m             �[38;5;28;01mraise�[39;00m �[38;5;167;01mTimeoutError�[39;00m(

E           �[1;32m    294�[0m                 �[38;5;124m"�[39m�[38;5;132;01m{}�[39;00m�[38;5;124m failed to start in �[39m�[38;5;132;01m{}�[39;00m�[38;5;124m seconds�[39m�[38;5;124m"�[39m�[38;5;241m.�[39mformat(

E           �[1;32m    295�[0m                     �[38;5;28mtype�[39m(�[38;5;28mself�[39m)�[38;5;241m.�[39m�[38;5;18m__name__�[39m, timeout

E           �[1;32m    296�[0m                 )

E           �[1;32m    297�[0m             )

E           �[1;32m    298�[0m     �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 299�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstart()

E           �[1;32m    300�[0m         �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mrunning

E           �[1;32m    301�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:347�[0m, in �[0;36mNanny.start�[0;34m(self)�[0m

E           �[1;32m    344�[0m     �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mplugin_add(plugin�[38;5;241m=�[39mplugin, name�[38;5;241m=�[39mname)

E           �[1;32m    346�[0m logger�[38;5;241m.�[39minfo(�[38;5;124m"�[39m�[38;5;124m        Start Nanny at: �[39m�[38;5;132;01m%r�[39;00m�[38;5;124m"�[39m, �[38;5;28mself�[39m�[38;5;241m.�[39maddress)

E           �[0;32m--> 347�[0m response �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39minstantiate()

E           �[1;32m    348�[0m �[38;5;28;01mif�[39;00m response �[38;5;241m==�[39m Status�[38;5;241m.�[39mrunning:

E           �[1;32m    349�[0m     �[38;5;28;01massert�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mworker_address

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:430�[0m, in �[0;36mNanny.instantiate�[0;34m(self)�[0m

E           �[1;32m    428�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    429�[0m     �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 430�[0m         result �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mprocess�[38;5;241m.�[39mstart()

E           �[1;32m    431�[0m     �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    432�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mclose()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:685�[0m, in �[0;36mWorkerProcess.start�[0;34m(self)�[0m

E           �[1;32m    683�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstatus

E           �[1;32m    684�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 685�[0m     msg �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_wait_until_connected(uid)

E           �[1;32m    686�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:

E           �[1;32m    687�[0m     �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mfailed

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:803�[0m, in �[0;36mWorkerProcess._wait_until_connected�[0;34m(self, uid)�[0m

E           �[1;32m    799�[0m �[38;5;28;01mif�[39;00m �[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m �[38;5;129;01min�[39;00m msg:

E           �[1;32m    800�[0m     logger�[38;5;241m.�[39merror(

E           �[1;32m    801�[0m         �[38;5;124m"�[39m�[38;5;124mFailed while trying to start worker process: �[39m�[38;5;132;01m%s�[39;00m�[38;5;124m"�[39m, msg[�[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m]

E           �[1;32m    802�[0m     )

E           �[0;32m--> 803�[0m     �[38;5;28;01mraise�[39;00m msg[�[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m]

E           �[1;32m    804�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    805�[0m     �[38;5;28;01mreturn�[39;00m msg

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:869�[0m, in �[0;36mrun�[0;34m()�[0m

E           �[1;32m    865�[0m �[38;5;124;03m"""�[39;00m

E           �[1;32m    866�[0m �[38;5;124;03mTry to start worker and inform parent of outcome.�[39;00m

E           �[1;32m    867�[0m �[38;5;124;03m"""�[39;00m

E           �[1;32m    868�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 869�[0m     �[38;5;28;01mawait�[39;00m worker

E           �[1;32m    870�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m �[38;5;28;01mas�[39;00m e:

E           �[1;32m    871�[0m     logger�[38;5;241m.�[39mexception(�[38;5;124m"�[39m�[38;5;124mFailed to start worker�[39m�[38;5;124m"�[39m)

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/core.py:299�[0m, in �[0;36m_�[0;34m()�[0m

E           �[1;32m    293�[0m             �[38;5;28;01mraise�[39;00m �[38;5;167;01mTimeoutError�[39;00m(

E           �[1;32m    294�[0m                 �[38;5;124m"�[39m�[38;5;132;01m{}�[39;00m�[38;5;124m failed to start in �[39m�[38;5;132;01m{}�[39;00m�[38;5;124m seconds�[39m�[38;5;124m"�[39m�[38;5;241m.�[39mformat(

E           �[1;32m    295�[0m                     �[38;5;28mtype�[39m(�[38;5;28mself�[39m)�[38;5;241m.�[39m�[38;5;18m__name__�[39m, timeout

E           �[1;32m    296�[0m                 )

E           �[1;32m    297�[0m             )

E           �[1;32m    298�[0m     �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 299�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstart()

E           �[1;32m    300�[0m         �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mrunning

E           �[1;32m    301�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/worker.py:1372�[0m, in �[0;36mstart�[0;34m()�[0m

E           �[1;32m   1370�[0m         �[38;5;28;01mfor�[39;00m exc �[38;5;129;01min�[39;00m plugins_exceptions:

E           �[1;32m   1371�[0m             logger�[38;5;241m.�[39merror(�[38;5;28mrepr�[39m(exc))

E           �[0;32m-> 1372�[0m     �[38;5;28;01mraise�[39;00m plugins_exceptions[�[38;5;241m0�[39m]

E           �[1;32m   1374�[0m �[38;5;28mself�[39m�[38;5;241m.�[39m_pending_plugins �[38;5;241m=�[39m ()

E           �[1;32m   1376�[0m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_register_with_scheduler()

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/worker.py:3248�[0m, in �[0;36mplugin_add�[0;34m()�[0m

E           �[1;32m   3246�[0m �[38;5;28;01mif�[39;00m �[38;5;28mhasattr�[39m(plugin, �[38;5;124m"�[39m�[38;5;124msetup�[39m�[38;5;124m"�[39m):

E           �[1;32m   3247�[0m     �[38;5;28;01mtry�[39;00m:

E           �[0;32m-> 3248�[0m         result �[38;5;241m=�[39m plugin�[38;5;241m.�[39msetup(worker�[38;5;241m=�[39m�[38;5;28mself�[39m)

E           �[1;32m   3249�[0m         �[38;5;28;01mif�[39;00m isawaitable(result):

E           �[1;32m   3250�[0m             result �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m result

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py:78�[0m, in �[0;36msetup�[0;34m()�[0m

E           �[1;32m     74�[0m     �[38;5;28;01mimport�[39;00m �[38;5;21;01mrmm�[39;00m

E           �[1;32m     76�[0m     pool_allocator �[38;5;241m=�[39m �[38;5;28;01mFalse�[39;00m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39minitial_pool_size �[38;5;129;01mis�[39;00m �[38;5;28;01mNone�[39;00m �[38;5;28;01melse�[39;00m �[38;5;28;01mTrue�[39;00m

E           �[0;32m---> 78�[0m     rmm�[38;5;241m.�[39mreinitialize(

E           �[1;32m     79�[0m         pool_allocator�[38;5;241m=�[39mpool_allocator,

E           �[1;32m     80�[0m         managed_memory�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mmanaged_memory,

E           �[1;32m     81�[0m         initial_pool_size�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39minitial_pool_size,

E           �[1;32m     82�[0m         maximum_pool_size�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mmaximum_pool_size,

E           �[1;32m     83�[0m         logging�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mlogging,

E           �[1;32m     84�[0m         log_file_name�[38;5;241m=�[39mget_rmm_log_file_name(

E           �[1;32m     85�[0m             worker, �[38;5;28mself�[39m�[38;5;241m.�[39mlogging, �[38;5;28mself�[39m�[38;5;241m.�[39mlog_directory

E           �[1;32m     86�[0m         ),

E           �[1;32m     87�[0m     )

E           �[1;32m     88�[0m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mrmm_track_allocations:

E           �[1;32m     89�[0m     �[38;5;28;01mimport�[39;00m �[38;5;21;01mrmm�[39;00m

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/rmm/rmm.py:85�[0m, in �[0;36mreinitialize�[0;34m()�[0m

E           �[1;32m     32�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mreinitialize�[39m(

E           �[1;32m     33�[0m     pool_allocator�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m,

E           �[1;32m     34�[0m     managed_memory�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m,

E           �[0;32m   (...)�[0m

E           �[1;32m     39�[0m     log_file_name�[38;5;241m=�[39m�[38;5;28;01mNone�[39;00m,

E           �[1;32m     40�[0m ):

E           �[1;32m     41�[0m     �[38;5;124;03m"""�[39;00m

E           �[1;32m     42�[0m �[38;5;124;03m    Finalizes and then initializes RMM using the options passed. Using memory�[39;00m

E           �[1;32m     43�[0m �[38;5;124;03m    from a previous initialization of RMM is undefined behavior and should be�[39;00m

E           �[0;32m   (...)�[0m

E           �[1;32m     83�[0m �[38;5;124;03m    corresponding to each device.�[39;00m

E           �[1;32m     84�[0m �[38;5;124;03m    """�[39;00m

E           �[0;32m---> 85�[0m     rmm�[38;5;241m.�[39mmr�[38;5;241m.�[39m_initialize(

E           �[1;32m     86�[0m         pool_allocator�[38;5;241m=�[39mpool_allocator,

E           �[1;32m     87�[0m         managed_memory�[38;5;241m=�[39mmanaged_memory,

E           �[1;32m     88�[0m         initial_pool_size�[38;5;241m=�[39minitial_pool_size,

E           �[1;32m     89�[0m         maximum_pool_size�[38;5;241m=�[39mmaximum_pool_size,

E           �[1;32m     90�[0m         devices�[38;5;241m=�[39mdevices,

E           �[1;32m     91�[0m         logging�[38;5;241m=�[39mlogging,

E           �[1;32m     92�[0m         log_file_name�[38;5;241m=�[39mlog_file_name,

E           �[1;32m     93�[0m     )

E

E           File �[0;32mmemory_resource.pyx:823�[0m, in �[0;36mrmm._lib.memory_resource._initialize�[0;34m()�[0m

E

E           File �[0;32mmemory_resource.pyx:883�[0m, in �[0;36mrmm._lib.memory_resource._initialize�[0;34m()�[0m

E

E           File �[0;32mmemory_resource.pyx:342�[0m, in �[0;36mrmm._lib.memory_resource.PoolMemoryResource.__cinit__�[0;34m()�[0m

E

E           �[0;31mMemoryError�[0m: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

E           MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError

----------------------------- Captured stderr call -----------------------------

2022-11-01 16:08:39,522 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-01 16:08:39,527 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-01 16:08:39,533 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-01 16:08:39,554 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-01 16:08:40,098 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-01 16:08:40,099 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-01 16:08:42,451 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-x38zsf6h', purging

2022-11-01 16:08:42,452 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-01 16:08:42,706 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-01 16:08:42,707 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-01 16:08:45,328 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-95wvmosr', purging

2022-11-01 16:08:45,329 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-01 16:08:45,424 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-01 16:08:45,426 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-01 16:08:45,513 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-01 16:08:45,662 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-01 16:08:45,663 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-01 16:08:45,866 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-01 16:08:45,867 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-01 16:08:45,876 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-01 16:08:45,877 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-01 16:08:45,998 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-01 16:08:45,999 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-01 16:08:48,421 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-44glbsaw', purging

2022-11-01 16:08:48,422 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-d0j6w9zl', purging

2022-11-01 16:08:48,422 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-ai08zbgy', purging

2022-11-01 16:08:48,422 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/output/criteo/test_dask/workdir/dask-worker-space/worker-az9xv679', purging

2022-11-01 16:08:48,423 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-01 16:08:48,675 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-11-01 16:08:48,676 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 57 leaked semaphore objects to clean up at shutdown

warnings.warn('resource_tracker: There appear to be %d '
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       355    217    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                      100     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     17    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        54     30    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       120     21    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     43    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    173    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    20      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      4    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     27    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     97    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      4    81%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          346    182    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          603    345    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     29    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     81    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     20    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      7    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    110    71%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49     15    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     75    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                114     55    52%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     169     90    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          245    130    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 244    116    52%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          421    170    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              170    117    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                104     69    34%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              288     91    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                60     36    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39     16    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          452    209    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    243    131    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               707    341    52%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              122     89    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               97     62    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    124    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     29    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       422    334    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         107     54    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    142    32%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     27    29%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31     10    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     33    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     43    52%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     44    77%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       209     55    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%
TOTAL                                                                                                 12751   6383    50%
==================== 2 failed, 1 passed in 64.23s (0:01:04) ====================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://github.com/gitapi/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins4516776451167368875.sh

nvidia-merlin-bot · 2022-11-02T14:44:06Z

Click to view CI Results

GitHub pull request #713 of commit daa80eb7aba14d8d4a024e604de9db9847d071cf, no merge conflicts.
Running as SYSTEM
Setting status of daa80eb7aba14d8d4a024e604de9db9847d071cf to PENDING with url https://10.20.13.93:8080/job/merlin_merlin/550/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/713/*:refs/remotes/origin/pr/713/* # timeout=10
 > git rev-parse daa80eb7aba14d8d4a024e604de9db9847d071cf^{commit} # timeout=10
Checking out Revision daa80eb7aba14d8d4a024e604de9db9847d071cf (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f daa80eb7aba14d8d4a024e604de9db9847d071cf # timeout=10
Commit message: "Merge branch 'main' into merlin-tf-int-test-bandaid"
 > git rev-list --no-walk a0a33fab989b46852696c3d8949bd9ac2d5a926c # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins15370793776933496935.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.0,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.0,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.8.2,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.42,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.0,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1840590713'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-2tna9pxr
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-2tna9pxr
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit d2737da6cc7e98ec8723731a7cc998aa143e6758
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+10.gd2737da) (0.3.0+12.g78ecddd)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+10.gd2737da) (2.22.0)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+10.gd2737da) (1.1.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (3.19.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.10.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.3.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.55.1)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (7.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (4.64.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.5.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.7.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (8.1.3)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 103.7 MB/s eta 0:00:00
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (65.4.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+10.gd2737da-py3-none-any.whl size=91051 sha256=1b013335209ed6900a2a54d70c8ec997a462d3be68646579f13e5b954fcf1188
  Stored in directory: /tmp/pip-ephem-wheel-cache-u1bqeucj/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+10.gd2737da
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/models.git

Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-kr__9vrk

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-kr__9vrk

Resolved https://github.com/NVIDIA-Merlin/models.git to commit 5e1a805e357d2e77d22f7e0dab668cd1bc9cfb0c

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+23.g5e1a805e) (0.3.0+12.g78ecddd)

Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (2022.3.0)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (3.19.5)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.2.5)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.10.0)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.3.5)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (21.3)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (0.55.1)

Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (2022.3.0)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (7.0.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (4.64.1)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (0.4.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (5.4.1)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (0.12.0)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.2.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (2.2.0)

Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (2022.5.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (3.1.2)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (6.1)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.0.4)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (5.8.0)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (2.4.0)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (2.0.0)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.7.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (8.1.3)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (0.38.1)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (65.4.1)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.20.3)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (2022.2.1)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.0.1)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (6.0.2)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (4.1.0)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (4.0.0)

Building wheels for collected packages: merlin-models

Building wheel for merlin-models (pyproject.toml): started

Building wheel for merlin-models (pyproject.toml): finished with status 'done'

Created wheel for merlin-models: filename=merlin_models-0.9.0+23.g5e1a805e-py3-none-any.whl size=356077 sha256=5a5c945074ee2b97901c2291e91c75ca152a2bda671181b77279a4aab75acffb

Stored in directory: /tmp/pip-ephem-wheel-cache-99pr7kjy/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5

Successfully built merlin-models

Installing collected packages: merlin-models

Attempting uninstall: merlin-models

Found existing installation: merlin-models 0.7.0+11.g280956aa4

Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu

Can't uninstall 'merlin-models'. No files were found to uninstall.

Successfully installed merlin-models-0.9.0+23.g5e1a805e
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git

Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-7_j1txz4

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-7_j1txz4

Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit 59579f2c46006fcb22795623ee9400c658166670

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+2.g59579f2c) (0.3.0+12.g78ecddd)

Collecting merlin-dataloader>=0.0.2

Downloading merlin-dataloader-0.0.2.tar.gz (44 kB)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.1/44.1 kB 1.6 MB/s eta 0:00:00

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+2.g59579f2c) (1.8.1)

Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (2022.3.0)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (3.19.5)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.2.5)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.10.0)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.3.5)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (21.3)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (0.55.1)

Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (2022.3.0)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (7.0.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (4.64.1)

Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+2.g59579f2c) (1.20.3)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (0.4.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (5.4.1)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (0.12.0)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.2.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (2.2.0)

Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (2022.5.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (3.1.2)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (6.1)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.0.4)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (5.8.0)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (2.4.0)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (2.0.0)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.7.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (8.1.3)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (0.38.1)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (65.4.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (2022.2.1)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.0.1)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (6.0.2)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (4.1.0)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (4.0.0)

Building wheels for collected packages: nvtabular, merlin-dataloader

Building wheel for nvtabular (pyproject.toml): started

Building wheel for nvtabular (pyproject.toml): finished with status 'done'

Created wheel for nvtabular: filename=nvtabular-1.6.0+2.g59579f2c-cp38-cp38-linux_x86_64.whl size=257750 sha256=e867f960781c3dab6266ac96721dc7e02cb0f61dfe7e52479e06c452569192db

Stored in directory: /tmp/pip-ephem-wheel-cache-0stf87tr/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc

Building wheel for merlin-dataloader (pyproject.toml): started

Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'

Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.2-py3-none-any.whl size=29205 sha256=4cc4ac4c479143a3c32aa285f2e07c461e4b83ebb447163952cc7ab9b61c431b

Stored in directory: /tmp/pip-ephem-wheel-cache-0stf87tr/wheels/d5/ce/8c/31476c01e0b5a2278110fe2092bdd911efb0e5b83d0d3550ca

Successfully built nvtabular merlin-dataloader

Installing collected packages: merlin-dataloader, nvtabular

Attempting uninstall: nvtabular

Found existing installation: nvtabular 1.1.1

Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu

Can't uninstall 'nvtabular'. No files were found to uninstall.

Successfully installed merlin-dataloader-0.0.2 nvtabular-1.6.0+2.g59579f2c
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/core.git

Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-_y2qa31f

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-y2qa31f

Resolved https://github.com/NVIDIA-Merlin/core.git to commit 563be4bf5ef675940d5fff2b5e4666424a7f7947

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (1.2.5)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (3.19.5)

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (2022.5.0)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (1.10.0)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (1.3.5)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (0.55.1)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (21.3)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (2022.3.0)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (7.0.0)

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (2022.3.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (4.64.1)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (0.4.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (1.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (5.4.1)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (0.12.0)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.2.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.2.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (3.1.2)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (6.1)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.0.4)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (5.8.0)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.4.0)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.0.0)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.7.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (8.1.3)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+5.g563be4b) (0.38.1)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+5.g563be4b) (65.4.1)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+5.g563be4b) (1.20.3)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+5.g563be4b) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+5.g563be4b) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+5.g563be4b) (2022.2.1)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+5.g563be4b) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+5.g563be4b) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+5.g563be4b) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.0.1)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (6.0.2)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (4.1.0)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (4.0.0)

Building wheels for collected packages: merlin-core

Building wheel for merlin-core (pyproject.toml): started

Building wheel for merlin-core (pyproject.toml): finished with status 'done'

Created wheel for merlin-core: filename=merlin_core-0.8.0+5.g563be4b-py3-none-any.whl size=118363 sha256=2af9c7cb228c6b312621d27a941bbbc45d8826b736196903348425edd2eaa294

Stored in directory: /tmp/pip-ephem-wheel-cache-lx41i_f/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84

Successfully built merlin-core

Installing collected packages: merlin-core

Attempting uninstall: merlin-core

Found existing installation: merlin-core 0.3.0+12.g78ecddd

Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu

Can't uninstall 'merlin-core'. No files were found to uninstall.

Successfully installed merlin-core-0.8.0+5.g563be4b
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin

plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0

collected 3 items
tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py F      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py F               [100%]
=================================== FAILURES ===================================

__________________________________ test_func ___________________________________
self = <testbook.client.TestbookNotebookClient object at 0x7f438639c0a0>

msg_id = '2c2150e2-39752790189e7de36e106d66_18229_32'

cell = {'id': '3e3d1425', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-02T14:41:36.485410Z',...b[38;5;241m=\x1b[39m \x1b[38;5;28mself\x1b[39m\x1b[38;5;241m.\x1b[39mpid\n', '\x1b[0;31mKeyboardInterrupt\x1b[0m: ']}]}

timeout = 120

task_poll_output_msg = <Task finished name='Task-379' coro=<NotebookClient._async_poll_output_msg() done, defined at /var/jenkins_home/.local/lib/python3.8/site-packages/nbclient/client.py:756> result=None>

task_poll_kernel_alive = <Task cancelled name='Task-378' coro=<NotebookClient._async_poll_kernel_alive() done, defined at /var/jenkins_home/.local/lib/python3.8/site-packages/nbclient/client.py:770>>
async def _async_poll_for_reply(
    self,
    msg_id: str,
    cell: NotebookNode,
    timeout: t.Optional[int],
    task_poll_output_msg: asyncio.Future,
    task_poll_kernel_alive: asyncio.Future,
) -> t.Dict:

    assert self.kc is not None
    new_timeout: t.Optional[float] = None
    if timeout is not None:
        deadline = monotonic() + timeout
        new_timeout = float(timeout)
    while True:
        try:


          msg = await ensure_async(self.kc.shell_channel.get_msg(timeout=new_timeout))


../../../.local/lib/python3.8/site-packages/nbclient/client.py:730:

obj = <coroutine object ZMQSocketChannel.get_msg at 0x7f43850718c0>
async def ensure_async(obj: Union[Awaitable, Any]) -> Any:
    """Convert a non-awaitable object to a coroutine if needed,
    and await it if it was not already awaited.
    """
    if inspect.isawaitable(obj):
        try:


          result = await obj


../../../.local/lib/python3.8/site-packages/nbclient/util.py:96:

self = <jupyter_client.channels.ZMQSocketChannel object at 0x7f44625f4160>

timeout = 120000.0
async def get_msg(self, timeout: t.Optional[float] = None) -> t.Dict[str, t.Any]:
    """Gets a message if there is one that is ready."""
    assert self.socket is not None
    if timeout is not None:
        timeout *= 1000  # seconds to ms
    ready = await self.socket.poll(timeout)

    if ready:
        res = await self._recv()
        return res
    else:


      raise Empty


E           _queue.Empty
/usr/local/lib/python3.8/dist-packages/jupyter_client/channels.py:230: Empty
During handling of the above exception, another exception occurred:
def test_func():
    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "01-Building-Recommender-Systems-with-Merlin.ipynb",
        execute=False,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["NUM_ROWS"] = "100000"
            os.system("mkdir -p /tmp/examples")
            os.environ["BASE_DIR"] = "/tmp/examples/"
            """
        )
        tb1.execute()
        assert os.path.isdir("/tmp/examples/dlrm")
        assert os.path.isdir("/tmp/examples/feature_repo")
        assert os.path.isdir("/tmp/examples/query_tower")
        assert os.path.isfile("/tmp/examples/item_embeddings.parquet")
        assert os.path.isfile("/tmp/examples/feature_repo/user_features.py")
        assert os.path.isfile("/tmp/examples/feature_repo/item_features.py")

    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "02-Deploying-multi-stage-RecSys-with-Merlin-Systems.ipynb",
        execute=False,
        timeout=120,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["BASE_DIR"] = "/tmp/examples/"
            os.environ["topk_retrieval"] = "20"
            """
        )
        NUM_OF_CELLS = len(tb2.cells)
        tb2.execute_cell(list(range(0, NUM_OF_CELLS - 3)))
        top_k = tb2.ref("top_k")
        outputs = tb2.ref("outputs")
        assert outputs[0] == "ordered_ids"


      tb2.inject(


            """
            import shutil
            from merlin.core.dispatch import get_lib
            from merlin.models.loader.tf_utils import configure_tensorflow
            configure_tensorflow()
            df_lib = get_lib()
            batch = df_lib.read_parquet(
                os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),
                num_rows=1,
                columns=["user_id_raw"],
            )
            from merlin.systems.triton.utils import run_ensemble_on_tritonserver
            response = run_ensemble_on_tritonserver(
                "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"
            )
            response = [x.tolist()[0] for x in response["ordered_ids"]]
            shutil.rmtree("/tmp/examples/", ignore_errors=True)
            """
        )

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:61:

../../../.local/lib/python3.8/site-packages/testbook/client.py:237: in inject

cell = TestbookNode(self.execute_cell(inject_idx)) if run else TestbookNode(code_cell)

../../../.local/lib/python3.8/site-packages/testbook/client.py:133: in execute_cell

cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)

../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped

return just_run(coro(*args, **kwargs))

../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run

return loop.run_until_complete(coro)

/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete

return future.result()

../../../.local/lib/python3.8/site-packages/nbclient/client.py:949: in async_execute_cell

exec_reply = await self.task_poll_for_reply

../../../.local/lib/python3.8/site-packages/nbclient/client.py:754: in _async_poll_for_reply

await self._async_handle_timeout(timeout, cell)

self = <testbook.client.TestbookNotebookClient object at 0x7f438639c0a0>

timeout = 120

cell = {'id': '3e3d1425', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-02T14:41:36.485410Z',...b[38;5;241m=\x1b[39m \x1b[38;5;28mself\x1b[39m\x1b[38;5;241m.\x1b[39mpid\n', '\x1b[0;31mKeyboardInterrupt\x1b[0m: ']}]}
async def _async_handle_timeout(
    self, timeout: int, cell: t.Optional[NotebookNode] = None
) -> None:

    self.log.error("Timeout waiting for execute reply (%is)." % timeout)
    if self.interrupt_on_timeout:
        self.log.error("Interrupting kernel")
        assert self.km is not None
        await ensure_async(self.km.interrupt_kernel())
    else:


      raise CellTimeoutError.error_from_timeout_and_cell(


            "Cell execution timed out", timeout, cell
        )

E           nbclient.exceptions.CellTimeoutError: A cell timed out while it was being executed, after 120 seconds.

E           The message was: Cell execution timed out.

E           Here is a preview of the cell contents:

E           -------------------

E           ['import shutil', 'from merlin.core.dispatch import get_lib', 'from merlin.models.loader.tf_utils import configure_tensorflow', 'configure_tensorflow()', 'df_lib = get_lib()']

E           ...

E           ['response = run_ensemble_on_tritonserver(', '    "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"', ')', 'response = [x.tolist()[0] for x in response["ordered_ids"]]', 'shutil.rmtree("/tmp/examples/", ignore_errors=True)']

E           -------------------
../../../.local/lib/python3.8/site-packages/nbclient/client.py:801: CellTimeoutError

----------------------------- Captured stdout call -----------------------------

Signal (2) received.

Signal (2) received.

Signal (15) received.

Signal (15) received.

Signal (15) received.

Signal (15) received.

Signal (15) received.

Signal (15) received.

Signal (15) received.

----------------------------- Captured stderr call -----------------------------

2022-11-02 14:39:41.746018: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-02 14:39:45.828947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-02 14:39:45.829782: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 13923 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-11-02 14:39:45.830392: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14247 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-11-02 14:39:45.830983: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 13923 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

WARNING clustering 462 points to 32 centroids: please provide at least 1248 training points

2022-11-02 14:41:29.904078: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-02 14:41:33.979635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-02 14:41:33.980518: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 13923 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-11-02 14:41:33.981142: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14247 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-11-02 14:41:33.981743: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 13923 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

I1102 14:41:36.814628 18786 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fe8e6000000' with size 268435456

I1102 14:41:36.815372 18786 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1102 14:41:36.823701 18786 model_lifecycle.cc:459] loading: 2_queryfaiss:1

I1102 14:41:36.823764 18786 model_lifecycle.cc:459] loading: 1_predicttensorflowtriton:1

I1102 14:41:36.823830 18786 model_lifecycle.cc:459] loading: 0_queryfeast:1

I1102 14:41:36.823864 18786 model_lifecycle.cc:459] loading: 3_queryfeast:1

I1102 14:41:36.823905 18786 model_lifecycle.cc:459] loading: 4_unrollfeatures:1

I1102 14:41:36.823959 18786 model_lifecycle.cc:459] loading: 5_predicttensorflowtriton:1

I1102 14:41:36.824062 18786 model_lifecycle.cc:459] loading: 6_softmaxsampling:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1102 14:41:44.931808 18786 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow

I1102 14:41:44.931852 18786 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10

I1102 14:41:44.931860 18786 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10

I1102 14:41:44.931866 18786 tensorflow.cc:2576] backend configuration:

{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}

I1102 14:41:44.931912 18786 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 2_queryfaiss (GPU device 0)

I1102 14:41:51.604773 18786 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 1_predicttensorflowtriton (version 1)

I1102 14:41:51.605933 18786 model_lifecycle.cc:693] successfully loaded '2_queryfaiss' version 1

2022-11-02 14:41:51.606059: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-02 14:41:51.610755: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-02 14:41:51.610788: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-02 14:41:51.610891: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-02 14:41:51.651835: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 2520 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-02 14:41:51.682098: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled

2022-11-02 14:41:51.684099: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-02 14:41:51.763879: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-02 14:41:51.790011: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 183968 microseconds.

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1102 14:42:23.404361 18786 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 5_predicttensorflowtriton (version 1)

2022-11-02 14:42:23.405809: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-02 14:42:23.425524: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-02 14:42:23.425569: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-02 14:42:23.427656: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 2520 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-02 14:42:23.461075: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-02 14:42:23.592256: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-02 14:42:23.643545: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 237751 microseconds.

I1102 14:42:23.657052 18786 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 1_predicttensorflowtriton (GPU device 0)

2022-11-02 14:42:23.658264: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-02 14:42:23.669968: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-02 14:42:23.670031: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-02 14:42:23.673107: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 2520 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-02 14:42:23.682419: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-02 14:42:23.735741: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-02 14:42:23.764645: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 106394 microseconds.

I1102 14:42:23.764850 18786 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_queryfeast (GPU device 0)

I1102 14:42:23.765087 18786 model_lifecycle.cc:693] successfully loaded '1_predicttensorflowtriton' version 1

I1102 14:42:30.178275 18786 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 3_queryfeast (GPU device 0)

I1102 14:42:30.180041 18786 model_lifecycle.cc:693] successfully loaded '0_queryfeast' version 1

I1102 14:42:36.566273 18786 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 4_unrollfeatures (GPU device 0)

I1102 14:42:36.566514 18786 model_lifecycle.cc:693] successfully loaded '3_queryfeast' version 1

I1102 14:42:42.578675 18786 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 6_softmaxsampling (GPU device 0)

I1102 14:42:42.580551 18786 model_lifecycle.cc:693] successfully loaded '4_unrollfeatures' version 1

I1102 14:42:48.549536 18786 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 5_predicttensorflowtriton (GPU device 0)

I1102 14:42:48.549823 18786 model_lifecycle.cc:693] successfully loaded '6_softmaxsampling' version 1

2022-11-02 14:42:48.550087: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-02 14:42:48.558774: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-02 14:42:48.558807: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-02 14:42:48.560740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 2520 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-02 14:42:48.581922: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-02 14:42:48.719272: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-02 14:42:48.771684: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 221603 microseconds.

I1102 14:42:48.772004 18786 model_lifecycle.cc:693] successfully loaded '5_predicttensorflowtriton' version 1

I1102 14:42:48.774931 18786 model_lifecycle.cc:459] loading: ensemble_model:1

I1102 14:42:48.775491 18786 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1

I1102 14:42:48.775640 18786 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+
I1102 14:42:48.775742 18786 server.cc:588]

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend    | Path                                                            | Config                                                                                                                                                                      |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python     | /opt/tritonserver/backends/python/libtriton_python.so           | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}}               |

| tensorflow | /opt/tritonserver/backends/tensorflow2/libtriton_tensorflow2.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}} |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1102 14:42:48.775872 18786 server.cc:631]

+---------------------------+---------+--------+

| Model                     | Version | Status |

+---------------------------+---------+--------+

| 0_queryfeast              | 1       | READY  |

| 1_predicttensorflowtriton | 1       | READY  |

| 2_queryfaiss              | 1       | READY  |

| 3_queryfeast              | 1       | READY  |

| 4_unrollfeatures          | 1       | READY  |

| 5_predicttensorflowtriton | 1       | READY  |

| 6_softmaxsampling         | 1       | READY  |

| ensemble_model            | 1       | READY  |

+---------------------------+---------+--------+
I1102 14:42:48.839525 18786 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1102 14:42:48.840392 18786 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/examples/poc_ensemble                                                                                                                                                                   |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1102 14:42:48.841781 18786 grpc_server.cc:4610] Started GRPCInferenceService at localhost:8001

I1102 14:42:48.841982 18786 http_server.cc:3316] Started HTTPService at 0.0.0.0:8000

I1102 14:42:48.882937 18786 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002

W1102 14:42:49.858804 18786 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1102 14:42:50.859014 18786 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1102 14:42:51.878784 18786 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

I1102 14:43:30.351689 18786 server.cc:262] Waiting for in-flight requests to complete.

I1102 14:43:30.351722 18786 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1102 14:43:30.351958 18786 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1

I1102 14:43:30.352048 18786 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1102 14:43:30.352100 18786 server.cc:293] All models are stopped, unloading models

I1102 14:43:30.352121 18786 server.cc:300] Timeout 30: Found 7 live models and 0 in-flight non-inference requests

I1102 14:43:30.352126 18786 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state

I1102 14:43:30.352231 18786 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1102 14:43:30.352414 18786 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state

I1102 14:43:30.365179 18786 model_lifecycle.cc:578] successfully unloaded '1_predicttensorflowtriton' version 1

I1102 14:43:30.369531 18786 model_lifecycle.cc:578] successfully unloaded '5_predicttensorflowtriton' version 1

I1102 14:43:31.352316 18786 server.cc:300] Timeout 29: Found 5 live models and 0 in-flight non-inference requests

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1102 14:43:32.352496 18786 server.cc:300] Timeout 28: Found 5 live models and 0 in-flight non-inference requests

I1102 14:43:32.360963 18786 model_lifecycle.cc:578] successfully unloaded '2_queryfaiss' version 1

I1102 14:43:32.489909 18786 model_lifecycle.cc:578] successfully unloaded '4_unrollfeatures' version 1

I1102 14:43:32.515890 18786 model_lifecycle.cc:578] successfully unloaded '6_softmaxsampling' version 1

I1102 14:43:33.352639 18786 server.cc:300] Timeout 27: Found 2 live models and 0 in-flight non-inference requests

I1102 14:43:34.352778 18786 server.cc:300] Timeout 26: Found 2 live models and 0 in-flight non-inference requests

I1102 14:43:35.352915 18786 server.cc:300] Timeout 25: Found 2 live models and 0 in-flight non-inference requests

I1102 14:43:36.353041 18786 server.cc:300] Timeout 24: Found 2 live models and 0 in-flight non-inference requests

I1102 14:43:37.353599 18786 server.cc:300] Timeout 23: Found 2 live models and 0 in-flight non-inference requests

I1102 14:43:38.353827 18786 server.cc:300] Timeout 22: Found 2 live models and 0 in-flight non-inference requests

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

------------------------------ Captured log call -------------------------------

ERROR    traitlets:client.py:795 Timeout waiting for execute reply (120s).

--------------------------- Captured stderr teardown ---------------------------

I1102 14:43:39.354162 18786 server.cc:300] Timeout 21: Found 2 live models and 0 in-flight non-inference requests

I1102 14:43:39.647468 18786 model_lifecycle.cc:578] successfully unloaded '0_queryfeast' version 1

__________________________________ test_func ___________________________________
def test_func():
    with testbook(
        REPO_ROOT / "examples" / "scaling-criteo" / "02-ETL-with-NVTabular.ipynb",
        execute=False,
        timeout=180,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["BASE_DIR"] = "/tmp/input/criteo/"
            os.environ["INPUT_DATA_DIR"] = "/tmp/input/criteo/"
            os.environ["OUTPUT_DATA_DIR"] = "/tmp/output/criteo/"
            os.system("mkdir -p /tmp/input/criteo")
            os.system("mkdir -p /tmp/output/criteo")

            from merlin.datasets.synthetic import generate_data

            train, valid = generate_data("criteo", int(100000), set_sizes=(0.7, 0.3))

            train.to_ddf().compute().to_parquet('/tmp/input/criteo/day_0.parquet')
            valid.to_ddf().compute().to_parquet('/tmp/input/criteo/day_1.parquet')
            """
        )


      tb1.execute()


tests/unit/examples/test_scaling_criteo_merlin_models.py:33:

../../../.local/lib/python3.8/site-packages/testbook/client.py:147: in execute

super().execute_cell(cell, index)

../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped

return just_run(coro(*args, **kwargs))

../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run

return loop.run_until_complete(coro)

/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete

return future.result()

../../../.local/lib/python3.8/site-packages/nbclient/client.py:965: in async_execute_cell

await self._check_raise_for_error(cell, cell_index, exec_reply)

self = <testbook.client.TestbookNotebookClient object at 0x7f4461c03eb0>

cell = {'id': 'fcd25787', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-02T14:43:58.536223Z',...ror at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory']}]}

cell_index = 27

exec_reply = {'buffers': [], 'content': {'ename': 'MemoryError', 'engine_info': {'engine_id': -1, 'engine_uuid': 'e6233e78-f6de-4a9...e, 'engine': 'e6233e78-f6de-4a96-b5ca-bd609eae6397', 'started': '2022-11-02T14:43:58.536545Z', 'status': 'error'}, ...}
async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:


      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)


E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E

E           import os

E           os.environ["BASE_DIR"] = "/tmp/input/criteo/"

E           os.environ["INPUT_DATA_DIR"] = "/tmp/input/criteo/"

E           os.environ["OUTPUT_DATA_DIR"] = "/tmp/output/criteo/"

E           os.system("mkdir -p /tmp/input/criteo")

E           os.system("mkdir -p /tmp/output/criteo")

E

E           from merlin.datasets.synthetic import generate_data

E

E           train, valid = generate_data("criteo", int(100000), set_sizes=(0.7, 0.3))

E

E           train.to_ddf().compute().to_parquet('/tmp/input/criteo/day_0.parquet')

E           valid.to_ddf().compute().to_parquet('/tmp/input/criteo/day_1.parquet')

E

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mMemoryError�[0m                               Traceback (most recent call last)

E           Cell �[0;32mIn [16], line 12�[0m

E           �[1;32m      8�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mdatasets�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01msynthetic�[39;00m �[38;5;28;01mimport�[39;00m generate_data

E           �[1;32m     10�[0m train, valid �[38;5;241m=�[39m generate_data(�[38;5;124m"�[39m�[38;5;124mcriteo�[39m�[38;5;124m"�[39m, �[38;5;28mint�[39m(�[38;5;241m100000�[39m), set_sizes�[38;5;241m=�[39m(�[38;5;241m0.7�[39m, �[38;5;241m0.3�[39m))

E           �[0;32m---> 12�[0m �[43mtrain�[49m�[38;5;241;43m.�[39;49m�[43mto_ddf�[49m�[43m(�[49m�[43m)�[49m�[38;5;241;43m.�[39;49m�[43mcompute�[49m�[43m(�[49m�[43m)�[49m�[38;5;241;43m.�[39;49m�[43mto_parquet�[49m�[43m(�[49m�[38;5;124;43m'�[39;49m�[38;5;124;43m/tmp/input/criteo/day_0.parquet�[39;49m�[38;5;124;43m'�[39;49m�[43m)�[49m

E           �[1;32m     13�[0m valid�[38;5;241m.�[39mto_ddf()�[38;5;241m.�[39mcompute()�[38;5;241m.�[39mto_parquet(�[38;5;124m'�[39m�[38;5;124m/tmp/input/criteo/day_1.parquet�[39m�[38;5;124m'�[39m)

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:5718�[0m, in �[0;36mDataFrame.to_parquet�[0;34m(self, path, args, **kwargs)�[0m

E           �[1;32m   5715�[0m �[38;5;124;03m"""{docstring}"""�[39;00m

E           �[1;32m   5716�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mcudf�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mio�[39;00m �[38;5;28;01mimport�[39;00m parquet

E           �[0;32m-> 5718�[0m �[38;5;28;01mreturn�[39;00m �[43mparquet�[49m�[38;5;241;43m.�[39;49m�[43mto_parquet�[49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[43m,�[49m�[43m �[49m�[43mpath�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[43margs�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m�[39;49m�[43mkwargs�[49m�[43m)�[49m

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101�[0m, in �[0;36mannotate.call..inner�[0;34m(args, **kwargs)�[0m

E           �[1;32m     98�[0m �[38;5;129m@wraps�[39m(func)

E           �[1;32m     99�[0m �[38;5;28;01mdef�[39;00m �[38;5;21minner�[39m(�[38;5;241m�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs):

E           �[1;32m    100�[0m     libnvtx_push_range(�[38;5;28mself�[39m�[38;5;241m.�[39mattributes, �[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)

E           �[0;32m--> 101�[0m     result �[38;5;241m=�[39m �[43mfunc�[49m�[43m(�[49m�[38;5;241;43m�[39;49m�[43margs�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m�[39;49m�[43mkwargs�[49m�[43m)�[49m

E           �[1;32m    102�[0m     libnvtx_pop_range(�[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)

E           �[1;32m    103�[0m     �[38;5;28;01mreturn�[39;00m result

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/cudf/io/parquet.py:657�[0m, in �[0;36mto_parquet�[0;34m(df, path, engine, compression, index, partition_cols, partition_file_name, partition_offsets, statistics, metadata_file_path, int96_timestamps, row_group_size_bytes, row_group_size_rows, args, **kwargs)�[0m

E           �[1;32m    649�[0m     �[38;5;28;01mif�[39;00m partition_offsets:

E           �[1;32m    650�[0m         kwargs[�[38;5;124m"�[39m�[38;5;124mpartitions_info�[39m�[38;5;124m"�[39m] �[38;5;241m=�[39m �[38;5;28mlist�[39m(

E           �[1;32m    651�[0m             �[38;5;28mzip�[39m(

E           �[1;32m    652�[0m                 partition_offsets,

E           �[1;32m    653�[0m                 np�[38;5;241m.�[39mroll(partition_offsets, �[38;5;241m-�[39m�[38;5;241m1�[39m) �[38;5;241m-�[39m partition_offsets,

E           �[1;32m    654�[0m             )

E           �[1;32m    655�[0m         )[:�[38;5;241m-�[39m�[38;5;241m1�[39m]

E           �[0;32m--> 657�[0m     �[38;5;28;01mreturn�[39;00m �[43m_write_parquet�[49m�[43m(�[49m

E           �[1;32m    658�[0m �[43m        �[49m�[43mdf�[49m�[43m,�[49m

E           �[1;32m    659�[0m �[43m        �[49m�[43mpaths�[49m�[38;5;241;43m=�[39;49m�[43mpath�[49m�[43m �[49m�[38;5;28;43;01mif�[39;49;00m�[43m �[49m�[43mis_list_like�[49m�[43m(�[49m�[43mpath�[49m�[43m)�[49m�[43m �[49m�[38;5;28;43;01melse�[39;49;00m�[43m �[49m�[43m[�[49m�[43mpath�[49m�[43m]�[49m�[43m,�[49m

E           �[1;32m    660�[0m �[43m        �[49m�[43mcompression�[49m�[38;5;241;43m=�[39;49m�[43mcompression�[49m�[43m,�[49m

E           �[1;32m    661�[0m �[43m        �[49m�[43mindex�[49m�[38;5;241;43m=�[39;49m�[43mindex�[49m�[43m,�[49m

E           �[1;32m    662�[0m �[43m        �[49m�[43mstatistics�[49m�[38;5;241;43m=�[39;49m�[43mstatistics�[49m�[43m,�[49m

E           �[1;32m    663�[0m �[43m        �[49m�[43mmetadata_file_path�[49m�[38;5;241;43m=�[39;49m�[43mmetadata_file_path�[49m�[43m,�[49m

E           �[1;32m    664�[0m �[43m        �[49m�[43mint96_timestamps�[49m�[38;5;241;43m=�[39;49m�[43mint96_timestamps�[49m�[43m,�[49m

E           �[1;32m    665�[0m �[43m        �[49m�[43mrow_group_size_bytes�[49m�[38;5;241;43m=�[39;49m�[43mrow_group_size_bytes�[49m�[43m,�[49m

E           �[1;32m    666�[0m �[43m        �[49m�[43mrow_group_size_rows�[49m�[38;5;241;43m=�[39;49m�[43mrow_group_size_rows�[49m�[43m,�[49m

E           �[1;32m    667�[0m �[43m        �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m�[39;49m�[43mkwargs�[49m�[43m,�[49m

E           �[1;32m    668�[0m �[43m    �[49m�[43m)�[49m

E           �[1;32m    670�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    671�[0m     �[38;5;28;01mif�[39;00m partition_offsets �[38;5;129;01mis�[39;00m �[38;5;129;01mnot�[39;00m �[38;5;28;01mNone�[39;00m:

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101�[0m, in �[0;36mannotate.call..inner�[0;34m(args, **kwargs)�[0m

E           �[1;32m     98�[0m �[38;5;129m@wraps�[39m(func)

E           �[1;32m     99�[0m �[38;5;28;01mdef�[39;00m �[38;5;21minner�[39m(�[38;5;241m�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs):

E           �[1;32m    100�[0m     libnvtx_push_range(�[38;5;28mself�[39m�[38;5;241m.�[39mattributes, �[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)

E           �[0;32m--> 101�[0m     result �[38;5;241m=�[39m �[43mfunc�[49m�[43m(�[49m�[38;5;241;43m�[39;49m�[43margs�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m�[39;49m�[43mkwargs�[49m�[43m)�[49m

E           �[1;32m    102�[0m     libnvtx_pop_range(�[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)

E           �[1;32m    103�[0m     �[38;5;28;01mreturn�[39;00m result

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/cudf/io/parquet.py:97�[0m, in �[0;36m_write_parquet�[0;34m(df, paths, compression, index, statistics, metadata_file_path, int96_timestamps, row_group_size_bytes, row_group_size_rows, partitions_info, **kwargs)�[0m

E           �[1;32m     93�[0m         write_parquet_res �[38;5;241m=�[39m libparquet�[38;5;241m.�[39mwrite_parquet(

E           �[1;32m     94�[0m             df, filepaths_or_buffers�[38;5;241m=�[39mfile_objs, �[38;5;241m�[39m�[38;5;241m�[39mcommon_args

E           �[1;32m     95�[0m         )

E           �[1;32m     96�[0m �[38;5;28;01melse�[39;00m:

E           �[0;32m---> 97�[0m     write_parquet_res �[38;5;241m=�[39m �[43mlibparquet�[49m�[38;5;241;43m.�[39;49m�[43mwrite_parquet�[49m�[43m(�[49m

E           �[1;32m     98�[0m �[43m        �[49m�[43mdf�[49m�[43m,�[49m�[43m �[49m�[43mfilepaths_or_buffers�[49m�[38;5;241;43m=�[39;49m�[43mpaths_or_bufs�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m*�[39;49m�[43mcommon_args�[49m

E           �[1;32m     99�[0m �[43m    �[49m�[43m)�[49m

E           �[1;32m    101�[0m �[38;5;28;01mreturn�[39;00m write_parquet_res

E

E           File �[0;32mcudf/_lib/parquet.pyx:287�[0m, in �[0;36mcudf._lib.parquet.write_parquet�[0;34m()�[0m

E

E           File �[0;32mcudf/_lib/parquet.pyx:397�[0m, in �[0;36mcudf._lib.parquet.write_parquet�[0;34m()�[0m

E

E           �[0;31mMemoryError�[0m: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

E           MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory
../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError

----------------------------- Captured stderr call -----------------------------

I1102 14:43:40.354328 18786 server.cc:300] Timeout 20: Found 1 live models and 0 in-flight non-inference requests

I1102 14:43:41.354449 18786 server.cc:300] Timeout 19: Found 1 live models and 0 in-flight non-inference requests

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1102 14:43:42.354563 18786 server.cc:300] Timeout 18: Found 1 live models and 0 in-flight non-inference requests

I1102 14:43:43.017937 18786 model_lifecycle.cc:578] successfully unloaded '3_queryfeast' version 1

I1102 14:43:43.354696 18786 server.cc:300] Timeout 17: Found 0 live models and 0 in-flight non-inference requests

2022-11-02 14:43:47,846 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-02 14:43:47,858 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-02 14:43:47,890 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-02 14:43:47,950 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 45 leaked semaphore objects to clean up at shutdown

warnings.warn('resource_tracker: There appear to be %d '
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       355    210    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                      100     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     17    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        54     19    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       125     23    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     41    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    158    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    22      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      2    83%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     26    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     97    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          346    164    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          603    345    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     29    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     81    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     20    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      1    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    114    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49     15    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     55    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                114     55    52%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     169     89    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          245    113    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 244    114    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          424    159    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              170    117    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                104     69    34%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              288     87    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                60     36    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39     16    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          452    157    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    243    125    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               707    325    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              122     89    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               97     62    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    110    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     24    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       425    337    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         107     54    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          85     12    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    141    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31      8    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     33    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     39    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     41    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       209     53    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 46     21    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     23      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                26      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                              9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py                               24      8    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py                                77      3    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py                               126     26    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             79      7    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py                       45     28    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py                     51      1    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           67     20    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py                      50      1    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py                         1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py                    11      2    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py                  1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py           61     12    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py                 140     47    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              47     17    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               268    234    13%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     14    81%
TOTAL                                                                                                 14113   6623    53%
=================== 2 failed, 1 passed in 274.75s (0:04:34) ====================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://github.com/gitapi/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins5815838804451703940.sh

nvidia-merlin-bot · 2022-11-02T16:38:17Z

Click to view CI Results

GitHub pull request #713 of commit 59f0e09c7e3eb5a0c80ebd6dbfbfd9aadde0d226, no merge conflicts.
Running as SYSTEM
Setting status of 59f0e09c7e3eb5a0c80ebd6dbfbfd9aadde0d226 to PENDING with url https://10.20.13.93:8080/job/merlin_merlin/555/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/713/*:refs/remotes/origin/pr/713/* # timeout=10
 > git rev-parse 59f0e09c7e3eb5a0c80ebd6dbfbfd9aadde0d226^{commit} # timeout=10
Checking out Revision 59f0e09c7e3eb5a0c80ebd6dbfbfd9aadde0d226 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 59f0e09c7e3eb5a0c80ebd6dbfbfd9aadde0d226 # timeout=10
Commit message: "Merge branch 'main' into merlin-tf-int-test-bandaid"
 > git rev-list --no-walk fbaee751454b67a03cef63dbdb70eaaf273e0e74 # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins15040607954697781586.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.0,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.0,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.8.2,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.42,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.0,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='555010715'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-h87ldj0n
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-h87ldj0n
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit d2737da6cc7e98ec8723731a7cc998aa143e6758
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+10.gd2737da) (0.3.0+12.g78ecddd)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+10.gd2737da) (1.1.1)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+10.gd2737da) (2.22.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.10.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.55.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (3.19.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (7.0.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (4.64.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.12.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.5.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.2.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.4.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (3.1.2)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (6.1)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 72.8 MB/s eta 0:00:00
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (65.4.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+10.gd2737da-py3-none-any.whl size=91051 sha256=29d6545e7f571b0a39dc350ba09840840fe210982932dcd0ab78e4f35ba86287
  Stored in directory: /tmp/pip-ephem-wheel-cache-tuwdk7te/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+10.gd2737da
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/models.git

Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-a3rh60rc

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-a3rh60rc

Resolved https://github.com/NVIDIA-Merlin/models.git to commit 5e1a805e357d2e77d22f7e0dab668cd1bc9cfb0c

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+23.g5e1a805e) (0.3.0+12.g78ecddd)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.10.0)

Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (2022.3.0)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (0.55.1)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.3.5)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.2.5)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (3.19.5)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (21.3)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (7.0.0)

Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (2022.3.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (4.64.1)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (0.4.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (5.4.1)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (0.12.0)

Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (2022.5.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (2.2.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (8.1.3)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (2.4.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (3.1.2)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (5.8.0)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (2.0.0)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.7.0)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.0.4)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (6.1)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (0.38.1)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (65.4.1)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.20.3)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (2022.2.1)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.2.0)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.52.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (1.0.1)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (4.1.0)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (6.0.2)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+23.g5e1a805e) (4.0.0)

Building wheels for collected packages: merlin-models

Building wheel for merlin-models (pyproject.toml): started

Building wheel for merlin-models (pyproject.toml): finished with status 'done'

Created wheel for merlin-models: filename=merlin_models-0.9.0+23.g5e1a805e-py3-none-any.whl size=356077 sha256=123b57a6a1fcb5db86db9cfc2d56a765f519ecc8fe73675889bc7707b4584cae

Stored in directory: /tmp/pip-ephem-wheel-cache-sbvi1uny/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5

Successfully built merlin-models

Installing collected packages: merlin-models

Attempting uninstall: merlin-models

Found existing installation: merlin-models 0.7.0+11.g280956aa4

Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu

Can't uninstall 'merlin-models'. No files were found to uninstall.

Successfully installed merlin-models-0.9.0+23.g5e1a805e
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git

Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-01pqv53h

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-01pqv53h

Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit 59579f2c46006fcb22795623ee9400c658166670

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+2.g59579f2c) (0.3.0+12.g78ecddd)

Collecting merlin-dataloader>=0.0.2

Downloading merlin-dataloader-0.0.2.tar.gz (44 kB)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.1/44.1 kB 1.6 MB/s eta 0:00:00

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+2.g59579f2c) (1.8.1)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.10.0)

Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (2022.3.0)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (0.55.1)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.3.5)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.2.5)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (3.19.5)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (21.3)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (7.0.0)

Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (2022.3.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (4.64.1)

Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+2.g59579f2c) (1.20.3)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (0.4.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (5.4.1)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (0.12.0)

Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (2022.5.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (2.2.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (8.1.3)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (2.4.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (3.1.2)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (5.8.0)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (2.0.0)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.7.0)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.0.4)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (6.1)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (0.38.1)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (65.4.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (2022.2.1)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.2.0)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.52.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (1.0.1)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (4.1.0)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (6.0.2)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+2.g59579f2c) (4.0.0)

Building wheels for collected packages: nvtabular, merlin-dataloader

Building wheel for nvtabular (pyproject.toml): started

Building wheel for nvtabular (pyproject.toml): finished with status 'done'

Created wheel for nvtabular: filename=nvtabular-1.6.0+2.g59579f2c-cp38-cp38-linux_x86_64.whl size=257750 sha256=3f10733604950073c2aeaea80a60c9e102d5acfc7b422786da700448a22b4621

Stored in directory: /tmp/pip-ephem-wheel-cache-m22nf8w1/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc

Building wheel for merlin-dataloader (pyproject.toml): started

Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'

Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.2-py3-none-any.whl size=29205 sha256=401711a51da9782853d073f1fea145dbcdd73558d544e8bae776d86c4fc1bf8b

Stored in directory: /tmp/pip-ephem-wheel-cache-m22nf8w1/wheels/d5/ce/8c/31476c01e0b5a2278110fe2092bdd911efb0e5b83d0d3550ca

Successfully built nvtabular merlin-dataloader

Installing collected packages: merlin-dataloader, nvtabular

Attempting uninstall: nvtabular

Found existing installation: nvtabular 1.1.1

Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu

Can't uninstall 'nvtabular'. No files were found to uninstall.

Successfully installed merlin-dataloader-0.0.2 nvtabular-1.6.0+2.g59579f2c
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/core.git

Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-htfgizmc

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-htfgizmc

Resolved https://github.com/NVIDIA-Merlin/core.git to commit 563be4bf5ef675940d5fff2b5e4666424a7f7947

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (1.10.0)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (0.55.1)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (1.3.5)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (1.2.5)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (21.3)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (3.19.5)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (7.0.0)

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (2022.3.0)

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (2022.5.0)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (2022.3.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (4.64.1)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (0.4.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (1.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (5.4.1)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (0.12.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.2.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (8.1.3)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.4.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (3.1.2)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (5.8.0)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.0.0)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.7.0)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.0.4)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (6.1)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+5.g563be4b) (0.38.1)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+5.g563be4b) (65.4.1)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+5.g563be4b) (1.20.3)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+5.g563be4b) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+5.g563be4b) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+5.g563be4b) (2022.2.1)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+5.g563be4b) (1.2.0)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+5.g563be4b) (1.52.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+5.g563be4b) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.0.1)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (4.1.0)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (6.0.2)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (4.0.0)

Building wheels for collected packages: merlin-core

Building wheel for merlin-core (pyproject.toml): started

Building wheel for merlin-core (pyproject.toml): finished with status 'done'

Created wheel for merlin-core: filename=merlin_core-0.8.0+5.g563be4b-py3-none-any.whl size=118363 sha256=db1d354781f5e6f5c6c98849904e5de72fc16abbf890aa42c6ea6ad075848419

Stored in directory: /tmp/pip-ephem-wheel-cache-9celzfkn/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84

Successfully built merlin-core

Installing collected packages: merlin-core

Attempting uninstall: merlin-core

Found existing installation: merlin-core 0.3.0+12.g78ecddd

Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu

Can't uninstall 'merlin-core'. No files were found to uninstall.

Successfully installed merlin-core-0.8.0+5.g563be4b
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin

plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0

collected 3 items
tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py .      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py .               [100%]
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       355    212    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                      100     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     15    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        54     19    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       125     23    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     24    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    146    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    22      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      4    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     27    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     93    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          346    157    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          603    344    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     20    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     78    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     19    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      1    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    113    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49      8    84%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     55    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                114     55    52%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     169     89    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          245    109    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 244    108    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          424    149    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              170    117    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                104     69    34%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              288     74    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                60     36    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39      4    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          452    149    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    243    125    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               707    310    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              122     89    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               97     62    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    108    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     22    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       425    337    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         107     54    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          85     12    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    141    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31      8    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     31    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     39    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     34    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       209     53    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 46     20    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     23      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                26      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                              9      4    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py                               24      8    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py                                77     27    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py                               126     53    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             79     32    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py                     51     12    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           67     22    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py                      50     12    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py                             72     26    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py                         1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py                    11      2    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py                  1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py           61     12    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py                 140     47    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              47     12    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py                           87     51    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               268    210    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     14    81%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py                            22     20     9%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py                               113     14    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py                             37     29    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py                             10      6    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py                          32     13    59%
TOTAL                                                                                                 14441   6688    54%
======================== 3 passed in 329.30s (0:05:29) =========================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

/usr/local/lib/python3.8/dist-packages/coverage/data.py:130: CoverageWarning: Data file '/var/jenkins_home/workspace/merlin_merlin/merlin/.coverage.10.20.17.231.30343.708972' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object

data._warn(str(exc))

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://github.com/gitapi/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins6333726446355766555.sh

add in reinstall for dask and distributed after feast install

f3ab757

jperez999 self-assigned this Nov 1, 2022

jperez999 added bug Something isn't working chore Infrastructure update ci labels Nov 1, 2022

jperez999 modified the milestones: Merlin 22.10, Merlin 22.11 Nov 1, 2022

karlhigley approved these changes Nov 1, 2022

View reviewed changes

Merge branch 'main' into merlin-tf-int-test-bandaid

daa80eb

Merge branch 'main' into merlin-tf-int-test-bandaid

59f0e09

jperez999 merged commit a4f82eb into NVIDIA-Merlin:main Nov 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add in reinstall for dask and distributed after feast install #713

add in reinstall for dask and distributed after feast install #713

jperez999 commented Nov 1, 2022

github-actions bot commented Nov 1, 2022

nvidia-merlin-bot commented Nov 1, 2022

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

nvidia-merlin-bot commented Nov 2, 2022

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

nvidia-merlin-bot commented Nov 2, 2022

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

add in reinstall for dask and distributed after feast install #713

add in reinstall for dask and distributed after feast install #713

Conversation

jperez999 commented Nov 1, 2022

github-actions bot commented Nov 1, 2022

Documentation preview

nvidia-merlin-bot commented Nov 1, 2022

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

nvidia-merlin-bot commented Nov 2, 2022

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

nvidia-merlin-bot commented Nov 2, 2022

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover