docs: Remove SM from the support matrix #721

mikemckiernan · 2022-11-04T13:08:01Z

The streaming multiprocessors (SM) value might be
more of an item for NVIDIA employees than Merlin
consumers.

In addition, the DLFW folks do not publish the SM
values for the TensorFlow and PyTorch containers
that we use as base containers.

The streaming multiprocessors (SM) value might be more of an item for NVIDIA employees than Merlin consumers. In addition, the DLFW folks do not publish the SM values for the TensorFlow and PyTorch containers that we use as base containers.

github-actions · 2022-11-04T13:10:03Z

Documentation preview

https://nvidia-merlin.github.io/Merlin/review/pr-721

nvidia-merlin-bot · 2022-11-04T13:15:54Z

Click to view CI Results

GitHub pull request #721 of commit 2922647005d7f8883a887a209f1cb577d9b23874, no merge conflicts.
Running as SYSTEM
Setting status of 2922647005d7f8883a887a209f1cb577d9b23874 to PENDING with url https://10.20.13.93:8080/job/merlin_merlin/566/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/721/*:refs/remotes/origin/pr/721/* # timeout=10
 > git rev-parse 2922647005d7f8883a887a209f1cb577d9b23874^{commit} # timeout=10
Checking out Revision 2922647005d7f8883a887a209f1cb577d9b23874 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 2922647005d7f8883a887a209f1cb577d9b23874 # timeout=10
Commit message: "docs: Remove SM from the support matrix"
 > git rev-list --no-walk 276127e29d857071f7a4b8f3dd7b0decf7980c04 # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins3723259405405226004.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.2,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.2,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.8.2,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.42,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.0,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='2005531779'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-yud8neoc
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-yud8neoc
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit d2737da6cc7e98ec8723731a7cc998aa143e6758
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+10.gd2737da) (0.3.0+12.g78ecddd)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+10.gd2737da) (1.1.1)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+10.gd2737da) (2.22.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.3.5)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (7.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (4.64.1)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.10.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.4.3)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.5.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.2.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (6.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.7.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (3.1.2)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.0.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (65.4.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.20.3)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 85.5 MB/s eta 0:00:00
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+10.gd2737da-py3-none-any.whl size=91051 sha256=8b9b44ec014c95618314524e218422eba530ef25056dec9c164b3aae42e479b9
  Stored in directory: /tmp/pip-ephem-wheel-cache-_psnvb_o/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+10.gd2737da
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/models.git

Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-95f9baj8

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-95f9baj8

Resolved https://github.com/NVIDIA-Merlin/models.git to commit ada3874d663592d215150ba3eadd04b075818b4f

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+29.gada3874d) (0.3.0+12.g78ecddd)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.3.5)

Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2022.3.0)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (7.0.0)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (21.3)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (4.64.1)

Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2022.3.0)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.2.5)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (0.55.1)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (3.19.5)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.10.0)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.2.0)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (0.4.3)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.2.0)

Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2022.5.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (0.12.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (5.4.1)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2.2.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (6.1)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.7.0)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (5.8.0)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2.4.0)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.0.4)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (8.1.3)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (3.1.2)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2.0.0)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (65.4.1)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.20.3)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (0.38.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2022.2.1)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.2.0)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.52.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.0.1)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (6.0.2)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (4.1.0)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (4.0.0)

Building wheels for collected packages: merlin-models

Building wheel for merlin-models (pyproject.toml): started

Building wheel for merlin-models (pyproject.toml): finished with status 'done'

Created wheel for merlin-models: filename=merlin_models-0.9.0+29.gada3874d-py3-none-any.whl size=356754 sha256=7d5a3fee8c645db3812cb9e0d9ea0616f45908adb6e1810058f92eea3e05d8a8

Stored in directory: /tmp/pip-ephem-wheel-cache-mqj1qa4p/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5

Successfully built merlin-models

Installing collected packages: merlin-models

Attempting uninstall: merlin-models

Found existing installation: merlin-models 0.7.0+11.g280956aa4

Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu

Can't uninstall 'merlin-models'. No files were found to uninstall.

Successfully installed merlin-models-0.9.0+29.gada3874d
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git

Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-eekjvaxy

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-eekjvaxy

Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit 8e7edbafd3006f56e73efdc0c01c4445ab57d028

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+3.g8e7edbaf) (0.3.0+12.g78ecddd)

Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+3.g8e7edbaf) (1.8.1)

Collecting merlin-dataloader>=0.0.2

Downloading merlin-dataloader-0.0.2.tar.gz (44 kB)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.1/44.1 kB 1.5 MB/s eta 0:00:00

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.3.5)

Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2022.3.0)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (7.0.0)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (21.3)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (4.64.1)

Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2022.3.0)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.2.5)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (0.55.1)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (3.19.5)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.10.0)

Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+3.g8e7edbaf) (1.20.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.2.0)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (0.4.3)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.2.0)

Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2022.5.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (0.12.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (5.4.1)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2.2.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (6.1)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.7.0)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (5.8.0)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2.4.0)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.0.4)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (8.1.3)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (3.1.2)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2.0.0)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (65.4.1)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (0.38.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2022.2.1)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.2.0)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.52.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.0.1)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (6.0.2)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (4.1.0)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (4.0.0)

Building wheels for collected packages: nvtabular, merlin-dataloader

Building wheel for nvtabular (pyproject.toml): started

Building wheel for nvtabular (pyproject.toml): finished with status 'done'

Created wheel for nvtabular: filename=nvtabular-1.6.0+3.g8e7edbaf-cp38-cp38-linux_x86_64.whl size=257744 sha256=7ccea96562010c767c5c204d25fc6a79f6da351e8669855591f857efff8775e2

Stored in directory: /tmp/pip-ephem-wheel-cache-il67kfgz/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc

Building wheel for merlin-dataloader (pyproject.toml): started

Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'

Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.2-py3-none-any.whl size=29205 sha256=6ed345a5c2ae24cece65cc7898ab782c9ed304cbcc1907c44ae915ce39ca57c1

Stored in directory: /tmp/pip-ephem-wheel-cache-il67kfgz/wheels/d5/ce/8c/31476c01e0b5a2278110fe2092bdd911efb0e5b83d0d3550ca

Successfully built nvtabular merlin-dataloader

Installing collected packages: merlin-dataloader, nvtabular

Attempting uninstall: nvtabular

Found existing installation: nvtabular 1.1.1

Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu

Can't uninstall 'nvtabular'. No files were found to uninstall.

Successfully installed merlin-dataloader-0.0.2 nvtabular-1.6.0+3.g8e7edbaf
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/core.git

Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-oixr98_e

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-oixr98_e

Resolved https://github.com/NVIDIA-Merlin/core.git to commit 563be4bf5ef675940d5fff2b5e4666424a7f7947

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (1.3.5)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (21.3)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (7.0.0)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (1.2.5)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (4.64.1)

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (2022.5.0)

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (2022.3.0)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (0.55.1)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (2022.3.0)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (3.19.5)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (1.10.0)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (1.2.0)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (0.4.3)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (0.12.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (5.4.1)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.2.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (6.1)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.7.0)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (5.8.0)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.4.0)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.0.4)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (8.1.3)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (3.1.2)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.0.0)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+5.g563be4b) (65.4.1)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+5.g563be4b) (1.20.3)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+5.g563be4b) (0.38.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+5.g563be4b) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+5.g563be4b) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+5.g563be4b) (2022.2.1)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+5.g563be4b) (1.2.0)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+5.g563be4b) (1.52.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+5.g563be4b) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.0.1)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (6.0.2)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (4.1.0)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (4.0.0)

Building wheels for collected packages: merlin-core

Building wheel for merlin-core (pyproject.toml): started

Building wheel for merlin-core (pyproject.toml): finished with status 'done'

Created wheel for merlin-core: filename=merlin_core-0.8.0+5.g563be4b-py3-none-any.whl size=118363 sha256=0d45a3184894b80b2bffb82820129609b27f4e1afc42670560fca70d549d93cf

Stored in directory: /tmp/pip-ephem-wheel-cache-j7fjrb70/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84

Successfully built merlin-core

Installing collected packages: merlin-core

Attempting uninstall: merlin-core

Found existing installation: merlin-core 0.3.0+12.g78ecddd

Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu

Can't uninstall 'merlin-core'. No files were found to uninstall.

Successfully installed merlin-core-0.8.0+5.g563be4b
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin

plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0

collected 3 items
tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py F      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py .               [100%]
=================================== FAILURES ===================================

__________________________________ test_func ___________________________________
self = <testbook.client.TestbookNotebookClient object at 0x7f824ddbc040>

msg_id = 'ee37896e-7c4122a0725a4c99a3a0542d_10085_32'

cell = {'id': '350e2c16', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-04T13:11:57.508164Z',...b[38;5;241m=\x1b[39m \x1b[38;5;28mself\x1b[39m\x1b[38;5;241m.\x1b[39mpid\n', '\x1b[0;31mKeyboardInterrupt\x1b[0m: ']}]}

timeout = 120

task_poll_output_msg = <Task finished name='Task-379' coro=<NotebookClient._async_poll_output_msg() done, defined at /var/jenkins_home/.local/lib/python3.8/site-packages/nbclient/client.py:756> result=None>

task_poll_kernel_alive = <Task cancelled name='Task-378' coro=<NotebookClient._async_poll_kernel_alive() done, defined at /var/jenkins_home/.local/lib/python3.8/site-packages/nbclient/client.py:770>>
async def _async_poll_for_reply(
    self,
    msg_id: str,
    cell: NotebookNode,
    timeout: t.Optional[int],
    task_poll_output_msg: asyncio.Future,
    task_poll_kernel_alive: asyncio.Future,
) -> t.Dict:

    assert self.kc is not None
    new_timeout: t.Optional[float] = None
    if timeout is not None:
        deadline = monotonic() + timeout
        new_timeout = float(timeout)
    while True:
        try:


          msg = await ensure_async(self.kc.shell_channel.get_msg(timeout=new_timeout))


../../../.local/lib/python3.8/site-packages/nbclient/client.py:730:

obj = <coroutine object ZMQSocketChannel.get_msg at 0x7f824ca483c0>
async def ensure_async(obj: Union[Awaitable, Any]) -> Any:
    """Convert a non-awaitable object to a coroutine if needed,
    and await it if it was not already awaited.
    """
    if inspect.isawaitable(obj):
        try:


          result = await obj


../../../.local/lib/python3.8/site-packages/nbclient/util.py:96:

self = <jupyter_client.channels.ZMQSocketChannel object at 0x7f832a00e0d0>

timeout = 120000.0
async def get_msg(self, timeout: t.Optional[float] = None) -> t.Dict[str, t.Any]:
    """Gets a message if there is one that is ready."""
    assert self.socket is not None
    if timeout is not None:
        timeout *= 1000  # seconds to ms
    ready = await self.socket.poll(timeout)

    if ready:
        res = await self._recv()
        return res
    else:


      raise Empty


E           _queue.Empty
/usr/local/lib/python3.8/dist-packages/jupyter_client/channels.py:230: Empty
During handling of the above exception, another exception occurred:
def test_func():
    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "01-Building-Recommender-Systems-with-Merlin.ipynb",
        execute=False,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["NUM_ROWS"] = "100000"
            os.system("mkdir -p /tmp/examples")
            os.environ["BASE_DIR"] = "/tmp/examples/"
            """
        )
        tb1.execute()
        assert os.path.isdir("/tmp/examples/dlrm")
        assert os.path.isdir("/tmp/examples/feature_repo")
        assert os.path.isdir("/tmp/examples/query_tower")
        assert os.path.isfile("/tmp/examples/item_embeddings.parquet")
        assert os.path.isfile("/tmp/examples/feature_repo/user_features.py")
        assert os.path.isfile("/tmp/examples/feature_repo/item_features.py")

    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "02-Deploying-multi-stage-RecSys-with-Merlin-Systems.ipynb",
        execute=False,
        timeout=120,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["BASE_DIR"] = "/tmp/examples/"
            os.environ["topk_retrieval"] = "20"
            """
        )
        NUM_OF_CELLS = len(tb2.cells)
        tb2.execute_cell(list(range(0, NUM_OF_CELLS - 3)))
        top_k = tb2.ref("top_k")
        outputs = tb2.ref("outputs")
        assert outputs[0] == "ordered_ids"


      tb2.inject(


            """
            import shutil
            from merlin.core.dispatch import get_lib
            from merlin.models.loader.tf_utils import configure_tensorflow
            configure_tensorflow()
            df_lib = get_lib()
            batch = df_lib.read_parquet(
                os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),
                num_rows=1,
                columns=["user_id_raw"],
            )
            from merlin.systems.triton.utils import run_ensemble_on_tritonserver
            response = run_ensemble_on_tritonserver(
                "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"
            )
            response = [x.tolist()[0] for x in response["ordered_ids"]]
            shutil.rmtree("/tmp/examples/", ignore_errors=True)
            """
        )

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:61:

../../../.local/lib/python3.8/site-packages/testbook/client.py:237: in inject

cell = TestbookNode(self.execute_cell(inject_idx)) if run else TestbookNode(code_cell)

../../../.local/lib/python3.8/site-packages/testbook/client.py:133: in execute_cell

cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)

../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped

return just_run(coro(*args, **kwargs))

../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run

return loop.run_until_complete(coro)

/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete

return future.result()

../../../.local/lib/python3.8/site-packages/nbclient/client.py:949: in async_execute_cell

exec_reply = await self.task_poll_for_reply

../../../.local/lib/python3.8/site-packages/nbclient/client.py:754: in _async_poll_for_reply

await self._async_handle_timeout(timeout, cell)

self = <testbook.client.TestbookNotebookClient object at 0x7f824ddbc040>

timeout = 120

cell = {'id': '350e2c16', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-04T13:11:57.508164Z',...b[38;5;241m=\x1b[39m \x1b[38;5;28mself\x1b[39m\x1b[38;5;241m.\x1b[39mpid\n', '\x1b[0;31mKeyboardInterrupt\x1b[0m: ']}]}
async def _async_handle_timeout(
    self, timeout: int, cell: t.Optional[NotebookNode] = None
) -> None:

    self.log.error("Timeout waiting for execute reply (%is)." % timeout)
    if self.interrupt_on_timeout:
        self.log.error("Interrupting kernel")
        assert self.km is not None
        await ensure_async(self.km.interrupt_kernel())
    else:


      raise CellTimeoutError.error_from_timeout_and_cell(


            "Cell execution timed out", timeout, cell
        )

E           nbclient.exceptions.CellTimeoutError: A cell timed out while it was being executed, after 120 seconds.

E           The message was: Cell execution timed out.

E           Here is a preview of the cell contents:

E           -------------------

E           ['import shutil', 'from merlin.core.dispatch import get_lib', 'from merlin.models.loader.tf_utils import configure_tensorflow', 'configure_tensorflow()', 'df_lib = get_lib()']

E           ...

E           ['response = run_ensemble_on_tritonserver(', '    "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"', ')', 'response = [x.tolist()[0] for x in response["ordered_ids"]]', 'shutil.rmtree("/tmp/examples/", ignore_errors=True)']

E           -------------------
../../../.local/lib/python3.8/site-packages/nbclient/client.py:801: CellTimeoutError

----------------------------- Captured stdout call -----------------------------

Signal (2) received.

Signal (2) received.

Signal (15) received.

Signal (15) received.

Signal (15) received.

Signal (15) received.

Signal (15) received.

Signal (15) received.

Signal (15) received.

----------------------------- Captured stderr call -----------------------------

2022-11-04 13:10:07.265142: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-04 13:10:11.300165: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-04 13:10:11.300934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15149 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-11-04 13:10:11.301556: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 15149 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-11-04 13:10:11.302168: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 15149 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

WARNING clustering 456 points to 32 centroids: please provide at least 1248 training points

2022-11-04 13:11:50.920219: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-04 13:11:54.962118: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-04 13:11:54.962889: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15149 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-11-04 13:11:54.963508: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 15149 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-11-04 13:11:54.964122: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 15149 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

I1104 13:11:57.854329 10648 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fe672000000' with size 268435456

I1104 13:11:57.855119 10648 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1104 13:11:57.863277 10648 model_lifecycle.cc:459] loading: 1_predicttensorflowtriton:1

I1104 13:11:57.863357 10648 model_lifecycle.cc:459] loading: 0_queryfeast:1

I1104 13:11:57.863404 10648 model_lifecycle.cc:459] loading: 2_queryfaiss:1

I1104 13:11:57.863448 10648 model_lifecycle.cc:459] loading: 3_queryfeast:1

I1104 13:11:57.863503 10648 model_lifecycle.cc:459] loading: 4_unrollfeatures:1

I1104 13:11:57.863540 10648 model_lifecycle.cc:459] loading: 5_predicttensorflowtriton:1

I1104 13:11:57.863614 10648 model_lifecycle.cc:459] loading: 6_softmaxsampling:1

I1104 13:11:58.143988 10648 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow

I1104 13:11:58.144033 10648 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10

I1104 13:11:58.144044 10648 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10

I1104 13:11:58.144050 10648 tensorflow.cc:2576] backend configuration:

{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}

I1104 13:11:58.144092 10648 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 1_predicttensorflowtriton (version 1)

2022-11-04 13:11:58.144723: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:11:58.149263: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-04 13:11:58.149291: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:11:58.149387: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-04 13:11:58.556383: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-04 13:11:58.586344: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled

2022-11-04 13:11:58.588352: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-04 13:11:58.638546: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:11:58.662093: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 517382 microseconds.

I1104 13:11:58.672810 10648 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 1_predicttensorflowtriton (GPU device 0)

2022-11-04 13:11:58.673314: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:11:58.676732: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-04 13:11:58.676755: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:11:58.678604: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-04 13:11:58.686116: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-04 13:11:58.738423: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:11:58.763338: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 90028 microseconds.

I1104 13:11:58.763645 10648 model_lifecycle.cc:693] successfully loaded '1_predicttensorflowtriton' version 1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1104 13:12:37.893650 10648 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 5_predicttensorflowtriton (version 1)

2022-11-04 13:12:37.895285: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:12:37.911124: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-04 13:12:37.911168: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:12:37.913317: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-04 13:12:37.943809: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-04 13:12:38.080064: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:12:38.131426: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 236152 microseconds.

I1104 13:12:38.148345 10648 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_queryfeast (GPU device 0)

I1104 13:12:44.501902 10648 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 2_queryfaiss (GPU device 0)

I1104 13:12:44.502254 10648 model_lifecycle.cc:693] successfully loaded '0_queryfeast' version 1

I1104 13:12:50.728289 10648 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 3_queryfeast (GPU device 0)

I1104 13:12:50.728481 10648 model_lifecycle.cc:693] successfully loaded '2_queryfaiss' version 1

I1104 13:12:57.058057 10648 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 4_unrollfeatures (GPU device 0)

I1104 13:12:57.060173 10648 model_lifecycle.cc:693] successfully loaded '3_queryfeast' version 1

I1104 13:13:02.971765 10648 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 6_softmaxsampling (GPU device 0)

I1104 13:13:02.972301 10648 model_lifecycle.cc:693] successfully loaded '4_unrollfeatures' version 1

I1104 13:13:08.936432 10648 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 5_predicttensorflowtriton (GPU device 0)

I1104 13:13:08.936715 10648 model_lifecycle.cc:693] successfully loaded '6_softmaxsampling' version 1

2022-11-04 13:13:08.937395: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:13:08.956516: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-04 13:13:08.956564: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:13:08.958753: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-04 13:13:08.983452: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-04 13:13:09.133364: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:13:09.187330: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 249950 microseconds.

I1104 13:13:09.187628 10648 model_lifecycle.cc:693] successfully loaded '5_predicttensorflowtriton' version 1

I1104 13:13:09.190522 10648 model_lifecycle.cc:459] loading: ensemble_model:1

I1104 13:13:09.191075 10648 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1

I1104 13:13:09.191220 10648 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+
I1104 13:13:09.191316 10648 server.cc:588]

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend    | Path                                                            | Config                                                                                                                                                                      |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| tensorflow | /opt/tritonserver/backends/tensorflow2/libtriton_tensorflow2.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}} |

| python     | /opt/tritonserver/backends/python/libtriton_python.so           | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}}               |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1104 13:13:09.191432 10648 server.cc:631]

+---------------------------+---------+--------+

| Model                     | Version | Status |

+---------------------------+---------+--------+

| 0_queryfeast              | 1       | READY  |

| 1_predicttensorflowtriton | 1       | READY  |

| 2_queryfaiss              | 1       | READY  |

| 3_queryfeast              | 1       | READY  |

| 4_unrollfeatures          | 1       | READY  |

| 5_predicttensorflowtriton | 1       | READY  |

| 6_softmaxsampling         | 1       | READY  |

| ensemble_model            | 1       | READY  |

+---------------------------+---------+--------+
I1104 13:13:09.254380 10648 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1104 13:13:09.255224 10648 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/examples/poc_ensemble                                                                                                                                                                   |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1104 13:13:09.256519 10648 grpc_server.cc:4610] Started GRPCInferenceService at localhost:8001

I1104 13:13:09.257012 10648 http_server.cc:3316] Started HTTPService at 0.0.0.0:8000

I1104 13:13:09.298249 10648 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002

W1104 13:13:10.272213 10648 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1104 13:13:11.272412 10648 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1104 13:13:12.291028 10648 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

I1104 13:13:56.388071 10648 server.cc:262] Waiting for in-flight requests to complete.

I1104 13:13:56.388110 10648 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1104 13:13:56.388345 10648 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1

I1104 13:13:56.388440 10648 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1104 13:13:56.388570 10648 server.cc:293] All models are stopped, unloading models

I1104 13:13:56.388611 10648 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state

I1104 13:13:56.388609 10648 server.cc:300] Timeout 30: Found 7 live models and 0 in-flight non-inference requests

I1104 13:13:56.388671 10648 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1104 13:13:56.388984 10648 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state

I1104 13:13:56.401901 10648 model_lifecycle.cc:578] successfully unloaded '1_predicttensorflowtriton' version 1

I1104 13:13:56.411897 10648 model_lifecycle.cc:578] successfully unloaded '5_predicttensorflowtriton' version 1

I1104 13:13:57.388841 10648 server.cc:300] Timeout 29: Found 5 live models and 0 in-flight non-inference requests

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1104 13:13:58.353005 10648 model_lifecycle.cc:578] successfully unloaded '6_softmaxsampling' version 1

I1104 13:13:58.389450 10648 server.cc:300] Timeout 28: Found 4 live models and 0 in-flight non-inference requests

I1104 13:13:58.426679 10648 model_lifecycle.cc:578] successfully unloaded '4_unrollfeatures' version 1

I1104 13:13:58.518559 10648 model_lifecycle.cc:578] successfully unloaded '2_queryfaiss' version 1

I1104 13:13:59.389670 10648 server.cc:300] Timeout 27: Found 2 live models and 0 in-flight non-inference requests

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

I1104 13:14:00.389995 10648 server.cc:300] Timeout 26: Found 2 live models and 0 in-flight non-inference requests

------------------------------ Captured log call -------------------------------

ERROR    traitlets:client.py:795 Timeout waiting for execute reply (120s).
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       355    210    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                      100     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     15    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        54     19    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       125     23    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     24    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    136    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    22      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      2    83%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     26    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     93    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          346    157    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          603    344    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     20    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     78    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     16    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      1    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    113    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49      8    84%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     55    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                114     55    52%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     169     89    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          241    107    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 244    108    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          426    151    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              170    117    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                104     69    34%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              280     71    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                60     36    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39      4    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          458    154    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    245    127    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               708    310    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              122     89    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               97     62    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    108    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     22    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       435    346    20%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         158     75    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          85     12    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    141    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31     10    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     31    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     39    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     34    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       209     52    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 46     20    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     23      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                26      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                              9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py                               24      8    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py                                77      3    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py                               126     26    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             79      7    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py                       45     28    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py                     51      1    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           67     18    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py                      50      1    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py                             72     26    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py                         1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py                    11      2    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py                  1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py           61     11    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py                 140     47    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              47     12    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py                           87     51    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               268    210    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     13    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py                            22     20     9%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py                               113     14    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py                             37     29    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py                             10      6    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py                          32     13    59%
TOTAL                                                                                                 14546   6625    54%
=================== 1 failed, 2 passed in 356.92s (0:05:56) ====================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://github.com/gitapi/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins17226043567743113438.sh

mikemckiernan · 2022-11-04T13:43:46Z

rerun tests

nvidia-merlin-bot · 2022-11-04T13:50:43Z

Click to view CI Results

GitHub pull request #721 of commit 2922647005d7f8883a887a209f1cb577d9b23874, no merge conflicts.
Running as SYSTEM
Setting status of 2922647005d7f8883a887a209f1cb577d9b23874 to PENDING with url https://10.20.13.93:8080/job/merlin_merlin/567/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/721/*:refs/remotes/origin/pr/721/* # timeout=10
 > git rev-parse 2922647005d7f8883a887a209f1cb577d9b23874^{commit} # timeout=10
Checking out Revision 2922647005d7f8883a887a209f1cb577d9b23874 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 2922647005d7f8883a887a209f1cb577d9b23874 # timeout=10
Commit message: "docs: Remove SM from the support matrix"
 > git rev-list --no-walk 2922647005d7f8883a887a209f1cb577d9b23874 # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins16230830522194375041.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.2,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.2,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.8.2,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.42,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.0,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3002368347'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-8grij_ob
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-8grij_ob
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit d2737da6cc7e98ec8723731a7cc998aa143e6758
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+10.gd2737da) (2.22.0)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+10.gd2737da) (1.1.1)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+10.gd2737da) (0.3.0+12.g78ecddd)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (3.19.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.55.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.10.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (4.64.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (7.0.0)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.3.0)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.3.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.4.3)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.2.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.5.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (5.4.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (8.1.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (65.4.1)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 17.9 MB/s eta 0:00:00
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+10.gd2737da-py3-none-any.whl size=91051 sha256=98126a8089f33f314a7212d5cfdebd26edc34f0de1690e93f220731e2df526c2
  Stored in directory: /tmp/pip-ephem-wheel-cache-0fd0o3n4/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+10.gd2737da
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/models.git

Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-r_qy4bmx

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-r_qy4bmx

Resolved https://github.com/NVIDIA-Merlin/models.git to commit ada3874d663592d215150ba3eadd04b075818b4f

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+29.gada3874d) (0.3.0+12.g78ecddd)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (3.19.5)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.3.5)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.2.5)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (0.55.1)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (21.3)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.10.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (4.64.1)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (7.0.0)

Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2022.3.0)

Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2022.3.0)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.2.0)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (0.4.3)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2.2.0)

Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2022.5.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (0.12.0)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (5.4.1)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (5.8.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (3.1.2)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2.4.0)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.7.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (6.1)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.0.4)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2.0.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (8.1.3)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (65.4.1)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (0.38.1)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.20.3)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (3.0.9)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2022.2.1)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2.8.2)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.0.1)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (4.1.0)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (6.0.2)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (4.0.0)

Building wheels for collected packages: merlin-models

Building wheel for merlin-models (pyproject.toml): started

Building wheel for merlin-models (pyproject.toml): finished with status 'done'

Created wheel for merlin-models: filename=merlin_models-0.9.0+29.gada3874d-py3-none-any.whl size=356754 sha256=4037784b0403d81aa532aa12670fe29a9d49cfe3a68dd563804258d204a6656a

Stored in directory: /tmp/pip-ephem-wheel-cache-i97fbeuh/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5

Successfully built merlin-models

Installing collected packages: merlin-models

Attempting uninstall: merlin-models

Found existing installation: merlin-models 0.7.0+11.g280956aa4

Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu

Can't uninstall 'merlin-models'. No files were found to uninstall.

Successfully installed merlin-models-0.9.0+29.gada3874d
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git

Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-z0_up6kx

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-z0_up6kx

Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit 8e7edbafd3006f56e73efdc0c01c4445ab57d028

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+3.g8e7edbaf) (1.8.1)

Collecting merlin-dataloader>=0.0.2

Downloading merlin-dataloader-0.0.2.tar.gz (44 kB)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.1/44.1 kB 1.6 MB/s eta 0:00:00

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+3.g8e7edbaf) (0.3.0+12.g78ecddd)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (3.19.5)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.3.5)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.2.5)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (0.55.1)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (21.3)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.10.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (4.64.1)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (7.0.0)

Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2022.3.0)

Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2022.3.0)

Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+3.g8e7edbaf) (1.20.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.2.0)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (0.4.3)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2.2.0)

Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2022.5.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (0.12.0)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (5.4.1)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (5.8.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (3.1.2)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2.4.0)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.7.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (6.1)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.0.4)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2.0.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (8.1.3)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (65.4.1)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (0.38.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (3.0.9)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2022.2.1)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2.8.2)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.0.1)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (4.1.0)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (6.0.2)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (4.0.0)

Building wheels for collected packages: nvtabular, merlin-dataloader

Building wheel for nvtabular (pyproject.toml): started

Building wheel for nvtabular (pyproject.toml): finished with status 'done'

Created wheel for nvtabular: filename=nvtabular-1.6.0+3.g8e7edbaf-cp38-cp38-linux_x86_64.whl size=257744 sha256=e2e58b00c23c417ad66cb0dfa5f965d936b0c7d43000617199463a9a50446f62

Stored in directory: /tmp/pip-ephem-wheel-cache-2ksbcprn/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc

Building wheel for merlin-dataloader (pyproject.toml): started

Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'

Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.2-py3-none-any.whl size=29205 sha256=b26aa3f7fba5df158125ff463b4f1984254f5c5c6a9cf045444a6942d8910cb2

Stored in directory: /tmp/pip-ephem-wheel-cache-2ksbcprn/wheels/d5/ce/8c/31476c01e0b5a2278110fe2092bdd911efb0e5b83d0d3550ca

Successfully built nvtabular merlin-dataloader

Installing collected packages: merlin-dataloader, nvtabular

Attempting uninstall: nvtabular

Found existing installation: nvtabular 1.1.1

Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu

Can't uninstall 'nvtabular'. No files were found to uninstall.

Successfully installed merlin-dataloader-0.0.2 nvtabular-1.6.0+3.g8e7edbaf
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/core.git

Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-c8ghpiu5

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-c8ghpiu5

Resolved https://github.com/NVIDIA-Merlin/core.git to commit 563be4bf5ef675940d5fff2b5e4666424a7f7947

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (2022.3.0)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (3.19.5)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (1.2.5)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (1.3.5)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (2022.3.0)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (0.55.1)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (1.10.0)

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (2022.5.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (4.64.1)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (7.0.0)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (21.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (1.2.0)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (0.4.3)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (0.12.0)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (5.4.1)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (5.8.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (3.1.2)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.4.0)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.7.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (6.1)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.0.4)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.0.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (8.1.3)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+5.g563be4b) (65.4.1)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+5.g563be4b) (0.38.1)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+5.g563be4b) (1.20.3)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+5.g563be4b) (3.0.9)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+5.g563be4b) (2022.2.1)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+5.g563be4b) (2.8.2)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+5.g563be4b) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+5.g563be4b) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+5.g563be4b) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.0.1)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (4.1.0)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (6.0.2)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (4.0.0)

Building wheels for collected packages: merlin-core

Building wheel for merlin-core (pyproject.toml): started

Building wheel for merlin-core (pyproject.toml): finished with status 'done'

Created wheel for merlin-core: filename=merlin_core-0.8.0+5.g563be4b-py3-none-any.whl size=118363 sha256=bc3b3f336b29a8e4f168e63e0b4678f06ba6ce61785438487135024a2813ed05

Stored in directory: /tmp/pip-ephem-wheel-cache-bi7i7p9x/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84

Successfully built merlin-core

Installing collected packages: merlin-core

Attempting uninstall: merlin-core

Found existing installation: merlin-core 0.3.0+12.g78ecddd

Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu

Can't uninstall 'merlin-core'. No files were found to uninstall.

Successfully installed merlin-core-0.8.0+5.g563be4b
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin

plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0

collected 3 items
tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py F      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py .               [100%]
=================================== FAILURES ===================================

__________________________________ test_func ___________________________________
self = <testbook.client.TestbookNotebookClient object at 0x7f3ae2fd80a0>

cell = [55], kwargs = {}, cell_indexes = [55], executed_cells = [], idx = 55
def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:


          cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)


../../../.local/lib/python3.8/site-packages/testbook/client.py:133:

args = (<testbook.client.TestbookNotebookClient object at 0x7f3ae2fd80a0>, {'id': '67c0141e', 'cell_type': 'code', 'metadata'...[38;5;241m.\x1b[39mis_server_ready()\n', '\x1b[0;31mRuntimeError\x1b[0m: Tritonserver failed to start (ret=1)']}]}, 55)

kwargs = {}
def wrapped(*args, **kwargs):


  return just_run(coro(*args, **kwargs))


../../../.local/lib/python3.8/site-packages/nbclient/util.py:84:

coro = <coroutine object NotebookClient.async_execute_cell at 0x7f3ae2472e40>
def just_run(coro: Awaitable) -> Any:
    """Make the coroutine run, even if there is an event loop running (using nest_asyncio)"""
    # original from vaex/asyncio.py
    loop = asyncio._get_running_loop()
    if loop is None:
        had_running_loop = False
        try:
            loop = asyncio.get_event_loop()
        except RuntimeError:
            # we can still get 'There is no current event loop in ...'
            loop = asyncio.new_event_loop()
            asyncio.set_event_loop(loop)
    else:
        had_running_loop = True
    if had_running_loop:
        # if there is a running loop, we patch using nest_asyncio
        # to have reentrant event loops
        check_ipython()
        import nest_asyncio

        nest_asyncio.apply()
        check_patch_tornado()


  return loop.run_until_complete(coro)


../../../.local/lib/python3.8/site-packages/nbclient/util.py:62:

self = <_UnixSelectorEventLoop running=False closed=False debug=False>

future = <Task finished name='Task-377' coro=<NotebookClient.async_execute_cell() done, defined at /var/jenkins_home/.local/lib...0;31mRuntimeError\x1b[0m: Tritonserver failed to start (ret=1)\nRuntimeError: Tritonserver failed to start (ret=1)\n')>
def run_until_complete(self, future):
    """Run until the Future is done.

    If the argument is a coroutine, it is wrapped in a Task.

    WARNING: It would be disastrous to call run_until_complete()
    with the same coroutine twice -- it would wrap it in two
    different Tasks and that can't be good.

    Return the Future's result, or raise its exception.
    """
    self._check_closed()
    self._check_running()

    new_task = not futures.isfuture(future)
    future = tasks.ensure_future(future, loop=self)
    if new_task:
        # An exception is raised if the future didn't complete, so there
        # is no need to log the "destroy pending task" message
        future._log_destroy_pending = False

    future.add_done_callback(_run_until_complete_cb)
    try:
        self.run_forever()
    except:
        if new_task and future.done() and not future.cancelled():
            # The coroutine raised a BaseException. Consume the exception
            # to not log a warning, the caller doesn't have access to the
            # local task.
            future.exception()
        raise
    finally:
        future.remove_done_callback(_run_until_complete_cb)
    if not future.done():
        raise RuntimeError('Event loop stopped before Future completed.')


  return future.result()


/usr/lib/python3.8/asyncio/base_events.py:616:

self = <testbook.client.TestbookNotebookClient object at 0x7f3ae2fd80a0>

cell = {'id': '67c0141e', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-04T13:47:36.392986Z',...t\x1b[38;5;241m.\x1b[39mis_server_ready()\n', '\x1b[0;31mRuntimeError\x1b[0m: Tritonserver failed to start (ret=1)']}]}

cell_index = 55, execution_count = None, store_history = True
async def async_execute_cell(
    self,
    cell: NotebookNode,
    cell_index: int,
    execution_count: t.Optional[int] = None,
    store_history: bool = True,
) -> NotebookNode:
    """
    Executes a single code cell.

    To execute all cells see :meth:`execute`.

    Parameters
    ----------
    cell : nbformat.NotebookNode
        The cell which is currently being processed.
    cell_index : int
        The position of the cell within the notebook object.
    execution_count : int
        The execution count to be assigned to the cell (default: Use kernel response)
    store_history : bool
        Determines if history should be stored in the kernel (default: False).
        Specific to ipython kernels, which can store command histories.

    Returns
    -------
    output : dict
        The execution output payload (or None for no output).

    Raises
    ------
    CellExecutionError
        If execution failed and should raise an exception, this will be raised
        with defaults about the failure.

    Returns
    -------
    cell : NotebookNode
        The cell which was just processed.
    """
    assert self.kc is not None

    await run_hook(self.on_cell_start, cell=cell, cell_index=cell_index)

    if cell.cell_type != 'code' or not cell.source.strip():
        self.log.debug("Skipping non-executing cell %s", cell_index)
        return cell

    if self.skip_cells_with_tag in cell.metadata.get("tags", []):
        self.log.debug("Skipping tagged cell %s", cell_index)
        return cell

    if self.record_timing:  # clear execution metadata prior to execution
        cell['metadata']['execution'] = {}

    self.log.debug("Executing cell:\n%s", cell.source)

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors or "raises-exception" in cell.metadata.get("tags", [])
    )

    await run_hook(self.on_cell_execute, cell=cell, cell_index=cell_index)
    parent_msg_id = await ensure_async(
        self.kc.execute(
            cell.source, store_history=store_history, stop_on_error=not cell_allows_errors
        )
    )
    await run_hook(self.on_cell_complete, cell=cell, cell_index=cell_index)
    # We launched a code cell to execute
    self.code_cells_executed += 1
    exec_timeout = self._get_timeout(cell)

    cell.outputs = []
    self.clear_before_next_output = False

    task_poll_kernel_alive = asyncio.ensure_future(self._async_poll_kernel_alive())
    task_poll_output_msg = asyncio.ensure_future(
        self._async_poll_output_msg(parent_msg_id, cell, cell_index)
    )
    self.task_poll_for_reply = asyncio.ensure_future(
        self._async_poll_for_reply(
            parent_msg_id, cell, exec_timeout, task_poll_output_msg, task_poll_kernel_alive
        )
    )
    try:
        exec_reply = await self.task_poll_for_reply
    except asyncio.CancelledError:
        # can only be cancelled by task_poll_kernel_alive when the kernel is dead
        task_poll_output_msg.cancel()
        raise DeadKernelError("Kernel died")
    except Exception as e:
        # Best effort to cancel request if it hasn't been resolved
        try:
            # Check if the task_poll_output is doing the raising for us
            if not isinstance(e, CellControlSignal):
                task_poll_output_msg.cancel()
        finally:
            raise

    if execution_count:
        cell['execution_count'] = execution_count


  await self._check_raise_for_error(cell, cell_index, exec_reply)


../../../.local/lib/python3.8/site-packages/nbclient/client.py:965:

self = <testbook.client.TestbookNotebookClient object at 0x7f3ae2fd80a0>

cell = {'id': '67c0141e', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-04T13:47:36.392986Z',...t\x1b[38;5;241m.\x1b[39mis_server_ready()\n', '\x1b[0;31mRuntimeError\x1b[0m: Tritonserver failed to start (ret=1)']}]}

cell_index = 55

exec_reply = {'buffers': [], 'content': {'ename': 'RuntimeError', 'engine_info': {'engine_id': -1, 'engine_uuid': 'ef57a0de-abf1-49...e, 'engine': 'ef57a0de-abf1-49dd-a127-458c31fac505', 'started': '2022-11-04T13:47:36.393320Z', 'status': 'error'}, ...}
async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:


      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)


E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E

E           import shutil

E           from merlin.core.dispatch import get_lib

E           from merlin.models.loader.tf_utils import configure_tensorflow

E           configure_tensorflow()

E           df_lib = get_lib()

E           batch = df_lib.read_parquet(

E               os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),

E               num_rows=1,

E               columns=["user_id_raw"],

E           )

E           from merlin.systems.triton.utils import run_ensemble_on_tritonserver

E           response = run_ensemble_on_tritonserver(

E               "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"

E           )

E           response = [x.tolist()[0] for x in response["ordered_ids"]]

E           shutil.rmtree("/tmp/examples/", ignore_errors=True)

E

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mRuntimeError�[0m                              Traceback (most recent call last)

E           Cell �[0;32mIn [32], line 12�[0m

E           �[1;32m      6�[0m batch �[38;5;241m=�[39m df_lib�[38;5;241m.�[39mread_parquet(

E           �[1;32m      7�[0m     os�[38;5;241m.�[39mpath�[38;5;241m.�[39mjoin(�[38;5;124m"�[39m�[38;5;124m/tmp/data/processed_nvt/�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mtrain�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mpart_0.parquet�[39m�[38;5;124m"�[39m),

E           �[1;32m      8�[0m     num_rows�[38;5;241m=�[39m�[38;5;241m1�[39m,

E           �[1;32m      9�[0m     columns�[38;5;241m=�[39m[�[38;5;124m"�[39m�[38;5;124muser_id_raw�[39m�[38;5;124m"�[39m],

E           �[1;32m     10�[0m )

E           �[1;32m     11�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01msystems�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtriton�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mutils�[39;00m �[38;5;28;01mimport�[39;00m run_ensemble_on_tritonserver

E           �[0;32m---> 12�[0m response �[38;5;241m=�[39m �[43mrun_ensemble_on_tritonserver�[49m�[43m(�[49m

E           �[1;32m     13�[0m �[43m    �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m/tmp/examples/poc_ensemble�[39;49m�[38;5;124;43m"�[39;49m�[43m,�[49m�[43m �[49m�[43mensemble�[49m�[38;5;241;43m.�[39;49m�[43mgraph�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[43m,�[49m�[43m  �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43mensemble_model�[39;49m�[38;5;124;43m"�[39;49m

E           �[1;32m     14�[0m �[43m)�[49m

E           �[1;32m     15�[0m response �[38;5;241m=�[39m [x�[38;5;241m.�[39mtolist()[�[38;5;241m0�[39m] �[38;5;28;01mfor�[39;00m x �[38;5;129;01min�[39;00m response[�[38;5;124m"�[39m�[38;5;124mordered_ids�[39m�[38;5;124m"�[39m]]

E           �[1;32m     16�[0m shutil�[38;5;241m.�[39mrmtree(�[38;5;124m"�[39m�[38;5;124m/tmp/examples/�[39m�[38;5;124m"�[39m, ignore_errors�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)

E

E           File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:138�[0m, in �[0;36mrun_ensemble_on_tritonserver�[0;34m(tmpdir, schema, df, output_columns, model_name)�[0m

E           �[1;32m    115�[0m �[38;5;124;03m"""Starts up a Triton server instance, loads up the ensemble model,�[39;00m

E           �[1;32m    116�[0m �[38;5;124;03mprepares the inference request and returns the unparsed inference�[39;00m

E           �[1;32m    117�[0m �[38;5;124;03mresponse.�[39;00m

E           �[0;32m   (...)�[0m

E           �[1;32m    135�[0m �[38;5;124;03m    the results of the prediction, parsed by output column name.�[39;00m

E           �[1;32m    136�[0m �[38;5;124;03m"""�[39;00m

E           �[1;32m    137�[0m response �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m

E           �[0;32m--> 138�[0m �[38;5;28;01mwith�[39;00m run_triton_server(tmpdir) �[38;5;28;01mas�[39;00m client:

E           �[1;32m    139�[0m     response �[38;5;241m=�[39m send_triton_request(

E           �[1;32m    140�[0m         schema, df, output_columns, client�[38;5;241m=�[39mclient, triton_model�[38;5;241m=�[39mmodel_name

E           �[1;32m    141�[0m     )

E           �[1;32m    143�[0m �[38;5;28;01mreturn�[39;00m response

E

E           File �[0;32m/usr/lib/python3.8/contextlib.py:113�[0m, in �[0;36m_GeneratorContextManager.__enter__�[0;34m(self)�[0m

E           �[1;32m    111�[0m �[38;5;28;01mdel�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39margs, �[38;5;28mself�[39m�[38;5;241m.�[39mkwds, �[38;5;28mself�[39m�[38;5;241m.�[39mfunc

E           �[1;32m    112�[0m �[38;5;28;01mtry�[39;00m:

E           �[0;32m--> 113�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28;43mnext�[39;49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mgen�[49m�[43m)�[49m

E           �[1;32m    114�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mStopIteration�[39;00m:

E           �[1;32m    115�[0m     �[38;5;28;01mraise�[39;00m �[38;5;167;01mRuntimeError�[39;00m(�[38;5;124m"�[39m�[38;5;124mgenerator didn�[39m�[38;5;124m'�[39m�[38;5;124mt yield�[39m�[38;5;124m"�[39m) �[38;5;28;01mfrom�[39;00m �[38;5;28mNone�[39m

E

E           File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:81�[0m, in �[0;36mrun_triton_server�[0;34m(model_repository, grpc_host, grpc_port, backend_config)�[0m

E           �[1;32m     79�[0m �[38;5;28;01mif�[39;00m process�[38;5;241m.�[39mpoll() �[38;5;129;01mis�[39;00m �[38;5;129;01mnot�[39;00m �[38;5;28;01mNone�[39;00m:

E           �[1;32m     80�[0m     retcode �[38;5;241m=�[39m process�[38;5;241m.�[39mreturncode

E           �[0;32m---> 81�[0m     �[38;5;28;01mraise�[39;00m �[38;5;167;01mRuntimeError�[39;00m(�[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mTritonserver failed to start (ret=�[39m�[38;5;132;01m{�[39;00mretcode�[38;5;132;01m}�[39;00m�[38;5;124m)�[39m�[38;5;124m"�[39m)

E           �[1;32m     83�[0m �[38;5;28;01mtry�[39;00m:

E           �[1;32m     84�[0m     ready �[38;5;241m=�[39m client�[38;5;241m.�[39mis_server_ready()

E

E           �[0;31mRuntimeError�[0m: Tritonserver failed to start (ret=1)

E           RuntimeError: Tritonserver failed to start (ret=1)
../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError
During handling of the above exception, another exception occurred:
def test_func():
    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "01-Building-Recommender-Systems-with-Merlin.ipynb",
        execute=False,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["NUM_ROWS"] = "100000"
            os.system("mkdir -p /tmp/examples")
            os.environ["BASE_DIR"] = "/tmp/examples/"
            """
        )
        tb1.execute()
        assert os.path.isdir("/tmp/examples/dlrm")
        assert os.path.isdir("/tmp/examples/feature_repo")
        assert os.path.isdir("/tmp/examples/query_tower")
        assert os.path.isfile("/tmp/examples/item_embeddings.parquet")
        assert os.path.isfile("/tmp/examples/feature_repo/user_features.py")
        assert os.path.isfile("/tmp/examples/feature_repo/item_features.py")

    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "02-Deploying-multi-stage-RecSys-with-Merlin-Systems.ipynb",
        execute=False,
        timeout=120,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["BASE_DIR"] = "/tmp/examples/"
            os.environ["topk_retrieval"] = "20"
            """
        )
        NUM_OF_CELLS = len(tb2.cells)
        tb2.execute_cell(list(range(0, NUM_OF_CELLS - 3)))
        top_k = tb2.ref("top_k")
        outputs = tb2.ref("outputs")
        assert outputs[0] == "ordered_ids"


      tb2.inject(


            """
            import shutil
            from merlin.core.dispatch import get_lib
            from merlin.models.loader.tf_utils import configure_tensorflow
            configure_tensorflow()
            df_lib = get_lib()
            batch = df_lib.read_parquet(
                os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),
                num_rows=1,
                columns=["user_id_raw"],
            )
            from merlin.systems.triton.utils import run_ensemble_on_tritonserver
            response = run_ensemble_on_tritonserver(
                "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"
            )
            response = [x.tolist()[0] for x in response["ordered_ids"]]
            shutil.rmtree("/tmp/examples/", ignore_errors=True)
            """
        )

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:61:

../../../.local/lib/python3.8/site-packages/testbook/client.py:237: in inject

cell = TestbookNode(self.execute_cell(inject_idx)) if run else TestbookNode(code_cell)

self = <testbook.client.TestbookNotebookClient object at 0x7f3ae2fd80a0>

cell = [55], kwargs = {}, cell_indexes = [55], executed_cells = [], idx = 55
def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:
            cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)
        except CellExecutionError as ce:


          raise TestbookRuntimeError(ce.evalue, ce, self._get_error_class(ce.ename))


E               testbook.exceptions.TestbookRuntimeError: An error occurred while executing the following cell:

E               ------------------

E

E               import shutil

E               from merlin.core.dispatch import get_lib

E               from merlin.models.loader.tf_utils import configure_tensorflow

E               configure_tensorflow()

E               df_lib = get_lib()

E               batch = df_lib.read_parquet(

E                   os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),

E                   num_rows=1,

E                   columns=["user_id_raw"],

E               )

E               from merlin.systems.triton.utils import run_ensemble_on_tritonserver

E               response = run_ensemble_on_tritonserver(

E                   "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"

E               )

E               response = [x.tolist()[0] for x in response["ordered_ids"]]

E               shutil.rmtree("/tmp/examples/", ignore_errors=True)

E

E               ------------------

E

E               �[0;31m---------------------------------------------------------------------------�[0m

E               �[0;31mRuntimeError�[0m                              Traceback (most recent call last)

E               Cell �[0;32mIn [32], line 12�[0m

E               �[1;32m      6�[0m batch �[38;5;241m=�[39m df_lib�[38;5;241m.�[39mread_parquet(

E               �[1;32m      7�[0m     os�[38;5;241m.�[39mpath�[38;5;241m.�[39mjoin(�[38;5;124m"�[39m�[38;5;124m/tmp/data/processed_nvt/�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mtrain�[39m�[38;5;124m"�[39m, �[38;5;124m"�[39m�[38;5;124mpart_0.parquet�[39m�[38;5;124m"�[39m),

E               �[1;32m      8�[0m     num_rows�[38;5;241m=�[39m�[38;5;241m1�[39m,

E               �[1;32m      9�[0m     columns�[38;5;241m=�[39m[�[38;5;124m"�[39m�[38;5;124muser_id_raw�[39m�[38;5;124m"�[39m],

E               �[1;32m     10�[0m )

E               �[1;32m     11�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01msystems�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtriton�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mutils�[39;00m �[38;5;28;01mimport�[39;00m run_ensemble_on_tritonserver

E               �[0;32m---> 12�[0m response �[38;5;241m=�[39m �[43mrun_ensemble_on_tritonserver�[49m�[43m(�[49m

E               �[1;32m     13�[0m �[43m    �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m/tmp/examples/poc_ensemble�[39;49m�[38;5;124;43m"�[39;49m�[43m,�[49m�[43m �[49m�[43mensemble�[49m�[38;5;241;43m.�[39;49m�[43mgraph�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[43m,�[49m�[43m �[49m�[43moutputs�[49m�[43m,�[49m�[43m  �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43mensemble_model�[39;49m�[38;5;124;43m"�[39;49m

E               �[1;32m     14�[0m �[43m)�[49m

E               �[1;32m     15�[0m response �[38;5;241m=�[39m [x�[38;5;241m.�[39mtolist()[�[38;5;241m0�[39m] �[38;5;28;01mfor�[39;00m x �[38;5;129;01min�[39;00m response[�[38;5;124m"�[39m�[38;5;124mordered_ids�[39m�[38;5;124m"�[39m]]

E               �[1;32m     16�[0m shutil�[38;5;241m.�[39mrmtree(�[38;5;124m"�[39m�[38;5;124m/tmp/examples/�[39m�[38;5;124m"�[39m, ignore_errors�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)

E

E               File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:138�[0m, in �[0;36mrun_ensemble_on_tritonserver�[0;34m(tmpdir, schema, df, output_columns, model_name)�[0m

E               �[1;32m    115�[0m �[38;5;124;03m"""Starts up a Triton server instance, loads up the ensemble model,�[39;00m

E               �[1;32m    116�[0m �[38;5;124;03mprepares the inference request and returns the unparsed inference�[39;00m

E               �[1;32m    117�[0m �[38;5;124;03mresponse.�[39;00m

E               �[0;32m   (...)�[0m

E               �[1;32m    135�[0m �[38;5;124;03m    the results of the prediction, parsed by output column name.�[39;00m

E               �[1;32m    136�[0m �[38;5;124;03m"""�[39;00m

E               �[1;32m    137�[0m response �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m

E               �[0;32m--> 138�[0m �[38;5;28;01mwith�[39;00m run_triton_server(tmpdir) �[38;5;28;01mas�[39;00m client:

E               �[1;32m    139�[0m     response �[38;5;241m=�[39m send_triton_request(

E               �[1;32m    140�[0m         schema, df, output_columns, client�[38;5;241m=�[39mclient, triton_model�[38;5;241m=�[39mmodel_name

E               �[1;32m    141�[0m     )

E               �[1;32m    143�[0m �[38;5;28;01mreturn�[39;00m response

E

E               File �[0;32m/usr/lib/python3.8/contextlib.py:113�[0m, in �[0;36m_GeneratorContextManager.__enter__�[0;34m(self)�[0m

E               �[1;32m    111�[0m �[38;5;28;01mdel�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39margs, �[38;5;28mself�[39m�[38;5;241m.�[39mkwds, �[38;5;28mself�[39m�[38;5;241m.�[39mfunc

E               �[1;32m    112�[0m �[38;5;28;01mtry�[39;00m:

E               �[0;32m--> 113�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28;43mnext�[39;49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mgen�[49m�[43m)�[49m

E               �[1;32m    114�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mStopIteration�[39;00m:

E               �[1;32m    115�[0m     �[38;5;28;01mraise�[39;00m �[38;5;167;01mRuntimeError�[39;00m(�[38;5;124m"�[39m�[38;5;124mgenerator didn�[39m�[38;5;124m'�[39m�[38;5;124mt yield�[39m�[38;5;124m"�[39m) �[38;5;28;01mfrom�[39;00m �[38;5;28mNone�[39m

E

E               File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py:81�[0m, in �[0;36mrun_triton_server�[0;34m(model_repository, grpc_host, grpc_port, backend_config)�[0m

E               �[1;32m     79�[0m �[38;5;28;01mif�[39;00m process�[38;5;241m.�[39mpoll() �[38;5;129;01mis�[39;00m �[38;5;129;01mnot�[39;00m �[38;5;28;01mNone�[39;00m:

E               �[1;32m     80�[0m     retcode �[38;5;241m=�[39m process�[38;5;241m.�[39mreturncode

E               �[0;32m---> 81�[0m     �[38;5;28;01mraise�[39;00m �[38;5;167;01mRuntimeError�[39;00m(�[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mTritonserver failed to start (ret=�[39m�[38;5;132;01m{�[39;00mretcode�[38;5;132;01m}�[39;00m�[38;5;124m)�[39m�[38;5;124m"�[39m)

E               �[1;32m     83�[0m �[38;5;28;01mtry�[39;00m:

E               �[1;32m     84�[0m     ready �[38;5;241m=�[39m client�[38;5;241m.�[39mis_server_ready()

E

E               �[0;31mRuntimeError�[0m: Tritonserver failed to start (ret=1)

E               RuntimeError: Tritonserver failed to start (ret=1)
../../../.local/lib/python3.8/site-packages/testbook/client.py:135: TestbookRuntimeError

----------------------------- Captured stderr call -----------------------------

2022-11-04 13:45:42.901758: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-04 13:45:46.955262: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-04 13:45:46.956144: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 13875 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-11-04 13:45:46.956752: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 13875 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-11-04 13:45:46.957352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 13875 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

WARNING clustering 449 points to 32 centroids: please provide at least 1248 training points

2022-11-04 13:47:29.188467: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-04 13:47:33.682615: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-04 13:47:33.683294: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14500 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-11-04 13:47:33.683850: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14500 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-11-04 13:47:33.684521: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14500 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

2022-11-04 13:47:33.689392: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 7.95G (8534360064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

2022-11-04 13:47:33.690748: I tensorflow/stream_executor/cuda/cuda_driver.cc:739] failed to allocate 7.15G (7680923648 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

I1104 13:47:36.705115 19199 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f1afe000000' with size 268435456

I1104 13:47:36.705902 19199 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1104 13:47:36.714185 19199 model_lifecycle.cc:459] loading: 1_predicttensorflowtriton:1

I1104 13:47:36.714266 19199 model_lifecycle.cc:459] loading: 0_queryfeast:1

I1104 13:47:36.714299 19199 model_lifecycle.cc:459] loading: 2_queryfaiss:1

I1104 13:47:36.714369 19199 model_lifecycle.cc:459] loading: 3_queryfeast:1

I1104 13:47:36.714425 19199 model_lifecycle.cc:459] loading: 4_unrollfeatures:1

I1104 13:47:36.714489 19199 model_lifecycle.cc:459] loading: 5_predicttensorflowtriton:1

I1104 13:47:36.714528 19199 model_lifecycle.cc:459] loading: 6_softmaxsampling:1

I1104 13:47:37.038397 19199 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow

I1104 13:47:37.038436 19199 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10

I1104 13:47:37.038444 19199 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10

I1104 13:47:37.038451 19199 tensorflow.cc:2576] backend configuration:

{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}

I1104 13:47:37.038493 19199 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 1_predicttensorflowtriton (version 1)

2022-11-04 13:47:37.039149: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:47:37.043864: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-04 13:47:37.043896: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:47:37.043995: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-04 13:47:37.451399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6856 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-04 13:47:37.491626: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled

2022-11-04 13:47:37.493830: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-04 13:47:37.545869: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:47:37.570877: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 531740 microseconds.

I1104 13:47:37.580283 19199 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 1_predicttensorflowtriton (GPU device 0)

2022-11-04 13:47:37.580853: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:47:37.584388: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-04 13:47:37.584414: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:47:37.586399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6856 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-04 13:47:37.593747: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-04 13:47:37.647470: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:47:37.670859: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 90011 microseconds.

I1104 13:47:37.671144 19199 model_lifecycle.cc:693] successfully loaded '1_predicttensorflowtriton' version 1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

1104 13:48:03.834074 19519 pb_stub.cc:241] Failed to initialize Python stub for auto-complete: CUDARuntimeError: cudaErrorInitializationError: initialization error
At:

/usr/local/lib/python3.8/dist-packages/rmm/_cuda/gpu.py(101): getDeviceCount

/usr/local/lib/python3.8/dist-packages/cudf/utils/gpu_utils.py(55): validate_setup

/usr/local/lib/python3.8/dist-packages/cudf/init.py(5): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py(52): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py(22): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py(18): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py(18): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

(219): _call_with_frames_removed

(961): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py(26): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py(19): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

(219): _call_with_frames_removed

(961): _find_and_load_unlocked

(991): _find_and_load

/tmp/examples/poc_ensemble/6_softmaxsampling/1/model.py(35): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load
E1104 13:48:03.861799 19199 model_lifecycle.cc:596] failed to load '6_softmaxsampling' version 1: Internal: CUDARuntimeError: cudaErrorInitializationError: initialization error
At:

/usr/local/lib/python3.8/dist-packages/rmm/_cuda/gpu.py(101): getDeviceCount

/usr/local/lib/python3.8/dist-packages/cudf/utils/gpu_utils.py(55): validate_setup

/usr/local/lib/python3.8/dist-packages/cudf/init.py(5): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py(52): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py(22): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py(18): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py(18): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

(219): _call_with_frames_removed

(961): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py(26): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py(19): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

(219): _call_with_frames_removed

(961): _find_and_load_unlocked

(991): _find_and_load

/tmp/examples/poc_ensemble/6_softmaxsampling/1/model.py(35): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1104 13:48:11.726133 19199 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 5_predicttensorflowtriton (version 1)

2022-11-04 13:48:11.726997: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:48:11.747251: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-04 13:48:11.747317: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:48:11.750037: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6856 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-04 13:48:11.799255: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-04 13:48:11.940879: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:48:11.995043: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 268052 microseconds.

I1104 13:48:12.010594 19199 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_queryfeast (GPU device 0)

I1104 13:48:18.519717 19199 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 2_queryfaiss (GPU device 0)

I1104 13:48:18.520298 19199 model_lifecycle.cc:693] successfully loaded '0_queryfeast' version 1

I1104 13:48:24.778629 19199 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 3_queryfeast (GPU device 0)

I1104 13:48:24.779788 19199 model_lifecycle.cc:693] successfully loaded '2_queryfaiss' version 1

I1104 13:48:31.239461 19199 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 4_unrollfeatures (GPU device 0)

I1104 13:48:31.240255 19199 model_lifecycle.cc:693] successfully loaded '3_queryfeast' version 1

I1104 13:48:37.247937 19199 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 5_predicttensorflowtriton (GPU device 0)

I1104 13:48:37.248298 19199 model_lifecycle.cc:693] successfully loaded '4_unrollfeatures' version 1

2022-11-04 13:48:37.249809: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:48:37.265998: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-04 13:48:37.266048: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:48:37.268122: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6856 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-04 13:48:37.293497: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-04 13:48:37.442270: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-04 13:48:37.497140: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 247348 microseconds.

I1104 13:48:37.497477 19199 model_lifecycle.cc:693] successfully loaded '5_predicttensorflowtriton' version 1

E1104 13:48:37.497581 19199 model_repository_manager.cc:481] Invalid argument: ensemble 'ensemble_model' depends on '6_softmaxsampling' which has no loaded version

I1104 13:48:37.497718 19199 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+
I1104 13:48:37.497818 19199 server.cc:588]

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend    | Path                                                            | Config                                                                                                                                                                      |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| tensorflow | /opt/tritonserver/backends/tensorflow2/libtriton_tensorflow2.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}} |

| python     | /opt/tritonserver/backends/python/libtriton_python.so           | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}}               |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1104 13:48:37.498133 +---------------------------+ | Model +---------------------------+ | 0_queryfeast | 1_predicttensorflowtriton | 1 | 2_queryfaiss | 3_queryfeast | 4_unrollfeatures | 5_predicttensorflowtriton | 1 | 6_softmaxsampling | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | +---------------------------+

19199 server.cc:631]
 ---------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
 | Version | Status                                                                                                                                                 |
 ---------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
 | 1       | READY                                                                                                                                                  |
 | READY                                                                                                                                                  |
 | 1       | READY                                                                                                                                                  |
 | 1       | READY                                                                                                                                                  |
 | 1       | READY                                                                                                                                                  |
 | READY                                                                                                                                                  |
 | 1       | UNAVAILABLE: Internal: CUDARuntimeError: cudaErrorInitializationError: initialization error                                                            |
 |         |                                                                                                                                                        |
 |         | At:                                                                                                                                                    |
 |         |   /usr/local/lib/python3.8/dist-packages/rmm/_cuda/gpu.py(101): getDeviceCount                                                                         |
 |         |   /usr/local/lib/python3.8/dist-packages/cudf/utils/gpu_utils.py(55): validate_setup                                                                   |
 |         |   /usr/local/lib/python3.8/dist-packages/cudf/init.py(5):                                                                                  |
 |         |   (219): _call_with_frames_removed                                                                                        |
 |         |   (848): exec_module                                                                                             |
 |         |   (686): _load_unlocked                                                                                                   |
 |         |   (975): _find_and_load_unlocked                                                                                          |
 |         |   (991): _find_and_load                                                                                                   |
 |         |   /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py(52):                      |
 |         |   (219): _call_with_frames_removed                                                                                        |
 |         |   (848): exec_module                                                                                             |
 |         |   (686): _load_unlocked                                                                                                   |
 |         |   (975): _find_and_load_unlocked                                                                                          |
 |         |   (991): _find_and_load                                                                                                   |
 |         |   /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py(22):                      |
 |         |   (219): _call_with_frames_removed                                                                                        |
 |         |   (848): exec_module                                                                                             |
 |         |   (686): _load_unlocked                                                                                                   |
 |         |   (975): _find_and_load_unlocked                                                                                          |
 |         |   (991): _find_and_load                                                                                                   |
 |         |   /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py(18):  |
 |         |   (219): _call_with_frames_removed                                                                                        |
 |         |   (848): exec_module                                                                                             |
 |         |   (686): _load_unlocked                                                                                                   |
 |         |   (975): _find_and_load_unlocked                                                                                          |
 |         |   (991): _find_and_load                                                                                                   |
 |         |   /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py(18):      |
 |         |   (219): _call_with_frames_removed                                                                                        |
 |         |   (848): exec_module                                                                                             |
 |         |   (686): _load_unlocked                                                                                                   |
 |         |   (975): _find_and_load_unlocked                                                                                          |
 |         |   (991): _find_and_load                                                                                                   |
 |         |   (219): _call_with_frames_removed                                                                                        |
 |         |   (961): _find_and_load_unlocked                                                                                          |
 |         |   (991): _find_and_load                                                                                                   |
 |         |   /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py(26):               |
 |         |   (219): _call_with_frames_removed                                                                                        |
 |         |   (848): exec_module                                                                                             |
 |         |   (686): _load_unlocked                                                                                                   |
 |         |   (975): _find_and_load_unlocked                                                                                          |
 |         |   (991): _find_and_load                                                                                                   |
 |         |   /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py(19):               |
 |         |   (219): _call_with_frames_removed                                                                                        |
 |         |   (848): exec_module                                                                                             |
 |         |   (686): _load_unlocked                                                                                                   |
 |         |   (975): _find_and_load_unlocked                                                                                          |
 |         |   (991): _find_and_load                                                                                                   |
 |         |   (219): _call_with_frames_removed                                                                                        |
 |         |   (961): _find_and_load_unlocked                                                                                          |
 |         |   (991): _find_and_load                                                                                                   |
 |         |   /tmp/examples/poc_ensemble/6_softmaxsampling/1/model.py(35):                                                                                 |
 |         |   (219): _call_with_frames_removed                                                                                        |
 |         |   (848): exec_module                                                                                             |
 |         |   (686): _load_unlocked                                                                                                   |
 |         |   (975): _find_and_load_unlocked                                                                                          |
 |         |   (991): _find_and_load                                                                                                   |
 ---------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
I1104 13:48:37.567046 19199 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1104 13:48:37.567897 19199 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/examples/poc_ensemble                                                                                                                                                                   |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1104 13:48:37.567936 19199 server.cc:262] Waiting for in-flight requests to complete.

I1104 13:48:37.567952 19199 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1104 13:48:37.568117 19199 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1104 13:48:37.568144 19199 server.cc:293] All models are stopped, unloading models

I1104 13:48:37.568161 19199 server.cc:300] Timeout 30: Found 6 live models and 0 in-flight non-inference requests

I1104 13:48:37.568185 19199 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1104 13:48:37.568264 19199 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state

I1104 13:48:37.568606 19199 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state

I1104 13:48:37.584759 19199 model_lifecycle.cc:578] successfully unloaded '1_predicttensorflowtriton' version 1

I1104 13:48:37.585841 19199 model_lifecycle.cc:578] successfully unloaded '5_predicttensorflowtriton' version 1

I1104 13:48:38.568252 19199 server.cc:300] Timeout 29: Found 4 live models and 0 in-flight non-inference requests

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

W1104 13:48:38.593221 19199 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1104 13:48:39.468702 19199 model_lifecycle.cc:578] successfully unloaded '4_unrollfeatures' version 1

I1104 13:48:39.515089 19199 model_lifecycle.cc:578] successfully unloaded '0_queryfeast' version 1

I1104 13:48:39.568410 19199 server.cc:300] Timeout 28: Found 2 live models and 0 in-flight non-inference requests

W1104 13:48:39.593421 19199 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

I1104 13:48:39.709030 19199 model_lifecycle.cc:578] successfully unloaded '2_queryfaiss' version 1

I1104 13:48:40.568557 19199 server.cc:300] Timeout 27: Found 1 live models and 0 in-flight non-inference requests

W1104 13:48:40.619620 19199 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

I1104 13:48:41.568687 19199 server.cc:300] Timeout 26: Found 1 live models and 0 in-flight non-inference requests

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1104 13:48:42.501290 19199 model_lifecycle.cc:578] successfully unloaded '3_queryfeast' version 1

I1104 13:48:42.568817 19199 server.cc:300] Timeout 25: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       355    212    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                      100     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     15    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        54     30    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       125     23    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     24    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    147    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    22      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      4    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     27    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     93    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          346    157    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          603    344    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     20    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     78    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     16    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      1    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    113    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49      8    84%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     55    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                114     55    52%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     169     89    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          241    107    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 244    108    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          426    151    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              170    117    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                104     69    34%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              280     71    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                60     36    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39      4    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          458    154    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    245    127    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               708    310    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              122     89    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               97     62    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    108    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     22    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       435    346    20%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         158     75    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          85     12    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    141    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31      8    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     31    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     39    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     34    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       209     53    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 46     20    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     23      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                26      7    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                              9      4    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py                               24      8    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py                                77     31    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py                               126     71    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             79     32    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           67     22    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py                      50     23    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py                             72     26    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py                         1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py                    11      2    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py                  1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py           61     12    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py                 140     47    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              47     12    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py                           87     51    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               268    210    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     14    81%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py                            22     20     9%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py                               113     14    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py                             37     29    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py                             10      6    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py                          32     13    59%
TOTAL                                                                                                 14450   6758    53%
=================== 1 failed, 2 passed in 310.21s (0:05:10) ====================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://github.com/gitapi/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins9013166034043925950.sh

jperez999 · 2022-11-04T14:54:37Z

rerun tests

nvidia-merlin-bot · 2022-11-04T15:02:24Z

Click to view CI Results

GitHub pull request #721 of commit 2922647005d7f8883a887a209f1cb577d9b23874, no merge conflicts.
Running as SYSTEM
Setting status of 2922647005d7f8883a887a209f1cb577d9b23874 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_merlin/568/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/721/*:refs/remotes/origin/pr/721/* # timeout=10
 > git rev-parse 2922647005d7f8883a887a209f1cb577d9b23874^{commit} # timeout=10
Checking out Revision 2922647005d7f8883a887a209f1cb577d9b23874 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 2922647005d7f8883a887a209f1cb577d9b23874 # timeout=10
Commit message: "docs: Remove SM from the support matrix"
 > git rev-list --no-walk 2922647005d7f8883a887a209f1cb577d9b23874 # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins11060706010192862784.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.2,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.2,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.8.2,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.42,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.0,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1558478917'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-eaqpc7bm
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-eaqpc7bm
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit d2737da6cc7e98ec8723731a7cc998aa143e6758
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+10.gd2737da) (2.22.0)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+10.gd2737da) (1.1.1)
Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+10.gd2737da) (0.3.0+12.g78ecddd)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.55.1)
Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (7.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (4.64.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.10.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.5)
Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.3.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.12.0)
Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.5.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (5.4.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.0.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.0.4)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.7.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (8.1.3)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (65.4.1)
Collecting llvmlite<0.39,>=0.38.0rc1
  Downloading llvmlite-0.38.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.5/34.5 MB 61.1 MB/s eta 0:00:00
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+10.gd2737da) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+10.gd2737da-py3-none-any.whl size=91051 sha256=9ead89f9b32790a5f66fb4c9dcd7cadb9ca2af75381e63ea09c1a66ddecd39cd
  Stored in directory: /tmp/pip-ephem-wheel-cache-0cjucv95/wheels/d3/db/b8/99d510a979c278774eda4142f1c0643c93b7b2674aff321c16
Successfully built merlin-systems
Installing collected packages: llvmlite, merlin-systems
  Attempting uninstall: llvmlite
    Found existing installation: llvmlite 0.39.1
    Not uninstalling llvmlite at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'llvmlite'. No files were found to uninstall.
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.5.0+4.g15074ad
    Not uninstalling merlin-systems at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu
    Can't uninstall 'merlin-systems'. No files were found to uninstall.
Successfully installed llvmlite-0.38.1 merlin-systems-0.7.0+10.gd2737da
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/models.git

Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-axaenap1

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-axaenap1

Resolved https://github.com/NVIDIA-Merlin/models.git to commit ada3874d663592d215150ba3eadd04b075818b4f

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-models==0.9.0+29.gada3874d) (0.3.0+12.g78ecddd)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (0.55.1)

Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2022.3.0)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (7.0.0)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (21.3)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (4.64.1)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (3.19.5)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.10.0)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.2.5)

Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2022.3.0)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.3.5)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (0.4.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.2.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (0.12.0)

Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2022.5.0)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (5.4.1)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (5.8.0)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2.0.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (6.1)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (3.1.2)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2.4.0)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.0.4)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.7.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (8.1.3)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.20.3)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (65.4.1)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (0.38.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (3.0.9)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2022.2.1)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2.8.2)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.2.0)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.52.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (1.0.1)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (4.1.0)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (6.0.2)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+29.gada3874d) (4.0.0)

Building wheels for collected packages: merlin-models

Building wheel for merlin-models (pyproject.toml): started

Building wheel for merlin-models (pyproject.toml): finished with status 'done'

Created wheel for merlin-models: filename=merlin_models-0.9.0+29.gada3874d-py3-none-any.whl size=356754 sha256=1f590b1bbcd63fd824f05c361c97a5911dc850df968cdb771dae3c0896a55bdc

Stored in directory: /tmp/pip-ephem-wheel-cache-m9u78g0e/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5

Successfully built merlin-models

Installing collected packages: merlin-models

Attempting uninstall: merlin-models

Found existing installation: merlin-models 0.7.0+11.g280956aa4

Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu

Can't uninstall 'merlin-models'. No files were found to uninstall.

Successfully installed merlin-models-0.9.0+29.gada3874d
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git

Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-b_xijf8q

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-b_xijf8q

Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit 8e7edbafd3006f56e73efdc0c01c4445ab57d028

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+3.g8e7edbaf) (1.8.1)

Collecting merlin-dataloader>=0.0.2

Downloading merlin-dataloader-0.0.2.tar.gz (44 kB)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.1/44.1 kB 1.5 MB/s eta 0:00:00

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: merlin-core>=0.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from nvtabular==1.6.0+3.g8e7edbaf) (0.3.0+12.g78ecddd)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (0.55.1)

Requirement already satisfied: distributed>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2022.3.0)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (7.0.0)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (21.3)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (4.64.1)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (3.19.5)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.10.0)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.2.5)

Requirement already satisfied: dask>=2021.11.2 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2022.3.0)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.3.5)

Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+3.g8e7edbaf) (1.20.3)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (0.4.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.2.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (0.12.0)

Requirement already satisfied: fsspec>=0.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2022.5.0)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (5.4.1)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (5.8.0)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2.0.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (6.1)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (3.1.2)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2.4.0)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.0.4)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.7.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (8.1.3)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (65.4.1)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (0.38.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (3.0.9)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2022.2.1)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2.8.2)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.2.0)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.52.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (1.0.1)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (4.1.0)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (6.0.2)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2021.11.2->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+3.g8e7edbaf) (4.0.0)

Building wheels for collected packages: nvtabular, merlin-dataloader

Building wheel for nvtabular (pyproject.toml): started

Building wheel for nvtabular (pyproject.toml): finished with status 'done'

Created wheel for nvtabular: filename=nvtabular-1.6.0+3.g8e7edbaf-cp38-cp38-linux_x86_64.whl size=257744 sha256=ef9380e9ee1fc72f6eb8ec5ec227221430e8bfb52061b0be2431acccf96a0185

Stored in directory: /tmp/pip-ephem-wheel-cache-7tl4i280/wheels/c2/16/76/39994bff39d812513de5b5572bff0903b9eb8f6c645b44cedc

Building wheel for merlin-dataloader (pyproject.toml): started

Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'

Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.2-py3-none-any.whl size=29205 sha256=7c9650a73643b775dfafd789046cab5460ce4bcc1dd1609e9b0024c170f7e119

Stored in directory: /tmp/pip-ephem-wheel-cache-7tl4i280/wheels/d5/ce/8c/31476c01e0b5a2278110fe2092bdd911efb0e5b83d0d3550ca

Successfully built nvtabular merlin-dataloader

Installing collected packages: merlin-dataloader, nvtabular

Attempting uninstall: nvtabular

Found existing installation: nvtabular 1.1.1

Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu

Can't uninstall 'nvtabular'. No files were found to uninstall.

Successfully installed merlin-dataloader-0.0.2 nvtabular-1.6.0+3.g8e7edbaf
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/core.git

Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-6nzeou2v

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-6nzeou2v

Resolved https://github.com/NVIDIA-Merlin/core.git to commit 563be4bf5ef675940d5fff2b5e4666424a7f7947

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (2022.5.0)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (0.55.1)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (7.0.0)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (21.3)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (4.64.1)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (2022.3.0)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (3.19.5)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (1.10.0)

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (2022.3.0)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+5.g563be4b) (1.2.5)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+5.g563be4b) (1.3.5)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (0.4.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (1.2.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (0.12.0)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (5.4.1)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (5.8.0)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.0.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (6.1)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (3.1.2)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.4.0)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.0.4)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.7.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (8.1.3)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+5.g563be4b) (1.20.3)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+5.g563be4b) (65.4.1)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+5.g563be4b) (0.38.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+5.g563be4b) (3.0.9)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+5.g563be4b) (2022.2.1)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+5.g563be4b) (2.8.2)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+5.g563be4b) (1.2.0)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+5.g563be4b) (1.52.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+5.g563be4b) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (1.0.1)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (4.1.0)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (6.0.2)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+5.g563be4b) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+5.g563be4b) (4.0.0)

Building wheels for collected packages: merlin-core

Building wheel for merlin-core (pyproject.toml): started

Building wheel for merlin-core (pyproject.toml): finished with status 'done'

Created wheel for merlin-core: filename=merlin_core-0.8.0+5.g563be4b-py3-none-any.whl size=118363 sha256=2b54dc2cbebdca374aa378b636fb4c2258a23f7a41977eb06969357e0fb2242d

Stored in directory: /tmp/pip-ephem-wheel-cache-o9k4vowk/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84

Successfully built merlin-core

Installing collected packages: merlin-core

Attempting uninstall: merlin-core

Found existing installation: merlin-core 0.3.0+12.g78ecddd

Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu

Can't uninstall 'merlin-core'. No files were found to uninstall.

Successfully installed merlin-core-0.8.0+5.g563be4b
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin

plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0

collected 3 items
tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py F      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py .               [100%]
=================================== FAILURES ===================================

__________________________________ test_func ___________________________________
self = <testbook.client.TestbookNotebookClient object at 0x7ff424988fd0>

msg_id = '21b131e2-5a0c0588804284995d9221a3_8282_32'

cell = {'id': '45c976e3', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-04T14:58:28.517363Z',...b[38;5;241m=\x1b[39m \x1b[38;5;28mself\x1b[39m\x1b[38;5;241m.\x1b[39mpid\n', '\x1b[0;31mKeyboardInterrupt\x1b[0m: ']}]}

timeout = 120

task_poll_output_msg = <Task finished name='Task-379' coro=<NotebookClient._async_poll_output_msg() done, defined at /var/jenkins_home/.local/lib/python3.8/site-packages/nbclient/client.py:756> result=None>

task_poll_kernel_alive = <Task cancelled name='Task-378' coro=<NotebookClient._async_poll_kernel_alive() done, defined at /var/jenkins_home/.local/lib/python3.8/site-packages/nbclient/client.py:770>>
async def _async_poll_for_reply(
    self,
    msg_id: str,
    cell: NotebookNode,
    timeout: t.Optional[int],
    task_poll_output_msg: asyncio.Future,
    task_poll_kernel_alive: asyncio.Future,
) -> t.Dict:

    assert self.kc is not None
    new_timeout: t.Optional[float] = None
    if timeout is not None:
        deadline = monotonic() + timeout
        new_timeout = float(timeout)
    while True:
        try:


          msg = await ensure_async(self.kc.shell_channel.get_msg(timeout=new_timeout))


../../../.local/lib/python3.8/site-packages/nbclient/client.py:730:

obj = <coroutine object ZMQSocketChannel.get_msg at 0x7ff500baf0c0>
async def ensure_async(obj: Union[Awaitable, Any]) -> Any:
    """Convert a non-awaitable object to a coroutine if needed,
    and await it if it was not already awaited.
    """
    if inspect.isawaitable(obj):
        try:


          result = await obj


../../../.local/lib/python3.8/site-packages/nbclient/util.py:96:

self = <jupyter_client.channels.ZMQSocketChannel object at 0x7ff500b960a0>

timeout = 120000.0
async def get_msg(self, timeout: t.Optional[float] = None) -> t.Dict[str, t.Any]:
    """Gets a message if there is one that is ready."""
    assert self.socket is not None
    if timeout is not None:
        timeout *= 1000  # seconds to ms
    ready = await self.socket.poll(timeout)

    if ready:
        res = await self._recv()
        return res
    else:


      raise Empty


E           _queue.Empty
/usr/local/lib/python3.8/dist-packages/jupyter_client/channels.py:230: Empty
During handling of the above exception, another exception occurred:
def test_func():
    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "01-Building-Recommender-Systems-with-Merlin.ipynb",
        execute=False,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["NUM_ROWS"] = "100000"
            os.system("mkdir -p /tmp/examples")
            os.environ["BASE_DIR"] = "/tmp/examples/"
            """
        )
        tb1.execute()
        assert os.path.isdir("/tmp/examples/dlrm")
        assert os.path.isdir("/tmp/examples/feature_repo")
        assert os.path.isdir("/tmp/examples/query_tower")
        assert os.path.isfile("/tmp/examples/item_embeddings.parquet")
        assert os.path.isfile("/tmp/examples/feature_repo/user_features.py")
        assert os.path.isfile("/tmp/examples/feature_repo/item_features.py")

    with testbook(
        REPO_ROOT
        / "examples"
        / "Building-and-deploying-multi-stage-RecSys"
        / "02-Deploying-multi-stage-RecSys-with-Merlin-Systems.ipynb",
        execute=False,
        timeout=120,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["DATA_FOLDER"] = "/tmp/data/"
            os.environ["BASE_DIR"] = "/tmp/examples/"
            os.environ["topk_retrieval"] = "20"
            """
        )
        NUM_OF_CELLS = len(tb2.cells)
        tb2.execute_cell(list(range(0, NUM_OF_CELLS - 3)))
        top_k = tb2.ref("top_k")
        outputs = tb2.ref("outputs")
        assert outputs[0] == "ordered_ids"


      tb2.inject(


            """
            import shutil
            from merlin.core.dispatch import get_lib
            from merlin.models.loader.tf_utils import configure_tensorflow
            configure_tensorflow()
            df_lib = get_lib()
            batch = df_lib.read_parquet(
                os.path.join("/tmp/data/processed_nvt/", "train", "part_0.parquet"),
                num_rows=1,
                columns=["user_id_raw"],
            )
            from merlin.systems.triton.utils import run_ensemble_on_tritonserver
            response = run_ensemble_on_tritonserver(
                "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"
            )
            response = [x.tolist()[0] for x in response["ordered_ids"]]
            shutil.rmtree("/tmp/examples/", ignore_errors=True)
            """
        )

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:61:

../../../.local/lib/python3.8/site-packages/testbook/client.py:237: in inject

cell = TestbookNode(self.execute_cell(inject_idx)) if run else TestbookNode(code_cell)

../../../.local/lib/python3.8/site-packages/testbook/client.py:133: in execute_cell

cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)

../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped

return just_run(coro(*args, **kwargs))

../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run

return loop.run_until_complete(coro)

/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete

return future.result()

../../../.local/lib/python3.8/site-packages/nbclient/client.py:949: in async_execute_cell

exec_reply = await self.task_poll_for_reply

../../../.local/lib/python3.8/site-packages/nbclient/client.py:754: in _async_poll_for_reply

await self._async_handle_timeout(timeout, cell)

self = <testbook.client.TestbookNotebookClient object at 0x7ff424988fd0>

timeout = 120

cell = {'id': '45c976e3', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-11-04T14:58:28.517363Z',...b[38;5;241m=\x1b[39m \x1b[38;5;28mself\x1b[39m\x1b[38;5;241m.\x1b[39mpid\n', '\x1b[0;31mKeyboardInterrupt\x1b[0m: ']}]}
async def _async_handle_timeout(
    self, timeout: int, cell: t.Optional[NotebookNode] = None
) -> None:

    self.log.error("Timeout waiting for execute reply (%is)." % timeout)
    if self.interrupt_on_timeout:
        self.log.error("Interrupting kernel")
        assert self.km is not None
        await ensure_async(self.km.interrupt_kernel())
    else:


      raise CellTimeoutError.error_from_timeout_and_cell(


            "Cell execution timed out", timeout, cell
        )

E           nbclient.exceptions.CellTimeoutError: A cell timed out while it was being executed, after 120 seconds.

E           The message was: Cell execution timed out.

E           Here is a preview of the cell contents:

E           -------------------

E           ['import shutil', 'from merlin.core.dispatch import get_lib', 'from merlin.models.loader.tf_utils import configure_tensorflow', 'configure_tensorflow()', 'df_lib = get_lib()']

E           ...

E           ['response = run_ensemble_on_tritonserver(', '    "/tmp/examples/poc_ensemble", ensemble.graph.input_schema, batch, outputs,  "ensemble_model"', ')', 'response = [x.tolist()[0] for x in response["ordered_ids"]]', 'shutil.rmtree("/tmp/examples/", ignore_errors=True)']

E           -------------------
../../../.local/lib/python3.8/site-packages/nbclient/client.py:801: CellTimeoutError

----------------------------- Captured stdout call -----------------------------

Signal (2) received.

Signal (2) received.

Signal (15) received.

Signal (15) received.

Signal (15) received.

Signal (15) received.

Signal (15) received.

Signal (15) received.

Signal (15) received.

----------------------------- Captured stderr call -----------------------------

2022-11-04 14:56:34.839087: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-04 14:56:38.836722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-04 14:56:38.837495: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15149 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-11-04 14:56:38.838131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 15149 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-11-04 14:56:38.838772: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 15149 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

WARNING clustering 463 points to 32 centroids: please provide at least 1248 training points

2022-11-04 14:58:22.018760: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-04 14:58:26.031381: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-04 14:58:26.032173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15149 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-11-04 14:58:26.032817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 15149 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-11-04 14:58:26.033450: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 15149 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

I1104 14:58:28.831465 8838 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f4f84000000' with size 268435456

I1104 14:58:28.832253 8838 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1104 14:58:28.840506 8838 model_lifecycle.cc:459] loading: 1_predicttensorflowtriton:1

I1104 14:58:28.840585 8838 model_lifecycle.cc:459] loading: 0_queryfeast:1

I1104 14:58:28.840630 8838 model_lifecycle.cc:459] loading: 2_queryfaiss:1

I1104 14:58:28.840675 8838 model_lifecycle.cc:459] loading: 3_queryfeast:1

I1104 14:58:28.840739 8838 model_lifecycle.cc:459] loading: 4_unrollfeatures:1

I1104 14:58:28.840775 8838 model_lifecycle.cc:459] loading: 5_predicttensorflowtriton:1

I1104 14:58:28.840858 8838 model_lifecycle.cc:459] loading: 6_softmaxsampling:1

I1104 14:58:29.130596 8838 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow

I1104 14:58:29.130636 8838 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10

I1104 14:58:29.130643 8838 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10

I1104 14:58:29.130649 8838 tensorflow.cc:2576] backend configuration:

{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}

I1104 14:58:29.130689 8838 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 1_predicttensorflowtriton (version 1)

2022-11-04 14:58:29.131319: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-04 14:58:29.135768: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-04 14:58:29.135796: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-04 14:58:29.135886: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-11-04 14:58:29.535976: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-04 14:58:29.579568: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled

2022-11-04 14:58:29.581679: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-04 14:58:29.633270: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-04 14:58:29.657673: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 526364 microseconds.

I1104 14:58:29.668704 8838 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 1_predicttensorflowtriton (GPU device 0)

2022-11-04 14:58:29.669241: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-04 14:58:29.672922: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-04 14:58:29.672946: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-04 14:58:29.674859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-04 14:58:29.682329: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-04 14:58:29.735176: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/1_predicttensorflowtriton/1/model.savedmodel

2022-11-04 14:58:29.759754: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 90518 microseconds.

I1104 14:58:29.760082 8838 model_lifecycle.cc:693] successfully loaded '1_predicttensorflowtriton' version 1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1104 14:59:08.800058 8838 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 5_predicttensorflowtriton (version 1)

2022-11-04 14:59:08.801393: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-04 14:59:08.821000: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-04 14:59:08.821048: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-04 14:59:08.823092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-04 14:59:08.852721: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-04 14:59:09.001742: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-04 14:59:09.054053: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 252673 microseconds.

I1104 14:59:09.070244 8838 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_queryfeast (GPU device 0)

I1104 14:59:15.397980 8838 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 2_queryfaiss (GPU device 0)

I1104 14:59:15.400289 8838 model_lifecycle.cc:693] successfully loaded '0_queryfeast' version 1

I1104 14:59:21.626164 8838 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 3_queryfeast (GPU device 0)

I1104 14:59:21.627256 8838 model_lifecycle.cc:693] successfully loaded '2_queryfaiss' version 1

I1104 14:59:27.964326 8838 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 4_unrollfeatures (GPU device 0)

I1104 14:59:27.967330 8838 model_lifecycle.cc:693] successfully loaded '3_queryfeast' version 1

I1104 14:59:33.862272 8838 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 6_softmaxsampling (GPU device 0)

I1104 14:59:33.864141 8838 model_lifecycle.cc:693] successfully loaded '4_unrollfeatures' version 1

I1104 14:59:39.801118 8838 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 5_predicttensorflowtriton (GPU device 0)

I1104 14:59:39.801424 8838 model_lifecycle.cc:693] successfully loaded '6_softmaxsampling' version 1

2022-11-04 14:59:39.801687: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-04 14:59:39.811115: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-11-04 14:59:39.811146: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-04 14:59:39.813107: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6383 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-11-04 14:59:39.836341: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-11-04 14:59:39.982183: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/examples/poc_ensemble/5_predicttensorflowtriton/1/model.savedmodel

2022-11-04 14:59:40.037662: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 235983 microseconds.

I1104 14:59:40.037989 8838 model_lifecycle.cc:693] successfully loaded '5_predicttensorflowtriton' version 1

I1104 14:59:40.040341 8838 model_lifecycle.cc:459] loading: ensemble_model:1

I1104 14:59:40.040617 8838 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1

I1104 14:59:40.040720 8838 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+
I1104 14:59:40.040810 8838 server.cc:588]

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend    | Path                                                            | Config                                                                                                                                                                      |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| tensorflow | /opt/tritonserver/backends/tensorflow2/libtriton_tensorflow2.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}} |

| python     | /opt/tritonserver/backends/python/libtriton_python.so           | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}}               |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1104 14:59:40.040919 8838 server.cc:631]

+---------------------------+---------+--------+

| Model                     | Version | Status |

+---------------------------+---------+--------+

| 0_queryfeast              | 1       | READY  |

| 1_predicttensorflowtriton | 1       | READY  |

| 2_queryfaiss              | 1       | READY  |

| 3_queryfeast              | 1       | READY  |

| 4_unrollfeatures          | 1       | READY  |

| 5_predicttensorflowtriton | 1       | READY  |

| 6_softmaxsampling         | 1       | READY  |

| ensemble_model            | 1       | READY  |

+---------------------------+---------+--------+
I1104 14:59:40.104228 8838 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1104 14:59:40.105108 8838 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/examples/poc_ensemble                                                                                                                                                                   |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1104 14:59:40.106470 8838 grpc_server.cc:4610] Started GRPCInferenceService at localhost:8001

I1104 14:59:40.106677 8838 http_server.cc:3316] Started HTTPService at 0.0.0.0:8000

I1104 14:59:40.147599 8838 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002

W1104 14:59:41.122031 8838 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1104 14:59:42.122242 8838 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1104 14:59:43.141287 8838 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

I1104 15:00:25.385906 8838 server.cc:262] Waiting for in-flight requests to complete.

I1104 15:00:25.385946 8838 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1104 15:00:25.386198 8838 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1

I1104 15:00:25.386281 8838 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1104 15:00:25.386404 8838 server.cc:293] All models are stopped, unloading models

I1104 15:00:25.386412 8838 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state

I1104 15:00:25.386446 8838 server.cc:300] Timeout 30: Found 7 live models and 0 in-flight non-inference requests

I1104 15:00:25.386571 8838 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1104 15:00:25.386879 8838 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state

I1104 15:00:25.401580 8838 model_lifecycle.cc:578] successfully unloaded '1_predicttensorflowtriton' version 1

I1104 15:00:25.406999 8838 model_lifecycle.cc:578] successfully unloaded '5_predicttensorflowtriton' version 1

I1104 15:00:26.386642 8838 server.cc:300] Timeout 29: Found 5 live models and 0 in-flight non-inference requests

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1104 15:00:27.381405 8838 model_lifecycle.cc:578] successfully unloaded '6_softmaxsampling' version 1

I1104 15:00:27.386778 8838 server.cc:300] Timeout 28: Found 4 live models and 0 in-flight non-inference requests

I1104 15:00:27.483363 8838 model_lifecycle.cc:578] successfully unloaded '4_unrollfeatures' version 1

I1104 15:00:27.585391 8838 model_lifecycle.cc:578] successfully unloaded '2_queryfaiss' version 1

I1104 15:00:28.386902 8838 server.cc:300] Timeout 27: Found 2 live models and 0 in-flight non-inference requests

I1104 15:00:29.387462 8838 server.cc:300] Timeout 26: Found 2 live models and 0 in-flight non-inference requests

I1104 15:00:30.387575 8838 server.cc:300] Timeout 25: Found 2 live models and 0 in-flight non-inference requests

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

I1104 15:00:31.387858 8838 server.cc:300] Timeout 24: Found 2 live models and 0 in-flight non-inference requests

------------------------------ Captured log call -------------------------------

ERROR    traitlets:client.py:795 Timeout waiting for execute reply (120s).
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover
.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                          10      4    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       355    210    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/protocols.py                                      100     45    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    108    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          5      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   122     15    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/dictarray.py                                        54     19    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       125     23    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     24    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            344    136    60%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               17      1    94%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    22      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      2    83%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                        101     26    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/init.py                                     2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/init.py                           3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/aliccp/dataset.py                   141    102    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/init.py                  0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/ecommerce/dressipi/dataset.py                  45     37    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py                                  147     56    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181     97    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     38    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          346    161    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37      8    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          603    344    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     29    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     81    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/api.py                                           14      5    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     16    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/io.py                                            15      1    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    113    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   70      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49      8    84%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                        108     55    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                114     55    52%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     169     89    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30      4    87%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29      7    76%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     12    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          241    107    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 244    108    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          426    151    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                              170    117    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                104     69    34%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            50     19    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              280     71    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                60     36    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          39      4    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          458    154    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/loader.py                                    245    127    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      1    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           115     57    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              198     82    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               708    310    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             67     43    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           78     44    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/init.py                            0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/base.py                              122     89    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/classification.py                     91     51    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/contrastive.py                       147    107    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/regression.py                          9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/base.py                      78     41    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/in_batch.py                  37     22    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/sampling/popularity.py                27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/outputs/topk.py                               97     62    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     207    108    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     22    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     31    58%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/init.py                       0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/block.py                        101     54    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transformers/transforms.py                    63     14    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                           111     77    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       435    346    20%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/sequence.py                       282    212    25%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                         158     75    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          85     12    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            209    141    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 38     18    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dependencies.py                            26     19    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/example_utils.py                           31      8    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/nvt_utils.py                               27     24    11%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     31    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     39    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      4    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       190     34    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       209     52    75%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      6    93%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 46     20    57%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     23      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                26      1    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                              9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/compat.py                               24      8    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py                                77      3    96%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/feast.py                               126     26    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             79      7    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/session_filter.py                       45     28    38%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/softmax_sampling.py                     51      1    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           67     18    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/unroll_features.py                      50      1    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py                             72     26    64%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py                         1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py                    11      2    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/init.py                  1      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/init.py              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/operator.py             11      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/ops/tensorflow.py           61     11    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/triton/runtime.py                 140     47    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              47     12    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py                           87     51    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               268    210    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/models/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     13    82%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py                            22     20     9%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py                               113     14    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py                             37     29    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py                             10      6    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py                          32     13    59%
TOTAL                                                                                                 14546   6643    54%
=================== 1 failed, 2 passed in 359.68s (0:05:59) ====================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://github.com/gitapi/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins10715576655739620373.sh

docs: Remove SM from the support matrix

2922647

The streaming multiprocessors (SM) value might be more of an item for NVIDIA employees than Merlin consumers. In addition, the DLFW folks do not publish the SM values for the TensorFlow and PyTorch containers that we use as base containers.

mikemckiernan self-assigned this Nov 4, 2022

mikemckiernan added the documentation Improvements or additions to documentation label Nov 4, 2022

alaiacano approved these changes Nov 4, 2022

View reviewed changes

nv-alaiacano approved these changes Nov 4, 2022

View reviewed changes

mikemckiernan merged commit 183d8f3 into main Nov 4, 2022

mikemckiernan deleted the docs-remove-sm branch November 4, 2022 15:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: Remove SM from the support matrix #721

docs: Remove SM from the support matrix #721

mikemckiernan commented Nov 4, 2022

github-actions bot commented Nov 4, 2022

nvidia-merlin-bot commented Nov 4, 2022

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

mikemckiernan commented Nov 4, 2022

nvidia-merlin-bot commented Nov 4, 2022

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

jperez999 commented Nov 4, 2022

nvidia-merlin-bot commented Nov 4, 2022

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

docs: Remove SM from the support matrix #721

docs: Remove SM from the support matrix #721

Conversation

mikemckiernan commented Nov 4, 2022

github-actions bot commented Nov 4, 2022

Documentation preview

nvidia-merlin-bot commented Nov 4, 2022

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

mikemckiernan commented Nov 4, 2022

nvidia-merlin-bot commented Nov 4, 2022

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

jperez999 commented Nov 4, 2022

nvidia-merlin-bot commented Nov 4, 2022

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover