Fix "Unsupported type_id conversion" in triton inference #839
Conversation
We're hitting an error like " Unsupported type_id conversion to cudf " when running any strings through nvtabular inference with triton. Fix by converting the dtypes as appropiate in _convert_tensor. Also add a basic unittest that would have caught this error, launching tritonserver in a subprocess and communicating through grpc
Click to view CI ResultsGitHub pull request #839 of commit 8ad832f35d923e1f356973218191833828226391, no merge conflicts.
Running as SYSTEM
Setting status of 8ad832f35d923e1f356973218191833828226391 to PENDING with url http://10.20.13.93:8080/job/nvtabular_tests/2458/ and message: 'Pending'
Using context: Jenkins Unit Test Run
Building in workspace /var/jenkins_home/workspace/nvtabular_tests
using credential nvidia-merlin-bot
Cloning the remote Git repository
Cloning repository https://github.com/NVIDIA/NVTabular.git
> git init /var/jenkins_home/workspace/nvtabular_tests/nvtabular # timeout=10
Fetching upstream changes from https://github.com/NVIDIA/NVTabular.git
> git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA/NVTabular.git +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://github.com/NVIDIA/NVTabular.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://github.com/NVIDIA/NVTabular.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA/NVTabular.git
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA/NVTabular.git +refs/pull/839/*:refs/remotes/origin/pr/839/* # timeout=10
> git rev-parse 8ad832f35d923e1f356973218191833828226391^{commit} # timeout=10
Checking out Revision 8ad832f35d923e1f356973218191833828226391 (detached)
> git config core.sparsecheckout # timeout=10
> git checkout -f 8ad832f35d923e1f356973218191833828226391 # timeout=10
Commit message: "Fix "Unsupported type_id conversion" in triton inference"
> git rev-list --no-walk 69de2793a807781079b04a89541ac0b3d9d3bc70 # timeout=10
First time build. Skipping changelog.
[nvtabular_tests] $ /bin/bash /tmp/jenkins2131137861674969113.sh
Installing NVTabular
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Obtaining file:///var/jenkins_home/workspace/nvtabular_tests/nvtabular
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel metadata: started
Preparing wheel metadata: finished with status 'done'
Installing collected packages: nvtabular
Running setup.py develop for nvtabular
Successfully installed nvtabular
WARNING: You are using pip version 21.0.1; however, version 21.1.1 is available.
You should consider upgrading via the '/usr/bin/python -m pip install --upgrade pip' command.
Running black --check
All done! ✨ 🍰 ✨
106 files would be left unchanged.
Running flake8
Running isort
/usr/local/lib/python3.8/dist-packages/isort/main.py:141: UserWarning: Likely recursive symlink detected to /var/jenkins_home/workspace/nvtabular_tests/nvtabular/images
warn(f"Likely recursive symlink detected to {resolved_path}")
/usr/local/lib/python3.8/dist-packages/isort/main.py:141: UserWarning: Likely recursive symlink detected to /var/jenkins_home/workspace/nvtabular_tests/nvtabular/examples/scaling-criteo/imgs
warn(f"Likely recursive symlink detected to {resolved_path}")
Skipped 1 files
Running bandit
Running pylint
tmpdir = local('/tmp/pytest-of-jenkins/pytest-2/test_tritonserver_inference_st0')
tests/unit/test_triton_inference.py:63: /usr/lib/python3.8/contextlib.py:113: in enter modelpath = local('/tmp/pytest-of-jenkins/pytest-2/test_tritonserver_inference_st0')
E RuntimeError: Timed out waiting for tritonserver to become ready tests/unit/test_triton_inference.py:42: RuntimeError I0521 20:21:37.362323 23670 server.cc:527] I0521 20:21:37.362392 23670 server.cc:570] I0521 20:21:37.362649 23670 tritonserver.cc:1658] I0521 20:21:37.373131 23670 grpc_server.cc:3983] Started GRPCInferenceService at 0.0.0.0:8001 -- Docs: https://docs.pytest.org/en/stable/warnings.html ----------- coverage: platform linux, python 3.8.5-final-0 -----------
|
Click to view CI ResultsGitHub pull request #839 of commit 8f7225381a4371243cc4134e0437f2f4c9fb944d, no merge conflicts.
Running as SYSTEM
Setting status of 8f7225381a4371243cc4134e0437f2f4c9fb944d to PENDING with url http://10.20.13.93:8080/job/nvtabular_tests/2459/ and message: 'Pending'
Using context: Jenkins Unit Test Run
Building in workspace /var/jenkins_home/workspace/nvtabular_tests
using credential nvidia-merlin-bot
Cloning the remote Git repository
Cloning repository https://github.com/NVIDIA/NVTabular.git
> git init /var/jenkins_home/workspace/nvtabular_tests/nvtabular # timeout=10
Fetching upstream changes from https://github.com/NVIDIA/NVTabular.git
> git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA/NVTabular.git +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://github.com/NVIDIA/NVTabular.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://github.com/NVIDIA/NVTabular.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA/NVTabular.git
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA/NVTabular.git +refs/pull/839/*:refs/remotes/origin/pr/839/* # timeout=10
> git rev-parse 8f7225381a4371243cc4134e0437f2f4c9fb944d^{commit} # timeout=10
Checking out Revision 8f7225381a4371243cc4134e0437f2f4c9fb944d (detached)
> git config core.sparsecheckout # timeout=10
> git checkout -f 8f7225381a4371243cc4134e0437f2f4c9fb944d # timeout=10
Commit message: "increase tritonserver timeout"
> git rev-list --no-walk 8ad832f35d923e1f356973218191833828226391 # timeout=10
[nvtabular_tests] $ /bin/bash /tmp/jenkins1133839267040135247.sh
Installing NVTabular
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Obtaining file:///var/jenkins_home/workspace/nvtabular_tests/nvtabular
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel metadata: started
Preparing wheel metadata: finished with status 'done'
Installing collected packages: nvtabular
Running setup.py develop for nvtabular
Successfully installed nvtabular
WARNING: You are using pip version 21.0.1; however, version 21.1.1 is available.
You should consider upgrading via the '/usr/bin/python -m pip install --upgrade pip' command.
Running black --check
All done! ✨ 🍰 ✨
106 files would be left unchanged.
Running flake8
Running isort
/usr/local/lib/python3.8/dist-packages/isort/main.py:141: UserWarning: Likely recursive symlink detected to /var/jenkins_home/workspace/nvtabular_tests/nvtabular/images
warn(f"Likely recursive symlink detected to {resolved_path}")
/usr/local/lib/python3.8/dist-packages/isort/main.py:141: UserWarning: Likely recursive symlink detected to /var/jenkins_home/workspace/nvtabular_tests/nvtabular/examples/scaling-criteo/imgs
warn(f"Likely recursive symlink detected to {resolved_path}")
Skipped 1 files
Running bandit
Running pylint
-- Docs: https://docs.pytest.org/en/stable/warnings.html ----------- coverage: platform linux, python 3.8.5-final-0 -----------
|
Click to view CI ResultsGitHub pull request #839 of commit 3af6bfa93435c0c528c26a382ffaecd9892aba29, no merge conflicts.
Running as SYSTEM
Setting status of 3af6bfa93435c0c528c26a382ffaecd9892aba29 to PENDING with url http://10.20.13.93:8080/job/nvtabular_tests/2462/ and message: 'Pending'
Using context: Jenkins Unit Test Run
Building in workspace /var/jenkins_home/workspace/nvtabular_tests
using credential nvidia-merlin-bot
Cloning the remote Git repository
Cloning repository https://github.com/NVIDIA/NVTabular.git
> git init /var/jenkins_home/workspace/nvtabular_tests/nvtabular # timeout=10
Fetching upstream changes from https://github.com/NVIDIA/NVTabular.git
> git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA/NVTabular.git +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://github.com/NVIDIA/NVTabular.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://github.com/NVIDIA/NVTabular.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA/NVTabular.git
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA/NVTabular.git +refs/pull/839/*:refs/remotes/origin/pr/839/* # timeout=10
> git rev-parse 3af6bfa93435c0c528c26a382ffaecd9892aba29^{commit} # timeout=10
Checking out Revision 3af6bfa93435c0c528c26a382ffaecd9892aba29 (detached)
> git config core.sparsecheckout # timeout=10
> git checkout -f 3af6bfa93435c0c528c26a382ffaecd9892aba29 # timeout=10
Commit message: "Merge branch 'main' into triton_string_fix"
> git rev-list --no-walk f7750f979f3eed6d322ca34d2fd37e9ad7cc0ac8 # timeout=10
First time build. Skipping changelog.
[nvtabular_tests] $ /bin/bash /tmp/jenkins2954899999427953040.sh
Installing NVTabular
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Obtaining file:///var/jenkins_home/workspace/nvtabular_tests/nvtabular
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel metadata: started
Preparing wheel metadata: finished with status 'done'
Installing collected packages: nvtabular
Running setup.py develop for nvtabular
Successfully installed nvtabular
WARNING: You are using pip version 21.0.1; however, version 21.1.1 is available.
You should consider upgrading via the '/usr/bin/python -m pip install --upgrade pip' command.
Running black --check
All done! ✨ 🍰 ✨
106 files would be left unchanged.
Running flake8
Running isort
/usr/local/lib/python3.8/dist-packages/isort/main.py:141: UserWarning: Likely recursive symlink detected to /var/jenkins_home/workspace/nvtabular_tests/nvtabular/images
warn(f"Likely recursive symlink detected to {resolved_path}")
/usr/local/lib/python3.8/dist-packages/isort/main.py:141: UserWarning: Likely recursive symlink detected to /var/jenkins_home/workspace/nvtabular_tests/nvtabular/examples/scaling-criteo/imgs
warn(f"Likely recursive symlink detected to {resolved_path}")
Skipped 1 files
Running bandit
Running pylint
|
We're hitting an error like " Unsupported type_id conversion to cudf " when running any strings through nvtabular inference with triton. Fix by converting the dtypes as appropiate in _convert_tensor. Also add a basic unittest that would have caught this error, launching tritonserver in a subprocess and communicating through grpc
We're hitting an error like " Unsupported type_id conversion to cudf " when running any strings
through nvtabular inference with triton. Fix by converting the dtypes as appropiate in
_convert_tensor.
Also add a basic unittest that would have caught this error, launching tritonserver in a
subprocess and communicating through grpc