Introduce SOKEmbedding using Sparse Operation Kit #863

WonderingWJ · 2022-11-07T03:08:44Z

Fixes # (issue)

Goals ⚽

Integrate SOK into merlin-models

Implementation Details 🚧

New a SOK Embedding class to be as the backend of InputBlock and Embeddings

Testing Details 🔍

Currently only test sok dynamic variables

nvidia-merlin-bot · 2022-11-07T03:09:15Z

Click to view CI Results

GitHub pull request #863 of commit 475145a583de8734ca3b2b03888459f149000c6a, no merge conflicts.
Running as SYSTEM
Setting status of 475145a583de8734ca3b2b03888459f149000c6a to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_models/1748/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/863/*:refs/remotes/origin/pr/863/* # timeout=10
 > git rev-parse 475145a583de8734ca3b2b03888459f149000c6a^{commit} # timeout=10
Checking out Revision 475145a583de8734ca3b2b03888459f149000c6a (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 475145a583de8734ca3b2b03888459f149000c6a # timeout=10
Commit message: "test sok dynamic variable"
 > git rev-list --no-walk 0d48d9b357e1b6b9cbc7d42b5a5c880d50346cd0 # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins7190934815795726225.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.5.0)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.8)
Requirement already satisfied: traitlets>=5.2.2 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (5.4.0)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.5)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.16.1)
Requirement already satisfied: jupyter_core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.11.1)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.16.0)
Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.9.0)
Requirement already satisfied: pkgutil-resolve-name>=1.3.10 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (1.3.10)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (22.1.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (24.0.0)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: tornado>=6.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.2)
Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-3.0.2, cov-4.0.0
collected 167 items / 49 errors
==================================== ERRORS ====================================

_________________ ERROR collecting tests/unit/tf/test_core.py __________________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/test_core.py:4: in 

import merlin.models.tf as ml

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

________________ ERROR collecting tests/unit/tf/test_loader.py _________________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/test_loader.py:26: in 

import merlin.models.tf as mm

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

______________ ERROR collecting tests/unit/tf/test_public_api.py _______________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/test_public_api.py:19: in 

import merlin.models.tf as ml

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

_____________ ERROR collecting tests/unit/tf/blocks/test_cross.py ______________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/blocks/test_cross.py:20: in 

import merlin.models.tf as mm

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

______________ ERROR collecting tests/unit/tf/blocks/test_dlrm.py ______________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/blocks/test_dlrm.py:19: in 

import merlin.models.tf as mm

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

__________ ERROR collecting tests/unit/tf/blocks/test_interactions.py __________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/blocks/test_interactions.py:19: in 

import merlin.models.tf as mm

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

______________ ERROR collecting tests/unit/tf/blocks/test_mlp.py _______________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/blocks/test_mlp.py:21: in 

import merlin.models.tf as ml

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

___________ ERROR collecting tests/unit/tf/blocks/test_optimizer.py ____________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/blocks/test_optimizer.py:22: in 

import merlin.models.tf as ml

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

_________ ERROR collecting tests/unit/tf/blocks/retrieval/test_base.py _________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/blocks/retrieval/test_base.py:18: in 

from merlin.models.tf.blocks.retrieval.base import ItemRetrievalScorer

merlin/models/tf/blocks/retrieval/base.py:27: in 

from merlin.models.tf.models.base import ModelBlock

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

_ ERROR collecting tests/unit/tf/blocks/retrieval/test_matrix_factorization.py _

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py:20: in 

import merlin.models.tf as ml

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

______ ERROR collecting tests/unit/tf/blocks/retrieval/test_two_tower.py _______

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/blocks/retrieval/test_two_tower.py:22: in 

import merlin.models.tf as ml

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

___________ ERROR collecting tests/unit/tf/core/test_aggregation.py ____________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/core/test_aggregation.py:21: in 

import merlin.models.tf as ml

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

_______________ ERROR collecting tests/unit/tf/core/test_base.py _______________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/core/test_base.py:19: in 

import merlin.models.tf as ml

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

___________ ERROR collecting tests/unit/tf/core/test_combinators.py ____________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/core/test_combinators.py:6: in 

import merlin.models.tf as mm

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

_____________ ERROR collecting tests/unit/tf/core/test_encoder.py ______________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/core/test_encoder.py:6: in 

import merlin.models.tf as mm

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

______________ ERROR collecting tests/unit/tf/core/test_index.py _______________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/core/test_index.py:19: in 

import merlin.models.tf as mm

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

____________ ERROR collecting tests/unit/tf/core/test_prediction.py ____________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/core/test_prediction.py:7: in 

import merlin.models.tf as mm

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

_____________ ERROR collecting tests/unit/tf/core/test_tabular.py ______________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/core/test_tabular.py:1: in 

import merlin.models.tf as mm

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

___________ ERROR collecting tests/unit/tf/inputs/test_continuous.py ___________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/inputs/test_continuous.py:19: in 

import merlin.models.tf as ml

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

___________ ERROR collecting tests/unit/tf/inputs/test_embedding.py ____________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:159: in exec_module

source_stat, co = _rewrite_test(fn, self.config)

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:337: in _rewrite_test

tree = ast.parse(source, filename=strfn)

/usr/lib/python3.8/ast.py:47: in parse

return compile(source, filename, mode, flags,

E     File "/var/jenkins_home/workspace/merlin_models/models/tests/unit/tf/inputs/test_embedding.py", line 346

E       local_indices = []

E                        ^

E   IndentationError: unindent does not match any outer indentation level

____________ ERROR collecting tests/unit/tf/inputs/test_tabular.py _____________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/inputs/test_tabular.py:21: in 

import merlin.models.tf as ml

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

_____________ ERROR collecting tests/unit/tf/layers/test_queue.py ______________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/layers/test_queue.py:19: in 

import merlin.models.tf as ml

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

_____________ ERROR collecting tests/unit/tf/losses/test_losses.py _____________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/losses/test_losses.py:20: in 

import merlin.models.tf as ml

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

______ ERROR collecting tests/unit/tf/metrics/test_metrics_popularity.py _______

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/metrics/test_metrics_popularity.py:20: in 

from merlin.models.tf.metrics.evaluation import ItemCoverageAt, NoveltyAt, PopularityBiasAt

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

_________ ERROR collecting tests/unit/tf/metrics/test_metrics_topk.py __________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/metrics/test_metrics_topk.py:23: in 

from merlin.models.tf.metrics.topk import (

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

______________ ERROR collecting tests/unit/tf/models/test_base.py ______________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/models/test_base.py:24: in 

import merlin.models.tf as mm

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

___________ ERROR collecting tests/unit/tf/models/test_benchmark.py ____________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/models/test_benchmark.py:19: in 

import merlin.models.tf as ml

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

____________ ERROR collecting tests/unit/tf/models/test_ranking.py _____________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.dict)

tests/unit/tf/models/test_ranking.py:22: in 

import merlin.models.tf as mm

merlin/models/tf/init.py:34: in 

from merlin.models.tf.blocks.dlrm import DLRMBlock

merlin/models/tf/blocks/dlrm.py:19: in 

from merlin.models.tf.blocks.interaction import DotProductInteraction

merlin/models/tf/blocks/interaction.py:26: in 

from merlin.models.tf.inputs.base import InputBlockV2

merlin/models/tf/inputs/base.py:26: in 

from merlin.models.tf.inputs.embedding import (

E     File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py", line 177

E       use_dynamic_variable = False,

E       ^

E   SyntaxError: invalid syntax

___________ ERROR collecting tests/unit/tf/models/test_retrieval.py ____________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:168: in exec_module

exec(co, module.

nvidia-merlin-bot · 2022-11-07T05:48:18Z

Click to view CI Results

GitHub pull request #863 of commit 4ef7a9e281fdb8f84aae202ca3b2b90d1882ce6c, no merge conflicts.
Running as SYSTEM
Setting status of 4ef7a9e281fdb8f84aae202ca3b2b90d1882ce6c to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_models/1749/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/863/*:refs/remotes/origin/pr/863/* # timeout=10
 > git rev-parse 4ef7a9e281fdb8f84aae202ca3b2b90d1882ce6c^{commit} # timeout=10
Checking out Revision 4ef7a9e281fdb8f84aae202ca3b2b90d1882ce6c (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4ef7a9e281fdb8f84aae202ca3b2b90d1882ce6c # timeout=10
Commit message: "bug fix comma"
 > git rev-list --no-walk 475145a583de8734ca3b2b03888459f149000c6a # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins8968674212784808091.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.8)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.5.0)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: traitlets>=5.2.2 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (5.4.0)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.5)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.16.0)
Requirement already satisfied: jupyter_core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.11.1)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.16.1)
Requirement already satisfied: pkgutil-resolve-name>=1.3.10 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (1.3.10)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (22.1.0)
Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.9.0)
Requirement already satisfied: tornado>=6.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (24.0.0)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-3.0.2, cov-4.0.0
collected 734 items / 1 error
==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/tf/inputs/test_embedding.py ____________

/usr/local/lib/python3.8/dist-packages/_pytest/python.py:608: in _importtestmodule

mod = import_path(self.path, mode=importmode, root=self.config.rootpath)

/usr/local/lib/python3.8/dist-packages/_pytest/pathlib.py:533: in import_path

importlib.import_module(module_name)

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

:1014: in _gcd_import

???

:991: in _find_and_load

???

:975: in _find_and_load_unlocked

???

:671: in _load_unlocked

???

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:159: in exec_module

source_stat, co = _rewrite_test(fn, self.config)

/usr/local/lib/python3.8/dist-packages/_pytest/assertion/rewrite.py:337: in _rewrite_test

tree = ast.parse(source, filename=strfn)

/usr/lib/python3.8/ast.py:47: in parse

return compile(source, filename, mode, flags,

E     File "/var/jenkins_home/workspace/merlin_models/models/tests/unit/tf/inputs/test_embedding.py", line 346

E       local_indices = []

E                        ^

E   IndentationError: unindent does not match any outer indentation level

=============================== warnings summary ===============================

../../../../../usr/lib/python3/dist-packages/requests/init.py:89

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!

========================= 7 warnings, 1 error in 6.68s =========================

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://github.com/gitapi/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins16694818952972757126.sh

merlin/models/tf/inputs/embedding.py

marcromeyn · 2022-11-07T08:48:27Z

merlin/models/tf/inputs/embedding.py

+            When it's string, it specifies the initializer used to generate initial values.
+            When it's list of numpy.array, its shape must be [vocab_size[i], embedding_vec_size],
+            and will be used as the initial indices and value.
+    use_dynamic_variable: bool = "False" use sok.DynamicVariable or sok.Variable


Would be great to elaborate a bit more about what the difference is between the 2 types of variables.

marcromeyn · 2022-11-07T08:49:37Z

merlin/models/tf/inputs/embedding.py

@@ -150,6 +150,182 @@ def from_config(cls, config):

        return cls(dim, *schema, **config)

+@tf.keras.utils.register_keras_serializable(package="merlin.models")
+class SOKEmbedding(EmbeddingTableBase):
+    """


A one-line summary of the class would be great here.

marcromeyn · 2022-11-07T08:50:35Z

merlin/models/tf/inputs/embedding.py

+                self._trainable_weights.append(v)
+                self._vars.append(v)
+
+    def call(self, inputs, training=True):


Should we throw in exception when a dense-tensor is provided here?

sok.lookup_sparse can accept Sparse and dense-tensor, but both of them should have a combiner

292 def lookup_sparse(params, sp_ids, combiners): 293 """ 294 Abbreviated as ``sok.experiment.lookup_sparse``. 295 296 Peform fused sparse lookup on the given embedding ``params``. This function 297 is similar to the ``tf.nn.embedding_lookup_sparse``, but with two differences: 298 299 - It can do distributed lookup. 300 - It can accept multiple params and multiple sp_ids to do fused lookup at once, 301 which brings performance benifits. 302 303 Parameters 304 ---------- 305 params: list, tuple 306 a list or tuple of trainable *sok.Variable*. 307 sp_ids: list, tuple 308 a list or tuple of tf.SparseTensor or tf.RaggedTensor. 309 combiners: list, tuple 310 a list or tuple of string to specify the combiner of each lookup.

nvidia-merlin-bot · 2022-11-16T12:58:24Z

Click to view CI Results

GitHub pull request #863 of commit 720eacf80e9b8ca049a742e2f070e70d45c0d41f, no merge conflicts.
Running as SYSTEM
Setting status of 720eacf80e9b8ca049a742e2f070e70d45c0d41f to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_models/1857/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/863/*:refs/remotes/origin/pr/863/* # timeout=10
 > git rev-parse 720eacf80e9b8ca049a742e2f070e70d45c0d41f^{commit} # timeout=10
Checking out Revision 720eacf80e9b8ca049a742e2f070e70d45c0d41f (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 720eacf80e9b8ca049a742e2f070e70d45c0d41f # timeout=10
Commit message: "add some comments and test distributed var"
 > git rev-list --no-walk 6287ef08ab172a4fce1bb1de221fa1bd6ad7dfcb # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins3772631357973278548.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.8)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.5.0)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: traitlets>=5.2.2 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (5.4.0)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.5)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.16.0)
Requirement already satisfied: jupyter_core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.11.1)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.16.1)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (22.1.0)
Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.9.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: pkgutil-resolve-name>=1.3.10 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (1.3.10)
Requirement already satisfied: tornado>=6.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (24.0.0)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
ERROR: tox config file (either pyproject.toml, tox.ini, setup.cfg) not found
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://github.com/gitapi/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log" 
[merlin_models] $ /bin/bash /tmp/jenkins339503894282912997.sh

nvidia-merlin-bot · 2022-11-17T06:09:28Z

Click to view CI Results

GitHub pull request #863 of commit 7a3a177a07522985a29e2d4875120e50e5f4295a, no merge conflicts.
Running as SYSTEM
Setting status of 7a3a177a07522985a29e2d4875120e50e5f4295a to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_models/1876/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/863/*:refs/remotes/origin/pr/863/* # timeout=10
 > git rev-parse 7a3a177a07522985a29e2d4875120e50e5f4295a^{commit} # timeout=10
Checking out Revision 7a3a177a07522985a29e2d4875120e50e5f4295a (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7a3a177a07522985a29e2d4875120e50e5f4295a # timeout=10
Commit message: "format the comments"
 > git rev-list --no-walk a189a59e2cdd2ab8c72352cae720c3eb7b7c42f7 # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins1099958190960743523.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.5.0)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.8)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.5)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: traitlets>=5.2.2 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (5.4.0)
Requirement already satisfied: jupyter_core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.11.1)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.16.0)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.16.1)
Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.9.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (22.1.0)
Requirement already satisfied: pkgutil-resolve-name>=1.3.10 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (1.3.10)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (24.0.0)
Requirement already satisfied: tornado>=6.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.2)
Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
ERROR: tox config file (either pyproject.toml, tox.ini, setup.cfg) not found
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://github.com/gitapi/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log" 
[merlin_models] $ /bin/bash /tmp/jenkins2561788083667233881.sh

bschifferer · 2022-11-22T13:09:59Z

Before we finish the integration, can someone provide a high-level explanation how the feature will work / will look like?
How does it work with the current horovod implementation for multi-gpu data parallel training?

It should be easy for the user to change between TF embeddings and SOK embeddings.

github-actions · 2022-12-14T14:10:28Z

Documentation preview

https://nvidia-merlin.github.io/models/review/pr-863

edknv · 2022-12-14T14:15:17Z

I moved the class to a separate file.

@WonderingWJ Can you help complete this? Some variables are undefined:

merlin/models/tf/distributed/embedding.py:58:42: F821 undefined name 'trainable'
merlin/models/tf/distributed/embedding.py:58:70: F821 undefined name 'dtype'
merlin/models/tf/distributed/embedding.py:77:20: F821 undefined name '_use_dynamic_variable'
merlin/models/tf/distributed/embedding.py:170:13: F821 undefined name 'dim'

and in the unit test, sok doesn't seem to have an init() method (Should this be captailzaed, i.e., Init?)

    @pytest.mark.parametrize("dim", [16, 32])
    def test_distributed_variables(self, dim):
        hvd.init()
>       sok.init()
E       AttributeError: module 'sparse_operation_kit' has no attribute 'init'

merlin/models/tf/distributed/embedding.py

edknv · 2023-03-06T17:14:12Z

@WonderingWJ Sorry, it looks like I was indeed talking about a CI error in horovod (cpu). Sorry for the confusion. I was trying to install sok in horovod (cpu) (which doesn't have GPUs) and was running into installation issues. But let's not worry about that now. I think there are other major issues to take care of, which you can see by running the tests locally. Or hopefully #1009 is merged soon so you can use the CI to see the errors.

WonderingWJ · 2023-03-07T03:13:57Z

tests

@edknv Thank you! After the MR is merged, I will test it. And I will try to run the test locally and try to find the problem

edknv · 2023-03-08T18:25:17Z

@WonderingWJ #1009 has been merged. Do we need to do anything other than pip install sparse_operation_kit to install SOK? I pushed a commit for pip-installing SOK in b716adc and we are getting the following:

tensorflow.python.framework.errors_impl.NotFoundError: /home/gha-user/gha1/models/models/.tox/py38-multi-gpu/lib/python3.8/site-packages/sparse_operation_kit/lib/libsparse_operation_kit.so: undefined symbol: cusparseDestroy

(link to GHA logs)
Maybe it's an issue with tox and we need to pass some environment variable or something.

Also some errors due to missing variables:

[1,1]<stdout>:>       return cls(
[1,1]<stdout>:            dim,
[1,1]<stdout>:            col_schema,
[1,1]<stdout>:            name=name,
[1,1]<stdout>:            initializer=weights,
[1,1]<stdout>:            use_dynamic_variable=use_dynamic_variable,
[1,1]<stdout>:            localized=localized,
[1,1]<stdout>:            trainable=trainable,
[1,1]<stdout>:            **kwargs,
[1,1]<stdout>:        )
[1,1]<stdout>:E       TypeError: __init__() missing 1 required keyword-only argument: 'vocab_sizes'
[1,1]<stdout>:
[1,1]<stdout>:merlin/models/tf/distributed/embedding.py:179: TypeError

(link to another GHA logs)

edknv · 2023-03-08T18:35:37Z

@WonderingWJ If it's not so straightforward to install sok in the CI, we could also manually test this PR as described in my previous comment, merge this PR as soon as the manual testing passes, and tackle the CI in a separate PR.

WonderingWJ · 2023-03-09T02:34:00Z

@WonderingWJ #1009 has been merged. Do we need to do anything other than pip install sparse_operation_kit to install SOK? I pushed a commit for pip-installing SOK in b716adc and we are getting the following:
tensorflow.python.framework.errors_impl.NotFoundError: /home/gha-user/gha1/models/models/.tox/py38-multi-gpu/lib/python3.8/site-packages/sparse_operation_kit/lib/libsparse_operation_kit.so: undefined symbol: cusparseDestroy
(link to GHA logs) Maybe it's an issue with tox and we need to pass some environment variable or something.

Also some errors due to missing variables:
[1,1]<stdout>:>       return cls(
[1,1]<stdout>:            dim,
[1,1]<stdout>:            col_schema,
[1,1]<stdout>:            name=name,
[1,1]<stdout>:            initializer=weights,
[1,1]<stdout>:            use_dynamic_variable=use_dynamic_variable,
[1,1]<stdout>:            localized=localized,
[1,1]<stdout>:            trainable=trainable,
[1,1]<stdout>:            **kwargs,
[1,1]<stdout>:        )
[1,1]<stdout>:E       TypeError: __init__() missing 1 required keyword-only argument: 'vocab_sizes'
[1,1]<stdout>:
[1,1]<stdout>:merlin/models/tf/distributed/embedding.py:179: TypeError
(link to another GHA logs)

Thank you for your work. the install of SOK should be straightforward. Let me check about the install and your running error. I will finish it this week.

WonderingWJ · 2023-03-12T16:00:19Z

@edknv
I have debug the errors and resolved the issue with two tests. But one test still failes. Can you give me a hand?

merlin/models/tf/distributed/embedding.py:224: in from_config
    return cls(dim, *schema, vocab_size, initializer, use_dynamic_variable, localized, **config)
merlin/models/tf/distributed/embedding.py:66: in __init__
    super(SOKEmbedding, self).__init__(
merlin/models/tf/inputs/embedding.py:80: in __init__
    self.add_feature(col_schema)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = SOKEmbedding(
  (features): Dict(
    (item_id): ColumnSchema(name='item_id', tags={<Tags.CATEGORICAL: 'categorical'>}...rties={'domain': {'min': 0, 'max': 10, 'name': 'item_id'}}, dtype=dtype('int32'), is_list=False, is_ragged=False)
  )
)
col_schema = List(
  (0): 10
)

    def add_feature(self, col_schema: ColumnSchema) -> None:
        """Add a feature to the table.

        Adding more than one feature enables the table to lookup and return embeddings
        for more than one feature when called with tabular data (Dict[str, TensorLike]).

        Additional column schemas must have an int domain that matches the existing ones.

        Parameters
        ----------
        col_schema : ColumnSchema
        """
>       if not col_schema.int_domain:
E       AttributeError: 'ListWrapper' object has no attribute 'int_domain'

merlin/models/tf/inputs/embedding.py:116: AttributeError

Can you help me with the error? I don't know what's wrong with the col_schema. Thank you!

edknv · 2023-03-14T00:34:15Z

It looks to me like the wrong values are being passed to col_schema because of the variable-length argument *schema in

merlin/models/tf/distributed/embedding.py:224: in from_config
    return cls(dim, *schema, vocab_size, initializer, use_dynamic_variable, localized, **config)

I think the following in the log is incorrect:

col_schema = List(
  (0): 10
)

and [10] should actually be passed to vocab_size?

Maybe use the keyword arguments so that the variable-length argument *schema can be parsed correctly?

return cls(dim, *schema, vocab_size=vocab_size, initializer=initializer, use_dynamic_variable=use_dynamic_variable, localized=localized, **config)

WonderingWJ · 2023-03-15T09:51:01Z

length

Thank you for your reply, it helps. All tests are passed. Thank you

edknv · 2023-03-20T07:07:50Z

@marcromeyn I'm okay with merging this if you are.

edknv · 2023-03-21T15:45:11Z

The implementation is pretty much isolated and independent of the rest of the library, so it should be good to merge. I will go ahead and merge this. If there are any concerns, we can address them later.

zhuwenjing added 4 commits November 4, 2022 20:13

new sok class

0141ad4

new sok class

d7ad9ba

test sok dynamic variable

7825963

test sok dynamic variable

475145a

WonderingWJ requested a review from marcromeyn November 7, 2022 03:08

bug fix comma

4ef7a9e

WonderingWJ changed the title ~~Fea sok integration wj~~ feature::Fea sok integration wj Nov 7, 2022

WonderingWJ changed the title ~~feature::Fea sok integration wj~~ feature sok integration wj Nov 7, 2022

WonderingWJ added the enhancement New feature or request label Nov 7, 2022

marcromeyn reviewed Nov 7, 2022

View reviewed changes

merlin/models/tf/inputs/embedding.py Outdated Show resolved Hide resolved

marcromeyn reviewed Nov 7, 2022

View reviewed changes

merlin/models/tf/inputs/embedding.py Outdated Show resolved Hide resolved

marcromeyn reviewed Nov 7, 2022

View reviewed changes

add some comments and test distributed var

720eacf

format the comments

7a3a177

wenjingz and others added 4 commits December 5, 2022 23:59

assert condition in sok lookup sparse

54f795b

Merge branch 'main' into fea-sok-integration-wj

f3136b7

Move SOKEmbedding to a separate file

7555c6f

Clean up

d0111b1

Clean up

debb6e2

marcromeyn reviewed Dec 15, 2022

View reviewed changes

merlin/models/tf/distributed/embedding.py Outdated Show resolved Hide resolved

fix some import and param bug

a4e3ffc

rnyak and others added 4 commits March 8, 2023 07:05

Merge branch 'main' into fea-sok-integration-wj

296a151

Merge branch 'main' into fea-sok-integration-wj

ef1c0bc

pip install sparse_operation_kit in tox.ini

b716adc

fix spelling

b1c1031

wenjingz added 6 commits March 12, 2023 07:10

resolve init method issue

91f25d1

fix schema issue

d0671c9

add indices and weights in from_pretrained

0eb266d

schema issue in from_pretrained

e9cc754

init method in DET

5532ab7

tensor type in sok assing

c9d3baf

wenjingz and others added 6 commits March 15, 2023 02:59

resolve config passing

33abba0

Merge branch 'main' into fea-sok-integration-wj

4a9465d

lint

b2a01c2

Merge branch 'main' into fea-sok-integration-wj

b862a2a

install sok in tox

0fd059a

Merge branch 'main' into fea-sok-integration-wj

fcaefc3

edknv changed the title ~~feature sok integration wj~~ Introduce SOKEmbedding using Sparse Operation Kit Mar 20, 2023

edknv approved these changes Mar 20, 2023

View reviewed changes

edknv merged commit 13e7fcb into main Mar 21, 2023

edknv mentioned this pull request Mar 23, 2023

Skip SOK tests if SOK is not installed #1030

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce SOKEmbedding using Sparse Operation Kit #863

Introduce SOKEmbedding using Sparse Operation Kit #863

WonderingWJ commented Nov 7, 2022 •

edited

Loading

nvidia-merlin-bot commented Nov 7, 2022

nvidia-merlin-bot commented Nov 7, 2022

marcromeyn Nov 7, 2022

marcromeyn Nov 7, 2022

marcromeyn Nov 7, 2022

WonderingWJ Nov 13, 2022 •

edited

Loading

nvidia-merlin-bot commented Nov 16, 2022

nvidia-merlin-bot commented Nov 17, 2022

bschifferer commented Nov 22, 2022 •

edited

Loading

github-actions bot commented Dec 14, 2022

edknv commented Dec 14, 2022

edknv commented Mar 6, 2023

WonderingWJ commented Mar 7, 2023 •

edited

Loading

edknv commented Mar 8, 2023

edknv commented Mar 8, 2023

WonderingWJ commented Mar 9, 2023

WonderingWJ commented Mar 12, 2023 •

edited

Loading

edknv commented Mar 14, 2023

WonderingWJ commented Mar 15, 2023 •

edited

Loading

edknv commented Mar 20, 2023

edknv commented Mar 21, 2023

Introduce SOKEmbedding using Sparse Operation Kit #863

Introduce SOKEmbedding using Sparse Operation Kit #863

Conversation

WonderingWJ commented Nov 7, 2022 • edited Loading

Goals ⚽

Implementation Details 🚧

Testing Details 🔍

nvidia-merlin-bot commented Nov 7, 2022

nvidia-merlin-bot commented Nov 7, 2022

marcromeyn Nov 7, 2022

Choose a reason for hiding this comment

marcromeyn Nov 7, 2022

Choose a reason for hiding this comment

marcromeyn Nov 7, 2022

Choose a reason for hiding this comment

WonderingWJ Nov 13, 2022 • edited Loading

Choose a reason for hiding this comment

nvidia-merlin-bot commented Nov 16, 2022

nvidia-merlin-bot commented Nov 17, 2022

bschifferer commented Nov 22, 2022 • edited Loading

github-actions bot commented Dec 14, 2022

Documentation preview

edknv commented Dec 14, 2022

edknv commented Mar 6, 2023

WonderingWJ commented Mar 7, 2023 • edited Loading

edknv commented Mar 8, 2023

edknv commented Mar 8, 2023

WonderingWJ commented Mar 9, 2023

WonderingWJ commented Mar 12, 2023 • edited Loading

edknv commented Mar 14, 2023

WonderingWJ commented Mar 15, 2023 • edited Loading

edknv commented Mar 20, 2023

edknv commented Mar 21, 2023

WonderingWJ commented Nov 7, 2022 •

edited

Loading

WonderingWJ Nov 13, 2022 •

edited

Loading

bschifferer commented Nov 22, 2022 •

edited

Loading

WonderingWJ commented Mar 7, 2023 •

edited

Loading

WonderingWJ commented Mar 12, 2023 •

edited

Loading

WonderingWJ commented Mar 15, 2023 •

edited

Loading