Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Revert to working tritonserver call in notebook using testbooks #619

Merged
merged 6 commits into from
Sep 14, 2022
Merged

Revert to working tritonserver call in notebook using testbooks #619

merged 6 commits into from
Sep 14, 2022

Conversation

jperez999
Copy link
Collaborator

@jperez999 jperez999 commented Sep 14, 2022 •
edited
Loading

Previously we replaced this code because of errors in starting tritonserver based on the tritonserver 22.07 container. Those errors are now resolved and we can run tritonserver from inside the notebook inside the testbooks context again. This is required because of changes in the systems library that now requires the input schema to run the run_ensemble_on_tritonserver function from merlin.systems.triton.utils.

@jperez999 jperez999 self-assigned this Sep 14, 2022
@jperez999 jperez999 added chore Infrastructure update ci labels Sep 14, 2022
@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #619 of commit b9d4433d003d8a3a8dd26e83769f2954b5b4fc30, no merge conflicts.
Running as SYSTEM
Setting status of b9d4433d003d8a3a8dd26e83769f2954b5b4fc30 to PENDING with url https://10.20.13.93:8080/job/merlin_merlin/431/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/619/*:refs/remotes/origin/pr/619/* # timeout=10
 > git rev-parse b9d4433d003d8a3a8dd26e83769f2954b5b4fc30^{commit} # timeout=10
Checking out Revision b9d4433d003d8a3a8dd26e83769f2954b5b4fc30 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b9d4433d003d8a3a8dd26e83769f2954b5b4fc30 # timeout=10
Commit message: "revert to now working tritonserver call in notebook using testbooks"
 > git rev-list --no-walk f3700c2143c6c786849c42b761ef1b83359e0d75 # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins168110364284359457.sh
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 3 items

tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py s      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py F               [100%]


=================================== FAILURES ===================================

__________________________________ test_func ___________________________________


self = <testbook.client.TestbookNotebookClient object at 0x7ffafb882be0>

cell = [31], kwargs = {}, cell_indexes = [31], executed_cells = [], idx = 31


def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:





          cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)





/usr/local/lib/python3.8/dist-packages/testbook/client.py:133:




args = (<testbook.client.TestbookNotebookClient object at 0x7ffafb882be0>, {'id': '8d44edff', 'cell_type': 'code', 'metadata'...n', '\x1b[0;31mTypeError\x1b[0m: run_ensemble_on_tritonserver() takes 4 positional arguments but 5 were given']}]}, 31)

kwargs = {}


def wrapped(*args, **kwargs):





  return just_run(coro(*args, **kwargs))





/usr/local/lib/python3.8/dist-packages/nbclient/util.py:85:




coro = <coroutine object NotebookClient.async_execute_cell at 0x7ffafc20b340>


def just_run(coro: Awaitable) -> Any:
    """Make the coroutine run, even if there is an event loop running (using nest_asyncio)"""
    try:
        loop = asyncio.get_running_loop()
    except RuntimeError:
        loop = None
    if loop is None:
        had_running_loop = False
        loop = asyncio.new_event_loop()
        asyncio.set_event_loop(loop)
    else:
        had_running_loop = True
    if had_running_loop:
        # if there is a running loop, we patch using nest_asyncio
        # to have reentrant event loops
        check_ipython()
        import nest_asyncio

        nest_asyncio.apply()
        check_patch_tornado()





  return loop.run_until_complete(coro)





/usr/local/lib/python3.8/dist-packages/nbclient/util.py:60:




self = <_UnixSelectorEventLoop running=False closed=False debug=False>

future = <Task finished name='Task-228' coro=<NotebookClient.async_execute_cell() done, defined at /usr/local/lib/python3.8/dis...rguments but 5 were given\nTypeError: run_ensemble_on_tritonserver() takes 4 positional arguments but 5 were given\n')>


def run_until_complete(self, future):
    """Run until the Future is done.

    If the argument is a coroutine, it is wrapped in a Task.

    WARNING: It would be disastrous to call run_until_complete()
    with the same coroutine twice -- it would wrap it in two
    different Tasks and that can't be good.

    Return the Future's result, or raise its exception.
    """
    self._check_closed()
    self._check_running()

    new_task = not futures.isfuture(future)
    future = tasks.ensure_future(future, loop=self)
    if new_task:
        # An exception is raised if the future didn't complete, so there
        # is no need to log the "destroy pending task" message
        future._log_destroy_pending = False

    future.add_done_callback(_run_until_complete_cb)
    try:
        self.run_forever()
    except:
        if new_task and future.done() and not future.cancelled():
            # The coroutine raised a BaseException. Consume the exception
            # to not log a warning, the caller doesn't have access to the
            # local task.
            future.exception()
        raise
    finally:
        future.remove_done_callback(_run_until_complete_cb)
    if not future.done():
        raise RuntimeError('Event loop stopped before Future completed.')





  return future.result()





/usr/lib/python3.8/asyncio/base_events.py:616:




self = <testbook.client.TestbookNotebookClient object at 0x7ffafb882be0>

cell = {'id': '8d44edff', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-09-14T02:30:40.549087Z',...00m)\n', '\x1b[0;31mTypeError\x1b[0m: run_ensemble_on_tritonserver() takes 4 positional arguments but 5 were given']}]}

cell_index = 31, execution_count = None, store_history = True


async def async_execute_cell(
    self,
    cell: NotebookNode,
    cell_index: int,
    execution_count: t.Optional[int] = None,
    store_history: bool = True,
) -> NotebookNode:
    """
    Executes a single code cell.

    To execute all cells see :meth:`execute`.

    Parameters
    ----------
    cell : nbformat.NotebookNode
        The cell which is currently being processed.
    cell_index : int
        The position of the cell within the notebook object.
    execution_count : int
        The execution count to be assigned to the cell (default: Use kernel response)
    store_history : bool
        Determines if history should be stored in the kernel (default: False).
        Specific to ipython kernels, which can store command histories.

    Returns
    -------
    output : dict
        The execution output payload (or None for no output).

    Raises
    ------
    CellExecutionError
        If execution failed and should raise an exception, this will be raised
        with defaults about the failure.

    Returns
    -------
    cell : NotebookNode
        The cell which was just processed.
    """
    assert self.kc is not None

    await run_hook(self.on_cell_start, cell=cell, cell_index=cell_index)

    if cell.cell_type != 'code' or not cell.source.strip():
        self.log.debug("Skipping non-executing cell %s", cell_index)
        return cell

    if self.skip_cells_with_tag in cell.metadata.get("tags", []):
        self.log.debug("Skipping tagged cell %s", cell_index)
        return cell

    if self.record_timing:  # clear execution metadata prior to execution
        cell['metadata']['execution'] = {}

    self.log.debug("Executing cell:\n%s", cell.source)

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors or "raises-exception" in cell.metadata.get("tags", [])
    )

    await run_hook(self.on_cell_execute, cell=cell, cell_index=cell_index)
    parent_msg_id = await ensure_async(
        self.kc.execute(
            cell.source, store_history=store_history, stop_on_error=not cell_allows_errors
        )
    )
    await run_hook(self.on_cell_complete, cell=cell, cell_index=cell_index)
    # We launched a code cell to execute
    self.code_cells_executed += 1
    exec_timeout = self._get_timeout(cell)

    cell.outputs = []
    self.clear_before_next_output = False

    task_poll_kernel_alive = asyncio.ensure_future(self._async_poll_kernel_alive())
    task_poll_output_msg = asyncio.ensure_future(
        self._async_poll_output_msg(parent_msg_id, cell, cell_index)
    )
    self.task_poll_for_reply = asyncio.ensure_future(
        self._async_poll_for_reply(
            parent_msg_id, cell, exec_timeout, task_poll_output_msg, task_poll_kernel_alive
        )
    )
    try:
        exec_reply = await self.task_poll_for_reply
    except asyncio.CancelledError:
        # can only be cancelled by task_poll_kernel_alive when the kernel is dead
        task_poll_output_msg.cancel()
        raise DeadKernelError("Kernel died")
    except Exception as e:
        # Best effort to cancel request if it hasn't been resolved
        try:
            # Check if the task_poll_output is doing the raising for us
            if not isinstance(e, CellControlSignal):
                task_poll_output_msg.cancel()
        finally:
            raise

    if execution_count:
        cell['execution_count'] = execution_count
    await run_hook(
        self.on_cell_executed, cell=cell, cell_index=cell_index, execute_reply=exec_reply
    )





  await self._check_raise_for_error(cell, cell_index, exec_reply)





/usr/local/lib/python3.8/dist-packages/nbclient/client.py:1025:




self = <testbook.client.TestbookNotebookClient object at 0x7ffafb882be0>

cell = {'id': '8d44edff', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-09-14T02:30:40.549087Z',...00m)\n', '\x1b[0;31mTypeError\x1b[0m: run_ensemble_on_tritonserver() takes 4 positional arguments but 5 were given']}]}

cell_index = 31

exec_reply = {'buffers': [], 'content': {'ename': 'TypeError', 'engine_info': {'engine_id': -1, 'engine_uuid': 'bc298d31-d51b-4211-...e, 'engine': 'bc298d31-d51b-4211-9471-87cfc8c64772', 'started': '2022-09-14T02:30:40.549309Z', 'status': 'error'}, ...}


async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(
        self.on_cell_error, cell=cell, cell_index=cell_index, execute_reply=exec_reply
    )
    if not cell_allows_errors:





      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)





E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E

E           import shutil

E           from merlin.systems.triton.utils import run_ensemble_on_tritonserver

E           outputs = ensemble.graph.output_schema.column_names

E           response = run_ensemble_on_tritonserver(

E               "/tmp/output/criteo/ensemble/",workflow.input_schema, batch.fillna(0),

E               outputs, "ensemble_model"

E           )

E           response = [x.tolist()[0] for x in response["label/binary_classification_task"]]

E           shutil.rmtree("/tmp/input/criteo", ignore_errors=True)

E           shutil.rmtree("/tmp/output/criteo", ignore_errors=True)

E

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mTypeError�[0m                                 Traceback (most recent call last)

E           Input �[0;32mIn [13]�[0m, in �[0;36m<cell line: 4>�[0;34m()�[0m

E           �[1;32m      2�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01msystems�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtriton�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mutils�[39;00m �[38;5;28;01mimport�[39;00m run_ensemble_on_tritonserver

E           �[1;32m      3�[0m outputs �[38;5;241m=�[39m ensemble�[38;5;241m.�[39mgraph�[38;5;241m.�[39moutput_schema�[38;5;241m.�[39mcolumn_names

E           �[0;32m----> 4�[0m response �[38;5;241m=�[39m �[43mrun_ensemble_on_tritonserver�[49m�[43m(�[49m

E           �[1;32m      5�[0m �[43m    �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m/tmp/output/criteo/ensemble/�[39;49m�[38;5;124;43m"�[39;49m�[43m,�[49m�[43mworkflow�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[38;5;241;43m.�[39;49m�[43mfillna�[49m�[43m(�[49m�[38;5;241;43m0�[39;49m�[43m)�[49m�[43m,�[49m

E           �[1;32m      6�[0m �[43m    �[49m�[43moutputs�[49m�[43m,�[49m�[43m �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43mensemble_model�[39;49m�[38;5;124;43m"�[39;49m

E           �[1;32m      7�[0m �[43m)�[49m

E           �[1;32m      8�[0m response �[38;5;241m=�[39m [x�[38;5;241m.�[39mtolist()[�[38;5;241m0�[39m] �[38;5;28;01mfor�[39;00m x �[38;5;129;01min�[39;00m response[�[38;5;124m"�[39m�[38;5;124mlabel/binary_classification_task�[39m�[38;5;124m"�[39m]]

E           �[1;32m      9�[0m shutil�[38;5;241m.�[39mrmtree(�[38;5;124m"�[39m�[38;5;124m/tmp/input/criteo�[39m�[38;5;124m"�[39m, ignore_errors�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)

E

E           �[0;31mTypeError�[0m: run_ensemble_on_tritonserver() takes 4 positional arguments but 5 were given

E           TypeError: run_ensemble_on_tritonserver() takes 4 positional arguments but 5 were given


/usr/local/lib/python3.8/dist-packages/nbclient/client.py:919: CellExecutionError


During handling of the above exception, another exception occurred:


def test_func():
    with testbook(
        REPO_ROOT / "examples" / "scaling-criteo" / "02-ETL-with-NVTabular.ipynb",
        execute=False,
        timeout=180,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["BASE_DIR"] = "/tmp/input/criteo/"
            os.environ["INPUT_DATA_DIR"] = "/tmp/input/criteo/"
            os.environ["OUTPUT_DATA_DIR"] = "/tmp/output/criteo/"
            os.system("mkdir -p /tmp/input/criteo")
            os.system("mkdir -p /tmp/output/criteo")

            from merlin.datasets.synthetic import generate_data

            train, valid = generate_data("criteo", int(100000), set_sizes=(0.7, 0.3))

            train.to_ddf().compute().to_parquet('/tmp/input/criteo/day_0.parquet')
            valid.to_ddf().compute().to_parquet('/tmp/input/criteo/day_1.parquet')
            """
        )
        tb1.execute()
        assert os.path.isfile("/tmp/output/criteo/train/part_0.parquet")
        assert os.path.isfile("/tmp/output/criteo/valid/part_0.parquet")
        assert os.path.isfile("/tmp/output/criteo/workflow/metadata.json")

    with testbook(
        REPO_ROOT
        / "examples"
        / "scaling-criteo"
        / "03-Training-with-Merlin-Models-TensorFlow.ipynb",
        execute=False,
        timeout=180,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["INPUT_DATA_DIR"] = "/tmp/output/criteo/"
            """
        )
        tb2.execute()
        metrics = tb2.ref("eval_metrics")
        assert set(metrics.keys()) == set(
            [
                "auc",
                "binary_accuracy",
                "loss",
                "precision",
                "recall",
                "regularization_loss",
            ]
        )
        assert os.path.isfile("/tmp/output/criteo/dlrm/saved_model.pb")

    with testbook(
        REPO_ROOT
        / "examples"
        / "scaling-criteo"
        / "04-Triton-Inference-with-Merlin-Models-TensorFlow.ipynb",
        execute=False,
        timeout=180,
    ) as tb3:
        tb3.inject(
            """
            import os
            os.environ["BASE_DIR"] = "/tmp/output/criteo/"
            os.environ["INPUT_FOLDER"] = "/tmp/input/criteo/"
            """
        )
        NUM_OF_CELLS = len(tb3.cells)
        tb3.execute_cell(list(range(0, NUM_OF_CELLS - 5)))





      tb3.inject(





            """
            import shutil
            from merlin.systems.triton.utils import run_ensemble_on_tritonserver
            outputs = ensemble.graph.output_schema.column_names
            response = run_ensemble_on_tritonserver(
                "/tmp/output/criteo/ensemble/",workflow.input_schema, batch.fillna(0),
                outputs, "ensemble_model"
            )
            response = [x.tolist()[0] for x in response["label/binary_classification_task"]]
            shutil.rmtree("/tmp/input/criteo", ignore_errors=True)
            shutil.rmtree("/tmp/output/criteo", ignore_errors=True)
            """
        )



tests/unit/examples/test_scaling_criteo_merlin_models.py:83:




/usr/local/lib/python3.8/dist-packages/testbook/client.py:237: in inject

cell = TestbookNode(self.execute_cell(inject_idx)) if run else TestbookNode(code_cell)




self = <testbook.client.TestbookNotebookClient object at 0x7ffafb882be0>

cell = [31], kwargs = {}, cell_indexes = [31], executed_cells = [], idx = 31


def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:
            cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)
        except CellExecutionError as ce:





          raise TestbookRuntimeError(ce.evalue, ce, self._get_error_class(ce.ename))





E               testbook.exceptions.TestbookRuntimeError: An error occurred while executing the following cell:

E               ------------------

E

E               import shutil

E               from merlin.systems.triton.utils import run_ensemble_on_tritonserver

E               outputs = ensemble.graph.output_schema.column_names

E               response = run_ensemble_on_tritonserver(

E                   "/tmp/output/criteo/ensemble/",workflow.input_schema, batch.fillna(0),

E                   outputs, "ensemble_model"

E               )

E               response = [x.tolist()[0] for x in response["label/binary_classification_task"]]

E               shutil.rmtree("/tmp/input/criteo", ignore_errors=True)

E               shutil.rmtree("/tmp/output/criteo", ignore_errors=True)

E

E               ------------------

E

E               �[0;31m---------------------------------------------------------------------------�[0m

E               �[0;31mTypeError�[0m                                 Traceback (most recent call last)

E               Input �[0;32mIn [13]�[0m, in �[0;36m<cell line: 4>�[0;34m()�[0m

E               �[1;32m      2�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01msystems�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mtriton�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mutils�[39;00m �[38;5;28;01mimport�[39;00m run_ensemble_on_tritonserver

E               �[1;32m      3�[0m outputs �[38;5;241m=�[39m ensemble�[38;5;241m.�[39mgraph�[38;5;241m.�[39moutput_schema�[38;5;241m.�[39mcolumn_names

E               �[0;32m----> 4�[0m response �[38;5;241m=�[39m �[43mrun_ensemble_on_tritonserver�[49m�[43m(�[49m

E               �[1;32m      5�[0m �[43m    �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m/tmp/output/criteo/ensemble/�[39;49m�[38;5;124;43m"�[39;49m�[43m,�[49m�[43mworkflow�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[38;5;241;43m.�[39;49m�[43mfillna�[49m�[43m(�[49m�[38;5;241;43m0�[39;49m�[43m)�[49m�[43m,�[49m

E               �[1;32m      6�[0m �[43m    �[49m�[43moutputs�[49m�[43m,�[49m�[43m �[49m�[38;5;124;43m"�[39;49m�[38;5;124;43mensemble_model�[39;49m�[38;5;124;43m"�[39;49m

E               �[1;32m      7�[0m �[43m)�[49m

E               �[1;32m      8�[0m response �[38;5;241m=�[39m [x�[38;5;241m.�[39mtolist()[�[38;5;241m0�[39m] �[38;5;28;01mfor�[39;00m x �[38;5;129;01min�[39;00m response[�[38;5;124m"�[39m�[38;5;124mlabel/binary_classification_task�[39m�[38;5;124m"�[39m]]

E               �[1;32m      9�[0m shutil�[38;5;241m.�[39mrmtree(�[38;5;124m"�[39m�[38;5;124m/tmp/input/criteo�[39m�[38;5;124m"�[39m, ignore_errors�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)

E

E               �[0;31mTypeError�[0m: run_ensemble_on_tritonserver() takes 4 positional arguments but 5 were given

E               TypeError: run_ensemble_on_tritonserver() takes 4 positional arguments but 5 were given


/usr/local/lib/python3.8/dist-packages/testbook/client.py:135: TestbookRuntimeError

----------------------------- Captured stderr call -----------------------------

2022-09-14 02:29:23,687 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-09-14 02:29:23,705 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(

/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(

Process Dask Worker process (from Nanny):

Traceback (most recent call last):

File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap

self.run()

File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run

self._target(*self._args, **self._kwargs)

File "/usr/local/lib/python3.8/dist-packages/distributed/process.py", line 175, in _run

target(*args, **kwargs)

File "/usr/local/lib/python3.8/dist-packages/distributed/nanny.py", line 918, in _run

loop.run_sync(do_stop)

File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 523, in run_sync

self.start()

File "/usr/local/lib/python3.8/dist-packages/tornado/platform/asyncio.py", line 215, in start

self.asyncio_loop.run_forever()

File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever

self._run_once()

File "/usr/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once

handle._run()

File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run

self._context.run(self._callback, *self._args)

File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 687, in 

lambda f: self._run_callback(functools.partial(callback, future))

File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 740, in _run_callback

ret = callback()

File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 764, in _discard_future_result

future.result()

File "/usr/local/lib/python3.8/dist-packages/distributed/nanny.py", line 911, in _run

loop.run_sync(run)

File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 523, in run_sync

self.start()

File "/usr/local/lib/python3.8/dist-packages/tornado/platform/asyncio.py", line 215, in start

self.asyncio_loop.run_forever()

File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever

self._run_once()

File "/usr/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once

handle._run()

File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run

self._context.run(self._callback, *self._args)

File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 921, in _run

val = self.callback()

File "/usr/local/lib/python3.8/dist-packages/distributed/system_monitor.py", line 115, in update

disk_ioc = psutil.disk_io_counters()

File "/usr/local/lib/python3.8/dist-packages/psutil/init.py", line 2064, in disk_io_counters

rawdict = _psplatform.disk_io_counters(**kwargs)

File "/usr/local/lib/python3.8/dist-packages/psutil/_pslinux.py", line 1157, in disk_io_counters

if not perdisk and not is_storage_device(name):

File "/usr/local/lib/python3.8/dist-packages/psutil/_pslinux.py", line 256, in is_storage_device

return os.access(path, os.F_OK)

KeyboardInterrupt

/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(

/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(

Process Dask Worker process (from Nanny):

Traceback (most recent call last):

File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap

self.run()

File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run

self._target(*self._args, **self._kwargs)

File "/usr/local/lib/python3.8/dist-packages/distributed/process.py", line 175, in _run

target(*args, **kwargs)

File "/usr/local/lib/python3.8/dist-packages/distributed/nanny.py", line 918, in _run

loop.run_sync(do_stop)

File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 523, in run_sync

self.start()

File "/usr/local/lib/python3.8/dist-packages/tornado/platform/asyncio.py", line 215, in start

self.asyncio_loop.run_forever()

File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever

self._run_once()

File "/usr/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once

handle._run()

File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run

self._context.run(self._callback, *self._args)

File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 687, in 

lambda f: self._run_callback(functools.partial(callback, future))

File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 740, in _run_callback

ret = callback()

File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 764, in _discard_future_result

future.result()

File "/usr/local/lib/python3.8/dist-packages/distributed/nanny.py", line 911, in _run

loop.run_sync(run)

File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 523, in run_sync

self.start()

File "/usr/local/lib/python3.8/dist-packages/tornado/platform/asyncio.py", line 215, in start

self.asyncio_loop.run_forever()

File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever

self._run_once()

File "/usr/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once

handle._run()

File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run

self._context.run(self._callback, *self._args)

File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 921, in _run

val = self.callback()

File "/usr/local/lib/python3.8/dist-packages/distributed/system_monitor.py", line 115, in update

disk_ioc = psutil.disk_io_counters()

File "/usr/local/lib/python3.8/dist-packages/psutil/init.py", line 2064, in disk_io_counters

rawdict = _psplatform.disk_io_counters(**kwargs)

File "/usr/local/lib/python3.8/dist-packages/psutil/_pslinux.py", line 1154, in disk_io_counters

for entry in gen:

File "/usr/local/lib/python3.8/dist-packages/psutil/_pslinux.py", line 1119, in read_procfs

(reads, reads_merged, rbytes, rtime, writes, writes_merged,

KeyboardInterrupt

/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 21 leaked semaphore objects to clean up at shutdown

warnings.warn('resource_tracker: There appear to be %d '

2022-09-14 02:29:39.499203: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-09-14 02:29:41.537988: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 0

2022-09-14 02:29:41.538139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1627 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-09-14 02:29:41.538930: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 1

2022-09-14 02:29:41.538986: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15153 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

2022-09-14 02:30:12.812130: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-09-14 02:30:14.877031: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 0

2022-09-14 02:30:14.877175: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1627 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-09-14 02:30:14.879055: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 1

2022-09-14 02:30:14.879109: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15153 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

=========================== short test summary info ============================

FAILED tests/unit/examples/test_scaling_criteo_merlin_models.py::test_func - ...

============== 1 failed, 1 passed, 1 skipped in 88.81s (0:01:28) ===============

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://github.com/gitapi/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins10153323276188947392.sh

@github-actions
Copy link

Documentation preview

https://nvidia-merlin.github.io/Merlin/review/pr-619

rnyak
rnyak reviewed Sep 14, 2022
View reviewed changes
tests/unit/examples/test_building_deploying_multi_stage_RecSys.py
from merlin.models.loader.tf_utils import configure_tensorflow
configure_tensorflow()
df_lib = get_lib()
batch = df_lib.read_parquet(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am changing the notebooks in this PR #618 to return the raw ids. so feeding this batch wont work. Either we can merge my PR first and then update unit test accordingly, or we can merge this PR, then merge my PR and update unit test again :)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should get what we had working, then you can modify as you see fit. :)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

regardless we should wait for the new CI container to be out. I will check on that now.

@jperez999
Copy link
Collaborator Author

rerun tests

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #619 of commit b9d4433d003d8a3a8dd26e83769f2954b5b4fc30, no merge conflicts.
Running as SYSTEM
Setting status of b9d4433d003d8a3a8dd26e83769f2954b5b4fc30 to PENDING with url https://10.20.13.93:8080/job/merlin_merlin/438/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/619/*:refs/remotes/origin/pr/619/* # timeout=10
 > git rev-parse b9d4433d003d8a3a8dd26e83769f2954b5b4fc30^{commit} # timeout=10
Checking out Revision b9d4433d003d8a3a8dd26e83769f2954b5b4fc30 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b9d4433d003d8a3a8dd26e83769f2954b5b4fc30 # timeout=10
Commit message: "revert to now working tritonserver call in notebook using testbooks"
 > git rev-list --no-walk 2a7f0ba32f73cd5ea147d0a6ae2e24e0db076a5f # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins13344365416157122515.sh
ERROR: tox config file (either pyproject.toml, tox.ini, setup.cfg) not found
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://github.com/gitapi/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log" 
[merlin_merlin] $ /bin/bash /tmp/jenkins8179197293173754003.sh

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #619 of commit 2cb3b3e26abb0080ed9745683094acd9972b1315, no merge conflicts.
Running as SYSTEM
Setting status of 2cb3b3e26abb0080ed9745683094acd9972b1315 to PENDING with url https://10.20.13.93:8080/job/merlin_merlin/439/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/619/*:refs/remotes/origin/pr/619/* # timeout=10
 > git rev-parse 2cb3b3e26abb0080ed9745683094acd9972b1315^{commit} # timeout=10
Checking out Revision 2cb3b3e26abb0080ed9745683094acd9972b1315 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 2cb3b3e26abb0080ed9745683094acd9972b1315 # timeout=10
Commit message: "Merge branch 'main' into test-rework"
 > git rev-list --no-walk b9d4433d003d8a3a8dd26e83769f2954b5b4fc30 # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins5562787332586201794.sh
ERROR: No pyproject.toml or setup.py file found. The expected locations are:
  /var/jenkins_home/workspace/merlin_merlin/merlin/pyproject.toml or /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
You can
  1. Create one:
     https://tox.readthedocs.io/en/latest/example/package.html
  2. Configure tox to avoid running sdist:
     https://tox.readthedocs.io/en/latest/example/general.html
  3. Configure tox to use an isolated_build
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://github.com/gitapi/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log" 
[merlin_merlin] $ /bin/bash /tmp/jenkins7628524624479769847.sh

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #619 of commit f92797ee019fbe7f18a977180c8df8efdb42bc96, no merge conflicts.
Running as SYSTEM
Setting status of f92797ee019fbe7f18a977180c8df8efdb42bc96 to PENDING with url https://10.20.13.93:8080/job/merlin_merlin/447/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/619/*:refs/remotes/origin/pr/619/* # timeout=10
 > git rev-parse f92797ee019fbe7f18a977180c8df8efdb42bc96^{commit} # timeout=10
Checking out Revision f92797ee019fbe7f18a977180c8df8efdb42bc96 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f f92797ee019fbe7f18a977180c8df8efdb42bc96 # timeout=10
Commit message: "Merge branch 'main' into test-rework"
 > git rev-list --no-walk 572d7989ce6594d123c9a52047e9d02ee37d210e # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins13857394556403249792.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.7,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,attrs==22.1.0,awscli==1.25.73,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.51,botocore==1.27.72,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,clang==5.0,click==8.1.3,cloudpickle==2.1.0,colorama==0.4.4,coverage==6.4.4,cuda-python==11.7.1,cudf==22.4.0,cupy-cuda116==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dask-cuda==22.4.0,dask-cudf==22.4.0,dbus-python==1.2.16,debugpy==1.6.2,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.3.0,distro==1.7.0,dm-tree==0.1.7,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==0.10.0,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.82.0,fastavro==1.6.0,fastcore==1.5.24,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.0,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.0,google-auth==2.11.0,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.2,grpcio==1.41.0,grpcio-channelz==1.47.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.4.0,hugectr2onnx==0.0.0,huggingface-hub==0.8.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.0,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.1,ipython==8.4.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.1.0,json5==0.9.9,jsonschema==4.9.1,jupyter-cache==0.4.3,jupyter-client==7.3.4,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyterlab==3.4.5,jupyterlab-pygments==0.2.2,jupyterlab-server==2.15.0,jupyterlab-widgets==1.1.0,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.0,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.5.3,matplotlib-inline==0.1.3,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0+untagged.112.gfbfcaf3,merlin-models==0.6.0+45.g5a345d9c1,merlin-systems==0+untagged.105.gf89cc51,mistune==0.8.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.6,nbconvert==6.5.3,nbdime==3.1.1,nbformat==5.4.0,nest-asyncio==1.5.5,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.0,numpy==1.21.5,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.3.3+15.g16e4e34e9),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.0,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.30,proto-plus==1.19.6,protobuf==3.19.4,psutil==5.9.1,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==6.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.12.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.2,pytest-cov==3.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.7,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==23.2.1,regex==2022.7.25,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rmm==21.12.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.0,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.4,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.2.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.1.1,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.4.0,starlette==0.19.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.26.0,tensorflow-metadata==1.9.0,termcolor==1.1.0,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.25.1,tqdm==4.64.0,traitlets==5.3.0,transformers==4.12.0,transformers4rec==0.1.11+10.g21a2a836a,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.22.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.11,uvicorn==0.18.3,uvloop==0.16.0,versioneer==0.20,virtualenv==20.16.4,wandb==0.13.1,watchfiles==0.16.1,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.3.3,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='2351098846'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-_5oge_2r
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-_5oge_2r
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit 15074adf11bcc9bb9da375ec40ab4e431f1c93e9
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: nvtabular>=1.0.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.5.0+4.g15074ad) (1.4.0+7.geed9172e)
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.5.0+4.g15074ad) (0.6.0+1.g5926fcf)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2022.5.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (3.19.4)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.9.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (21.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.2.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (6.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (4.64.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.3.5)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular>=1.0.0->merlin-systems==0.5.0+4.g15074ad) (1.9.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2.1.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.2.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (5.8.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (6.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (65.3.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (6.0.1)
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-xjhbz9oi
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-xjhbz9oi
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit 91fee2a85e309cbd5d338d9ba37d989b54594498
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-models==0.7.0+5.g91fee2a8) (0.6.0+1.g5926fcf)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2022.5.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (3.19.4)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.9.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (21.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.2.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (6.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (4.64.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.3.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2.1.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.2.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (5.8.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (6.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (65.3.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (6.0.1)
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-fdmngrhp
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-fdmngrhp
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit eed9172ec30a47e020b06ff60469d6d168eb865b
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from nvtabular==1.4.0+7.geed9172e) (0.6.0+1.g5926fcf)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.4.0+7.geed9172e) (1.9.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2022.5.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (3.19.4)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.9.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (21.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.2.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (6.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (4.64.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.3.5)
Requirement already satisfied: numpy<1.25.0,>=1.18.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.4.0+7.geed9172e) (1.20.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2.1.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.2.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (5.8.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (6.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (65.3.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (6.0.1)
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-liuwfszi
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-liuwfszi
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 5926fcfc0776c112680cbca28553c15211c2fda7
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.6.0+1.g5926fcf) (2022.5.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.6.0+1.g5926fcf) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.6.0+1.g5926fcf) (3.19.4)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.6.0+1.g5926fcf) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.6.0+1.g5926fcf) (1.9.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.6.0+1.g5926fcf) (21.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.6.0+1.g5926fcf) (1.2.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.6.0+1.g5926fcf) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.6.0+1.g5926fcf) (6.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.6.0+1.g5926fcf) (4.64.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.6.0+1.g5926fcf) (1.3.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.6.0+1.g5926fcf) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.6.0+1.g5926fcf) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (2.1.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (1.2.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (5.8.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (6.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.6.0+1.g5926fcf) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.6.0+1.g5926fcf) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.6.0+1.g5926fcf) (65.3.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.6.0+1.g5926fcf) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.6.0+1.g5926fcf) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.6.0+1.g5926fcf) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.6.0+1.g5926fcf) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.6.0+1.g5926fcf) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.6.0+1.g5926fcf) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.6.0+1.g5926fcf) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.6.0+1.g5926fcf) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.6.0+1.g5926fcf) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.6.0+1.g5926fcf) (6.0.1)
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 3 items

tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py s      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py F               [100%]


=================================== FAILURES ===================================

__________________________________ test_func ___________________________________


self = <testbook.client.TestbookNotebookClient object at 0x7f20823143d0>

cell = {'cell_type': 'markdown', 'id': '60486140', 'metadata': {}, 'source': 'We generate a Triton Inference Server request o...re not supported for int32 columns. As a workaround, we will NA values with 0. This will be updated in the future. '}

kwargs = {}, cell_indexes = [0, 1, 2, 3, 4, 5, ...]

executed_cells = [{'cell_type': 'code', 'execution_count': 2, 'id': '47c9026a', 'metadata': {'execution': {'iopub.status.busy': '2022-0...: 'markdown', 'id': '25a6ac04', 'metadata': {}, 'source': 'We define the path for the saved workflow and model.'}, ...]

idx = 24


def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:





          cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)





../../../.local/lib/python3.8/site-packages/testbook/client.py:133:




args = (<testbook.client.TestbookNotebookClient object at 0x7f20823143d0>, {'cell_type': 'code', 'execution_count': 12, 'id':...output_schema.column_names\noutputs = [\n    grpcclient.InferRequestedOutput(col)\n    for col in output_cols\n]'}, 24)

kwargs = {}


def wrapped(*args, **kwargs):





  return just_run(coro(*args, **kwargs))





../../../.local/lib/python3.8/site-packages/nbclient/util.py:84:




coro = <coroutine object NotebookClient.async_execute_cell at 0x7f20822ed9c0>


def just_run(coro: Awaitable) -> Any:
    """Make the coroutine run, even if there is an event loop running (using nest_asyncio)"""
    # original from vaex/asyncio.py
    loop = asyncio._get_running_loop()
    if loop is None:
        had_running_loop = False
        try:
            loop = asyncio.get_event_loop()
        except RuntimeError:
            # we can still get 'There is no current event loop in ...'
            loop = asyncio.new_event_loop()
            asyncio.set_event_loop(loop)
    else:
        had_running_loop = True
    if had_running_loop:
        # if there is a running loop, we patch using nest_asyncio
        # to have reentrant event loops
        check_ipython()
        import nest_asyncio

        nest_asyncio.apply()
        check_patch_tornado()





  return loop.run_until_complete(coro)





../../../.local/lib/python3.8/site-packages/nbclient/util.py:62:




self = <_UnixSelectorEventLoop running=False closed=False debug=False>

future = <Task finished name='Task-223' coro=<NotebookClient.async_execute_cell() done, defined at /var/jenkins_home/.local/lib...' object has no attribute 'column_schemas'\nAttributeError: 'list' object has no attribute 'column_schemas'\n')>


def run_until_complete(self, future):
    """Run until the Future is done.

    If the argument is a coroutine, it is wrapped in a Task.

    WARNING: It would be disastrous to call run_until_complete()
    with the same coroutine twice -- it would wrap it in two
    different Tasks and that can't be good.

    Return the Future's result, or raise its exception.
    """
    self._check_closed()
    self._check_running()

    new_task = not futures.isfuture(future)
    future = tasks.ensure_future(future, loop=self)
    if new_task:
        # An exception is raised if the future didn't complete, so there
        # is no need to log the "destroy pending task" message
        future._log_destroy_pending = False

    future.add_done_callback(_run_until_complete_cb)
    try:
        self.run_forever()
    except:
        if new_task and future.done() and not future.cancelled():
            # The coroutine raised a BaseException. Consume the exception
            # to not log a warning, the caller doesn't have access to the
            # local task.
            future.exception()
        raise
    finally:
        future.remove_done_callback(_run_until_complete_cb)
    if not future.done():
        raise RuntimeError('Event loop stopped before Future completed.')





  return future.result()





/usr/lib/python3.8/asyncio/base_events.py:616:




self = <testbook.client.TestbookNotebookClient object at 0x7f20823143d0>

cell = {'cell_type': 'code', 'execution_count': 12, 'id': 'b833997d', 'metadata': {'execution': {'iopub.status.busy': '2022-0...raph.output_schema.column_names\noutputs = [\n    grpcclient.InferRequestedOutput(col)\n    for col in output_cols\n]'}

cell_index = 24, execution_count = None, store_history = True


async def async_execute_cell(
    self,
    cell: NotebookNode,
    cell_index: int,
    execution_count: t.Optional[int] = None,
    store_history: bool = True,
) -> NotebookNode:
    """
    Executes a single code cell.

    To execute all cells see :meth:`execute`.

    Parameters
    ----------
    cell : nbformat.NotebookNode
        The cell which is currently being processed.
    cell_index : int
        The position of the cell within the notebook object.
    execution_count : int
        The execution count to be assigned to the cell (default: Use kernel response)
    store_history : bool
        Determines if history should be stored in the kernel (default: False).
        Specific to ipython kernels, which can store command histories.

    Returns
    -------
    output : dict
        The execution output payload (or None for no output).

    Raises
    ------
    CellExecutionError
        If execution failed and should raise an exception, this will be raised
        with defaults about the failure.

    Returns
    -------
    cell : NotebookNode
        The cell which was just processed.
    """
    assert self.kc is not None

    await run_hook(self.on_cell_start, cell=cell, cell_index=cell_index)

    if cell.cell_type != 'code' or not cell.source.strip():
        self.log.debug("Skipping non-executing cell %s", cell_index)
        return cell

    if self.skip_cells_with_tag in cell.metadata.get("tags", []):
        self.log.debug("Skipping tagged cell %s", cell_index)
        return cell

    if self.record_timing:  # clear execution metadata prior to execution
        cell['metadata']['execution'] = {}

    self.log.debug("Executing cell:\n%s", cell.source)

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors or "raises-exception" in cell.metadata.get("tags", [])
    )

    await run_hook(self.on_cell_execute, cell=cell, cell_index=cell_index)
    parent_msg_id = await ensure_async(
        self.kc.execute(
            cell.source, store_history=store_history, stop_on_error=not cell_allows_errors
        )
    )
    await run_hook(self.on_cell_complete, cell=cell, cell_index=cell_index)
    # We launched a code cell to execute
    self.code_cells_executed += 1
    exec_timeout = self._get_timeout(cell)

    cell.outputs = []
    self.clear_before_next_output = False

    task_poll_kernel_alive = asyncio.ensure_future(self._async_poll_kernel_alive())
    task_poll_output_msg = asyncio.ensure_future(
        self._async_poll_output_msg(parent_msg_id, cell, cell_index)
    )
    self.task_poll_for_reply = asyncio.ensure_future(
        self._async_poll_for_reply(
            parent_msg_id, cell, exec_timeout, task_poll_output_msg, task_poll_kernel_alive
        )
    )
    try:
        exec_reply = await self.task_poll_for_reply
    except asyncio.CancelledError:
        # can only be cancelled by task_poll_kernel_alive when the kernel is dead
        task_poll_output_msg.cancel()
        raise DeadKernelError("Kernel died")
    except Exception as e:
        # Best effort to cancel request if it hasn't been resolved
        try:
            # Check if the task_poll_output is doing the raising for us
            if not isinstance(e, CellControlSignal):
                task_poll_output_msg.cancel()
        finally:
            raise

    if execution_count:
        cell['execution_count'] = execution_count





  await self._check_raise_for_error(cell, cell_index, exec_reply)





../../../.local/lib/python3.8/site-packages/nbclient/client.py:965:




self = <testbook.client.TestbookNotebookClient object at 0x7f20823143d0>

cell = {'cell_type': 'code', 'execution_count': 12, 'id': 'b833997d', 'metadata': {'execution': {'iopub.status.busy': '2022-0...raph.output_schema.column_names\noutputs = [\n    grpcclient.InferRequestedOutput(col)\n    for col in output_cols\n]'}

cell_index = 24

exec_reply = {'buffers': [], 'content': {'ename': 'AttributeError', 'engine_info': {'engine_id': -1, 'engine_uuid': '41af5313-01ce-...e, 'engine': '41af5313-01ce-45c8-8093-4aa3ab79ac7e', 'started': '2022-09-14T19:45:54.304338Z', 'status': 'error'}, ...}


async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:





      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)





E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E

E           # create inputs and outputs

E           inputs = convert_df_to_triton_input(workflow.input_schema.column_names, batch.fillna(0), grpcclient.InferInput)

E           output_cols = ensemble.graph.output_schema.column_names

E           outputs = [

E               grpcclient.InferRequestedOutput(col)

E               for col in output_cols

E           ]

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mAttributeError�[0m                            Traceback (most recent call last)

E           Input �[0;32mIn [12]�[0m, in �[0;36m<cell line: 2>�[0;34m()�[0m

E           �[1;32m      1�[0m �[38;5;66;03m# create inputs and outputs�[39;00m

E           �[0;32m----> 2�[0m inputs �[38;5;241m=�[39m �[43mconvert_df_to_triton_input�[49m�[43m(�[49m�[43mworkflow�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[38;5;241;43m.�[39;49m�[43mcolumn_names�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[38;5;241;43m.�[39;49m�[43mfillna�[49m�[43m(�[49m�[38;5;241;43m0�[39;49m�[43m)�[49m�[43m,�[49m�[43m �[49m�[43mgrpcclient�[49m�[38;5;241;43m.�[39;49m�[43mInferInput�[49m�[43m)�[49m

E           �[1;32m      3�[0m output_cols �[38;5;241m=�[39m ensemble�[38;5;241m.�[39mgraph�[38;5;241m.�[39moutput_schema�[38;5;241m.�[39mcolumn_names

E           �[1;32m      4�[0m outputs �[38;5;241m=�[39m [

E           �[1;32m      5�[0m     grpcclient�[38;5;241m.�[39mInferRequestedOutput(col)

E           �[1;32m      6�[0m     �[38;5;28;01mfor�[39;00m col �[38;5;129;01min�[39;00m output_cols

E           �[1;32m      7�[0m ]

E

E           File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py:57�[0m, in �[0;36mconvert_df_to_triton_input�[0;34m(schema, batch, input_class, dtype)�[0m

E           �[1;32m     37�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mconvert_df_to_triton_input�[39m(schema, batch, input_class�[38;5;241m=�[39mgrpcclient�[38;5;241m.�[39mInferInput, dtype�[38;5;241m=�[39m�[38;5;124m"�[39m�[38;5;124mint32�[39m�[38;5;124m"�[39m):

E           �[1;32m     38�[0m     �[38;5;124;03m"""�[39;00m

E           �[1;32m     39�[0m �[38;5;124;03m    Convert a dataframe to a set of Triton inputs�[39;00m

E           �[1;32m     40�[0m

E           �[0;32m   (...)�[0m

E           �[1;32m     55�[0m �[38;5;124;03m        A list of Triton inputs of the requested input class�[39;00m

E           �[1;32m     56�[0m �[38;5;124;03m    """�[39;00m

E           �[0;32m---> 57�[0m     df_dict �[38;5;241m=�[39m �[43m_convert_df_to_dict�[49m�[43m(�[49m�[43mschema�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[43m,�[49m�[43m �[49m�[43mdtype�[49m�[43m)�[49m

E           �[1;32m     58�[0m     inputs �[38;5;241m=�[39m [

E           �[1;32m     59�[0m         _convert_column_to_triton_input(col_name, col_values, input_class)

E           �[1;32m     60�[0m         �[38;5;28;01mfor�[39;00m col_name, col_values �[38;5;129;01min�[39;00m df_dict�[38;5;241m.�[39mitems()

E           �[1;32m     61�[0m     ]

E           �[1;32m     62�[0m     �[38;5;28;01mreturn�[39;00m inputs

E

E           File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py:67�[0m, in �[0;36m_convert_df_to_dict�[0;34m(schema, batch, dtype)�[0m

E           �[1;32m     65�[0m �[38;5;28;01mdef�[39;00m �[38;5;21m_convert_df_to_dict�[39m(schema, batch, dtype�[38;5;241m=�[39m�[38;5;124m"�[39m�[38;5;124mint32�[39m�[38;5;124m"�[39m):

E           �[1;32m     66�[0m     df_dict �[38;5;241m=�[39m {}

E           �[0;32m---> 67�[0m     �[38;5;28;01mfor�[39;00m col_name, col_schema �[38;5;129;01min�[39;00m �[43mschema�[49m�[38;5;241;43m.�[39;49m�[43mcolumn_schemas�[49m�[38;5;241m.�[39mitems():

E           �[1;32m     68�[0m         col �[38;5;241m=�[39m batch[col_name]

E           �[1;32m     69�[0m         shape �[38;5;241m=�[39m col_schema�[38;5;241m.�[39mproperties�[38;5;241m.�[39mget(�[38;5;124m"�[39m�[38;5;124mshape�[39m�[38;5;124m"�[39m, �[38;5;28;01mNone�[39;00m) �[38;5;129;01mor�[39;00m [�[38;5;241m-�[39m�[38;5;241m1�[39m, �[38;5;241m1�[39m]

E

E           �[0;31mAttributeError�[0m: 'list' object has no attribute 'column_schemas'

E           AttributeError: 'list' object has no attribute 'column_schemas'


../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError


During handling of the above exception, another exception occurred:


def test_func():
    with testbook(
        REPO_ROOT / "examples" / "scaling-criteo" / "02-ETL-with-NVTabular.ipynb",
        execute=False,
        timeout=180,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["BASE_DIR"] = "/tmp/input/criteo/"
            os.environ["INPUT_DATA_DIR"] = "/tmp/input/criteo/"
            os.environ["OUTPUT_DATA_DIR"] = "/tmp/output/criteo/"
            os.system("mkdir -p /tmp/input/criteo")
            os.system("mkdir -p /tmp/output/criteo")

            from merlin.datasets.synthetic import generate_data

            train, valid = generate_data("criteo", int(100000), set_sizes=(0.7, 0.3))

            train.to_ddf().compute().to_parquet('/tmp/input/criteo/day_0.parquet')
            valid.to_ddf().compute().to_parquet('/tmp/input/criteo/day_1.parquet')
            """
        )
        tb1.execute()
        assert os.path.isfile("/tmp/output/criteo/train/part_0.parquet")
        assert os.path.isfile("/tmp/output/criteo/valid/part_0.parquet")
        assert os.path.isfile("/tmp/output/criteo/workflow/metadata.json")

    with testbook(
        REPO_ROOT
        / "examples"
        / "scaling-criteo"
        / "03-Training-with-Merlin-Models-TensorFlow.ipynb",
        execute=False,
        timeout=180,
    ) as tb2:
        tb2.inject(
            """
            import os
            os.environ["INPUT_DATA_DIR"] = "/tmp/output/criteo/"
            """
        )
        tb2.execute()
        metrics = tb2.ref("eval_metrics")
        assert set(metrics.keys()) == set(
            [
                "auc",
                "binary_accuracy",
                "loss",
                "precision",
                "recall",
                "regularization_loss",
            ]
        )
        assert os.path.isfile("/tmp/output/criteo/dlrm/saved_model.pb")

    with testbook(
        REPO_ROOT
        / "examples"
        / "scaling-criteo"
        / "04-Triton-Inference-with-Merlin-Models-TensorFlow.ipynb",
        execute=False,
        timeout=180,
    ) as tb3:
        tb3.inject(
            """
            import os
            os.environ["BASE_DIR"] = "/tmp/output/criteo/"
            os.environ["INPUT_FOLDER"] = "/tmp/input/criteo/"
            """
        )
        NUM_OF_CELLS = len(tb3.cells)





      tb3.execute_cell(list(range(0, NUM_OF_CELLS - 5)))





tests/unit/examples/test_scaling_criteo_merlin_models.py:82:




self = <testbook.client.TestbookNotebookClient object at 0x7f20823143d0>

cell = {'cell_type': 'markdown', 'id': '60486140', 'metadata': {}, 'source': 'We generate a Triton Inference Server request o...re not supported for int32 columns. As a workaround, we will NA values with 0. This will be updated in the future. '}

kwargs = {}, cell_indexes = [0, 1, 2, 3, 4, 5, ...]

executed_cells = [{'cell_type': 'code', 'execution_count': 2, 'id': '47c9026a', 'metadata': {'execution': {'iopub.status.busy': '2022-0...: 'markdown', 'id': '25a6ac04', 'metadata': {}, 'source': 'We define the path for the saved workflow and model.'}, ...]

idx = 24


def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:
            cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)
        except CellExecutionError as ce:





          raise TestbookRuntimeError(ce.evalue, ce, self._get_error_class(ce.ename))





E               testbook.exceptions.TestbookRuntimeError: An error occurred while executing the following cell:

E               ------------------

E

E               # create inputs and outputs

E               inputs = convert_df_to_triton_input(workflow.input_schema.column_names, batch.fillna(0), grpcclient.InferInput)

E               output_cols = ensemble.graph.output_schema.column_names

E               outputs = [

E                   grpcclient.InferRequestedOutput(col)

E                   for col in output_cols

E               ]

E               ------------------

E

E               �[0;31m---------------------------------------------------------------------------�[0m

E               �[0;31mAttributeError�[0m                            Traceback (most recent call last)

E               Input �[0;32mIn [12]�[0m, in �[0;36m<cell line: 2>�[0;34m()�[0m

E               �[1;32m      1�[0m �[38;5;66;03m# create inputs and outputs�[39;00m

E               �[0;32m----> 2�[0m inputs �[38;5;241m=�[39m �[43mconvert_df_to_triton_input�[49m�[43m(�[49m�[43mworkflow�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[38;5;241;43m.�[39;49m�[43mcolumn_names�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[38;5;241;43m.�[39;49m�[43mfillna�[49m�[43m(�[49m�[38;5;241;43m0�[39;49m�[43m)�[49m�[43m,�[49m�[43m �[49m�[43mgrpcclient�[49m�[38;5;241;43m.�[39;49m�[43mInferInput�[49m�[43m)�[49m

E               �[1;32m      3�[0m output_cols �[38;5;241m=�[39m ensemble�[38;5;241m.�[39mgraph�[38;5;241m.�[39moutput_schema�[38;5;241m.�[39mcolumn_names

E               �[1;32m      4�[0m outputs �[38;5;241m=�[39m [

E               �[1;32m      5�[0m     grpcclient�[38;5;241m.�[39mInferRequestedOutput(col)

E               �[1;32m      6�[0m     �[38;5;28;01mfor�[39;00m col �[38;5;129;01min�[39;00m output_cols

E               �[1;32m      7�[0m ]

E

E               File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py:57�[0m, in �[0;36mconvert_df_to_triton_input�[0;34m(schema, batch, input_class, dtype)�[0m

E               �[1;32m     37�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mconvert_df_to_triton_input�[39m(schema, batch, input_class�[38;5;241m=�[39mgrpcclient�[38;5;241m.�[39mInferInput, dtype�[38;5;241m=�[39m�[38;5;124m"�[39m�[38;5;124mint32�[39m�[38;5;124m"�[39m):

E               �[1;32m     38�[0m     �[38;5;124;03m"""�[39;00m

E               �[1;32m     39�[0m �[38;5;124;03m    Convert a dataframe to a set of Triton inputs�[39;00m

E               �[1;32m     40�[0m

E               �[0;32m   (...)�[0m

E               �[1;32m     55�[0m �[38;5;124;03m        A list of Triton inputs of the requested input class�[39;00m

E               �[1;32m     56�[0m �[38;5;124;03m    """�[39;00m

E               �[0;32m---> 57�[0m     df_dict �[38;5;241m=�[39m �[43m_convert_df_to_dict�[49m�[43m(�[49m�[43mschema�[49m�[43m,�[49m�[43m �[49m�[43mbatch�[49m�[43m,�[49m�[43m �[49m�[43mdtype�[49m�[43m)�[49m

E               �[1;32m     58�[0m     inputs �[38;5;241m=�[39m [

E               �[1;32m     59�[0m         _convert_column_to_triton_input(col_name, col_values, input_class)

E               �[1;32m     60�[0m         �[38;5;28;01mfor�[39;00m col_name, col_values �[38;5;129;01min�[39;00m df_dict�[38;5;241m.�[39mitems()

E               �[1;32m     61�[0m     ]

E               �[1;32m     62�[0m     �[38;5;28;01mreturn�[39;00m inputs

E

E               File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py:67�[0m, in �[0;36m_convert_df_to_dict�[0;34m(schema, batch, dtype)�[0m

E               �[1;32m     65�[0m �[38;5;28;01mdef�[39;00m �[38;5;21m_convert_df_to_dict�[39m(schema, batch, dtype�[38;5;241m=�[39m�[38;5;124m"�[39m�[38;5;124mint32�[39m�[38;5;124m"�[39m):

E               �[1;32m     66�[0m     df_dict �[38;5;241m=�[39m {}

E               �[0;32m---> 67�[0m     �[38;5;28;01mfor�[39;00m col_name, col_schema �[38;5;129;01min�[39;00m �[43mschema�[49m�[38;5;241;43m.�[39;49m�[43mcolumn_schemas�[49m�[38;5;241m.�[39mitems():

E               �[1;32m     68�[0m         col �[38;5;241m=�[39m batch[col_name]

E               �[1;32m     69�[0m         shape �[38;5;241m=�[39m col_schema�[38;5;241m.�[39mproperties�[38;5;241m.�[39mget(�[38;5;124m"�[39m�[38;5;124mshape�[39m�[38;5;124m"�[39m, �[38;5;28;01mNone�[39;00m) �[38;5;129;01mor�[39;00m [�[38;5;241m-�[39m�[38;5;241m1�[39m, �[38;5;241m1�[39m]

E

E               �[0;31mAttributeError�[0m: 'list' object has no attribute 'column_schemas'

E               AttributeError: 'list' object has no attribute 'column_schemas'


../../../.local/lib/python3.8/site-packages/testbook/client.py:135: TestbookRuntimeError

----------------------------- Captured stderr call -----------------------------

2022-09-14 19:44:36,572 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-09-14 19:44:36,580 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-09-14 19:44:36,597 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-09-14 19:44:36,598 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 48 leaked semaphore objects to clean up at shutdown

warnings.warn('resource_tracker: There appear to be %d '

2022-09-14 19:44:56.153163: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-09-14 19:45:00.248248: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 0

2022-09-14 19:45:00.248408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-09-14 19:45:00.249096: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 1

2022-09-14 19:45:00.249154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15153 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-09-14 19:45:00.249767: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 2

2022-09-14 19:45:00.249814: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 15153 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-09-14 19:45:00.250435: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 3

2022-09-14 19:45:00.250482: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 15153 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

2022-09-14 19:45:31.404836: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-09-14 19:45:35.462711: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 0

2022-09-14 19:45:35.462854: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-09-14 19:45:35.463675: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 1

2022-09-14 19:45:35.463728: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 15153 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-09-14 19:45:35.464393: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 2

2022-09-14 19:45:35.464441: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 15153 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-09-14 19:45:35.465002: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 3

2022-09-14 19:45:35.465048: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 15153 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover


.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                           9      4    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       346    239    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    119    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   115     24    79%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       111     92    17%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     37    63%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            342    170    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               18      7    61%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    20      2    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      4    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                         88     36    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181    153    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     47    23%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      4    81%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          346    238    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37     13    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          603    387    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     20    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     66    18%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190    152    20%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18     13    28%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     19    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    117    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   64      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49     11    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                         93     48    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                114     56    51%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     168    125    26%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30     18    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29     11    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     20    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          245    109    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 239    115    52%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          393    130    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                               73     47    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                104     68    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            25      2    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              271     71    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/dataset.py                                   213    108    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                56     44    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          33      3    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          432    237    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      2    85%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           113     55    51%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              184     83    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               545    249    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             69     42    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           32     15    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     199    107    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     23    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     48    34%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/predictions/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/predictions/base.py                          130     95    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/predictions/classification.py                154    100    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/predictions/dot_product.py                   120     73    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/predictions/regression.py                      9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/predictions/sampling/init.py               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/predictions/sampling/base.py                  67     32    52%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/predictions/sampling/in_batch.py              35     20    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/predictions/sampling/popularity.py            27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                            74     49    34%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       239    187    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                          91     47    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          81     56    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            186    118    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 33     24    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     31    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     62    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      6    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       191     43    77%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       205     56    73%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      8    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 78      8    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     20      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                21     15    29%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                              9      4    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             83     37    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           81     19    77%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py                             25      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              47     30    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               266    210    21%


TOTAL                                                                                                 11582   6060    48%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:16: 'NoneType' returning from faiss operator because of randomly generated data.

============== 1 failed, 1 passed, 1 skipped in 94.97s (0:01:34) ===============

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:794: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://github.com/gitapi/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins5825393342795341538.sh

@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #619 of commit 400b68e1d4f6a2cc895d233c0af8c30eacc9dd29, no merge conflicts.
Running as SYSTEM
Setting status of 400b68e1d4f6a2cc895d233c0af8c30eacc9dd29 to PENDING with url https://10.20.13.93:8080/job/merlin_merlin/448/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/619/*:refs/remotes/origin/pr/619/* # timeout=10
 > git rev-parse 400b68e1d4f6a2cc895d233c0af8c30eacc9dd29^{commit} # timeout=10
Checking out Revision 400b68e1d4f6a2cc895d233c0af8c30eacc9dd29 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 400b68e1d4f6a2cc895d233c0af8c30eacc9dd29 # timeout=10
Commit message: "Merge branch 'test-rework' of https://github.com/jperez999/merlin into test-rework"
 > git rev-list --no-walk f92797ee019fbe7f18a977180c8df8efdb42bc96 # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins14523786350432424176.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.7,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,attrs==22.1.0,awscli==1.25.73,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.51,botocore==1.27.72,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,clang==5.0,click==8.1.3,cloudpickle==2.1.0,colorama==0.4.4,coverage==6.4.4,cuda-python==11.7.1,cudf==22.4.0,cupy-cuda116==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dask-cuda==22.4.0,dask-cudf==22.4.0,dbus-python==1.2.16,debugpy==1.6.2,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.3.0,distro==1.7.0,dm-tree==0.1.7,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==0.10.0,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.82.0,fastavro==1.6.0,fastcore==1.5.24,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.0,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.0,google-auth==2.11.0,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.2,grpcio==1.41.0,grpcio-channelz==1.47.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.4.0,hugectr2onnx==0.0.0,huggingface-hub==0.8.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.0,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.1,ipython==8.4.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.1.0,json5==0.9.9,jsonschema==4.9.1,jupyter-cache==0.4.3,jupyter-client==7.3.4,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyterlab==3.4.5,jupyterlab-pygments==0.2.2,jupyterlab-server==2.15.0,jupyterlab-widgets==1.1.0,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.0,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.5.3,matplotlib-inline==0.1.3,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0+untagged.112.gfbfcaf3,merlin-models==0.6.0+45.g5a345d9c1,merlin-systems==0+untagged.105.gf89cc51,mistune==0.8.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.6,nbconvert==6.5.3,nbdime==3.1.1,nbformat==5.4.0,nest-asyncio==1.5.5,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.0,numpy==1.21.5,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.3.3+15.g16e4e34e9),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.0,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.30,proto-plus==1.19.6,protobuf==3.19.4,psutil==5.9.1,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==6.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.12.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.2,pytest-cov==3.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.7,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==23.2.1,regex==2022.7.25,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rmm==21.12.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.0,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.4,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.2.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.1.1,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.4.0,starlette==0.19.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.26.0,tensorflow-metadata==1.9.0,termcolor==1.1.0,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.25.1,tqdm==4.64.0,traitlets==5.3.0,transformers==4.12.0,transformers4rec==0.1.11+10.g21a2a836a,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.22.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.11,uvicorn==0.18.3,uvloop==0.16.0,versioneer==0.20,virtualenv==20.16.4,wandb==0.13.1,watchfiles==0.16.1,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.3.3,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='897638094'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-o_tcgnsr
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-o_tcgnsr
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit 15074adf11bcc9bb9da375ec40ab4e431f1c93e9
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.5.0+4.g15074ad) (0.6.0+1.g5926fcf)
Requirement already satisfied: nvtabular>=1.0.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.5.0+4.g15074ad) (1.4.0+7.geed9172e)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2022.5.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.9.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.2.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2022.3.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (6.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (21.3)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (3.19.4)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (0.55.1)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (4.64.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.3.5)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular>=1.0.0->merlin-systems==0.5.0+4.g15074ad) (1.9.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (0.4.3)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2.1.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.2.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.0.4)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.7.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2.4.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2.0.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (65.3.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (4.0.0)
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-fikqifa5
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-fikqifa5
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit 91fee2a85e309cbd5d338d9ba37d989b54594498
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-models==0.7.0+5.g91fee2a8) (0.6.0+1.g5926fcf)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2022.5.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.9.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.2.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2022.3.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (6.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (21.3)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (3.19.4)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (0.55.1)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (4.64.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.3.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (0.4.3)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2.1.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.2.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.0.4)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.7.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2.4.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2.0.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (65.3.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (4.0.0)
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-54h9mu2c
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-54h9mu2c
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit eed9172ec30a47e020b06ff60469d6d168eb865b
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from nvtabular==1.4.0+7.geed9172e) (0.6.0+1.g5926fcf)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.4.0+7.geed9172e) (1.9.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2022.5.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.9.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.2.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2022.3.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (6.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (21.3)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (3.19.4)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (0.55.1)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (4.64.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.3.5)
Requirement already satisfied: numpy<1.25.0,>=1.18.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.4.0+7.geed9172e) (1.20.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (0.4.3)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2.1.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.2.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.0.4)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.7.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2.4.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2.0.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (65.3.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (4.0.0)
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-oucr3thl
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-oucr3thl
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 5926fcfc0776c112680cbca28553c15211c2fda7
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.6.0+1.g5926fcf) (2022.5.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.6.0+1.g5926fcf) (1.9.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.6.0+1.g5926fcf) (1.2.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.6.0+1.g5926fcf) (2022.3.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.6.0+1.g5926fcf) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.6.0+1.g5926fcf) (6.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.6.0+1.g5926fcf) (21.3)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.6.0+1.g5926fcf) (3.19.4)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.6.0+1.g5926fcf) (0.55.1)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.6.0+1.g5926fcf) (4.64.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.6.0+1.g5926fcf) (1.3.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.6.0+1.g5926fcf) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.6.0+1.g5926fcf) (0.4.3)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (2.1.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (1.2.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (1.0.4)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (1.7.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (2.4.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (2.0.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.6.0+1.g5926fcf) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.6.0+1.g5926fcf) (65.3.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.6.0+1.g5926fcf) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.6.0+1.g5926fcf) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.6.0+1.g5926fcf) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.6.0+1.g5926fcf) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.6.0+1.g5926fcf) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.6.0+1.g5926fcf) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.6.0+1.g5926fcf) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.6.0+1.g5926fcf) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.6.0+1.g5926fcf) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.6.0+1.g5926fcf) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.6.0+1.g5926fcf) (4.0.0)
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 3 items

tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py s      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py F               [100%]WARNING: Failed to generate report: No data to report.


=================================== FAILURES ===================================

__________________________________ test_func ___________________________________


def test_func():
    with testbook(
        REPO_ROOT / "examples" / "scaling-criteo" / "02-ETL-with-NVTabular.ipynb",
        execute=False,
        timeout=180,
    ) as tb1:
        tb1.inject(
            """
            import os
            os.environ["BASE_DIR"] = "/tmp/input/criteo/"
            os.environ["INPUT_DATA_DIR"] = "/tmp/input/criteo/"
            os.environ["OUTPUT_DATA_DIR"] = "/tmp/output/criteo/"
            os.system("mkdir -p /tmp/input/criteo")
            os.system("mkdir -p /tmp/output/criteo")

            from merlin.datasets.synthetic import generate_data

            train, valid = generate_data("criteo", int(100000), set_sizes=(0.7, 0.3))

            train.to_ddf().compute().to_parquet('/tmp/input/criteo/day_0.parquet')
            valid.to_ddf().compute().to_parquet('/tmp/input/criteo/day_1.parquet')
            """
        )





      tb1.execute()





tests/unit/examples/test_scaling_criteo_merlin_models.py:33:




../../../.local/lib/python3.8/site-packages/testbook/client.py:147: in execute

super().execute_cell(cell, index)

../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped

return just_run(coro(*args, **kwargs))

../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run

return loop.run_until_complete(coro)

/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete

return future.result()

../../../.local/lib/python3.8/site-packages/nbclient/client.py:965: in async_execute_cell

await self._check_raise_for_error(cell, cell_index, exec_reply)




self = <testbook.client.TestbookNotebookClient object at 0x7fddc60cd0a0>

cell = {'id': 'd1d8f500', 'cell_type': 'code', 'metadata': {'execution': {'iopub.status.busy': '2022-09-14T19:51:28.873874Z',...ry: CUDA error at: /usr/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory']}]}

cell_index = 27

exec_reply = {'buffers': [], 'content': {'ename': 'MemoryError', 'engine_info': {'engine_id': -1, 'engine_uuid': 'e2f9c142-d719-49d...e, 'engine': 'e2f9c142-d719-49d6-a406-02f273fb6344', 'started': '2022-09-14T19:51:28.874345Z', 'status': 'error'}, ...}


async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:





      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)





E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E

E           import os

E           os.environ["BASE_DIR"] = "/tmp/input/criteo/"

E           os.environ["INPUT_DATA_DIR"] = "/tmp/input/criteo/"

E           os.environ["OUTPUT_DATA_DIR"] = "/tmp/output/criteo/"

E           os.system("mkdir -p /tmp/input/criteo")

E           os.system("mkdir -p /tmp/output/criteo")

E

E           from merlin.datasets.synthetic import generate_data

E

E           train, valid = generate_data("criteo", int(100000), set_sizes=(0.7, 0.3))

E

E           train.to_ddf().compute().to_parquet('/tmp/input/criteo/day_0.parquet')

E           valid.to_ddf().compute().to_parquet('/tmp/input/criteo/day_1.parquet')

E

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mMemoryError�[0m                               Traceback (most recent call last)

E           Input �[0;32mIn [16]�[0m, in �[0;36m<cell line: 10>�[0;34m()�[0m

E           �[1;32m      6�[0m os�[38;5;241m.�[39msystem(�[38;5;124m"�[39m�[38;5;124mmkdir -p /tmp/output/criteo�[39m�[38;5;124m"�[39m)

E           �[1;32m      8�[0m �[38;5;28;01mfrom�[39;00m �[38;5;21;01mmerlin�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01mdatasets�[39;00m�[38;5;21;01m.�[39;00m�[38;5;21;01msynthetic�[39;00m �[38;5;28;01mimport�[39;00m generate_data

E           �[0;32m---> 10�[0m train, valid �[38;5;241m=�[39m �[43mgenerate_data�[49m�[43m(�[49m�[38;5;124;43m"�[39;49m�[38;5;124;43mcriteo�[39;49m�[38;5;124;43m"�[39;49m�[43m,�[49m�[43m �[49m�[38;5;28;43mint�[39;49m�[43m(�[49m�[38;5;241;43m100000�[39;49m�[43m)�[49m�[43m,�[49m�[43m �[49m�[43mset_sizes�[49m�[38;5;241;43m=�[39;49m�[43m(�[49m�[38;5;241;43m0.7�[39;49m�[43m,�[49m�[43m �[49m�[38;5;241;43m0.3�[39;49m�[43m)�[49m�[43m)�[49m

E           �[1;32m     12�[0m train�[38;5;241m.�[39mto_ddf()�[38;5;241m.�[39mcompute()�[38;5;241m.�[39mto_parquet(�[38;5;124m'�[39m�[38;5;124m/tmp/input/criteo/day_0.parquet�[39m�[38;5;124m'�[39m)

E           �[1;32m     13�[0m valid�[38;5;241m.�[39mto_ddf()�[38;5;241m.�[39mcompute()�[38;5;241m.�[39mto_parquet(�[38;5;124m'�[39m�[38;5;124m/tmp/input/criteo/day_1.parquet�[39m�[38;5;124m'�[39m)

E

E           File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py:131�[0m, in �[0;36mgenerate_data�[0;34m(input, num_rows, set_sizes, min_session_length, max_session_length, device)�[0m

E           �[1;32m    128�[0m         start_i �[38;5;241m=�[39m end_i

E           �[1;32m    129�[0m         output_datasets�[38;5;241m.�[39mappend(set_df)

E           �[0;32m--> 131�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28mtuple�[39m([merlin�[38;5;241m.�[39mio�[38;5;241m.�[39mDataset(d, schema�[38;5;241m=�[39mschema) �[38;5;28;01mfor�[39;00m d �[38;5;129;01min�[39;00m output_datasets])

E           �[1;32m    133�[0m �[38;5;28;01mreturn�[39;00m merlin�[38;5;241m.�[39mio�[38;5;241m.�[39mDataset(df, schema�[38;5;241m=�[39mschema)

E

E           File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/datasets/synthetic.py:131�[0m, in �[0;36m�[0;34m(.0)�[0m

E           �[1;32m    128�[0m         start_i �[38;5;241m=�[39m end_i

E           �[1;32m    129�[0m         output_datasets�[38;5;241m.�[39mappend(set_df)

E           �[0;32m--> 131�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28mtuple�[39m([�[43mmerlin�[49m�[38;5;241;43m.�[39;49m�[43mio�[49m�[38;5;241;43m.�[39;49m�[43mDataset�[49m�[43m(�[49m�[43md�[49m�[43m,�[49m�[43m �[49m�[43mschema�[49m�[38;5;241;43m=�[39;49m�[43mschema�[49m�[43m)�[49m �[38;5;28;01mfor�[39;00m d �[38;5;129;01min�[39;00m output_datasets])

E           �[1;32m    133�[0m �[38;5;28;01mreturn�[39;00m merlin�[38;5;241m.�[39mio�[38;5;241m.�[39mDataset(df, schema�[38;5;241m=�[39mschema)

E

E           File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:262�[0m, in �[0;36mDataset.__init__�[0;34m(self, path_or_source, engine, npartitions, part_size, part_mem_fraction, storage_options, dtypes, client, cpu, base_dataset, schema, **kwargs)�[0m

E           �[1;32m    256�[0m npartitions �[38;5;241m=�[39m npartitions �[38;5;129;01mor�[39;00m �[38;5;241m1�[39m

E           �[1;32m    257�[0m �[38;5;28;01mif�[39;00m �[38;5;28misinstance�[39m(path_or_source, dask�[38;5;241m.�[39mdataframe�[38;5;241m.�[39mDataFrame) �[38;5;129;01mor�[39;00m is_dataframe_object(

E           �[1;32m    258�[0m     path_or_source

E           �[1;32m    259�[0m ):

E           �[1;32m    260�[0m     �[38;5;66;03m# User is passing in a <dask.dataframe|cudf|pd>.DataFrame�[39;00m

E           �[1;32m    261�[0m     �[38;5;66;03m# Use DataFrameDatasetEngine�[39;00m

E           �[0;32m--> 262�[0m     _path_or_source �[38;5;241m=�[39m �[43mconvert_data�[49m�[43m(�[49m

E           �[1;32m    263�[0m �[43m        �[49m�[43mpath_or_source�[49m�[43m,�[49m�[43m �[49m�[43mcpu�[49m�[38;5;241;43m=�[39;49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mcpu�[49m�[43m,�[49m�[43m �[49m�[43mto_collection�[49m�[38;5;241;43m=�[39;49m�[38;5;28;43;01mTrue�[39;49;00m�[43m,�[49m�[43m �[49m�[43mnpartitions�[49m�[38;5;241;43m=�[39;49m�[43mnpartitions�[49m

E           �[1;32m    264�[0m �[43m    �[49m�[43m)�[49m

E           �[1;32m    265�[0m     �[38;5;66;03m# Check if this is a collection that has now moved between host <-> device�[39;00m

E           �[1;32m    266�[0m     moved_collection �[38;5;241m=�[39m �[38;5;28misinstance�[39m(path_or_source, dask�[38;5;241m.�[39mdataframe�[38;5;241m.�[39mDataFrame) �[38;5;129;01mand�[39;00m (

E           �[1;32m    267�[0m         �[38;5;129;01mnot�[39;00m �[38;5;28misinstance�[39m(_path_or_source�[38;5;241m.�[39m_meta, �[38;5;28mtype�[39m(path_or_source�[38;5;241m.�[39m_meta))

E           �[1;32m    268�[0m     )

E

E           File �[0;32m~/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py:567�[0m, in �[0;36mconvert_data�[0;34m(x, cpu, to_collection, npartitions)�[0m

E           �[1;32m    565�[0m     _x �[38;5;241m=�[39m cudf�[38;5;241m.�[39mDataFrame�[38;5;241m.�[39mfrom_arrow(x)

E           �[1;32m    566�[0m �[38;5;28;01melif�[39;00m �[38;5;28misinstance�[39m(x, pd�[38;5;241m.�[39mDataFrame):

E           �[0;32m--> 567�[0m     _x �[38;5;241m=�[39m �[43mcudf�[49m�[38;5;241;43m.�[39;49m�[43mDataFrame�[49m�[38;5;241;43m.�[39;49m�[43mfrom_pandas�[49m�[43m(�[49m�[43mx�[49m�[43m)�[49m

E           �[1;32m    568�[0m �[38;5;66;03m# Output a collection if to_collection=True�[39;00m

E           �[1;32m    569�[0m �[38;5;28;01mreturn�[39;00m (

E           �[1;32m    570�[0m     dask_cudf�[38;5;241m.�[39mfrom_cudf(_x, sort�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m, npartitions�[38;5;241m=�[39mnpartitions)

E           �[1;32m    571�[0m     �[38;5;28;01mif�[39;00m to_collection

E           �[1;32m    572�[0m     �[38;5;28;01melse�[39;00m _x

E           �[1;32m    573�[0m )

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101�[0m, in �[0;36mannotate.call..inner�[0;34m(args, **kwargs)�[0m

E           �[1;32m     98�[0m �[38;5;129m@wraps�[39m(func)

E           �[1;32m     99�[0m �[38;5;28;01mdef�[39;00m �[38;5;21minner�[39m(�[38;5;241m�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs):

E           �[1;32m    100�[0m     libnvtx_push_range(�[38;5;28mself�[39m�[38;5;241m.�[39mattributes, �[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)

E           �[0;32m--> 101�[0m     result �[38;5;241m=�[39m �[43mfunc�[49m�[43m(�[49m�[38;5;241;43m�[39;49m�[43margs�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m*�[39;49m�[43mkwargs�[49m�[43m)�[49m

E           �[1;32m    102�[0m     libnvtx_pop_range(�[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)

E           �[1;32m    103�[0m     �[38;5;28;01mreturn�[39;00m result

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:4386�[0m, in �[0;36mDataFrame.from_pandas�[0;34m(cls, dataframe, nan_as_null)�[0m

E           �[1;32m   4382�[0m �[38;5;28;01mfor�[39;00m col_name, col_value �[38;5;129;01min�[39;00m dataframe�[38;5;241m.�[39mitems():

E           �[1;32m   4383�[0m     �[38;5;66;03m# necessary because multi-index can return multiple�[39;00m

E           �[1;32m   4384�[0m     �[38;5;66;03m# columns for a single key�[39;00m

E           �[1;32m   4385�[0m     �[38;5;28;01mif�[39;00m �[38;5;28mlen�[39m(col_value�[38;5;241m.�[39mshape) �[38;5;241m==�[39m �[38;5;241m1�[39m:

E           �[0;32m-> 4386�[0m         df[col_name] �[38;5;241m=�[39m �[43mcolumn�[49m�[38;5;241;43m.�[39;49m�[43mas_column�[49m�[43m(�[49m

E           �[1;32m   4387�[0m �[43m            �[49m�[43mcol_value�[49m�[38;5;241;43m.�[39;49m�[43marray�[49m�[43m,�[49m�[43m �[49m�[43mnan_as_null�[49m�[38;5;241;43m=�[39;49m�[43mnan_as_null�[49m

E           �[1;32m   4388�[0m �[43m        �[49m�[43m)�[49m

E           �[1;32m   4389�[0m     �[38;5;28;01melse�[39;00m:

E           �[1;32m   4390�[0m         vals �[38;5;241m=�[39m col_value�[38;5;241m.�[39mvalues�[38;5;241m.�[39mT

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/cudf/core/column/column.py:1983�[0m, in �[0;36mas_column�[0;34m(arbitrary, nan_as_null, dtype, length)�[0m

E           �[1;32m   1981�[0m     data �[38;5;241m=�[39m as_column(pa�[38;5;241m.�[39mArray�[38;5;241m.�[39mfrom_pandas(arbitrary), dtype�[38;5;241m=�[39marb_dtype)

E           �[1;32m   1982�[0m �[38;5;28;01melse�[39;00m:

E           �[0;32m-> 1983�[0m     data �[38;5;241m=�[39m �[43mas_column�[49m�[43m(�[49m

E           �[1;32m   1984�[0m �[43m        �[49m�[43mpa�[49m�[38;5;241;43m.�[39;49m�[43marray�[49m�[43m(�[49m

E           �[1;32m   1985�[0m �[43m            �[49m�[43marbitrary�[49m�[43m,�[49m

E           �[1;32m   1986�[0m �[43m            �[49m�[43mfrom_pandas�[49m�[38;5;241;43m=�[39;49m�[38;5;28;43;01mTrue�[39;49;00m�[43m �[49m�[38;5;28;43;01mif�[39;49;00m�[43m �[49m�[43mnan_as_null�[49m�[43m �[49m�[38;5;129;43;01mis�[39;49;00m�[43m �[49m�[38;5;28;43;01mNone�[39;49;00m�[43m �[49m�[38;5;28;43;01melse�[39;49;00m�[43m �[49m�[43mnan_as_null�[49m�[43m,�[49m

E           �[1;32m   1987�[0m �[43m        �[49m�[43m)�[49m�[43m,�[49m

E           �[1;32m   1988�[0m �[43m        �[49m�[43mnan_as_null�[49m�[38;5;241;43m=�[39;49m�[43mnan_as_null�[49m�[43m,�[49m

E           �[1;32m   1989�[0m �[43m    �[49m�[43m)�[49m

E           �[1;32m   1990�[0m �[38;5;28;01mif�[39;00m dtype �[38;5;129;01mis�[39;00m �[38;5;129;01mnot�[39;00m �[38;5;28;01mNone�[39;00m:

E           �[1;32m   1991�[0m     data �[38;5;241m=�[39m data�[38;5;241m.�[39mastype(dtype)

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/cudf/core/column/column.py:1776�[0m, in �[0;36mas_column�[0;34m(arbitrary, nan_as_null, dtype, length)�[0m

E           �[1;32m   1770�[0m �[38;5;28;01mif�[39;00m �[38;5;28misinstance�[39m(arbitrary, pa�[38;5;241m.�[39mlib�[38;5;241m.�[39mHalfFloatArray):

E           �[1;32m   1771�[0m     �[38;5;28;01mraise�[39;00m �[38;5;167;01mNotImplementedError�[39;00m(

E           �[1;32m   1772�[0m         �[38;5;124m"�[39m�[38;5;124mType casting from float16 to float32 is not �[39m�[38;5;124m"�[39m

E           �[1;32m   1773�[0m         �[38;5;124m"�[39m�[38;5;124myet supported in pyarrow, see: �[39m�[38;5;124m"�[39m

E           �[1;32m   1774�[0m         �[38;5;124m"�[39m�[38;5;124mhttps://issues.apache.org/jira/browse/ARROW-3802�[39m�[38;5;124m"�[39m

E           �[1;32m   1775�[0m     )

E           �[0;32m-> 1776�[0m col �[38;5;241m=�[39m �[43mColumnBase�[49m�[38;5;241;43m.�[39;49m�[43mfrom_arrow�[49m�[43m(�[49m�[43marbitrary�[49m�[43m)�[49m

E           �[1;32m   1778�[0m �[38;5;28;01mif�[39;00m �[38;5;28misinstance�[39m(arbitrary, pa�[38;5;241m.�[39mNullArray):

E           �[1;32m   1779�[0m     new_dtype �[38;5;241m=�[39m cudf�[38;5;241m.�[39mdtype(arbitrary�[38;5;241m.�[39mtype�[38;5;241m.�[39mto_pandas_dtype())

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/cudf/core/column/column.py:302�[0m, in �[0;36mColumnBase.from_arrow�[0;34m(cls, array)�[0m

E           �[1;32m    297�[0m �[38;5;28;01melif�[39;00m �[38;5;28misinstance�[39m(

E           �[1;32m    298�[0m     array�[38;5;241m.�[39mtype, pd�[38;5;241m.�[39mcore�[38;5;241m.�[39marrays�[38;5;241m.�[39m_arrow_utils�[38;5;241m.�[39mArrowIntervalType

E           �[1;32m    299�[0m ):

E           �[1;32m    300�[0m     �[38;5;28;01mreturn�[39;00m cudf�[38;5;241m.�[39mcore�[38;5;241m.�[39mcolumn�[38;5;241m.�[39mIntervalColumn�[38;5;241m.�[39mfrom_arrow(array)

E           �[0;32m--> 302�[0m result �[38;5;241m=�[39m �[43mlibcudf�[49m�[38;5;241;43m.�[39;49m�[43minterop�[49m�[38;5;241;43m.�[39;49m�[43mfrom_arrow�[49m�[43m(�[49m�[43mdata�[49m�[43m,�[49m�[43m �[49m�[43mdata�[49m�[38;5;241;43m.�[39;49m�[43mcolumn_names�[49m�[43m)�[49m[�[38;5;241m0�[39m][�[38;5;124m"�[39m�[38;5;124mNone�[39m�[38;5;124m"�[39m]

E           �[1;32m    304�[0m �[38;5;28;01mreturn�[39;00m result�[38;5;241m.�[39m_with_type_metadata(cudf_dtype_from_pa_type(array�[38;5;241m.�[39mtype))

E

E           File �[0;32mcudf/_lib/interop.pyx:167�[0m, in �[0;36mcudf._lib.interop.from_arrow�[0;34m()�[0m

E

E           �[0;31mMemoryError�[0m: std::bad_alloc: out_of_memory: CUDA error at: /usr/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

E           MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /usr/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory


../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError

----------------------------- Captured stderr call -----------------------------

2022-09-14 19:51:14,850 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-09-14 19:51:14,868 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-09-14 19:51:14,870 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-09-14 19:51:14,876 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-09-14 19:51:21,872 - distributed.worker - WARNING - Compute Failed

Key:       ('level_2-bc7f3266731e63ba01f7580e4f224e63', 20, 2)

Function:  _mid_level_groupby

args:      ([pyarrow.Table

C21: int64

C21_size: int32


C21: [[6,374,475,65,78,539,62,282,513,14,...,53,273,497,218,28,93,32,151,429,136]]

C21_size: [[2315,2,1,210,143,2,264,1,1,1830,...,348,2,1,6,945,97,751,24,1,40]]], <merlin.dag.selector.ColumnSelector object at 0x7f88814a1dc0>, None, FitOptions(col_groups=[<merlin.dag.selector.ColumnSelector object at 0x7f88815adaf0>, <merlin.dag.selector.ColumnSelector object at 0x7f895245a940>, <merlin.dag.selector.ColumnSelector object at 0x7f895245a9a0>, <merlin.dag.selector.ColumnSelector object at 0x7f895245a1f0>, <merlin.dag.selector.ColumnSelector object at 0x7f8880b35c10>, <merlin.dag.selector.ColumnSelector object at 0x7f88813cd0a0>, <merlin.dag.selector.ColumnSelector object at 0x7f88813cd400>, <merlin.dag.selector.ColumnSelector object at 0x7f88813cd070>, <merlin.dag.selector.ColumnSelector object at 0x7f88813cd460>, <merlin.dag.selector.ColumnSelector object at 0x7f88813cd670>, <merlin.dag.selector.ColumnSelector object at 0x7f88807dd880>, <me

kwargs:    {}

Exception: "RuntimeError('CUDA error encountered at: /opt/tritonserver/build-env/cpp/src/bitmask/null_mask.cu:93: 2 cudaErrorMemoryAllocation out of memory')"


2022-09-14 19:51:21,881 - distributed.worker - WARNING - Compute Failed

Key:       ('level_2-bc7f3266731e63ba01f7580e4f224e63', 23, 0)

Function:  _mid_level_groupby

args:      ([pyarrow.Table

C24: int64

C24_size: int32


C24: [[3475,2915,4377,1519,5603,3551,3872,8361,992,9642,...,8597,4871,4556,4619,5311,4281,6552,9920,6279,6726]]

C24_size: [[3,8,3,10,2,13,7,9,10,4,...,5,9,4,3,4,5,7,3,6,11]]], <merlin.dag.selector.ColumnSelector object at 0x7f88815206a0>, None, FitOptions(col_groups=[<merlin.dag.selector.ColumnSelector object at 0x7f88815ada30>, <merlin.dag.selector.ColumnSelector object at 0x7f895245a1f0>, <merlin.dag.selector.ColumnSelector object at 0x7f88813cd400>, <merlin.dag.selector.ColumnSelector object at 0x7f88813cd3a0>, <merlin.dag.selector.ColumnSelector object at 0x7f88813cd670>, <merlin.dag.selector.ColumnSelector object at 0x7f88813cd520>, <merlin.dag.selector.ColumnSelector object at 0x7f88813cd460>, <merlin.dag.selector.ColumnSelector object at 0x7f88807ddfa0>, <merlin.dag.selector.ColumnSelector object at 0x7f88807ddf70>, <merlin.dag.selector.ColumnSelector object at 0x7f88807dd850>, <merlin.dag.selector.ColumnSelector object at 0x7f8880

kwargs:    {}

Exception: "RuntimeError('CUDA error encountered at: /opt/tritonserver/build-env/cpp/src/bitmask/null_mask.cu:93: 2 cudaErrorMemoryAllocation out of memory')"


2022-09-14 19:51:26,245 - distributed.worker - WARNING - Compute Failed

Key:       ('write-processed-88c3f49110679979b73d96ee6d57c0f4-partition88c3f49110679979b73d96ee6d57c0f4', "('part_0.parquet',)")

Function:  _write_subgraph

args:      (<merlin.io.dask.DaskSubgraph object at 0x7f88813cd670>, ('part_0.parquet',), '/tmp/output/criteo/train/', <Shuffle.PER_PARTITION: 0>, <fsspec.implementations.local.LocalFileSystem object at 0x7f88806c3040>, ['C1', 'C2', 'C3', 'C4', 'C5', 'C6', 'C7', 'C8', 'C9', 'C10', 'C11', 'C12', 'C13', 'C14', 'C15', 'C16', 'C17', 'C18', 'C19', 'C20', 'C21', 'C22', 'C23', 'C24', 'C25', 'C26'], ['I1', 'I2', 'I3', 'I4', 'I5', 'I6', 'I7', 'I8', 'I9', 'I10', 'I11', 'I12', 'I13'], ['label'], 'parquet', 0, False, '')

kwargs:    {}

Exception: "RuntimeError('CUDA error encountered at: /opt/tritonserver/build-env/cpp/src/io/utilities/hostdevice_vector.hpp:57: 2 cudaErrorMemoryAllocation out of memory')"


/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(

Failed to transform operator <nvtabular.ops.categorify.Categorify object at 0x7f17bc31a610>

Traceback (most recent call last):

File "/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/nvtabular/ops/categorify.py", line 461, in transform

path = self.categories[storage_name]

KeyError: 'C1'


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

File "/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py", line 91, in transform

output_df = node.op.transform(selection, input_df)

File "/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py", line 101, in inner

result = func(*args, **kwargs)

File "/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/nvtabular/ops/categorify.py", line 483, in transform

raise RuntimeError(f"Failed to categorical encode column {name}") from e

RuntimeError: Failed to categorical encode column C1

2022-09-14 19:51:26,258 - distributed.worker - WARNING - Compute Failed

Key:       ('write-processed-88c3f49110679979b73d96ee6d57c0f4-partition88c3f49110679979b73d96ee6d57c0f4', "('part_2.parquet',)")

Function:  _write_subgraph

args:      (<merlin.io.dask.DaskSubgraph object at 0x7f1b2c4157c0>, ('part_2.parquet',), '/tmp/output/criteo/train/', <Shuffle.PER_PARTITION: 0>, <fsspec.implementations.local.LocalFileSystem object at 0x7f1b2c570fa0>, ['C1', 'C2', 'C3', 'C4', 'C5', 'C6', 'C7', 'C8', 'C9', 'C10', 'C11', 'C12', 'C13', 'C14', 'C15', 'C16', 'C17', 'C18', 'C19', 'C20', 'C21', 'C22', 'C23', 'C24', 'C25', 'C26'], ['I1', 'I2', 'I3', 'I4', 'I5', 'I6', 'I7', 'I8', 'I9', 'I10', 'I11', 'I12', 'I13'], ['label'], 'parquet', 0, False, '')

kwargs:    {}

Exception: "RuntimeError('Failed to categorical encode column C1')"


/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(

Failed to transform operator <nvtabular.ops.categorify.Categorify object at 0x7f4db97b1be0>

Traceback (most recent call last):

File "/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/nvtabular/ops/categorify.py", line 461, in transform

path = self.categories[storage_name]

KeyError: 'C1'


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

File "/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py", line 91, in transform

output_df = node.op.transform(selection, input_df)

File "/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py", line 101, in inner

result = func(*args, **kwargs)

File "/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/nvtabular/ops/categorify.py", line 483, in transform

raise RuntimeError(f"Failed to categorical encode column {name}") from e

RuntimeError: Failed to categorical encode column C1

/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.

warnings.warn(

Failed to transform operator <nvtabular.ops.categorify.Categorify object at 0x7fa40cf6d490>

Traceback (most recent call last):

File "/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/nvtabular/ops/categorify.py", line 461, in transform

path = self.categories[storage_name]

KeyError: 'C1'


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

File "/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py", line 91, in transform

output_df = node.op.transform(selection, input_df)

File "/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py", line 101, in inner

result = func(*args, **kwargs)

File "/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/nvtabular/ops/categorify.py", line 483, in transform

raise RuntimeError(f"Failed to categorical encode column {name}") from e

RuntimeError: Failed to categorical encode column C1

2022-09-14 19:51:26,262 - distributed.worker - WARNING - Compute Failed

Key:       ('write-processed-88c3f49110679979b73d96ee6d57c0f4-partition88c3f49110679979b73d96ee6d57c0f4', "('part_1.parquet',)")

Function:  _write_subgraph

args:      (<merlin.io.dask.DaskSubgraph object at 0x7fa40f8a7eb0>, ('part_1.parquet',), '/tmp/output/criteo/train/', <Shuffle.PER_PARTITION: 0>, <fsspec.implementations.local.LocalFileSystem object at 0x7fa8408c4d90>, ['C1', 'C2', 'C3', 'C4', 'C5', 'C6', 'C7', 'C8', 'C9', 'C10', 'C11', 'C12', 'C13', 'C14', 'C15', 'C16', 'C17', 'C18', 'C19', 'C20', 'C21', 'C22', 'C23', 'C24', 'C25', 'C26'], ['I1', 'I2', 'I3', 'I4', 'I5', 'I6', 'I7', 'I8', 'I9', 'I10', 'I11', 'I12', 'I13'], ['label'], 'parquet', 0, False, '')

kwargs:    {}

Exception: "RuntimeError('Failed to categorical encode column C1')"


2022-09-14 19:51:26,263 - distributed.worker - WARNING - Compute Failed

Key:       ('write-processed-88c3f49110679979b73d96ee6d57c0f4-partition88c3f49110679979b73d96ee6d57c0f4', "('part_3.parquet',)")

Function:  _write_subgraph

args:      (<merlin.io.dask.DaskSubgraph object at 0x7f4db9c1bf10>, ('part_3.parquet',), '/tmp/output/criteo/train/', <Shuffle.PER_PARTITION: 0>, <fsspec.implementations.local.LocalFileSystem object at 0x7f4e53bfcac0>, ['C1', 'C2', 'C3', 'C4', 'C5', 'C6', 'C7', 'C8', 'C9', 'C10', 'C11', 'C12', 'C13', 'C14', 'C15', 'C16', 'C17', 'C18', 'C19', 'C20', 'C21', 'C22', 'C23', 'C24', 'C25', 'C26'], ['I1', 'I2', 'I3', 'I4', 'I5', 'I6', 'I7', 'I8', 'I9', 'I10', 'I11', 'I12', 'I13'], ['label'], 'parquet', 0, False, '')

kwargs:    {}

Exception: "RuntimeError('Failed to categorical encode column C1')"


2022-09-14 19:51:28,397 - distributed.worker - WARNING - Compute Failed

Key:       ('write-processed-edf9fc2c393d7b5fb84a1f61807d162a-partitionedf9fc2c393d7b5fb84a1f61807d162a', "('part_0.parquet',)")

Function:  _write_subgraph

args:      (<merlin.io.dask.DaskSubgraph object at 0x7f8881544fa0>, ('part_0.parquet',), '/tmp/output/criteo/valid/', None, <fsspec.implementations.local.LocalFileSystem object at 0x7f88806c3040>, ['C1', 'C2', 'C3', 'C4', 'C5', 'C6', 'C7', 'C8', 'C9', 'C10', 'C11', 'C12', 'C13', 'C14', 'C15', 'C16', 'C17', 'C18', 'C19', 'C20', 'C21', 'C22', 'C23', 'C24', 'C25', 'C26'], ['I1', 'I2', 'I3', 'I4', 'I5', 'I6', 'I7', 'I8', 'I9', 'I10', 'I11', 'I12', 'I13'], ['label'], 'parquet', 0, False, '')

kwargs:    {}

Exception: "RuntimeError('CUDA error encountered at: /opt/tritonserver/build-env/cpp/src/io/utilities/hostdevice_vector.hpp:57: 2 cudaErrorMemoryAllocation out of memory')"


Failed to transform operator <nvtabular.ops.categorify.Categorify object at 0x7f17bc25ae20>

Traceback (most recent call last):

File "/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/nvtabular/ops/categorify.py", line 461, in transform

path = self.categories[storage_name]

KeyError: 'C1'


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

File "/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py", line 91, in transform

output_df = node.op.transform(selection, input_df)

File "/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py", line 101, in inner

result = func(*args, **kwargs)

File "/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/nvtabular/ops/categorify.py", line 483, in transform

raise RuntimeError(f"Failed to categorical encode column {name}") from e

RuntimeError: Failed to categorical encode column C1

Failed to transform operator <nvtabular.ops.categorify.Categorify object at 0x7f4db4e3f130>

Traceback (most recent call last):

File "/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/nvtabular/ops/categorify.py", line 461, in transform

path = self.categories[storage_name]

KeyError: 'C1'


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

File "/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py", line 91, in transform

output_df = node.op.transform(selection, input_df)

File "/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py", line 101, in inner

result = func(*args, **kwargs)

File "/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/nvtabular/ops/categorify.py", line 483, in transform

raise RuntimeError(f"Failed to categorical encode column {name}") from e

RuntimeError: Failed to categorical encode column C1

Failed to transform operator <nvtabular.ops.categorify.Categorify object at 0x7fa40ce529a0>

Traceback (most recent call last):

File "/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/nvtabular/ops/categorify.py", line 461, in transform

path = self.categories[storage_name]

KeyError: 'C1'


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

File "/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py", line 91, in transform

output_df = node.op.transform(selection, input_df)

File "/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py", line 101, in inner

result = func(*args, **kwargs)

File "/var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/lib/python3.8/site-packages/nvtabular/ops/categorify.py", line 483, in transform

raise RuntimeError(f"Failed to categorical encode column {name}") from e

RuntimeError: Failed to categorical encode column C1

2022-09-14 19:51:28,413 - distributed.worker - WARNING - Compute Failed

Key:       ('write-processed-edf9fc2c393d7b5fb84a1f61807d162a-partitionedf9fc2c393d7b5fb84a1f61807d162a', "('part_1.parquet',)")

Function:  _write_subgraph

args:      (<merlin.io.dask.DaskSubgraph object at 0x7f17bf780ac0>, ('part_1.parquet',), '/tmp/output/criteo/valid/', None, <fsspec.implementations.local.LocalFileSystem object at 0x7f1b2c570fa0>, ['C1', 'C2', 'C3', 'C4', 'C5', 'C6', 'C7', 'C8', 'C9', 'C10', 'C11', 'C12', 'C13', 'C14', 'C15', 'C16', 'C17', 'C18', 'C19', 'C20', 'C21', 'C22', 'C23', 'C24', 'C25', 'C26'], ['I1', 'I2', 'I3', 'I4', 'I5', 'I6', 'I7', 'I8', 'I9', 'I10', 'I11', 'I12', 'I13'], ['label'], 'parquet', 0, False, '')

kwargs:    {}

Exception: "RuntimeError('Failed to categorical encode column C1')"


2022-09-14 19:51:28,413 - distributed.worker - WARNING - Compute Failed

Key:       ('write-processed-edf9fc2c393d7b5fb84a1f61807d162a-partitionedf9fc2c393d7b5fb84a1f61807d162a', "('part_3.parquet',)")

Function:  _write_subgraph

args:      (<merlin.io.dask.DaskSubgraph object at 0x7f4db9a5d9a0>, ('part_3.parquet',), '/tmp/output/criteo/valid/', None, <fsspec.implementations.local.LocalFileSystem object at 0x7f4e53bfcac0>, ['C1', 'C2', 'C3', 'C4', 'C5', 'C6', 'C7', 'C8', 'C9', 'C10', 'C11', 'C12', 'C13', 'C14', 'C15', 'C16', 'C17', 'C18', 'C19', 'C20', 'C21', 'C22', 'C23', 'C24', 'C25', 'C26'], ['I1', 'I2', 'I3', 'I4', 'I5', 'I6', 'I7', 'I8', 'I9', 'I10', 'I11', 'I12', 'I13'], ['label'], 'parquet', 0, False, '')

kwargs:    {}

Exception: "RuntimeError('Failed to categorical encode column C1')"


2022-09-14 19:51:28,415 - distributed.worker - WARNING - Compute Failed

Key:       ('write-processed-edf9fc2c393d7b5fb84a1f61807d162a-partitionedf9fc2c393d7b5fb84a1f61807d162a', "('part_2.parquet',)")

Function:  _write_subgraph

args:      (<merlin.io.dask.DaskSubgraph object at 0x7fa40d02d370>, ('part_2.parquet',), '/tmp/output/criteo/valid/', None, <fsspec.implementations.local.LocalFileSystem object at 0x7fa8408c4d90>, ['C1', 'C2', 'C3', 'C4', 'C5', 'C6', 'C7', 'C8', 'C9', 'C10', 'C11', 'C12', 'C13', 'C14', 'C15', 'C16', 'C17', 'C18', 'C19', 'C20', 'C21', 'C22', 'C23', 'C24', 'C25', 'C26'], ['I1', 'I2', 'I3', 'I4', 'I5', 'I6', 'I7', 'I8', 'I9', 'I10', 'I11', 'I12', 'I13'], ['label'], 'parquet', 0, False, '')

kwargs:    {}

Exception: "RuntimeError('Failed to categorical encode column C1')"


/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 51 leaked semaphore objects to clean up at shutdown

warnings.warn('resource_tracker: There appear to be %d '


---------- coverage: platform linux, python 3.8.10-final-0 -----------


=========================== short test summary info ============================

SKIPPED [1] tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:16: 'NoneType' returning from faiss operator because of randomly generated data.

=================== 1 failed, 1 passed, 1 skipped in 31.28s ====================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:794: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

/usr/local/lib/python3.8/dist-packages/coverage/data.py:130: CoverageWarning: Data file '/var/jenkins_home/workspace/merlin_merlin/merlin/.coverage.10.20.17.231.22792.001634' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object

data._warn(str(exc))

/usr/local/lib/python3.8/dist-packages/pytest_cov/plugin.py:294: CovReportWarning: Failed to generate report: No data to report.


self.cov_controller.finish()

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://github.com/gitapi/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins12863144807747359961.sh

@jperez999
Copy link
Collaborator Author

rerun tests

@nvidia-merlin-bot
Copy link
Contributor

Click to view CI Results
GitHub pull request #619 of commit 400b68e1d4f6a2cc895d233c0af8c30eacc9dd29, no merge conflicts.
Running as SYSTEM
Setting status of 400b68e1d4f6a2cc895d233c0af8c30eacc9dd29 to PENDING with url https://10.20.13.93:8080/job/merlin_merlin/449/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_merlin
using credential systems-login
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Merlin # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Merlin
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Merlin +refs/pull/619/*:refs/remotes/origin/pr/619/* # timeout=10
 > git rev-parse 400b68e1d4f6a2cc895d233c0af8c30eacc9dd29^{commit} # timeout=10
Checking out Revision 400b68e1d4f6a2cc895d233c0af8c30eacc9dd29 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 400b68e1d4f6a2cc895d233c0af8c30eacc9dd29 # timeout=10
Commit message: "Merge branch 'test-rework' of https://github.com/jperez999/merlin into test-rework"
 > git rev-list --no-walk 400b68e1d4f6a2cc895d233c0af8c30eacc9dd29 # timeout=10
[merlin_merlin] $ /bin/bash /tmp/jenkins12117187023860776989.sh
GLOB sdist-make: /var/jenkins_home/workspace/merlin_merlin/merlin/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_merlin/merlin/.tox/.tmp/package/1/merlin-0.0.1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.7,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,attrs==22.1.0,awscli==1.25.73,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.51,botocore==1.27.72,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,clang==5.0,click==8.1.3,cloudpickle==2.1.0,colorama==0.4.4,coverage==6.4.4,cuda-python==11.7.1,cudf==22.4.0,cupy-cuda116==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dask-cuda==22.4.0,dask-cudf==22.4.0,dbus-python==1.2.16,debugpy==1.6.2,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.3.0,distro==1.7.0,dm-tree==0.1.7,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==0.10.0,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.82.0,fastavro==1.6.0,fastcore==1.5.24,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.0,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.0,google-auth==2.11.0,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.2,grpcio==1.41.0,grpcio-channelz==1.47.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.4.0,hugectr2onnx==0.0.0,huggingface-hub==0.8.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.0,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.1,ipython==8.4.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.1.0,json5==0.9.9,jsonschema==4.9.1,jupyter-cache==0.4.3,jupyter-client==7.3.4,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyterlab==3.4.5,jupyterlab-pygments==0.2.2,jupyterlab-server==2.15.0,jupyterlab-widgets==1.1.0,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.0,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.5.3,matplotlib-inline==0.1.3,mdit-py-plugins==0.2.8,merlin==0.0.1,merlin-core==0+untagged.112.gfbfcaf3,merlin-models==0.6.0+45.g5a345d9c1,merlin-systems==0+untagged.105.gf89cc51,mistune==0.8.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.6,nbconvert==6.5.3,nbdime==3.1.1,nbformat==5.4.0,nest-asyncio==1.5.5,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.0,numpy==1.21.5,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.3.3+15.g16e4e34e9),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.0,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.30,proto-plus==1.19.6,protobuf==3.19.4,psutil==5.9.1,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==6.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.12.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.2,pytest-cov==3.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.7,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==23.2.1,regex==2022.7.25,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rmm==21.12.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.0,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.4,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.2.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.1.1,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.4.0,starlette==0.19.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.26.0,tensorflow-metadata==1.9.0,termcolor==1.1.0,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.25.1,tqdm==4.64.0,traitlets==5.3.0,transformers==4.12.0,transformers4rec==0.1.11+10.g21a2a836a,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.22.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.11,uvicorn==0.18.3,uvloop==0.16.0,versioneer==0.20,virtualenv==20.16.4,wandb==0.13.1,watchfiles==0.16.1,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.3.3,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='479474319'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/systems.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/systems.git
  Cloning https://github.com/NVIDIA-Merlin/systems.git to /tmp/pip-req-build-z0xgp72c
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/systems.git /tmp/pip-req-build-z0xgp72c
  Resolved https://github.com/NVIDIA-Merlin/systems.git to commit 15074adf11bcc9bb9da375ec40ab4e431f1c93e9
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.5.0+4.g15074ad) (0.6.0+1.g5926fcf)
Requirement already satisfied: nvtabular>=1.0.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.5.0+4.g15074ad) (1.4.0+7.geed9172e)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (4.64.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (3.19.4)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2022.5.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.2.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (6.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.9.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.3.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2022.3.0)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular>=1.0.0->merlin-systems==0.5.0+4.g15074ad) (1.9.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (0.4.3)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2.1.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (3.1.2)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (8.1.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (65.3.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.5.0+4.g15074ad) (6.0.1)
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-jr0xxjh4
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-jr0xxjh4
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit 91fee2a85e309cbd5d338d9ba37d989b54594498
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-models==0.7.0+5.g91fee2a8) (0.6.0+1.g5926fcf)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (4.64.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (3.19.4)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2022.5.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.2.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (6.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.9.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.3.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2022.3.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (0.4.3)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2.1.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (3.1.2)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (8.1.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (65.3.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.7.0+5.g91fee2a8) (6.0.1)
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/NVTabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-om0a045o
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-om0a045o
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit eed9172ec30a47e020b06ff60469d6d168eb865b
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from nvtabular==1.4.0+7.geed9172e) (0.6.0+1.g5926fcf)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.4.0+7.geed9172e) (1.9.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (4.64.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (3.19.4)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2022.5.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.2.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (6.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.9.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.3.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2022.3.0)
Requirement already satisfied: numpy<1.25.0,>=1.18.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.4.0+7.geed9172e) (1.20.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (0.4.3)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2.1.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (3.1.2)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (8.1.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (65.3.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.4.0+7.geed9172e) (6.0.1)
test-gpu run-test: commands[3] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-xh_b6goc
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-xh_b6goc
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 5926fcfc0776c112680cbca28553c15211c2fda7
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.6.0+1.g5926fcf) (4.64.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.6.0+1.g5926fcf) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.6.0+1.g5926fcf) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.6.0+1.g5926fcf) (3.19.4)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.6.0+1.g5926fcf) (2022.5.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.6.0+1.g5926fcf) (1.2.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.6.0+1.g5926fcf) (6.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.6.0+1.g5926fcf) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.6.0+1.g5926fcf) (1.9.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.6.0+1.g5926fcf) (1.3.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.6.0+1.g5926fcf) (2022.3.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.6.0+1.g5926fcf) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.6.0+1.g5926fcf) (0.4.3)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (2.1.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (3.1.2)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (8.1.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.6.0+1.g5926fcf) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.6.0+1.g5926fcf) (65.3.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.6.0+1.g5926fcf) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.6.0+1.g5926fcf) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.6.0+1.g5926fcf) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.6.0+1.g5926fcf) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.6.0+1.g5926fcf) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.6.0+1.g5926fcf) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.6.0+1.g5926fcf) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.6.0+1.g5926fcf) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.6.0+1.g5926fcf) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.6.0+1.g5926fcf) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.6.0+1.g5926fcf) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.6.0+1.g5926fcf) (6.0.1)
test-gpu run-test: commands[4] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_merlin/merlin
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 3 items

tests/unit/test_version.py .                                             [ 33%]

tests/unit/examples/test_building_deploying_multi_stage_RecSys.py s      [ 66%]

tests/unit/examples/test_scaling_criteo_merlin_models.py .               [100%]


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                                                  Stmts   Miss  Cover


.tox/test-gpu/lib/python3.8/site-packages/merlin/core/init.py                                         2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/_version.py                                         4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/compat.py                                           9      4    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py                                       346    221    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py                                          195    119    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/init.py                                          4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/base_operator.py                                   115     20    83%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py                                       111     50    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py                                            99     35    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py                                            342    165    52%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/init.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/concat_columns.py                               18      6    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/selection.py                                    20      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subset_columns.py                               12      4    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/ops/subtraction.py                                  21     11    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/selector.py                                         88     28    68%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/init.py                                           4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/csv.py                                               57     38    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dask.py                                             181    117    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_engine.py                                  61     47    23%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataframe_iter.py                                    21      4    81%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py                                          346    234    32%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py                                    37     13    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/fsspec_utils.py                                     127    108    15%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/hugectr.py                                           45     35    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py                                          603    345    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/shuffle.py                                           38     20    47%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/worker.py                                            80     30    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer.py                                           190     78    59%

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/writer_factory.py                                    18      5    72%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/_version.py                                       4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/config/schema.py                                 62     19    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/init.py                                0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/backend.py                               379    117    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/dataframe_iter.py                         21     17    19%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/tf_utils.py                               57     27    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/loader/utils.py                                  40     15    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/init.py                                   64      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/cross.py                               44     28    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/dlrm.py                                49     11    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/experts.py                             99     75    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/interaction.py                         93     48    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/mlp.py                                114     56    51%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/optimizer.py                          173    127    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/base.py                     168    125    26%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/matrix_factorization.py      35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/retrieval/two_tower.py                 30     18    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/init.py                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/base.py                       29     11    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/cross_batch.py                46     31    33%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/in_batch.py                   35     20    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/blocks/sampling/queue.py                     115     99    14%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/init.py                               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/aggregation.py                          245    109    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/base.py                                 239    115    52%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/combinators.py                          393    130    67%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/encoder.py                               73     47    36%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/index.py                                104     68    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/prediction.py                            25      2    92%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/core/tabular.py                              271     71    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/dataset.py                                   213    108    49%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/base.py                                56     44    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/continuous.py                          33      3    91%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/inputs/embedding.py                          432    237    45%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/init.py                             4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/base.py                                 9      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/listwise.py                            13      2    85%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/losses/pairwise.py                           113     55    51%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/init.py                            2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/metrics/topk.py                              184     83    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/init.py                             0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/base.py                               545    249    54%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/benchmark.py                           16      6    62%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/ranking.py                             69     42    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/retrieval.py                           32     15    53%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/models/utils.py                               10      2    80%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/init.py                   0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/base.py                     199    107    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/classification.py            68     23    66%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/multi.py                      7      1    86%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/next_item.py                 59     33    44%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/regression.py                35     19    46%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/prediction_tasks/retrieval.py                 73     48    34%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/predictions/init.py                        0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/predictions/base.py                          130     95    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/predictions/classification.py                154    100    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/predictions/dot_product.py                   120     73    39%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/predictions/regression.py                      9      2    78%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/predictions/sampling/init.py               0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/predictions/sampling/base.py                  67     32    52%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/predictions/sampling/in_batch.py              35     20    43%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/predictions/sampling/popularity.py            27     17    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/init.py                         0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/bias.py                            74     49    34%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/features.py                       239    187    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/noise.py                           43     28    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/regularization.py                  17      6    65%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/transforms/tensor.py                          91     47    48%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/typing.py                                      7      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/init.py                              0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/batch_utils.py                          81     56    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/repr_utils.py                           69     48    30%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/search_utils.py                         34     22    35%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/tf/utils/tf_utils.py                            186    118    37%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/init.py                                 0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/constants.py                                3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/dataset.py                                 33     24    27%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/doc_utils.py                               10      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/misc_utils.py                             118     90    24%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/registry.py                               101     31    69%

.tox/test-gpu/lib/python3.8/site-packages/merlin/models/utils/schema_utils.py                            90     62    31%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/init.py                                       2      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/init.py                                    0      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/proto_utils.py                                20      6    70%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/schema_bp.py                                 306      7    98%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/io/tensorflow_metadata.py                       191     43    77%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/schema.py                                       205     54    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py                                          82      8    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/init.py                                      6      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/_version.py                                      4      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py                                  3      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py                                 78      8    90%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/node.py                                     20      1    95%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py                                21     15    29%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/init.py                              9      4    56%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py                             83     37    55%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/tensorflow.py                           81     19    77%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/workflow.py                             25      0   100%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/model_registry.py                               16      8    50%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/init.py                              47     12    74%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py                           87     51    41%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/export.py                               266    210    21%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/triton/utils.py                                 72     14    81%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/init.py                            22     20     9%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py                               113     14    88%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/hugectr.py                             37     29    22%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/pytorch.py                             10      6    40%

.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/workflow/tensorflow.py                          32     13    59%


TOTAL                                                                                                 11955   5905    51%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/examples/test_building_deploying_multi_stage_RecSys.py:16: 'NoneType' returning from faiss operator because of randomly generated data.

=================== 2 passed, 1 skipped in 102.68s (0:01:42) ===================

/usr/local/lib/python3.8/dist-packages/coverage/inorout.py:519: CoverageWarning: Module merlin was never imported. (module-not-imported)

self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

/usr/local/lib/python3.8/dist-packages/coverage/control.py:794: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

/usr/local/lib/python3.8/dist-packages/coverage/data.py:130: CoverageWarning: Couldn't use data file '/var/jenkins_home/workspace/merlin_merlin/merlin/.coverage.10.20.17.231.24722.145804': database disk image is malformed

data._warn(str(exc))

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://github.com/gitapi/repos/NVIDIA-Merlin/Merlin/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_merlin] $ /bin/bash /tmp/jenkins2677782907732095404.sh

@jperez999 jperez999 merged commit 9013783 into NVIDIA-Merlin:main Sep 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rnyak rnyak rnyak left review comments

@benfred benfred benfred approved these changes

@karlhigley karlhigley Awaiting requested review from karlhigley

@nv-alaiacano nv-alaiacano Awaiting requested review from nv-alaiacano

Assignees

@jperez999 jperez999

Labels
chore Infrastructure update ci
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jperez999 @nvidia-merlin-bot @benfred @rnyak