Quick-start for ranking with Merlin Models

[gabrielspmoreira]

This PR is a port of the #988 PR that was originally in models repo, then ported here as the quick-start involves different Merlin libraries: NVTabular, models, and systems

Fixes #916 , fixes #986 , fixes #918, fixes #680, fixes #681, fixes #666

Goals ⚽

This PR introduces a quick-start example for preprocessing, training, evaluating and deploying ranking models.
It is composed by a set of scripts and markdown documents. We use in the example the TenRec dataset, but the scripts are generic and can be used with customer own data, provided that they have the right shape: positive and potentially negative user-item events with tabular features.

Implementation Details 🚧

• preprocessing.py - Generic script for preprocessing with CLI arguments for preprocessing a raw dataset (CSV or parquet) with NVTabular. It contains arguments to configure input path and format, categorical and continuous features, configuring the features tagging (user_id, item_id, ...), to filter interactions by using min/max frequency for users or items and dataset split.
  Example command line for TenRec dataset:

python preprocessing.py --input_data_format=csv --csv_na_values=\\N --input_data_path /data/QK-video.csv --output_path=$OUT_DATASET_PATH --categorical_features=user_id,item_id,video_category,gender,age --binary_classif_targets=click,follow,like,share --regression_targets=watching_times --to_int32=user_id,item_id --to_int16=watching_times --to_int8=gender,age,video_category,click,follow,like,share --user_id_feature=user_id --item_id_feature=item_id --min_user_freq 5 --persist_intermediate_files --dataset_split_strategy=random --random_split_eval_perc=0.2 	

• ranking_train_eval.py - Generic script for training and evaluation of ranking models. It takes the preprocessed dataset from preprocessing.py and schema as input. You can set many different training and model hparams for train both single-task learning (MLP, DCN, DLRM, Wide&Deep, DeepFM) and multi-task learning specific models (e.g. MMOE, CGC, PLE).

python  ranking_train_eval.py --train_path $OUT_DATASET_PATH/final_dataset/train --eval_path $OUT_DATASET_PATH/final_dataset/eval --output_path ./outputs/ --tasks=click --stl_positive_class_weight 4 --model dlrm --embeddings_dim 64 --l2_reg 1e-5 --embeddings_l2_reg 1e-6 --dropout 0.05 --mlp_layers 64,32  --lr 1e-4 --lr_decay_rate 0.99 --lr_decay_steps 100 --train_batch_size 4096 --eval_batch_size 4096 --epochs 1 --train_steps_per_epoch 10 

Testing Details 🔍

• The preprocessing and training ranking scripts are going to be added as integration tests.

Tasks

Implementation

• [Task] Create a generic dataset preprocessing template for ranking models #916
• Update the research STL/MTL ranking training script to use the latest API (PredictionBlock instead of PredictionTask) #917
• Create a generic STL/MTL ranking training/eval script based on the research scripts #918
• Refine preprocessing.py to provide additional dataset split strategies (e.g. random_by_user, temporal).
• Adapt preprocessing.py to use Dask Distributed client for preprocessing larger/full dataset (single or multiple GPU)

Experimentation

• Run hypertuning experiments for the STL ranking models: MLP, Wide&Deep, DCN-v2, DeepFM, DLRM for Tenrec dataset #680
• Run hypertuning experiments for the MTL ranking models: MMoE, PLE for Tenrec dataset #681

Documentation

• Create documentation for Quick-Start for Ranking (CLI args, best practices and tutorials) #666 - Create a markdown document providing best practices on setting hyperparameters for ranking models based on the empirical results from our research experimentation (e.g. hparam optimization search space, best hparams found, comparison of the accuracy of STL and MTL models for each task)

You can check the Quick-start for ranking documentation starting from this main page

[github-actions]

Documentation preview

https://nvidia-merlin.github.io/Merlin/review/pr-915

[rnyak]

as we discussed, I think this should be optional, and users should be able to use CPU as well, which they can prefer with a small dataset. This is the case for the recsys23 competition dataset, it is not that big.

[gabrielspmoreira]

I changed the preprocess.py script to detect if there are GPU available, otherwise it configures Dataset(..., cpu=True). But when testing this setting in the Merlin TF container 23.02 without GPUs available it raised some errors when importing NVTabular due to a known issue with cuda-python 11.7.0 and earlier (used by cudf). According to @oliverholworthy, it seems we won't have this issue with the release 23.04 because we’re using a more recent version of cudf and cuda-python.

[gabrielspmoreira]

In addition to that, I created an option --enable_dask_cuda_cluster in the preprocess.py script to enable/disable the usage of Dask Cluster, as for smaller datasets not using LocalCUDACluster might be faster

[rnyak]

if one already split train and val sets at their end up front, in that case, data-path would be train set path right? and to avoid any further split, users should also set --dataset_split_strategy to None?. or it is default None? If it is default None, that's fine.

[gabrielspmoreira]

Yes, the default --dataset_split_strategy is None. You the eval and test set were already split, you just need to provide them in --eval_data_path and --test_data_path

[rnyak]

do you think this should go one level up? Like opening readme page right under Merlin/examples/quick_start/ folder?

[gabrielspmoreira]

It could be there, but maybe renamed to something like ranking.md?
The idea is that we gonna add other quick-start documents next, like Quick-start for session-based recommendation, for retrieval, for Two/Multiple Stages.
In that case, I think that should be a README.md that would work as an index for our quick-starts. What do you think?

[rnyak]

sounds good to me.

[gabrielspmoreira]

Have created a README.md with an intro to Quick-starts, and a link to the ranking one

[rnyak]

@gabrielspmoreira One think I think we can improve is the prediction step. I tested the script you shared with me for prediction but it retrains the model.. but is there a prediction script that user can feed the saved model path and then do the batch predict automatically without training again? It'd be better if we can provide an example code snippet how one can do the prediction.

[gabrielspmoreira]

@gabrielspmoreira One think I think we can improve is the prediction step. I tested the script you shared with me for prediction but it retrains the model.. but is there a prediction script that user can feed the saved model path and then do the batch predict automatically without training again? It'd be better if we can provide an example code snippet how one can do the prediction.

Indeed. Following your suggestion, I made it possible to save the trained model with --save_model_path, then run the script again providing --load_model_path, in this case not providing train_data_path but just --predict_data_path, so that the script loads the trained model and just perform the batch predict, saving them to --predict_output_path