-
Notifications
You must be signed in to change notification settings - Fork 50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add dataloader pre-trained embeddings support to Merlin Models #1083
Conversation
Documentation preview |
1f1857c
to
1d13cb5
Compare
50a3ddd
to
78264e4
Compare
…trained embeddings
…96 to 100, in order to speed-up tests
…ence_testing_data
…d item is equal the item id cardinality
bd63bf3
to
c535e83
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the PR @gabrielspmoreira! This looks good to me, I've just left some minor comments.
@@ -96,7 +96,12 @@ def MLPBlock( | |||
|
|||
for idx, dim in enumerate(dimensions): | |||
dropout_layer = None | |||
activation_idx = activation if isinstance(activation, str) else activation[idx] | |||
activation = activation or "linear" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I understand that if activation is None, we will use "linear" by default. If yes, can we set the default value of activation
in the method's args to None
?
merlin/models/tf/inputs/embedding.py
Outdated
aggregation: Optional[TabularAggregationType], optional | ||
Transformation block to apply for aggregating the inputs, by default None | ||
block_name: str, optional | ||
Name of the block, by default "embeddings" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Name of the block, by default "embeddings" | |
Name of the block, by default "pretrained_embeddings" |
merlin/models/tf/inputs/embedding.py
Outdated
@@ -1238,6 +1335,8 @@ def process_str_sequence_combiner( | |||
combiner = tf.keras.layers.Lambda(lambda x: tf.reduce_mean(x, axis=1)) | |||
elif combiner == "sum": | |||
combiner = tf.keras.layers.Lambda(lambda x: tf.reduce_sum(x, axis=1)) | |||
elif combiner == "max": | |||
combiner = tf.keras.layers.Lambda(lambda x: tf.reduce_max(x, axis=1)) | |||
else: | |||
raise ValueError( | |||
"Only 'mean' and 'sum' str combiners is implemented for dense" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"Only 'mean' and 'sum' str combiners is implemented for dense" | |
"Only 'mean', 'sum', and 'max' str combiners is implemented for dense" |
else: | ||
inputs = tf.linalg.l2_normalize(inputs, axis=axis) | ||
inputs = self._l2_norm(inputs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we might need to specify the axis parameter in this line
) -> Union[tf.Tensor, tf.SparseTensor, tf.RaggedTensor]: | ||
"""Computes L2-norm for a given axis, typically axis = -1. | ||
Equivalent to tf.linalg.l2_normalize(), but that function | ||
does not support tf.RaggedTensor |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good to know, thanks for the fix!
for col in schema: | ||
table_name = col.name | ||
|
||
tables[table_name] = NoOp() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this be configurable to allow users to do things like normalization?
Fixes #1070, Fixes #1071, Fixes #1068, Fixes #1072, Fixes #1073
Goals ⚽
EmbeddingOperator
transform from Merlin dataloader: [RMP] Support pre-trained vector embeddings as input features into a model via the dataloader Merlin#211Tasks
BroadcastFeatures
support expanding 2D pre-trained embeddings (contextual features) to concat with 3D sequence features #1073Implementation Details 🚧
PretrainedEmbeddings
block that takes pre-trained embedding features and optionally projects them to a dim with a linear layer, applies a sequence aggregator and normalizes (e.g. with l2-norm). All of these options are configurableInputBlockV2
was changed to accept an optionalpretrained_embeddings
argument, which by default selects features tagged with the EMBEDDING tag,PrepareListFeatures
was changed to define the shape of the last dim the pre-trained embeddings provided by the Loader, as that dim is None in graph mode.Testing Details 🔍