You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am using mittens with a pre-built cooccurrence matrix of domains with the hopes of clustering certain domains that are thematically related, close to each other. Using the non-vectorized glove implementation from https://github.com/stanfordnlp/GloVe, I get very strong results. The current initialization is:
Given these results, I wanted to use mittens for two reasons: take advantage of the vectorized implementation for speed, and harness the ability to extend glove into a retrofitted model. However, when I used a basic mittens (without retrofitting existing embeddings), the results come out quite poor, even when the same hyperparameters are used.
glove_mittens_50_50 = GloVe(n=50, max_iter=50, learning_rate=0.03)
cooccurance = np.array(matrix.todense()) # was sparse matrix for original glove
glove_mittens_trained_50_50 = glove_mittens_50_50.fit(cooccurance)
I built a pd dataframe with the resulting numpy matrix and incorporated the domains as the index before writing a function that would calculate the cosine distance in the same way that the original glove model does.
As you can see, the results are not at all as expected. Furthermore, while the original glove model will have converged and not change much (only very slightly) by increasing the number of iterations, the vectorized glove in this package will.
From the ReadMe file it says that we need pre-trained matrix in order to use mittens. This means you should have implied some kind of reweighting scheme to your co-occurrence matrix and then use Mittens.
I am using mittens with a pre-built cooccurrence matrix of domains with the hopes of clustering certain domains that are thematically related, close to each other. Using the non-vectorized glove implementation from https://github.com/stanfordnlp/GloVe, I get very strong results. The current initialization is:
Finding the nearest domains to
nintendo
using cosine distance yields good results.Given these results, I wanted to use
mittens
for two reasons: take advantage of the vectorized implementation for speed, and harness the ability to extend glove into a retrofitted model. However, when I used a basic mittens (without retrofitting existing embeddings), the results come out quite poor, even when the same hyperparameters are used.I built a pd dataframe with the resulting numpy matrix and incorporated the domains as the index before writing a function that would calculate the cosine distance in the same way that the original glove model does.
As you can see, the results are not at all as expected. Furthermore, while the original glove model will have converged and not change much (only very slightly) by increasing the number of iterations, the vectorized glove in this package will.
Is there a reason why this is the case? Am I doing anything wrong, or is there anything else you'd like me to try?
Thanks.
The text was updated successfully, but these errors were encountered: