LSTM node

[msperber]

This attempts to reduce the per-timestep LSTM memory consumption to a minimum, by avoiding creation of intermediate nodes. The memory allocated is now 6*hidden_dim for forward & backward pass respectively: hidden state, cell state, and the result of the 4 matrix multiplies + nonlinearities. In an ideal case we wouldn’t have to save the latter in the backward pass, but due to the complex LSTM dependencies I don’t see a good way to accomplish that without hardcoding the complete loop over the sequence, which would make things very inflexible.



Implemented is the vanilla LSTM, dropout, and weight noise. Dropout and weight noise require no additional memory. Preliminary experiments indicate smaller overall memory footprint at slightly slower speed.



The above is achieved via 3 new dynet nodes:

gates_t = vanilla_lstm_gates(x_t, h_tm1, Wx, Wh, b, dropout_mask_x, dropout_mask_h, weightnoise_std)

c_t = vanilla_lstm_c(c_tm1, gates_t)

h_t = vanilla_lstm_h(c_t, gates_t)

[jcyk]

Great! Excited about it!
@msperber What is the memory allocation in current version?

[msperber]

If I counted correctly, it should currently be at least 18hidden_dim, and 20hidden_dim if using dropout.

[pmichel31415]

Very minor comment: I think you can do .sigmoid() here

[pmichel31415]

Thanks matthias!

I have a quick question: any guess as to why is it slower? is this fixable?

[msperber]

Hmm, I don't see a particular reason why it should be slower, so yeah I assume that should be just a matter of doing some more profiling / code-level optimization. If anything we could probably make it slightly faster because we have more opportunity for batchings things together, such as the 3 sigmoids.

[msperber]

Actually, now that you made me think about it, Paul: A likely reason is that the i,f,o,g parts are no longer separated in memory, and so the pointwise operations are performed by striding over the memory. If that slows things down, it should be easy to fix by copying them to separate regions of scratch memory (same as the currently implemented LSTM is doing, except we release the memory afterwards).
In any case, I guess we can save this for a future commit as I won't have time to look into this for the next couple of weeks, and speed differences are not super big anyways.

[neubig]

Thanks, this is great! I'm going to merge this for now, and we can take a look at the remaining speed issues in the future.