Replacement for ConvRNN API (implementing custom ConvGRU2D Cell)

[dryglicki]

Environment:
Miniforge conda installation; pip installation via instructions on main Tensorflow page with additional workarounds described here

Python version: 3.11.8

New versions:
Tensorflow: 2.16.1
Keras: 3.1.1

Old versions:
Tensorflow: 2.14.0
Keras: 2.14.1

Situation:
I wrote a custom ConvGRU2DCell. It worked in my generator this way:

from keras.src.layers.rnn.base_conv_rnn import ConvRNN
...
        with options({"layout_optimizer" : False}):
            z = ConvRNN(rank=2, cell = ConvGRU2DCell(filters=ofilt, kernel_size=(3,3)), return_sequences = True)(z, initial_state = [skip])

Setting aside the "layout_optimizer" hack to get running, this worked. I couldn't get the regular RNN API to accept this properly at the time. Indeed, in the Keras RNN API, it looks like the RNN API is expecting a Tensor of shape [B, T, C]. Now, in the new version of Keras and TF, this ConvRNN wrapper is gone. Looking into ConvLSTM2D class, there is a ConvLSTM class that it inherits from that permits the "rank" argument.

Do I have any recourse here aside from coding it all from scratch or should I stick with the older versions of TF and Keras? Will this API return? Are we not meant to implement our own recurrent convolutional cells anymore? Did I screw something up when I first tried the RNN API?

[SuryanarayanaY]

Hi @dryglicki ,

To better understand your enquiry can I have more context and details which worked for Keras2 and which is not working now with Keras3 with a minimal code snippet. Thanks!

[dryglicki]

Hi Surya. Okay, I can provide a simpler example. The ConvGRU2DCell is a bit complicated, so I'll work on getting a simpler example that has all of its same parts but the call logic is reduced. Standby.

[dryglicki]

@SuryanarayanaY All right, I created a gist here: https://gist.github.com/dryglicki/5f198807d5cd14f9cdd8b52d90dac211

If I try running that, I get the following error:

Traceback (most recent call last):
  File "/home/dryglicki/code/RNN_example/example_old.py", line 135, in <module>
    x = KL.TimeDistributed(KL.Conv2D(1,kernel_size=3))(x)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data3/miniforge3_2024-01/envs/tensorflow_2d16/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 122, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/data3/miniforge3_2024-01/envs/tensorflow_2d16/lib/python3.11/site-packages/keras/src/ops/operation_utils.py", line 184, in compute_conv_output_shape
    raise ValueError(
ValueError: Kernel shape must have the same length as input, but received kernel of shape (3, 3, 32, 1) and input of shape (None, 32).

That GRU Cell is a minor modification of what I'm using in my development -- I took out Spectral Normalizations to make it easier to work with.

Anyway, I've been spending a lot of time looking at the ConvLSTMCell implementation here: https://github.com/keras-team/keras/blob/master/keras/layers/rnn/conv_lstm.py

It does look similar to what was in the older versions of Keras. I find this message to be alarming, however:

         if None in self.spatial_dims:
            raise ValueError(
                "ConvLSTM layers only support static "
                "input shapes for the spatial dimension. "
                f"Received invalid input shape: input_shape={inputs_shape}"

So does that mean ALL custom RNN-CNN cells have to follow this rule, or is it specific to this particular implementation of ConvLSTM? That would be a showstopper for me if that's the case where I can't set my inputs to include [None, None] in the [H, W] dimensions.

[dryglicki]

...small update in gist to change tensorflow.keras to regular keras. Same error.

[fchollet]

There are no plans to bring back this API at this time. You can either use Keras 2 (which is available as the tf_keras package, regularly updated) or you can write your own layer in the style of ConvLSTM2D, which permit higher input ranks.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No

[dryglicki]

@fchollet All right, well that's not the best news. Thanks for your attention. I have a new surprise in a new ticket!