Get Contributions Scores for Middle Layer

[EoinKenny]

Is it possible to use DeepExplain to get contribution scores for a layer in the middle of the network?

I presume you have to change this line

target_tensor = fModel(input_tensor)
But if I replace input_tensor with a conv layer like model.layers[-7].input (for example) I get the error message.

ValueError: number of input channels does not match corresponding dimension of filter

Is it possible to do this?

[marcoancona]

Yes, it possible to use an intermediate hidden layer as input to generate attributions (indeed, this is necessary for layers that stop gradient propagation, ie. embedding lookups)
You need to make sure the sample parameter when you call explain(method_name, target_tensor, input_tensor, samples, ...args) is the numeric input at that hidden layer (one that can be used to feed input_tensor).

For example, when you have an embedding lookup you need to run first your model up to the interest hidden layer, take the numerical result at that point, and use it when calling explain.

If that is not clear enough, it would be helpful to see a snippet of your code to check where the problem is.

[EoinKenny]

Thanks for the reply!

So this is the normal code which works fine

with DeepExplain(session=K.get_session()) as de:  # <-- init DeepExplain context
    input_tensor = final_model.input
    fModel = Model(inputs=input_tensor, outputs=final_model.layers[-2].output)
    target_tensor = fModel(input_tensor)

    xs = image_matrix
    ys = np.zeros((1000))
    ys[900] = 1

    x = de.explain('elrp', target_tensor * ys, input_tensor, xs)

And this is how I modify it to get the contributions at the conv layer at layer [-7]

with DeepExplain(session=K.get_session()) as de:  # <-- init DeepExplain context
    input_tensor = final_model.layers[-7].input
    fModel = Model(inputs=input_tensor, outputs=final_model.layers[-2].output)
    target_tensor = fModel(input_tensor)

    xs = conv_act
    ys = np.zeros((1000))
    ys[900] = 1

    x = de.explain('elrp', target_tensor * ys, input_tensor, xs)

The error message is

ValueError: Graph disconnected: cannot obtain value for tensor Tensor("input_1:0", shape=(?, 224, 224, 3), dtype=float32) at layer "input_1". The following previous layers were accessed without issue: []

For some background here, I'm using the pre-trained VGG16 model. I modified the last two layers to be a separate dense and activation layer (softmax) with the usual 1000 classes.

The variable conv_act is a matrix of shape (1, 14, 14, 512) - Which does match the conv layer input.

[marcoancona]

I don't see an obvious problem with your code. Seems you have some disconnected operations on the graph but that should not happen with within that snipped of code. Could you provide a minimal working example that I can run? Also, is the problem still there if you use grad*input as method?

[EoinKenny]

Thanks a lot for the help again. Here's a fully coded example with the error. The same error occurs with grad*input also. Is this because you need to separate out the activation part of all layers in the network maybe?

import keras.backend as K
import numpy as np

from deepexplain.tensorflow import DeepExplain
from keras.layers import Dense, Activation
from keras.applications.vgg16 import VGG16
from keras.models import Model, Sequential


# Load VGG16 model and change last layer to be split with dense and Activation
vgg = VGG16(include_top=True, weights='imagenet')
vgg.save_weights('imagenet.h5')
x = Dense(1000)(vgg.layers[-2].output)
x = Activation('softmax')(x)
vgg_new = Model(vgg.input, x)
vgg_new.load_weights('imagenet.h5')

# Just use an array of 0's for this example
image_instance = np.zeros((1,224,224,3))

# Get activations of layer [-7]
conv_act_model = Model(inputs=vgg_new.input, outputs=vgg_new.layers[-7].output)
conv_act = conv_act_model.predict(image_instance)

# Get contributions of convolutional layer [-7]
with DeepExplain(session=K.get_session()) as de:  # <-- init DeepExplain context
    input_tensor = vgg_new.layers[-7].input
    fModel = Model(inputs=input_tensor, outputs = vgg_new.layers[-2].output)
    target_tensor = fModel(input_tensor)

    xs = conv_act
    ys = np.zeros((1000))
    ys[900] = 1

    answer = de.explain('elrp', target_tensor * ys, input_tensor, xs)

[EoinKenny]

I'm just going to bump this in case! Thanks again for the help.