How to hidden layers work?

[pwang1092]

In the binary classification model, 3 linear layers were used along with a ReLU, with the forward method defined as:

def forward(self, x): return self.layer_3(self.relu(self.layer_2(self.relu(self.layer_1(x)))))

How does this work? Would we have 3 lines (linear layers), each with a bend in them (from the ReLUs)?

[LuluW8071]

Yes we would have 3 linear layers,

• 1st being input layer with ReLU
• 2nd as hidden layer with ReLU
• 3rd as output layer

ReLU works as activation function to introduce non-linearity for the layer; if total weight of a neuron is greater than 0 it fires(1) else doesnt fire(0)

[Luismbpr]

self.layer_3(self.relu(self.layer_2(self.relu(self.layer_1(x)))))
The code above is the same as having the layer_1(x), then passing that into layer_2 and then into layer_3 but on a single step which is more efficient.

[LuluW8071]

• Single Liner is more efficient
• Breaking down into layers and passing them gives readability and helps in debugging(shape mismatch error)

[mrdbourke]

All are correct.

More lines = more readability, however, wrapping them all together means that no extra memory is being assigned to a variable as the operations take place (e.g. each operation happens sequentially as it passing through the chain).