Visualising the training of an Artifical Neural Network (ANN) on a microcontroller using an LED matrix.
Neural network:
- Implemented from scratch.
- Multi-layer perceptron with 2 inputs, 2 hidden layers and 1 output in a 2-3-3-1 arrangement.
- Mini-batch gradient descent backpropagation.
- Sigmoid activation function.
- Learning rate scheduling.
- Random weight initialisation.
- A sample regression problem for training the model was generated using sklearn.datasets.make_regression().
Visualisation details:
- forwards pass:
- Each forward pulse represents the path taken by a single instance of the dataset passing through the network.
- The brightness of each node corresponds to the output value of that node.
- The brightness of each connection between nodes corresponds to the output from the previous node multiplied by its associated input weight.
- Brightnesses of nodes and connections scaled with each forward pass using min-max normalization to provide a better visualisation of the differences between different nodes and connections.
- 32 forward passes occur (the batch size) for each backwards pass.
- backwards pass:
- each backward pulse represents the updating of input weights using the average of the gradients collected over a mini-batch of instances.
- The brightness of each node corresponds to the sum of gradients propagating backwards through that node.
- The brightness of each connection corresponds to the value of the gradient that will be used to update the weight.
- min-max normalization is used for both nodes and connections to provide a better visualisation of differences.
- The toggleable value in the lower right of the display represents lowest loss achieved over a mini-batch during training.
- The toggleable coloured connections represents positive weights with red and negative weights with blue.
Hardware components:
Key resources: