Welcome to Fun Activations! This repository is dedicated to exploring various activation functions used in neural networks. Our goal is to understand when and how different activation functions are helpful in different scenarios.
Activation functions are a crucial component of neural networks, influencing how the network learns and performs. In this repository, we will dive into a variety of activation functions, analyze their properties, and compare their performance in different tasks.
We will cover a wide range of activation functions, including but not limited to:
- Sigmoid
- Tanh
- ReLU (Rectified Linear Unit)
- Leaky ReLU
(ongoing additions)
- PReLU (Parametric ReLU)
- ELU (Exponential Linear Unit)
- SELU (Scaled Exponential Linear Unit)
- Swish
- Mish
- Softmax
Each activation function will have its own detailed explanation, implementation, and use cases.
You can use this repository to:
- Learn about different activation functions and their properties.
- Experiment with various activation functions in your own projects.
- Compare the performance of different activation functions in different scenarios.
To get started, clone this repository to your local machine:
git clone https://github.com/PujanMotiwala/fun_activations.git
cd fun_activations
Install the required dependencies:
pip install -r requirements.txt
We provide several example scripts demonstrating the use of different activation functions. You can find them in the examples directory.
Here’s how you can run an example:
python examples/example_script.py
Each example script includes a detailed explanation and results analysis.
Check out our comparative analysis notebook to see how different activation functions perform on the same dataset. This notebook includes visualizations and performance metrics to help you understand the strengths and weaknesses of each activation function.
We welcome contributions from the community! If you have an idea for a new activation function or an improvement to an existing one, feel free to open an issue or submit a pull request.
1. Fork the repository.
2. Create a new branch (git checkout -b feature-branch).
3. Make your changes.
4. Commit your changes (git commit -am 'Add new feature').
5. Push to the branch (git push origin feature-branch).
6. Create a new Pull Request.
This project is licensed under the MIT License. See the LICENSE file for more details.