Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ACON activation function #2893

Merged
merged 2 commits into from
Apr 22, 2021
Merged

ACON activation function #2893

merged 2 commits into from
Apr 22, 2021

Commits on Apr 22, 2021

  1. ACON Activation Function 

    ## 🚀 Feature

There is a new activation function [ACON (CVPR 2021)](https://arxiv.org/pdf/2009.04759.pdf) that unifies ReLU and Swish. 
ACON is simple but very effective, code is here: https://github.com/nmaac/acon/blob/main/acon.py#L19

![image](https://user-images.githubusercontent.com/5032208/115676962-a38dfe80-a382-11eb-9883-61fa3216e3e6.png)

The improvements are very significant:
![image](https://user-images.githubusercontent.com/5032208/115680180-eac9be80-a385-11eb-9c7a-8643db552c69.png)

## Alternatives

It also has an enhanced version meta-ACON that uses a small network to learn beta explicitly, which may influence the speed a bit.

## Additional context

[Code](https://github.com/nmaac/acon) and [paper](https://arxiv.org/pdf/2009.04759.pdf).
    @glenn-jocher
    glenn-jocher committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    4353c8c View commit details
    Browse the repository at this point in the history

  2. Update activations.py

    @glenn-jocher
    glenn-jocher committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    a9ea248 View commit details
    Browse the repository at this point in the history