An unofficial pytorch implementation of funnel activation https://arxiv.org/pdf/2007.11824.pdf. Official implementation can be found here.

pip install -r requirements.txt

• Simply replace nn.ReLU with FReLU(num_channels), details can be found here.

from frelu import FReLU

conv = nn.Conv2d(in_channels, out_channels, 3, padding=1)
bn = nn.BatchNorm2d(out_channels)
frelu = FReLU(out_channels) # ⬅️

• ImageNet training & performance checking

• official implementation
• https://github.com/pytorch/examples/blob/master/imagenet/main.py
• https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py