Activation Function: Absolute Function,One Function Behaves more Individualized

Inspired by natural world mode, a activation function is proposed. It is absolute function. According to probability principle, nature word is normal distribution. Stimulation which happens frequently is low value, it is shown around zero in figure 1. Stimulation which happens accidentally is high value, it is shown far away from zero in figure 1. So the high value is the big stimulation, which is individualization. Through test on mnist dataset and fully-connected neural network and convolutional neural network, some conclusions are put forward. The line of accuracy of absolute function is a little shaken that is different from the line of accuracy of relu and leaky relu. The absolute function can keep the negative parts as equal as the positive parts, so the individualization is more active than relu and leaky relu function. In order to generalization, the individualization is the reason of shake, the accuracy may be good in some set and may be worse in some set. The absolute function is less likely to be over-fitting. The batch size is small, the individualization is clear, vice versa. If you want to change the individualization of absolute function, just change the batch size. Through one more test on mnist and autoencoder, It is that the leaky relu function can do classification task well, while the absolute function can do generation task well. Because the classification task needs more universality and generation task need more individualization.