A collection of activation function approximations for Flux.
In some models the accuracy of the sigmoid and tanh functions can be reduced without a loss of accuracy in the training process. Switching to an approximation can reduce training times significantly for some models.
For sigmoid we provide fitted approximations using Taylor and Pade curve fit models as well as an implementation which uses a fast exp imeplemention based on an approximation of the formula: exp(x) = limn->inf (1 + x/n)
| Fitted Functions | Fast Expr |
|---|---|
 |
 |
 |
There is also an implementation of TheanoFastSigmoid which is currently accepted in the Theano project. It is here mostly for comparison because it is both slower and less accurate than other Sigmoid approximations.
For tanh we provide fitted approximations using Taylor and Pade curve fit models as well as an implementation based on the continuous fraction approximation of tanh.
Additionally we also provide the serpentine function.
| Fitted Functions | Continuous Fraction | Serpentine |
|---|---|---|
 |
 |
 |
 |
 |