How do you create a custom activation function with Keras?

Sometimes the default standard activations like ReLU, tanh, softmax, ... and the advanced activations like LeakyReLU aren't enough. And it might also not be in keras-contrib.

How do you create your own activation function?

Solution 1:

Credits to this Github issue comment by Ritchie Ng.

# Creating a model
from keras.models import Sequential
from keras.layers import Dense

# Custom activation function
from keras.layers import Activation
from keras import backend as K
from keras.utils.generic_utils import get_custom_objects


def custom_activation(x):
    return (K.sigmoid(x) * 5) - 1

get_custom_objects().update({'custom_activation': Activation(custom_activation)})

# Usage
model = Sequential()
model.add(Dense(32, input_dim=784))
model.add(Activation(custom_activation, name='SpecialActivation'))
print(model.summary())

Please keep in mind that you have to import this function when you save and restore the model. See the note of keras-contrib.

Solution 2:

Slightly simpler than Martin Thoma's answer: you can just create a custom element-wise back-end function and use it as a parameter. You still need to import this function before loading your model.

from keras import backend as K

def custom_activation(x):
    return (K.sigmoid(x) * 5) - 1

model.add(Dense(32 , activation=custom_activation))

Solution 3:

Let's say you would like to add swish or gelu to keras, the previous methods are nice inline insertions. But you could also insert them in the set of keras activation functions, so that you call you custom fucntion as you would call ReLU. I tested this with keras 2.2.2 (any v2 would do). Append to this file $HOME/anaconda2/lib/python2.7/site-packages/keras/activations.py the definition of your custom function (can be different for you python and anaconda version).

In keras internal:

$HOME/anaconda2/lib/python2.7/site-packages/keras/activations.py

def swish(x):
    return (K.sigmoid(beta * x) * alpha *x)

Then in your python file:

$HOME/Documents/neural_nets.py

model = Sequential()
model.add(Activation('swish'))

Solution 4:

You can use the lambda keyword or a Lambda layer. Let's say your neural network without activation gives a bunch of 5:

import tensorflow as tf
import numpy as np

x = np.ones((5, 5))

model = tf.keras.Sequential([
    tf.keras.layers.Dense(1, kernel_initializer=tf.initializers.Ones)
])

model.build(input_shape=x.shape)

model(x)

<tf.Tensor: shape=(5, 1), dtype=float32, numpy=
array([[5.],
       [5.],
       [5.],
       [5.],
       [5.]], dtype=float32)>

And you want the activation function to divide by 5. You can add a Lambda layer:

model = tf.keras.Sequential([
    tf.keras.layers.Dense(1, kernel_initializer=tf.initializers.Ones),
    tf.keras.layers.Lambda(lambda x: x/5)
])

<tf.Tensor: shape=(5, 1), dtype=float32, numpy=
array([[1.],
       [1.],
       [1.],
       [1.],
       [1.]], dtype=float32)>

Or use the lambda keyword in the activation argument:

model = tf.keras.Sequential([
    tf.keras.layers.Dense(1, 
                          kernel_initializer=tf.initializers.Ones, 
                          activation=lambda x: x/5)
])

<tf.Tensor: shape=(5, 1), dtype=float32, numpy=
array([[1.],
       [1.],
       [1.],
       [1.],
       [1.]], dtype=float32)>