[Neural Network and Deep Learning] Derivatives of activation functions

This note is based on Coursera course by Andrew ng.

(It is just study note for me. It could be copied or awkward sometimes for sentence anything, because i am not native. But, i want to learn Deep Learning on English. So, everything will be bettter and better :))

 

 

 

 

 

When we implement back propagation for our neural network, we need to compute the derivative of the activation functions. So, let's take a look at our choices of activation functions and how we can compute the slope of these functions.

 

 

 

Here is a familiar sigmoid activation function. We know from calculus that it is the slope of g of x at z. And it we take the derivative of the sigmoid function, it is equal to that formula. The advantage of this formula, a(a-1), is that if we have already computed the value of a, then by using this expression, we can very quickly compute the value of the slope for g' as well. 

 

 

 

 

Let's now look at the Tanh, ReLU and leaky ReLU activation functions.

Tanh activation function

 

ReLU activation function

 

Leaky ReLU activation function