[Neural Network and Deep Learning] Building blocks of deep neural networks

This note is based on Coursera course by Andrew ng.

(It is just study note for me. It could be copied or awkward sometimes for sentence anything, because i am not native. But, i want to learn Deep Learning on English. So, everything will be bettter and better :))

 

 

INTRO

We have already seen the basic building blocks of forward propagation and back propagtaion, the key components we need to implement a deep neural network. Let's see how we can put these components together to build our deep net.

 

 

MAIN

Here is a network of a few layers. Let's pick one layer and look into the computations focusing on just that layer for now. So for layer l, we have some parameters w[l] and b[l], and for the forward prop, we will input the activations a[l-1] from previous layer and output a[l].

So the way we did this previously was we compute z[l] = w[l] * a[l-1] + b[l]. And then a[l] = g[l] ( z[l] ). This is how we go from the input a[l-1] to the output a[l]. And, it turns out that for later use it will be useful to also cache the value z[l]. Storing the value z[l] would be useful for backward prop.

 

And then for the backward prop, again focusing on computation for this layer l. We are going to implement a function that inputs da[l] and outputs da[l-1]. For more details, the inputs actually da[l] as well as the cache z[l], then in addition, outputing da[l-1] we bring output or the gradient we want in order to implement gradient descent for learning. 

 

This is the back structure of how we implement this forward step as well as backward step.

 

To summarize, in layer l, we are going to have the forward step and backward step. If we can implement these two functions, then the basic computation of the neural network will be as follows. So, this is one iteration of gradient descent for our neural network. 

 

 

We have seen the basic blocks of implementing a deep neural network. A forward propagation step for each layer, and a corresponding backward propagation step. For now, let's see how we can actually implement these steps.