Archive for category Neural Network

Online learning and Batch learning

On line learning or stochastic learning:

Stochastic learning occurs when the neuron weights are updated after each individual piece of data is passed through the system. The Feed Forward Neural Network therefore changes with every piece of data and is in a constant state of change during training.

Batch learning:

Batch Learning on the other hand stores each neuron weight change when it occurs, and only at the end of each epoch does it update the weights with the net change over the training set. This means the neural network will only update once at the end of each epoch.



, , ,

Leave a comment

Back Propagation Algorithm

The generalized delta rule  also  known  as back  propagation  algorithm is explained here briefly for feed forward Neural Network (NN). The explanation here is intended to give an outline of  the process involved in back propagation algorithm.

See the figure carefully and try to match with the explanation given below.


The NN explained here contains three  layers.  These are input, hidden, and output Layers.  During the training phase, the training data  is fed into to the input layer. The data is propagated  to the hidden  layer  and then  to the output  layer.  This  is called the  forward  pass of  the  back  propagation algorithm.  In  forward  pass, each node in hidden layer gets input from all the nodes from input layer, which are multiplied with appropriate weights  and then summed. The output of  the hidden  node  is  the non-linear  transformation of  the this  resulting sum.  Similarly each node in output  layer gets input from all the nodes from hidden layer, which are multiplied with appropriate weights and  then summed. The output of  this node is  the non-linear  transformation of  the resulting sum.

The output values of  the output layer are compared with the target output  values. The  target  output  values are those  that we attempt to teach our  network.  The error between actual output values and  target output values  is calculated and  propagated back  toward  hidden  layer.  This is called  the backward  pass of  the back  propagation algorithm.  The error  is used  to update  the connection strengths  between nodes,  i.e. weight matrices between input-hidden layers and hidden-output  layers are updated.

During  the  testing  phase, no  learning takes place  i.e.,  weight  matrices are  not changed.  Each  test vector is fed  into the input  layer. The feed forward of  the testing data is similar to the feed forward of  the training data.