AI - Lecture 13

Fast Recap:

• Gradient Descent
• Delta Rule

---

Recap:

MLP (Multilayer Perceptron) Learning Algorithm: Backpropagation:

1. Batch Criterion Function:

$$C(\tau ,w)=C(W)=\frac{1}{2}\sum _{j=1}^n\sum _{i=1}^m(\widehat{y_i}-y_i{)}^2$$

Where:

• $\tau =\{(\underline{x_1},\underline{y_1}),\dots ,(\underline{x_1},\underline{y_1})\}$ : is the training set (supervised)
• $w$ : weight
• $\widehat{y_i}$ : predicted output given by the ANN.
  

2. Online Criterion Function:

$$C(W)=\frac{1}{2}\sum _{j=1}^m(\widehat{y_i}-y_i{)}^2$$

The difference with the Batch Criterion is that in the online mode only one output is considered to calculate the Cost $C(W)$, while in the batch for each cost we consider $m$ outputs (a batch).

Gradient Descent: For a generic weight $w$ we apply the Gradient Descent:

$$\begin{array}{rl}& w^{\prime }=w+\Delta w\\ & \text{where:}\Delta w=-\eta \frac{\partial C}{\partial w}\end{array}$$

~Ex.: Online Mode, 1-Layer ANN 1-Layer ANN: 1 Input Layer, 1 Output Layer, we only have one set of weight.

• $C(W)=\frac{1}{2}{\sum }_{j=1}^m(\widehat{y_i}-y_i{)}^2$
• $y_i=f_i(a_i)$ $f_i$ : $i$-esim activation function
• $a_i={\sum }_k{w}_{ik}{o}_i$ $o_i$ : old layer, in this case the $i$-esim input.

After some calculation the result is:

$$\frac{\partial C}{\partial w}=\frac{\partial f(a_i)}{\partial a_i}{o}_i$$

~Ex.: Online Mode, 2-Layer ANN Having an Hidden Layer adds to the computation $o_i$ in the last equation is not the input layer but in this case it’s the output of the hidden layer, so it depends on some weights and on an activation function. The $\Delta w$ of the output layer is:

$$\Delta w=-\eta \frac{\partial C}{\partial w}=\eta \left(\sum _{i=1}^m{w}_{ij}{\delta }_i\right)\cdot f_j^{\prime }(a_j)\cdot o_k$$

Where:

• $f_j’(a_j)=\frac{\partial }{\partial a_j}{f}_j(a_j)$
  
• ${\delta }_j=\{\begin{array}{lc}(\widehat{y_j}-y_j)f_j^{\prime }(a_j)& \text{if}j\in L_l\\ \left({\sum }_{i\in L_{k+1}}{w}_{ij}{\delta }_i\right)\cdot f_j^{\prime }(a_j)& \text{if}j\in L_k\text{where:}k=l-1,\dots ,0\end{array}$
  
• This is known as the Generalized Delta Rule: $\Delta w_{jk}=\eta {\delta }_j{o}_k$
• Learning is iterated for a defined number of consecutive cycles called epochs, using only the training set.
• An epoch is a cycle of application of the delta-rule over all the data in $\tau$.

---

Original Files: