Backpropagation for a 2-Layer Network

As shown in Section 3.2.3, for standard backpropagation neurones (Section 2.4.4) the parameters of a $n$-$m$-$o$-network can be described by two real weight matrices $W^{(1)}$ and $W^{(2)}$ of the dimensions $(n+1) \times m$ and $(m+1) \times o$. Let the vectors $I$, $H$ and $O$ refer to the states of the input, hidden and output neurones and $I^{(p)}$ and $O^{(p)}$ to the actual training pattern. The weight updates $\Delta W^{(1)}$ and $\Delta W^{(2)}$ without impulse can then be calculated as follows.

$$\Delta w^{(1)}_{ij}=-\gamma\,I_i\delta^{(1)}_{j}, \quad \Delta w^{(2)}_{jk}=-\gamma\,H_j\delta^{(2)}_{k}$$

$$\delta^{(1)}_{j}=H_j(1-H_j)\, \sum_{k=1}^o w^{(2)}_{jk} \delta^{(2)}_{k}, \quad \delta^{(2)}_{k}=O_k(1-O_k)\, (O_k-O_k^{(p)})$$