The Neurone

The Propagation Function

Since neural networks can merely be seen a function networks, neurone types are classified by the type of their propagation function. The propagation function $f_k$ of a neurone $N_k$ is normally of the form

$$f_k(x)=g(h_k(x)), \qquad f_k: {\bf S}^n \rightarrow {\bf S}... ... g: {\bf R} \rightarrow {\bf S}, \qquad n=\vert{\bf I}_k\vert$$

The function $h_k$ is called the netto input function of the neurone $N_k$, and maps the input states onto a single real value. The function $g$ is called activation function, is usually the same for all neurones and maps the real netto input back onto ${\bf S}$.

The Nettoinput Functions

Since most learn algorithm train the network by iteratively changing the netto input functions of the neurones, they can be written as

$$h_k=h(P_k) \quad \mbox{and} \quad h_k(I_k)=h(P_k;I_k)$$

where $P_k$ is a vector of function parameters, which is to be determined by the learn algorithm. Normally, each input node $X_i^{(k)}$ of the neurone $N_k$ is associated with a parameter $w_i^{(k)}$ called its weight. Often, there is an extra parameter, called threshold or bias.

A very common definition of the netto input, is a weighted sum (a dot product) to which the bias is added.

$$h(I) = h(P,I) = \theta + \sum_{i=1}^n x_i w_i \quad \mbox{wit... ...I=(x_1,x_2, \ldots x_n), \quad P=(w_i,w_2, \ldots w_n; \theta)$$

A convenient way to store the weights for $n$-layer networks is by replacing the adjacence matrix $M_k$ by the weight matrix $W_k$, which contains the weights the input nodes or $0$ if there is no connection.

The Activation Function

The activation function $g$ is normally a monotone, nonlinear function to rescale the netto input to ${\bf S}$, which is usually a limited interval. If $\sup {\bf S} = 1$ and $\inf {\bf S} = 0$, then $g$ uses binary logic, if $\sup {\bf S} = 1$ and $\inf {\bf S} = -1$, then $g$ uses bipolar logic.

The two most commonly used activation functions for binary logic are the step- or $\Theta$-function and the sigmoide function $\sigma_c$.

$$\Theta(x) = \left\{ \begin{array}{cl} 0, & x\leq 0 \\ 1, ... ...ray} \right. , \qquad \sigma_c(x) = {1 \over 1+{\rm e}^{-cx}}$$

Common Neurone Types

In the following examples, ${\bf B}$ stands for the set $\{0,1\}$, ${\bf Z}$ is the set of integer numbers and ${\bf R}$ the set of real numbers. All neurones are assumed to have $n$ inputs. $I\cdot W$ stands for $\sum_{i=1}^n x_i w_i$.

McCulloch-Pitts-Cells

$${\bf S}={\bf B}, \quad P=(W; \theta), \quad W \in {\bf B}^n$$

$$h(P;I)= \left\{ \begin{array}{cl} I \cdot W-\theta, & (\for... ..._i=1 \wedge w_i=0 \end{array} \right. , \quad g(x)=\Theta(x)$$

Perceptron

$${\bf S}={\bf R}, \quad P=(W; \theta), \quad W \in {\bf R}^n \quad h(P;I)= I \cdot W + \theta, \quad g(x)=\Theta(x)$$

Standard Backpropagation Neurone

This type of neurone is very often used with the backpropagation algorithm and also the main type for this project. No explicit $\theta$ is defined, however, the effect of the bias can be achieved by simply adding an extra neurone to each layer with a constant propagation function of $0$.

$${\bf S}={\bf R}, \quad P=W, \quad W \in {\bf R}^n \quad h(P;I)= I \cdot W, \quad g(x)=\sigma_c(x)$$