Feedforward networks

The principal class of ANNs that we discuss are so-called feedforward networks. Of these, perhaps the most widely used is the multi-level perceptron (MLP) model.

Feedforward networks can give reasonable answers when presented with inputs that they have never seen, if properly trained.

A typical feedforward ANN consists of three or more inter-connected layers of nodes ― an input layer, one or more hidden intermediate layers (often just 1), and an output layer (Figure 8‑15). The arrows indicate the direction of information flow, feeding information forward from input to output. Note that there may be any number of nodes at each level of the network, and not all nodes need to be connected to every node in the next layer. For example, in Figure 8‑15 hidden node H2 only receives input from input nodes I1 and I3, and only provides output to node O1. This arrangement can be seen as a directed graph, or as a rather complex-looking function mapping.

Figure 8‑15 Simple 3-5-2 feedforward artificial neural network

The connections between the input layer and hidden layer can be described using a weight matrix, W, where the row/column entries wij are positive or negative real-valued weights, or 0 if no connection exists. Likewise, the connections between the hidden layer and the output layer can also be viewed as a weight matrix, Z say, again consisting of a set of weights, zjk. Positive weights in each case imply a reinforcement process associated with the source node or input, whilst negative weights correspond to inhibition. 

https://www.spatialanalysisonline.com/HTML/index.html?introduction_to_artificial_neu.htm


 

ANNs: Terminology

Inputs

Weights

Transfer functions

Epoch: Every loop is known as a epoch

Multi-layer perceptron (MLP)

In MLP each node is connected to every node in the next layer.Nodes in the hidden and output layers are connected to a bias node (feeding a constant value). Input data are weighted, then added up and finally transformed using an activation function. 

 Figure 8‑16 MLP 3-5-2 with bias nodes

When each node in the network is connected to every node in the next layer by a feedforward link, it is commonly referred to as a multi-level perceptron (MLP, Figure 8‑16). The perceptron part of this description is a reference to its early development in a simpler form as a model of neuron triggering in the retina of an eye.

By convention all nodes in the hidden layer and the output layer are also connected to a bias node. This feeds a constant input value, 1, into the set of weights, and acts in a similar manner to the constant term in a regression model. Bias node weights are treated in the same manner as per other node weights. In the MLP, with 1 bias node associated with each feedforward layer, n input nodes, one hidden layer with m hidden nodes and p output nodes, the W and Z weight matrices have dimensions (n+1)x(m) and (m+1)x(p). The MLP then has the architecture shown in Figure 8‑16. The effective number of parameters, λ, for such a network is the sum of the number of feedforward connections (including the bias connections), so in this example λ=20+12=32.

Input data are weighted according to their wij values and combined (typically summed) in the hidden layer. This weighted sum is then modified by what is known as an activation function. This two-step process of summation of inputs and then modification of this sum by an activation function, g, to create the output value can be illustrated at the node level as shown in Figure 8‑17.

Figure 8‑17 ANN hidden node structure

ANNs: Training/ Learning

Backpropagation training algorithm:

Forward phase – the inputs are presented to the network

Backpropagation phase – the outputs are compared with the targets and the weights are adjusted