Graphing the error landscape for the 1-1-1 network

The error landscape for the 1-1-1 network shows how the total error (or TSS) changes as the values of the weights in the network change. To graph the error equation for a network, you need several things:

1. Describe the parts of the network:
   1. The 1-1-1 network has one input, one hidden and one output unit.
   2. Let the units in the network be a1, a2, and a3.
   3. Let the weights be w1 and w2.
   4. Let the biases be b1 and b2. In this example, the biases are zero, so b1=b2=0
2. The independent variables (x and y axes) that will be graphed are the weights in the network (w1 and w2), and the dependent variable is the TSS (or total sum of square error) value for each set of weights. The TSS is calculated from:

   TSS = Σ (target_p - output_p)²

3. In the 1-1-1 network there are just two patterns to consider (p0 = 0, p1=1), and the target is the same as the input. So the TSS can be calculated as:
   

   TSS = (target_p0 - output_p0)² + (target_p1 - output_p1)²

   = (0 - output_p0)² + (1 - output_p1)²

   = (output_p0)² + (1 - output_p1)²

4. The outputs for each pattern are given by the activations of the output unit, a3.
   The equations for the hidden and output units are given by:

   a2 = f(a1*w1)

   a3 = f(a2*w2)

   where f(x) is the sigmoid equation given by:

   f(x) = 1/(1+e^-x)

5. Substituting in for each pattern gives the following:
   For pattern p0:

   a2 = f(p0*w1) = f(0*w1) = f(0) =1/(1+e⁰) = 0.5

   a3 = f(0.5*w2) = 1/(1+e^-0.5*w2)

   For pattern p1

   a2 = f(p1*w1) = f(1*w1) = f(w1) =1/(1+e^-w1)

   a3 = f(a2*w2) = 1/(1+e^{-(1/(1+e-w1))*w2})

6. Substituting back into the TSS equation gives the final equation to graph:

   TSS == output_p0² + (1 - output_p1)²

   = (1/(1+e^-0.5*w2))² + (1 - 1/(1+e^{-(1/(1+e-w1))*w2}))²

   = (1/(1+e^-w2/2))² + (1 - 1/(1+e^-w2/(1+e-w1)))²

7. When graphed, the axes are usually labelled x, y and z, so the final equation is:

   (1/(1+e^-y/2))² + (1 - 1/(1+e^-y/(1+e-x)))²

   Transforming this equation into the required form for the GA tutorial, yields:

   (1/(1+e^(-y/2)))^2 + (1-1/(1+e^(-y/(1+e^(-x)))))^2

   Or for use in gnuplot:

   (1/(1+exp(-y/2)))**2 + (1-1/(1+exp(-y/(1+exp(-x)))))**2