Enter your search above to see Results

Whitepapers

Neural Networks
Part 3: Back Propagation

Introduction

This is the third of a short series of posts to help the reader to understand what we mean by neural networks and how they work.  Our first post explained what we mean by a neuron and introduced the mathematics of how to calculate the numbers associated with it.

In our second post we gave you some simple code in R that illustrated the topics from the first post.

This post will explain the back propagation algorithm by deriving it mathematically.

This series of posts is accompanied by some example R scripts for you to run. You can download them from The Data Scientists GitHub page.

1 Features of a Neural Network

So far in this series we have considered single neurons.  We have experimented with having just one input and then two.  We also experimented with having one test case and then four.  So now we understand neurons we now introduce neural networks – networks of neurons. To get us started we will consider an extremely simple network as shown in the diagram below.

*** QuickLaTeX cannot compile formula:
\begin{tikzpicture}

% Comment Accordingly. Additionally. Afterward. Afterwards. Albeit. Also. Although. Altogether. Another. Basically. Because.
% Comment Accordingly. Additionally. Afterward. Afterwards. Albeit. Also. Although. Altogether. Another. Basically. Because.
% Comment Accordingly. Additionally. Afterward. Afterwards. Albeit. Also. Although. Altogether. Another. Basically. Because.
% Comment Accordingly. Additionally. Afterward. Afterwards. Albeit. Also. Although. Altogether. Another. Basically. Because.
% Comment Accordingly. Additionally. Afterward. Afterwards. Albeit. Also. Although. Altogether. Another. Basically. Because.
\foreach \m [count=\y] in {1,2}
\node [every neuron/.try, neuron \m/.try] (input-\m) at (0,5-\y*4) {$a_\m^{L-2}$};
\foreach \m [count=\y] in {1,2}
\node [every neuron/.try, neuron \m/.try ] (hidden-\m) at (4,2-\y*2) {$b_\m^{L-1}$};
\foreach \m [count=\y] in {1}
\node [every neuron/.try, neuron \m/.try ] (output-\m) at (8,2-\y*3) {$b^{L}$};
% Comment Accordingly. Additionally. Afterward. Afterwards. Albeit. Also. Although. Altogether. Another. Basically. Because.
\foreach \l [count=\i] in {1,2}
\draw [<-] (input-\i) -- ++(-4,0) node [midway, fill=white] {$I_{\l}$};
\foreach \l [count=\i] in {1} \draw [->] (output-\i) -- ++(2,0)
node [above, midway] {};
% Comment Accordingly. Additionally. Afterward. Afterwards. Albeit. Also. Although. Altogether. Another. Basically. Because.
\foreach \i in {1,2}
\foreach \j in {1,2}
\draw [->] (input-\i) -- (hidden-\j) node [midway, fill=white] {$w_{\j\i}^{L-1}$};
% Comment Accordingly. Additionally. Afterward. Afterwards. Albeit. Also. Although. Altogether. Another. Basically. Because.
\foreach \i in {1,2}
\foreach \j in {1}
\draw [->] (hidden-\i) -- (output-\j) node [midway, fill=white]{$w_\i^L$};
% Comment
\foreach \l [count=\x from 0] in {L-2, L-1, L}
\node [align=center, above] at (\x*4,2) {Layer \\ $\l$};
\end{tikzpicture}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

2 Explanation of the network

Before we talked about input layers and output layers whereas now we labelled the layers consecutively. This is because the above diagram could, in fact, represent neurons anywhere in a network.  In other words, the left hand nodes are not necessarily inputs and the right hand nodes are not necessarily outputs.  A layer of neurons in the middle of a network is called a hidden layer.  The first two nodes on the left therefore show no weights, biases or activation functions. We just know that for whatever connections and inputs exist to the left, there will be a value

*** QuickLaTeX cannot compile formula:
a^{L-2}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

entering the nodes in layer

*** QuickLaTeX cannot compile formula:
L-2

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

.

The remaining three nodes in layers

*** QuickLaTeX cannot compile formula:
L-1

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

and

*** QuickLaTeX cannot compile formula:
L

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

have weights (shown on the diagram as

*** QuickLaTeX cannot compile formula:
w

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

) they have biases (shown as

*** QuickLaTeX cannot compile formula:
b

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

) and they have activation functions

*** QuickLaTeX cannot compile formula:
a=\sigma(z)

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

which are not shown. Neither do we show the values of

*** QuickLaTeX cannot compile formula:
z

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

and

*** QuickLaTeX cannot compile formula:
a

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

but we understand that every node will have both of these variables.

Note that the indexes shown on the weights have a counter intuitive order. The digits refer firstly to the receiving node and then lastly to the sending node. Thus

*** QuickLaTeX cannot compile formula:
w^{L-1}_{21}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

is the weight that connects node 1 in layer

*** QuickLaTeX cannot compile formula:
L-2

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

to node 2 in layer

*** QuickLaTeX cannot compile formula:
L-1

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

. This may seem to be the wrong way round but all texts on Neural Networks adopt this approach and it is done deliberately. You will see why when we expand our network in the next section.

Expand the math

The example above is also accessible from a mathematical point of view. We can easily expand the output value of

*** QuickLaTeX cannot compile formula:
a^L

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

as follows:

*** QuickLaTeX cannot compile formula:
\begin{align*}
a^L&=\sigma(z^L) \\
&=\sigma(b^L+\sum_{i=1}^2w^L_ia^{L-1}_i) \\
% Comment
&=\sigma(b^L+\sum_{i=1}^2w^L_i\sigma(z^{L-1}_i)) \\
&=\sigma(b^L+\sum_{i=1}^2w^L_i\sigma(b^{L-1}_i+\sum_{j=1}^2w^{L-1}_{ij}a^{L-2}_j)) \\
\end{align*}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

3 Simplifying the math using matrices

The above formula is getting unwieldy and with all the nested sums and subscripts and superscripts this will be confusing!  Imagine the confusion if we had many more layers and inputs and outputs. Let us write the above equation a different way:

*** QuickLaTeX cannot compile formula:
\begin{align*}
a^L&=\sigma(z^L) \\
&=\sigma\Bigg(b^L +
\begin{bmatrix}
w^L_1 & w^L_2
% Comment
\end{bmatrix} \cdot
\begin{bmatrix}
a^{L-1}_1 \\ a^{L-1}_2
\end{bmatrix}
\Bigg)\\
&=\sigma\Bigg(b^L +
\begin{bmatrix}
w^L_1 & w^L_2
\end{bmatrix} \cdot \sigma \Bigg(
\begin{bmatrix}
z^{L-1}_1 \\ z^{L-1}_2
\end{bmatrix} \Bigg)
\Bigg)\\
&=\sigma\Bigg(b^L +
\begin{bmatrix}
w^L_1 & w^L_2
\end{bmatrix} \cdot \sigma \Bigg(
\begin{bmatrix}
b^{L-1}_1 \\ b^{L-1}_2
\end{bmatrix} +
\begin{bmatrix}
w^{L-1}_{11} & w^{L-1}_{12} \\
w^{L-1}_{21} & w^{L-1}_{22}
\end{bmatrix}
\begin{bmatrix}
a^{L-2}_1 \\ a^{L-2}_2
\end{bmatrix}
\Bigg)
\Bigg)\\
\end{align*}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

This is still a bit messy but we can now see how we can describe the network recursively by one very small and simple equation:

*** QuickLaTeX cannot compile formula:
\begin{equation*}
\bold a^{l+1}=\sigma(\bold b^{l+1}+\bold w^{l+1}\cdot\bold a^{l})\text{ where }0\leqslant l \leqslant L
\end{equation*}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

Starting at

*** QuickLaTeX cannot compile formula:
l=0

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

where the matrix

*** QuickLaTeX cannot compile formula:
\bold a^0

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

represents our inputs we can recursively compute

*** QuickLaTeX cannot compile formula:
\bold a^0, \bold a^1, \bold a^2, \hdots, \bold a^L

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

in a forward pass through the network from input to output.  We can also now see why we adopted the counter intuitive numbering. When the weights are placed in a matrix the numbering represents rows and then columns as we would expect.

In our diagram

*** QuickLaTeX cannot compile formula:
a^L

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

is a scalar because we only have one node but we are more likely to have many nodes.  For this reason we tend to always use matrix notation safe in the knowledge that this will also encompass scalars which are just a special case of the set of matrices (a one by one matrix).

We have already introduced the concept of a cost function which compares the output (

*** QuickLaTeX cannot compile formula:
a^L

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

) to a known value (

*** QuickLaTeX cannot compile formula:
y

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

) and quantifies how close the two are. The cost function is dependent on both output and known value, however when training the known value, y, is a constant and so for taking derivatives we can consider the cost to only be a function of

*** QuickLaTeX cannot compile formula:
a^L

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

. Using

*** QuickLaTeX cannot compile formula:
f

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

to denote the cost function the network definition can be completed by adding:

*** QuickLaTeX cannot compile formula:
\begin{equation*}
c=f(\bold a^L,\bold  y)
\end{equation*}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

4 The Back Propagation Algorithm

We now introduce a new measure

*** QuickLaTeX cannot compile formula:
\delta^L

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

to show how the final cost varies when we make changes to the input sum applied to the last neuron in our diagram.

More precisely:

*** QuickLaTeX cannot compile formula:
\[\delta^L=\frac{\partial c}{\partial z^L}\]

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37
*** QuickLaTeX cannot compile formula:
\begin{equation*} 
\delta^L=f'(a^L)\sigma'(z^L)
\end{equation*}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

To see why this new measure is so useful we can use it to tune the output neuron to minimize the cost by adjusting its weights and bias. Previously we calculated the slope of the cost function with respect to weight and bias in order to use gradient decent. If we repeat that exercise we see that both of these slopes are easily obtained from our new measure

*** QuickLaTeX cannot compile formula:
\delta^L

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

.

*** QuickLaTeX cannot compile formula:
\begin{align*}
\frac{\partial c}{\partial b^L} &= \frac{\partial c}{\partial z^L} \times \frac{\partial z^L}{\partial b^L} \\
&=\delta^L \times 1 \\
&=\delta^L
\end{align*}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

Equation 4.2 tells us that the gradient of the bias is equal to our new metric.

*** QuickLaTeX cannot compile formula:
\begin{align*}
\frac{\partial c}{\partial w_j^L} &= \frac{\partial c}{\partial z^L} \times \frac{\partial z^L}{\partial w_j^L} \\
&=\delta^L \times a_j^{L-1} \\
&=\delta^L a_j^{L-1}
\end{align*}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

Equation 4.3 tells us the gradient of the weight is proportional to our new metric and we just need to multiply by

*** QuickLaTeX cannot compile formula:
a_j^{L-1}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

which is a value we calculated during the forward recursive pass.

More Outputs

If we have more than one output neuron then the metric

*** QuickLaTeX cannot compile formula:
\boldsymbol{\delta}^L

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

will no longer be a scalar.  There will be a value

*** QuickLaTeX cannot compile formula:
\delta_i^L

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

for each neuron

*** QuickLaTeX cannot compile formula:
i

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

in the output layer.

*** QuickLaTeX cannot compile formula:
\begin{tikzpicture}

% Comment Accordingly. Additionally. Afterward. Afterwards. Albeit. Also. Although. Altogether. Another. Basically. Because.
\foreach \m [count=\y] in {1,2} {
\node [every neuron/.try, neuron \m/.try] (input-\m) at (0,5-\y*4) {$a_\m^{L-1}$};
\node [every neuron/.try, neuron \m/.try ] (hidden-\m) at (4,5-\y*4) {$b_\m^{L}$};
}
% Comment Accordingly. Additionally. Afterward. Afterwards. Albeit. Also. Although. Altogether. Another. Basically. Because.
% Comment Accordingly. Additionally. Afterward. Afterwards. Albeit. Also. Although. Altogether. Another. Basically. Because.
% Comment Accordingly. Additionally. Afterward. Afterwards. Albeit. Also. Although. Altogether. Another. Basically. Because.
\foreach \i in {1,2}
\foreach \j in {1,2}
\draw [->] (input-\i) -- (hidden-\j) node [pos=.25, fill=white] {$w_{\j\i}^{L}$};
\foreach \l [count=\x from 0] in {L-1, L}
\node [align=center, above] at (\x*4,2) {Layer \\ $\l$};
\end{tikzpicture}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

This is not an issue at all and we can write the three equations (4.1, 4.2, 4.3) for matrices.

*** QuickLaTeX cannot compile formula:
\[ \begin{bmatrix}
\delta^L_1\\[6pt]
\delta^L_2\\[6pt]
\vdots \\[6pt]
\delta^L_K
\end{bmatrix} =
\begin{bmatrix}
f'(a^L_1)\sigma'(z^L_1)\\[6pt]
% Comment
f'(a^L_2)\sigma'(z^L_2)\\[6pt]
\vdots\\[6pt]
f'(a^L_K)\sigma'(z^L_K)
\end{bmatrix} = \begin{bmatrix}
f'(a^L_1)\\[6pt]
% Comment
f'(a^L_2)\\[6pt]
\vdots\\[6pt]
f'(a^L_K)
\end{bmatrix} \odot
% Comment
\begin{bmatrix}
\sigma'(z^L_1)\\[6pt]
\sigma'(z^L_2)\\[6pt]
\vdots\\[6pt]
\sigma'(z^L_K)
\end{bmatrix}\]

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37
*** QuickLaTeX cannot compile formula:
\begin{equation*} 
\boldsymbol{\delta}^L=f'(\bold a^L) \odot \sigma'(\bold z^L)
\end{equation*}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

Where

*** QuickLaTeX cannot compile formula:
\odot

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

is the hadamard product of element wise multiplication.


*** QuickLaTeX cannot compile formula:
\[
\begin{bmatrix}
\frac{\partial c}{\partial b_1^L} \\[6pt]
\vdots\\[6pt]
\frac{\partial c}{\partial b_k^L}
\end{bmatrix} =
\begin{bmatrix}
\frac{\partial c}{\partial z_1^L} \times \frac{\partial z_1^L}{\partial b_1^L}\\[6pt]
\vdots\\[6pt]
\frac{\partial c}{\partial z_k^L} \times \frac{\partial z_k^L}{\partial b_k^L}
\end{bmatrix} =
\begin{bmatrix}
\delta_1^L\\[6pt]\vdots\\[6pt]\delta_k^L
\end{bmatrix}\odot
\begin{bmatrix}
1\\[6pt]\vdots\\[6pt]1
\end{bmatrix}\]

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37
*** QuickLaTeX cannot compile formula:
\begin{align*}
\frac{\partial c}{\partial \bold b^L} &= \frac{\partial c}{\partial \bold z^L} \times \frac{\partial \bold z^L}{\partial \bold b^L} \\
&=\boldsymbol{\delta}^L \odot 1 \\
&=\boldsymbol{\delta}^L
\end{align*}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37
*** QuickLaTeX cannot compile formula:
\begin{equation*} 
\nabla_{\bold b}^L c=\boldsymbol {\delta}^L
\end{equation*}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

*** QuickLaTeX cannot compile formula:
\begin{align*}
\begin{bmatrix}
\frac{\partial c}{\partial w_{11}^L} & \frac{\partial c}{\partial w_{12}^L} \\[8pt]
\frac{\partial c}{\partial w_{21}^L} & \frac{\partial c}{\partial w_{22}^L}
\end{bmatrix} &=
\begin{bmatrix}
\frac{\partial c}{\partial z_1^L} \times
\frac{\partial z_1^L}{\partial w_{11}^L} &
% Readability
\frac{\partial c}{\partial z_1^L} \times
\frac{\partial z_1^L}{\partial w_{12}^L} \\[8pt]
% Readability
\frac{\partial c}{\partial z_2^L} \times
\frac{\partial z_2^L}{\partial w_{21}^L} &
% Readability
\frac{\partial c}{\partial z_2^L} \times
\frac{\partial z_2^L}{\partial w_{22}^L}
\end{bmatrix} \\&=
\begin{bmatrix}
\delta_1^L a_1^{L-1} &
% Comment
\delta_1^L a_2^{L-1} \\[8pt]
\delta_2^L a_1^{L-1} & \delta_2^L a_2^{L-1}
\end{bmatrix}\\&=
\begin{bmatrix}
\delta_1^L \\[8pt]
\delta_2^L
\end{bmatrix} \cdot
\begin{bmatrix}
a_1^{L-1} & a_2^{L-1}
\end{bmatrix}
\end{align*}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37
*** QuickLaTeX cannot compile formula:
\begin{align*}
\frac{\partial c}{\partial \bold w^L}
&=\boldsymbol{\delta}^L \cdot [\bold a^{L-1}]^T \\
\end{align*}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37
*** QuickLaTeX cannot compile formula:
\begin{equation*} 
\nabla_{\bold w}^Lc=\boldsymbol{\delta}^L \cdot [\bold a^{L-1}]^T
\end{equation*}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

Now that we have a very simple calculation for tuning our output neuron let us think about the neurons in prior layers, specifically layer

*** QuickLaTeX cannot compile formula:
L-1

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

.

*** QuickLaTeX cannot compile formula:
\[\delta^{L-1}_i=\frac {\partial c}{\partial z^{L-1}_i}\]

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

The cost will be the total of the costs from each output neuron and each neuron is influenced by all the neurons in the previous layer.  To help us analyse this let us imagine that we have

*** QuickLaTeX cannot compile formula:
K

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

neurons in layer

*** QuickLaTeX cannot compile formula:
L

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

and

*** QuickLaTeX cannot compile formula:
J

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

neurons in layer

*** QuickLaTeX cannot compile formula:
L-1

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

.

*** QuickLaTeX cannot compile formula:
\[c=\sum^K_{k=1} f(a^L_k)
=\sum^K_{k=1} f(\sigma(z^L_k))\]

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37
*** QuickLaTeX cannot compile formula:
\[\delta^{L-1}_i=\sum^K_{k=1} f'(a^L_k)
\sigma'(z^L_k)
\frac{\partial z^L_k}{\partial z^{L-1}_i}\]

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37
*** QuickLaTeX cannot compile formula:
\[\delta^{L-1}_i=\sum^K_{k=1} f'(a^L_k)\sigma'(z^L_k)\frac{\partial}{\partial z^{L-1}_i}\bigg[b^L_k+\sum_{j=1}^J w^L_{kj}a^{L-1}_j\bigg]\]

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37
*** QuickLaTeX cannot compile formula:
\[\delta^{L-1}_i=\sum^K_{k=1} f'(a^L_k)\sigma'(z^L_k)\sum_{j=1}^J\Big[w^L_{kj}\sigma'(z^{L-1}_j)\frac{\partial z^{L-1}_j}{\partial z^{L-1}_i}\Big]\]

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

Where

*** QuickLaTeX cannot compile formula:
i=j

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

the final partial derivative becomes one and conversely where

*** QuickLaTeX cannot compile formula:
i \ne j

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

it is zero. We can therefore simplify the second sum.

*** QuickLaTeX cannot compile formula:
\[\delta^{L-1}_i=\sum^K_{k=1} f'(a^L_k)\sigma'(z^L_k)w^L_{ki}\sigma'(z^{L-1}_i)\]

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

But equation 4.1 allows us to substitute

*** QuickLaTeX cannot compile formula:
\delta_k^L

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

for

*** QuickLaTeX cannot compile formula:
f'(a^L_k)\sigma'(z^L_k)

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37
*** QuickLaTeX cannot compile formula:
\begin{equation*}  
\delta^{L-1}_i=\sum^K_{k=1} \delta^L_kw^L_{ki}\sigma'(z^{L-1}_i)
\end{equation*}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

The Heart of Back Propagation

This equation has expressed

*** QuickLaTeX cannot compile formula:
\delta^{L-1}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

in terms of

*** QuickLaTeX cannot compile formula:
\delta^L

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

. This is again a recursive format that means that once we have the

*** QuickLaTeX cannot compile formula:
\delta^L

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

for the last layer we can then compute recursively from the output of the network back to the input of the network.  This is the back propagation algorithm which helps to speed up machine learning in neural networks.  This will be the final equation in the set of four which describe back propagation.  Our other equations have all been written in matrix format so let us now do the same for 4.4.

*** QuickLaTeX cannot compile formula:
\[
\begin{bmatrix}
\delta^{L-1}_1 \\[8pt]
\delta^{L-1}_2 \\[8pt]
\vdots \\[8pt]
\delta^{L-1}_J
\end{bmatrix} =
\begin{bmatrix}
\delta^L_1w^L_{11} \sigma'(z^{L-1}_1) + \delta^L_2w^L_{21} \sigma'(z^{L-1}_1) + \hdots +
\delta^L_Kw^L_{K1} \sigma'(z^{L-1}_1) \\[8pt]
\delta^L_1w^L_{12} \sigma'(z^{L-1}_2) +
\delta^L_2w^L_{22} \sigma'(z^{L-1}_2) + \hdots +
\delta^L_Kw^L_{K2} \sigma'(z^{L-1}_2) \\[8pt]
\vdots \\[8pt]
\delta^L_1w^L_{1J}\sigma'(z^{L-1}_J) + \delta^L_2w^L_{2J}\sigma'(z^{L-1}_J) + \hdots +
\delta^L_Kw^L_{KJ}\sigma'(z^{L-1}_J)
\end{bmatrix}\]

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37
*** QuickLaTeX cannot compile formula:
\[
\begin{bmatrix}
\delta^{L-1}_1 \\[8pt]
\delta^{L-1}_2 \\[8pt]
\vdots \\[8pt]
\delta^{L-1}_J
\end{bmatrix} =
\begin{bmatrix}
\delta^L_1w^L_{11}+\delta^L_2w^L_{21}+\hdots+\delta^L_Kw^L_{K1} \\[8pt]
\delta^L_1w^L_{12}+\delta^L_2w^L_{22}+\hdots+\delta^L_2w^L_{K2} \\[8pt]
\vdots \\[8pt]
\delta^L_1w^L_{1J}+\delta^L_2w^L_{2J}+\hdots+\delta^L_Kw^L_{KJ}
\end{bmatrix} \odot
\begin{bmatrix}
\sigma'(z^{L-1}_1) \\[8pt]
\sigma'(z^{L-1}_2) \\[8pt]
\vdots \\[8pt]
\sigma'(z^{L-1}_J)
\end{bmatrix}\]

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37
*** QuickLaTeX cannot compile formula:
\[\begin{bmatrix}
\delta^{L-1}_1 \\[8pt]
\delta^{L-1}_2 \\[8pt]
\vdots \\[8pt]
\delta^{L-1}_J
\end{bmatrix} =
\begin{bmatrix}
w^L_{11} & w^L_{21} & \hdots & w^L_{K1} \\[8pt]
w^L_{12} & w^L_{22} & \hdots & w^L_{K2} \\[8pt]
\vdots & \vdots & \hdots & \vdots \\[8pt]
w^L_{1J} & w^L_{2J} & \hdots & w^L_{KJ}
\end{bmatrix}
\begin{bmatrix}
\delta^L_1 \\[8pt]
\delta^L_2 \\[8pt]
\vdots \\[8pt]
\delta^L_J
\end{bmatrix} \odot
\begin{bmatrix}
\sigma'(z^{L-1}_1) \\[8pt]
\sigma'(z^{L-1}_2) \\[8pt]
\vdots \\[8pt]
\sigma'(z^{L-1}_3)
\end{bmatrix}\]

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

From here we can now see a convenient matrix format for our recursive formula:

*** QuickLaTeX cannot compile formula:
\begin{equation*}  
\boldsymbol{\delta}^{L-1}=[\bold w^{L}]^T \odot \boldsymbol{\delta}^{L} \odot \sigma'(\bold z^{L-1})
\end{equation*}

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

Final Thoughts

We are very close to having the final back propagation equations. However BP2 and BP3 refer to gradients in the final layer

*** QuickLaTeX cannot compile formula:
L

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

. What about the other layers? In fact the metric

*** QuickLaTeX cannot compile formula:
\delta

*** Error message:
Cannot connect to QuickLaTeX server: cURL error 60: The certificate issuer's certificate has expired.  Check your system date and time.
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

works in the same way as the final layer and so these two equations can work in any layer.

Latest Articles

JTA: The Data Scientists attends ENEMath’21 Power BI Best Practices Daniela Costa is a finalist for PWIT’s Data & Analytics Expert Award Ana and Daniela nominated for the PWIT Data & Analytics Expert Award

Categories

Latest News Case Studies Computer Conservation Events Latest News Whitepapers / Publications
Enquiry

See how we can make your data speak

Send an enquiry to us below

reCAPTCHA