Geoscience Reference
In-Depth Information
but the most usual form comprises a non-linear, non-decreasing, bounded, piecewise, differentiable
function (fixed within finite asymptotic limits). With regard to computational overheads, it is also
desirable that its derivative should not be difficult to calculate.
If the input data are continuous and have values that range between 0 to +1, then the logistic
function is a common choice for
i
:
1
F
i
()
v
=
(13.5)
i
(
)
1
+
exp
−β
v
i
where β denotes the slope parameter which has to be chosen
a priori
. In the limit, as β approaches
infinity, the logistic function becomes a simple threshold function producing ones and zeros.
13.5 NETWORK TOPOLOGIES
In this section, we examine the pattern of the connections that exist between the PEs, often referred
to as the network architecture, and where it is possible to make a major distinction between
•
Feedforward CNNs
•
Recurrent CNNs
Feedforward CNNs comprise those architectural configurations in which the networks do not con-
tain directed cycles, that is, the flow of information all goes in one direction from the input nodes
(start) - via various intermediaries - to the output nodes (finish). There are no data feedback loops
whatsoever. It is often convenient to organise our arrangement of nodes within each feedforward
CNN into a number of distinct layers and to label each layer according to the following rule. We
define an
L
-layer feedforward network as being a CNN wherein the PEs are grouped into
L
+ 1 lay-
ers (subsets)
L
0
,
L
1
, …,
L
L
such that if unit
u
in layer
L
a
is connected to unit
u
i
in layer
L
b
,
then
a
<
b
,
that is, the layers are numbered in ascending order in accordance with the direction of our data flow.
For a strict
L
-layer network, we would also require that the output links from the PEs in one layer
are only connected to units in the next layer, that is,
b
=
a
+ 1 (as opposed to
a
+ 2,
a
+ 3, etc.). All
units in layer
L
0
are input units, all units in layers
L
1
, …,
L
L
are trainable PEs, and all units in layer
L
L
are also output devices.
Figure 13.3 comprises a pair of architectural diagrams and is intended to illustrate various fea-
tures concerning the basic layout of a multilayered feedforward CNN. Both networks have a single
hidden layer and can therefore be referred to as single-hidden-layer feedforward networks. The
shorthand notation for describing both such multilayered items would be to refer to them as 8:4:2
networks since, going from left to right, there are eight input nodes, four hidden PEs and two out-
put units. The CNN in Figure 13.3a is said to be
fully connected
in the sense that each and every
node in the network is connected to each and every node in the adjacent forward layer. If some of
the connections are missing from the network, then the network is said to be
partially connected
.
Figure 13.3b is an example of a partially connected CNN where the input nodes and PEs in the hid-
den layer are connected to a partial (limited) set of PEs in the immediate forward neighbourhood.
The set of localised nodes feeding an individual PE is said to constitute the
receptive field
of the
PE. Although the CNN in Figure 13.3b is noted to have an identical number of input nodes, hidden
units and output units to that of Figure 13.3a, the pattern of its connections nevertheless forms a
specialised structure. In real-world applications, when specialised structures of this nature are built
into the design of a feedforward CNN, it would be to reflect prior information about the problem
that has been targeted for analysis.
A recurrent (feedback) CNN distinguishes itself from a feedforward CNN in that it contains data
processing cycles (i.e. feedback connections or data processing loops). The data are not only fed
forward in the usual manner but can also be fed backward from output units to input units, from
