Information Technology Reference
In-Depth Information
Family Trees Learning Curves
We are not so interested in raw learning speed for
its own sake, but more in the facilitation of learning
in deep networks from the additional biases or con-
straints imposed by combining model and task learn-
ing. Figure 6.9 shows that these additional constraints
facilitate learning in deep networks; the purely task-
driven BP network learns relatively slowly, whereas the
Leabra network, with both Hebbian learning and in-
hibitory competition, learns relatively quickly. In this
exploration (as in the previous one), we will manipulate
the contribution of Hebbian learning. To do this, we can
run a network without Hebbian learning and compare
the learning times.
100
BP lr .39
Leabra
75
50
25
0
0
20
40
60
80
Epochs
Figure 6.9:
Learning curves for family trees task, showing
that combination of Hebbian (model) and error-driven (task)
learning in Leabra results in faster learning in deep networks
compared to pure error-driven learning in a backpropagation
network (BP).
If you want to wait while the network trains, set
learn_rule
to
PURE_ERR
, and
Apply
. You will see
the
lrn.hebb
change to 0. Then
Run
again.
Again, if you do not want to wait for the network to
train, you can just load the results from a fully trained
network.
,
!
In
this
case,
go
to
the
graph
log,
and
Your network should train in around 40 to 50 epochs
using the initial default parameters. This may take a
while. You can either wait, or you can instead load the
results from a fully trained network.
select
LogFile/Load File
,
and
choose
,
!
family_trees.pure_err.epc.log
.
Hebbian learning clearly facilitates learning in deep
networks, as demonstrated by the network taking longer
to learn without it (81 epochs in this case compared
to 41; repeated runs of the networks with different pa-
rameters substantiate this effect). Further, kWTA ac-
tivation constraints play an important facilitatory role
in learning as well. The benefits of kWTA activation
constraints are somewhat obscured in comparing the
purely error-driven Leabra network with the backpropa-
gation (BP) network shown in figure 6.9, because of the
very high learning rate used for finding the best perfor-
mance of the BP network. The benefits of kWTA activa-
tion constraints are particularly clear in comparing the
purely error-driven Leabra network to a bidirectionally-
connected error-driven (GeneRec) network that does
not have the kWTA activation constraints, which takes
around 300 or more epochs to learn at its fastest
(O'Reilly, 1996b).
Now, let's see what pure Hebbian learning can do in
this task.
If you decide to wait while your network trains to
completion, do
Object/SaveAs
in the network window
after training is done, and save it with some descriptive
name — we will want to come back to it later.
,
!
If you instead opt to load the results from a fully
trained network, press
Stop
in the control panel. To load
the network's learning curve into the graph log, go to the
graph log and select menu
LogFile/Load File
, and
choose
family_trees.hebb_err.epc.log
.
The network took 41 epochs to learn, which is indi-
cated in the network file name. Note that the graph log
goes from 0 to 40 (41 total).
The 41 epochs it took for the default network to
learn the problem is actually relatively rapid learning
for a deep network like this one. For example, fig-
ure 6.9 shows a comparison of a typical learning curve
in Leabra versus the fastest standard feedforward back-
propagation network (BP, see box 5.2), which took
about 77 epochs to learn and required a very large learn-
ing rate of .39 compared to the standard .01 for the
Leabra network (O'Reilly, 1996b).
,
!
Select
PURE_HEBB
for
learn_rule
, and re-run the
network. You can
Stop
it after 10 epochs or so.
This network isn't going to improve at all. You can
,
!
Search WWH ::
Custom Search