Information Technology Reference
In-Depth Information
a similar Bayesian interpretation in terms of comparing
the thresholded V m value to the constant null hypothe-
sis represented arbitrarily by the number 1, as described
in section 2.7). This equation can also be written some-
what more simply as:
(2.20)
which makes clear the relationship between this func-
tion and the standard sigmoidal activation function
(equation 2.12) commonly used in artificial neural net-
work models, as discussed previously.
Equation 2.20 by itself does not provide a particu-
larly good fit to actual discrete spiking rates, because
it fails to take into account the presence of noise in the
spiking model. Although our simulated spiking neuron
will fire spikes at completely regular intervals with a
constant input (figure 2.12), the detailed spike timing is
very sensitive to small fluctuations in membrane poten-
tial, as we will see in explorations described later. Thus,
once any noise enters into a network of spiking neu-
rons, the resulting fluctuations in membrane potential
will tend to propagate the noise throughout the network.
This is consistent with the finding that neural spike tim-
ing indeed appears to be quite random (e.g., Shadlen &
Newsome, 1994). Note however that just because the
detailed timing appears to be random, the firing rate av-
eraged over a suitable time window can nevertheless be
relatively steady, as we will see.
One effect that this kind of noise would have is that
the neuron would sometimes fire even when its average
membrane potential is below threshold, due to random
noise fluctuations. Thus, we might expect that the very
sharp threshold in the XX1 function would be softened
by the presence of noise. We will see that this is indeed
what happens.
Perhaps the most direct way to capture the effects of
noise in our rate code model would be to literally add
noise to the membrane potential, resulting in a model
that exhibits stochastic (noisy) behavior. However, this
slows processing and results in the need for averag-
ing over large samples to obtain reliable effects, which
are some of the things we had hoped to avoid by us-
ing a rate code output in the first place. Thus, we in-
stead use a modified rate-code activation function that
Vm
Θ
Figure 2.13: Illustration of convolution, where each point
in the the new function (dashed line) is produced by adding
up the values of a normalized Gaussian centered on that point
multiplied times the original function (solid line).
Noisy XX1 Activation Function
1.0
0.8
0.6
0.4
Noise = .005
Noise = 0
0.2
0.0
−0.02
0.00
0.02
0.04
Diff from Threshold: Vm − Q
Figure 2.14: Noisy XX1 (X/X+1) activation function
(threshold written as Q instead of ￿ ), showing effects of con-
volving with Gaussian noise of ￿ = :005 to the case with no
noise. The gain ( ￿ ) is the standard 600 for membrane poten-
tials in the 0 ￿￿1 range.
directly incorporates the average effect of noise. The
result is that we still have deterministic (nonstochastic,
perfectly predictable) units, which nevertheless reflect
the expected or average effects of noise.
The averaging of the noise into the activation func-
tion is done by convolving a Gaussian-distributed noise
function with the XX1 activation function given by
equation 2.20. Convolving, illustrated in figure 2.13,
simply amounts to multiplying a bell-shaped Gaussian
noise function times a neighborhood of points surround-
Search WWH ::




Custom Search