Biomedical Engineering Reference
In-Depth Information
[20]) can be computed by some Turing machine. Before a Turing machine gives its
output, it goes through a series of internal computation steps, the number of which
depends on the specific input and the difficulty of the computational task (therefore
it is called an
offline computation
). This may not be inadequate for modeling human
reasoning about chess end games, but most cognitive tasks are closer related to real-
time computations on continuous input streams, where online responses are needed
within specific (typically very short) time windows, regardless of the complexity of
the input. In this case the domain
D
and range
R
consist of time-varying functions
u
(with analog inputs and outputs), rather than of static character strings. We
propose here an alternative computational model that is more adequate for analyzing
parallel real-time computations on analog input streams, such as those occurring in
generic cognitive information processing tasks. Furthermore, we present a theoreti-
cal result which implies that within this framework the computational units of a pow-
erful computational system can be quite arbitrary, provided that sufficiently diverse
units are available (see the separation property and approximation property discussed
in Section 18.4). It also is not necessary to
construct
circuits to achieve substantial
computational power. Instead sufficiently large and complex
found
circuits tend to
have already large computational power for real-time computing, provided that the
reservoir from which their units are chosen is sufficiently diverse.
Our approach is based on the following observations. If one excites a sufficiently
complex recurrent circuit (or other medium) with a continuous input stream
u
(
·
)
,
y
(
·
)
(
s
)
,
and looks at a later time
t
>
s
at the current internal state
x
(
t
)
of the circuit, then
x
(
t
)
is likely to hold a substantial amount of information about recent inputs
u
(for the
case of neural circuit models this was first demonstrated by [4]). We as human ob-
servers may not be able to understand the
code
by which this information about
u
(
s
)
(
s
)
is encoded in the current circuit state
x
, but that is obviously not essential. Essen-
tial is whether a readout neuron that has to extract such information at time
t
for a
specific task can accomplish this. But this amounts to a classical pattern recognition
problem, since the temporal dynamics of the input stream
u
(
t
)
(
)
s
has been transformed
by the recurrent circuit into a high dimensional spatial pattern
x
. This pattern clas-
sification problem tends to be relatively easy to learn, even by a memoryless readout,
provided the desired information is present in the circuit state
x
(
t
)
.Furthermore,if
the recurrent neural circuit is sufficiently large, it may support this learning task by
acting like a kernel for support vector machines (see [25]), which presents a large
number of nonlinear combinations of components of the preceding input stream to
the readout. Such nonlinear projection of the original input stream
u
(
t
)
into a high
dimensional space tends to facilitate the extraction of information about this input
stream at later times
t
, since it boosts the power of
linear
readouts for classification
and regression tasks. Linear readouts are not only better models for the readout ca-
pabilities of a biological neuron than for example multi-layer-perceptrons, but their
training is much easier and robust because it cannot get stuck in local minima of
the error function (see [25] and [7]). These considerations suggest new hypotheses
regarding the computational function of generic recurrent neural circuits: to serve as
general-purpose temporal integrators, and simultaneously as kernels (i.e., nonlinear
projections into a higher dimensional space) to facilitate subsequent linear readout of
(
·
)
Search WWH ::
Custom Search