Be at Odds? Deep and Hierarchical Neural Networks for Classification and Regression of Conflict in Speech - Conflict and Multimodal Communication: Social Research and Machine Intelligence - page 400

including time-delay neural networks (Waibel et al. 1989 ), hierarchical sequence
compression (Schmidhuber 1992 ), or echo state networks (Jaeger 2001 ; Jaeger et al.
2007 ).
One of the most effective models, however, is the LSTM architecture (Hochreiter
and Schmidhuber 1997 ;Gersetal. 2002 ). LSTM networks can be interpreted as
RNNs in which the hidden neurons are replaced by special, linear memory blocks.
Similar to the cyclic connections in RNNs, these memory blocks are recurrently
connected. Every memory block consists of self-connected memory cells and
three multiplicative units, the input, output, and forget gates, which control the
information flow inside the memory block. The surrounding network can only
interact with the memory cells via the gates. Since these gates allow for write,
read, and reset operations within a memory block, an LSTM block can be seen
as a (differentiable) memory chip in a digital computer. The overall effect of the
gate units is that the LSTM memory cells can store and access information over
long periods of time and thus avoid the vanishing gradient problem. For instance,
as long as the input gate remains closed (corresponding to an input gate activation
close to zero), the activation of the cell will not be overwritten by new inputs and
can therefore be made available to the net much later in the sequence by opening
the output gate. This allows to bridge long time lags between relevant inputs and
outputs, which would not be possible with standard RNNs. Figure 19.3 illustrates a
single LSTM memory block containing one memory cell.
Fig. 19.3 Long Short-Term Memory block containing one memory cell. The state of cell c.t/ is
controlled by input gate i.t/ , its own state via a recursive connection modulated by the forget gate
f.t/ , and the cell input g i .x.t/;h.t 1/ . The cell output emitted by the memory block, h.t/ ,is
modulated by the output gate o.t/ .The dashed arrows are termed “peephole” connections (see text
for details)