Neural Networks for Handwriting Recognition - Computational Intelligence Paradigms in Advanced Pattern Classification - page 13

of past context. Context plays an important role in handwriting recognition, as il-
lustrated in Figure 4. Another important advantage of recurrency is that the rate of
change of the internal state can be finely modulated by the recurrent weights,
which builds in robustness to localized distortions of the input data.
Fig. 4 Importance of context. The characters “ur” would be hard to recognize without the
context of the word “entourage”.
3.2 Long Short-Term Memory (LSTM)
Unfortunately, the range of contextual information that standard RNNs can access
is quite limited. The problem is that the influence of a given input on the hidden
layer, and therefore on the network output, either decays or blows up exponential-
ly as it cycles around the network's recurrent connections, and is repeatedly scaled
by the connection weights. In practice this shortcoming (referred to in the litera-
ture as the vanishing gradient problem) makes it hard for an RNN to bridge gaps
of more than about 10 time steps between relevant input and target events.
Long Short-Term Memory (LSTM) is an RNN architecture specifically de-
signed to address the vanishing gradient problem. An LSTM hidden layer consists
of multiple recurrently connected subnets, known as memory blocks. Each block
contains a set of internal units, or cells, whose activation is controlled by three
multiplicative units: the input gate, forget gate and output gate. Figure 5 provides
a detailed illustration of an LSTM memory block with a single cell.
The effect of the gates is to allow the cells to store and access information over
long periods of time. For example, as long as the input gate remains closed (i.e.,
has an activation close to 0), the activation of the cell will not be overwritten by
the new inputs arriving in the network. Similarly, the cell activation is only avail-
able to the rest of the network when the output gate is open, and the cell's recur-
rent connection is switched on and off by the forget gate.