Information Technology Reference
In-Depth Information
Fig. 5 LSTM memory block with one cell
The mathematical background of LSTM is described in depth in [40,41,34]. A
short description follows. A conventional recurrent multi-layer Perceptron net-
work (MLP) contains a hidden layer where all neurons of the hidden layer are ful-
ly connected with all neurons of the same layer (the recurrent connections). The
activation of a single cell at the timestamp t is a weighted sum of the inputs x i t plus
the weighted sum of the outputs of the previous timestamp b h (t-1) . This can be ex-
pressed as follows (or in a matrix form):
t
t
i
t
h
1
t
t
1
a
=
w
x
+
w
b
=
W
X
+
W
B
i
h
i
h
t
t
b
=
f
(
a
)
t
t
B
=
f
(
A
)
Since the outputs of the previous timestamp are just calculated by the squashing
function of the corresponding cell activations, the influence of the network input
in the previous time stamp can be considered as smaller, since it has been
weighted already a second time. Thus the overall network activation can be rough-
ly rewritten as:
2
t
t
t
t
1
t
2
1
A
=
g
(
X
,
W
X
,
W
X
,
,
W
X
)
h
h
h
Search WWH ::




Custom Search