Be at Odds? Deep and Hierarchical Neural Networks for Classification and Regression of Conflict in Speech - Conflict and Multimodal Communication: Social Research and Machine Intelligence - page 393

19.2 Deep Neural Networks

DNNs composed of multiple hidden layers were already proposed decades ago, but
they were difficult to train. Neural networks are typically trained using stochastic
gradient descent (SGD), e.g. using the backpropagation algorithm. However, with
large initial weights the network parameters converge towards poor local minima,
while with small initial weights the gradients in the lower layers vanish, making it
infeasible to train networks with many hidden layers. Moreover, deep networks with
many hidden layers and many hidden units in each layer tend to overfit the data sets,
especially in the case of the relatively small data sets often found in computational
paralinguistics.
In order to overcome these difficulties Hinton et al. ( 2006 ) proposed an efficient
method to pre-train DNNs, one layer at a time, using an undirected graphical model
called a Restricted Boltzmann Machine . One of the most interesting properties of
this approach is that this pre-training is fully accomplished in an unsupervised
fashion, i.e. without the need for target labels, which require (often costly) annotated
data.
The principal task of this pre-training stage is to move the network parameters
near to a local optimum in parameter space. The network parameters can then further
be optimized by running a few iterations of SGD on the pre-trained network to
discriminatively fine-tune the network to the task at hand.

19.2.1 Restricted Boltzmann Machines

An RBM is an undirected graphical model with a layer of observed ( visible )
variables and a layer of latent ( hidden ) variables, each layer forming one part of
a bipartite graph; i.e., each visible unit (node) is connected to each hidden unit,
but there are no intra-visible or intra-hidden connections. The graph of an RBM is
depicted in Fig. 19.1 .

Fig. 19.1 Undirected acyclic graph of a Restricted Boltzmann Machine. W denote the weights
between the visible layer v and the hidden layer h