Neural Identification of Controlled Dynamical Systems and Recurrent Networks - Neural Networks: Methodology and Applications - page 287

Consider now cases (c) and (d) of Fig. 4.21. Pictures are representing
recurrent network architectures. In the two cases, the architectures are similar
with respect to static characteristics and differ only by one delay, which has
been added in case (d). At time 2, the state of unit 3 is different: in case (c), it
depends on initial states of units 2 and 4, whereas, in case (d), it depends on
initial states of units 2 and 1. This difference is propagated at the following
time step to unit 4 then to unit 1 and so on. Finally, the network states are
completely different.
For recurrent networks, the state of units do not stabilize across time even
when the network is fed by static inputs. The state dynamics depend strongly
on the delay distribution. The update order plays a key role in the dynamics.
References
A list of references is given at the end of Chap. 5 for Chaps. 4 and 5.