Information Technology Reference
In-Depth Information
tivity level over the input layer. Because most of our
line stimuli have 5 units active, and there are 25 units in
the input layer, this ￿ value is set to .2. Let's explore
this issue using an environment where the features have
zero correlation with the receiving unit, and see how the
renormalization results in weight values of .5 for this
case.
be represented by a given hidden unit, then we can
set savg_cor appropriately so that the .5 level corre-
sponds roughly to this prior expectation. For example,
if we know that the units should have relatively selec-
tive representations (e.g., one or two input features per
unit), then we might want to set savg_cor to .5 or
even less, because the full correction for the input layer
will result in larger weights for features that are rela-
tively weakly correlated compared to this expected level
of selectivity. If units are expected to represent a num-
ber of input features, then a value of savg_cor closer
to 1 is more appropriate. We will revisit this issue.
Now, let's explore the contrast enhancement sig-
moid function of the effective weights. The parameters
wt_gain and wt_off in the control panel control the
gain and offset of the sigmoid function. First, we will
plot the shape of the contrast enhancement function for
different values of these parameters.
Set env_type in the control panel to the
FIVE_HORIZ lines environment ( Apply ). Do View and
EVENTS to see this environment.
You should see that the environment contains 5 hori-
zontal lines, each of which is presented with equal prob-
ability (i.e., 1/5 or .2). Thus, these line features rep-
resent the zero correlation case, because they each co-
occur with the receiving unit with the same probability
as the expected activity level over the input layer (.2).
In other words, you would expect this same level of co-
occurrence if you simply activated input units at random
such that the overall activity level on the input was at .2.
, !
First set wt_gain to 6 instead of 1. Click the
PlotEffWt button in the control panel, which will bring
up a graph log.
You should see a sigmoidal function (the shape of the
resulting effective weights function) plotted in a graph
log window at the bottom of your screen. The hori-
zontal axis represents the raw linear weight value, and
the vertical axis represents the contrast enhanced weight
value. The increase in wt_gain results in substantial
contrast enhancement.
Click on r.wt in the network window (and select the
hidden unit), and then Run the network.
You will see that because of the very literal behav-
ior of the unmodified CPCA algorithm in reflecting the
conditional probabilities, the weights are all around .2 at
the end of learning. Thus, if we were interpreting these
weights in terms of the standard meaning of conditional
probabilities (i.e., where .5 represents zero correlation),
we would conclude that the input units are anticorre-
lated with the receiving unit. However, we know that
this is not correct given the sparse activity levels in the
input.
, !
, !
Try setting wt_gain to various different values, and
then clicking the PlotEffWt button to observe the ef-
fect on the shape of this function.
We next see the effects of wt_gain on learning.
, !
Now, set savg_cor (which is the q m parameter in
equation 4.20) in the control panel to a value of 1 instead
of 0.
This means that we will now be applying the full cor-
rection for the average activity level in the sending (in-
put) layer.
First, select Defaults , change savg_cor to 1,
SelectEnv the THREE_LINES environment, and Run .
This run provides a baseline for comparison. You
should see a somewhat bloblike representation in the
weights, where the right lines are a bit more strong than
the left lines, but not dramatically so.
, !
, !
Run the network again.
You should observe that the weights now hover
around .5, which is the correct value for expressing the
lack of correlation.
Although the ability to fully correct for sparse send-
ing activations is useful, one does not always want to
do this. In particular, if we have any prior expecta-
tion about how many individual input patterns should
Now increase wt_gain from 1 to 6, and Run again.
You should very clearly see that only the right lines
are represented, and with relatively strong weights.
Thus, the contrast enhancement allows the network to
represent the reality of the distinct underlying left and
right categories of features even when it is imperfectly
selective (.7) to these features. This effect will be espe-
, !
, !
Search WWH ::




Custom Search