TOTAL LEAST SQUARES PROBLEMS - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 17

to a factor of proportionality, the data [ A ; B ] can be transformed to the new data

A ∗ ; B ∗ = [ A ; B ] C − 1 , where C is a square root ( Cholesky factor ) of ( =
C T C ) , such that the error covariance matrix of the transformed data is now diag-
onal with equal error variances. Then the TLS algorithm can be used on the new
set, and finally, its solution must be converted to a solution of the original set of
equations [ 69,70 ] .

Remark 35 (Covariance Knowledge) The knowledge of the form of the error
covariance matrix up to a constant scalar multiple may still appear too restrictive,
as this type of information is not always available to an experimenter. Assuming
that independent repeated observations are available for each variable observed
with error, this type of replication provides enough information about the error
covariance matrix to derive consistent unbiased estimates of [ 60,62 ] .Using
these consistent estimates instead of does not change the consistency properties
of the parameter estimators [ 60 ] .

The TLS estimators in both the intercept and no-intercept models are asymp-
totically normally distributed . For the unidimensional case, the covariance matrix
of the TLS estimator x is larger than the covariance matrix of the LS estimator
x ,evenif A is noisy [98].
Summarizing, a comparison about the accuracy of the TLS and LS solution
with respect to their bias, total variance, and mean squared error (MSE; total
variance + squared bias) gives:

• The bias of TLS is much smaller than the bias of LS and decreases with
increasing m (i.e., increasing the degree of overdetermination).
• The total variance of TLS is larger than that of LS.
• At the smallest noise variances, MSE is comparable for TLS and LS; by
increasing the noise in the data, the differences in MSE are greater, showing
the better performance of TLS; the TLS solution will be more accurate,
especially when the set of equations are more overdetermined, but the better
performance is already good for moderate sample size m .

All these conclusions are true even if the errors are not Gaussian but are
exponentially distributed or t -distributed with 3 degrees of freedom.

1.10.1 Outliers

The TLS, which is characterized by larger variances, is less stable than LS. This
instability has quite serious implications for the accuracy of the estimates in the
presence of outliers in the data (i.e., large errors in the measurements) [20,105].
In this case a dramatic deterioration of the TLS estimates happens. Also, the
LS estimates encounter serious stability problems, although less dramatically;
efficient robust procedures should be considered, which are quite efficient and
rather insensitive to outliers (e.g., by down-weighting measurement samples that