All-Sky Satellite Radiance Data Assimilation: Methodology and Challenges - Data Assimilation for Atmospheric, Oceanic and Hydrologic Applications - page 471

Discontinuity of observation operator creates an obvious problem for variational
methods since they are commonly using gradient-based minimization (e.g., Nocedal
1980 ; Navon 1986 ). A non-existent gradient at the point of transition from clear-
sky to cloudy conditions prevents correct performance of minimization, eventually
resulting in incorrect minimizing solution. Since KF can be described in terms
of gradient-based minimization, (e.g., Jazwinski 1970 ), it has similar problems as
variational methods. This implies that non-differentiability of all-sky observation
operator is a problem of data assimilation in general.
There are many ways to deal with discontinuity, most obvious being: (1)
neglect it, (2) apply smoothing, (3) use non-differentiable minimization algorithms.
Although the first option may not be mathematically correct, it does not require any
additional effort. Since the discontinuity point is in the area of transition from cloudy
to clear-sky conditions, the discontinuity problem may be confined to only those
geographical areas, allowing minimization to perform well in the rest of the domain.
However, since discontinuity also impacts the line-search algorithm (i.e. finding the
optimal step-size), its influence can be more pronounced. Therefore, neglecting the
discontinuity problem may be acceptable in some situations, but not in general and
definitely not in operational practice. The option (2) has been successfully applied
within 4d-Var methods in cases of parameterization schemes (e.g., Zupanski and
Mesinger 1995 ). The approach is to change the original operator by introducing a
smooth function in the place of an on-off switch, thus preventing code branching. In
choosing the adequate smoothing function and parameters it is important to maintain
approximately the same skill and accuracy of the original operator. This could be
difficult and it requires an extensive preparation of the code. The third option (3) is
the most correct approach, since it does not alter the original observation operator
and it addresses the true problem, which is the minimization algorithm performance.
There are numerous minimization algorithms that can address non-differentiability,
some of those developed as an extension of gradient-based algorithms ( Haarala
et al. 2004 ; Karmitsa et al. 2012 ; Steward et al. 2012 ). Encouraging results obtained
using this approach in data assimilation have been reported by Steward et al. ( 2012 ).
It is likely that the choice of approach will depend on the actual assimilation
problem and the amount of work required to implement the changes. Important
message from this section is that non-differentiability of all-sky observation operator
should not be overlooked. Once the problem is identified, one can proceed to
solutions (1)-(3), or take an alternative approach.

19.3.4 Non-Gaussian Errors

Current data assimilation methodologies are generally designed to address only
Gaussian errors. It is also understood that there are numerous applications with
non-Gaussian errors and that a non-Gaussian data assimilation framework may
be necessary (e.g., Abramov and Majda 2004 ; Fletcher and Zupanski 2006a ,b;
Bocquet et al. 2010 ). Satellite radiance observation error statistics indicates a