Mathematical test criteria for filtering complex systems: Plentiful observations

An important emerging scientific issue is the real time filtering through observations of noisy turbulent signals for complex systems as well as the statistical accuracy of spatio-temporal discretizations for such systems. These issues are addressed here in detail for the setting with plentiful observations for a scalar field through explicit mathematical test criteria utilizing a recent theory [A.J. Majda, M.J. Grote, Explicit off-line criteria for stable accurate time filtering of strongly unstable spatially extended systems, Proceedings of the National Academy of Sciences 104 (4) (2007) 1124-1129]. For plentiful observations, the number of observations equals the number of mesh points. These test criteria involve much simpler decoupled complex scalar filtering test problems with explicit formulas and elementary numerical experiments which are developed here as guidelines for filter performance. The theory includes information criteria to avoid filter divergence with large model errors, asymptotic Kalman gain, filter stability, and accurate filtering with small ensemble size as well as rigorous results delineating the role of various turbulent spectra for filtering under mesh refinement. These guidelines are also applied to discrete approximations for filtering the stochastically forced dissipative advection equation with very turbulent and noisy signals with either an equipartition of energy or -5/3 turbulent spectrum with infrequent observations as severe test problems. The theory and companion simulations demonstrate accurate statistical filtering in this context with implicit schemes with large time step with very small ensemble sizes and even with unstable explicit schemes under appropriate circumstances provided the filtering strategies are guided by the off-line theoretical criteria. The surprising failure of other strongly stable filtering strategies is also explained through these off-line criteria.