Novel metrics for evaluation of ensemble forecasts of tropical cyclone structure

Predicting extratropical transition (ET) of a tropical cyclone poses a significant challenge to numerical forecast models because the storm evolution depends on both the timing of the phasing between the tropical cyclone and midlatitude weather systems and the structures of each system. Ensemble prediction systems offer the potential for assessing confidence in numerical guidance during ET cases. Thus, forecasts of storm structure changes during ET from the European Centre for Medium-Range Weather Forecasts (ECMWF) ensemble prediction system (EPS) are explored using two novel validation approaches. The evolution of the (initially tropical) storm structure is characterized in the framework of the cyclone phase space (CPS) and the validation metrics are based on separation between the EPS forecasts and verifying analyses in the CPS. The first validation approach utilizes two metrics and most closely resembles traditional forecast validation techniques. The second approach involves clustering the ensemble member initializations and operational analyses during the life cycles of each tropical cyclone to provide a reference structure evolution against which to evaluate the EPS forecasts. Application of these metrics is demonstrated for two case studies of ET in the western North Pacific: Typhoons Tokage (2004) and Maemi (2003). Both validation approaches identify a decline in EPS structure forecast accuracy for all valid times coinciding with ET onset and beyond, as well as during a weakening tropical stage prior to ET for Tokage. While track forecast errors contribute to structure errors in the EPS forecasts, they are not an overwhelming factor. The two validation approaches highlight the inability of ensemble member forecasts to appropriately weaken the warm core prior to and during ET, and the effects this has on forecasts of ET timing. The analyses adopted in this study provide a basis for future assessments of ensemble forecast skill of cyclone structure during ET.