Weather Logistics completed a successful collaboration with the National Physical Laboratory (NPL) in April 2022. As part of its deliverable, NPL validated our short-term climate forecasts and provided us with a framework for quality assessment of probabilistic weather predictions. We used these techniques to ensure the reliability of our Climate API product delivery.
Running our software on the Google Compute Engine we produced a set of high-resolution seasonal climate forecasts. Climate models cannot easily be tested in real-time; therefore we made use of past observations to predict conditions at future dates. These hindcasts, or ‘reforecasts’, spanned the past 5 years (2017 to 2021 inclusive) and provided a testbed to assess our model’s predictive skill. Most importantly for pricing climate risk our seasonal forecasts of weather hazards have proven reliable, well-calibrated and bias-corrected. This is a major breakthrough in seasonal climate prediction, with forecasts from many other model vendors not satisfying these essential components.
Unique to our seasonal climate data is a town or city scale stochastic prediction of daily precipitation events. This dataset extends current daily weather predictions out to several months. It also provides users with a well-calibrated indication into the probabilities of weather hazards of different intensities for flood risk management on operational timescales. The newly released Climate API will help develop better models for surface water flooding, river basin planning, or groundwater modelling teams. Re/insurance, finance or climate-related financial disclosure teams can now purchase these quarterly forecasts as a monthly subscription.
Direct Comparisons of Seasonal Forecasts Against Observations
The conditional quantile plots provided below are from the appendix of our NPL ‘Analysis for Innovators’ report. They show the consistency between daily observation quantiles (in 10% likelihood intervals) versus the conditional quantiles of seasonal precipitation forecasts. The first quantile shows q(observed) = 0.1 against q(forecast) = 0.1. Results were promising, demonstrating a good fit between our seasonal forecasts and coincident observations, and a very good fit indicated by lines close to the diagonal dashed lines in the plots when bias corrections are applied.**
The validation report results provided in the links below show the complete results for 96 UK cities, covering retrospective seasonal climate forecasts generated for the years 2018 to 2022. For the late winter assessment Weather Logistics’ forecasts cover the January to March period (t+1 to 3 months ahead) as they would be issued on 10th December of the previous year. These were compared directly against an ERA5-land assimilation of point observations for the same months. The charts show the distribution of daily precipitation events of different intensities and frequency of occurrences, see ‘Late Winter Forecast Analysis’.
Similarly, NPL’s report also provided direct comparisons of daily precipitation events for the late summer issued on 10th June that cover the July to September periods from 2018 to 2022, see ‘Late Summer Forecast Analysis’. The results show that for both the winter and summer forecasts the conditional quantile analyses demonstrate a good overall seasonal consistency between observations and retrospective seasonal weather forecasts at the city scale.
**Following NPL's validation assessment, a bias correction (reduction) of 10 to 15% was applied to the seasonal climate forecasts, not shown in the results presented, which corrects a modelling offset introduced in pre-April 2022 software releases. Credits to UKRI, the A4I programme, and NPL.
Figure 1. Three month daily precipitation ensembles for 96 UK towns and cities were compared against the ERA5-land 9km Reanalysis datasets. The Manchester precipitation forecast for late winter shows the consistency between Weather Logistics' seasonal forecasts at predicting the correct likelihood and intensities of daily events over the quarterly prediction window. This work was funded by UKRI via the Analysis for Innovators (A4I) programme. Reliability charts show predictions at 1-, 2- and 3-month lead time for a seasonal climate ‘forecast’ issued from Weather Logistics Ltd.
Key learnings from our NPL collaboration were the development of a code base for validation assessment, and appropriate skill score methods for climate data. These methods are transferrable to assessment of any ensemble-based (probabilistic) climate output from any model vendor in the marketplace. Seasonal climate predictions share common attributes to climate projections in that they present likelihoods of occurrence for weather events of different intensities linked to hazards e.g., flood, wind damage or heat waves.
For all our probabilistic forecast assessments we apply a similar quantile or category-based scoring approach to assess the consistency of predicted weather events of different intensity categories.
Receiver Operating Characteristic (ROC)
Figure 2. ROC assessments for daily precipitation forecasts for January 2019. Ensemble members were collected in quantiles to calculate the likelihood of daily precipitation occurrence of different intensities. They were then compared to the observed frequency of daily precipitation events from the ECMWF ReAnalysis (ERA5-land) product[1]. The shift to top-left indicates that the forecast detection rate classifies more true than false positives and is deemed skilful.
Figure 3. Receiver Operating Characteristic (ROC) curve with False Positive Rate and True Positive Rate. The diagonal shows the performance of a random classifier. Three classifiers (blue, orange, green) are shown. Drawn by CMG Lee based on http://commons.wikimedia.org/wiki/File:roc-draft-xkcd-style.svg
Seasonal Forecast Skill Scores
The data analysis team at NPL also generated a UK-wide chart showing the Brier skill score classification, which was adapted for probabilistic/ ensemble weather prediction data. While previous results showed that Weather Logistics’ seasonal forecasts were well-calibrated and unbiased, the Brier score assessment also provides strong evidence to support their skilfulness. Covering 96 town and city locations in the UK [Figure 4], the NPL team reported a skill score of 0.75 +/- 0.04. This is an improvement of 0.14 units compared to a Gamma-distributed random selection from past precipitation observations. For this analysis, daily precipitation events were assessed in equal quartiles (0th-25th, 25th-50th, 50th-75th, 75th-100th centiles) representing well-below, below, above, well-above average intervals of daily precipitation totals. This provided a more complete assessment that was reflective of the seasonal climate prediction performance for both the extremes as well as near-normal precipitation events.
With flooding the hazard most affecting industry, NPL data specialists have helped demonstrate a valuable product that will help organisations forecast floods and calculate their risk exposure.
Figure 4. Brier Score Map for January 2019, comparing the ratio of daily precipitation quartiles where the seasonal forecast was in the same confidence interval banding as the observations (0th to 25th, 25th to 75th, or 75th to 100th). Scores less than 1.0 are skilful, and a score of 0.0 indicates a perfect forecast.
Brier Score/ Skilfulness Assessment Process
- Compute quantiles of cumulative probability density of observations
- Calculate in each quantile number of forecast values matching the quantile of the observed values
- Calculate in each quantile number of observed values not matched by the forecast
- Calculate the Brier score
Overview of Results/ Model Improvements
**We tested the assumption that our precipitation forecasts are well-calibrated, and this was found to correct. Once correcting for an observed 10 to 15% wet bias, the retrospective runs of our seasonal climate forecasts were extremely consistent with ERA5-land reanalysis histories between the years 2017 to 2021. We also identified and updated our forecast software to account for future biases in April 2022.
Spatially correlated daily precipitation extremes are now incorporated into our modelling approach, accounting for weekly/ monthly patterns in seasonal climate variability.
Weather Logistics is also proud to announce that its know-how and validation tools are being used to perform skill score assessments for other climate data products in the risk marketplace. This validation work promises to deliver quality assessments through evaluation and ranking of datasets from a variety of model vendors.
Notes/ Credits
The National Physical Laboratory (NPL) is the UK’s National Metrology Institute, providing measurement capability that underpins the UK’s prosperity and quality of life.
Building on over a century’s worth of expertise, our science, engineering and technology provides this foundation. NPL is a national laboratory whose advice is always impartial and independent, meaning consumers, investors, policymakers and entrepreneurs can always rely on the work we do.
The Analysis for Innovators (A4I) Competition has been running since 2016. A4I is a very different type of programme from Innovate UK’s usual grant funding competitions. It is focused on helping individual companies solve tricky and, perhaps, long running technical problems affecting existing processes, products or services.
Innovate UK is the UK’s innovation agency who drive productivity and economic growth by supporting businesses to develop and realise the potential of new ideas. It connects businesses to the partners, customers and investors that can help them turn ideas into commercially successful products and services and business growth.
Innovate UK is part of UK Research and Innovation. For more information visit www.innovateuk.ukri.org
