Quantifying uncertainty in European climate projections using combined performance-independence weighting

This study is based on all available CMIP5 models which contain the variables required to inform the weighting (37 models with a total of 79 realizations; see figure S6 in the supplement for a full list). We use monthly data, regridded to a regular 2.5° × 2.5° grid using bi-linear remapping. Throughout the study, we base our weighting approach on the base-period 1995–2014 in the combined historical and Representative Concentration Pathway 8.5 (RCP8.5) (van Vuuren et al 2011) forcing runs, and apply the weights to RCP8.5 forcing runs for the future projections. The base-period is selected to (i) best represent present day conditions and (ii) in anticipation of and to be easily comparable to the upcoming CMIP6 which uses historical forcing up to 2014 (Eyring et al 2016).

To account for uncertainties in the observational records, the performance weighting is based on observational estimates from two reanalysis datasets and one combined observational dataset. We use the interim reanalysis from the European Centre for Medium-Range Weather Forecasts (ERA-Interim) (Dee et al 2011) and the Modern-Era Retrospective analysis for Research and Applications version 2 from the National Aeronautics and Space Administration (MERRA2) (Gelaro et al 2017). In addition, we use a combined observational dataset, which is based on the ensemble version 17 of E-OBS (Cornes et al 2018) (for temperature, precipitation, sea level pressure) and the CERES EBAF Surface Ed2.8 (Kato et al 2013) (for radiation variables).

We focus on changes in summer (JJA) temperature and precipitation in the three European SREX (IPCC 2012) regions: Northern Europe (NEU), Central Europe (CEU), and the Mediterranean (MED) as well as in the combined European domain (EUR). In addition, we test the performance of the method on four single grid cells as listed in table 1. A geographical map showing all regions as well as the model resolution and the applied land-sea mask can be found in the supplement (figure S1 is available online at stacks.iop.org/ERL/14/124010/mmedia). The effects of the weighting on future projections for two 20-year periods representing mid-century (2041–2060) and end-of-century (2081–2100) conditions are investigated.

Weights are calculated for each model following the approach presented by Lorenz et al (2018), which is based on earlier work from Knutti et al (2017) and Sanderson et al (2015a, 2015b). Each weight w_i is a combination of the observational distance D_i (informing the performance weighting) and the model distance S_ij (informing the independence weighting):

$$\begin{eqnarray}&&{w}_{i}=\displaystyle \frac{{e}^{-\tfrac{{D}_{i}}{{\sigma }_{D}}}}{1+{\displaystyle \sum }_{j\ne i}^{M}{e}^{-\tfrac{{S}_{{ij}}}{{\sigma }_{S}}}},\end{eqnarray} \tag{ 1 }$$

with the total number of model runs M and the shape parameters σ_D and σ_S. The shape parameters define the strength of the weighting and the relative importance of performance and independence, large values will lead to an approximation of equal weighting, while small values will lead to aggressive weighting, giving a few models most of the weight. The shape parameters used in this study are summarized in table 1 and their calculation is described by Lorenz et al (2018) and in the supplement.

The observational distance D_i, is based on six diagnostics (see table 2). A diagnostic can be based on any CMIP5 output variable for which observations are available. The calculation of D_i follows a straight-forward approach: (i) variable, region, time period, and season are selected, (ii) the mean (CLIM) or standard deviation (STD) over the time period is calculated, (iii) the point-to-point distance between model and the center of the observational spread is calculated, and finally (iv) the area-weighted root mean squared error is calculated over the selected region.

To account for uncertainty in our knowledge of observed climate we use three observational estimates in the calculation of the point-to-point distance (iii). Several methods of combining these estimates and their influence on the weighting are investigated. Distances are tested with regard to each of the estimates individually as well as with regard to mean, median, and full spread. A detailed description of all steps and the different methods of combining observations can be found in the supplement.

The independence weighting is informed by diagnostics which can be based on any CMIP5 output variable which is available for all models. In practice, we use the same diagnostics as for the performance weighting. To compute the model distance S_ij, the point-to-point difference (step (iii) above) is calculated between each model pair.

To select the most informative diagnostics, different combinations and numbers of diagnostics are tested and evaluated. The aim is to select diagnostics which are, on the one hand, relevant for the target and, on the other hand, add additional information to the already used diagnostics (Gleckler et al 2008, Lorenz et al 2018). We test a pool of variables (table 2) which we judge to be relevant for predicting our targets and for which observations are available. Indeed, the availability of observations of sufficient duration, spatial coverage, and quality is a considerable constraint for the pool of tested variables, particularly since we here aim at using multiple data sets in order to include observational uncertainty. Using the same approach as Lorenz et al (2018), we select six diagnostics, which is found to be a good number to differentiate between high and low performing models (performance tends to converge with an increasing number of diagnostics) while not being overconfident in a model's ability to reproduce observed climate (which becomes an issue with a low number of diagnostics) (Lorenz et al 2018).

A model-as-truth test (or perfect model test) picks each model from a multi-model ensemble in turn and treats it as the true representation of the climate system. Weights for the remaining models are computed using historical information from this perfect model as 'pseudo-observations'. This allows for an evaluation of the impact of the weighting in the future based on each model representing the truth once. Here this approach is used for two applications: (i) to estimate the ideal shape parameter for the performance weighting σ_D (as detailed in the supplement) and (ii) to evaluate the impact of the weighting on the skill of the future projections. As a measure for the skill we use the continuous ranked probability score (CRPS) as detailed in Hersbach (2000). For each perfect model the CRPS basically represents the mean absolute error between the distribution of all other models and the perfect model. We define the skill of the weighting as the relative change in CRPS in the end-of-century period:

$$\begin{eqnarray}&&{\rm{skill}}=\displaystyle \frac{{{\rm{CRPS}}}_{\mathrm{unweighted}}-{{\rm{CRPS}}}_{\mathrm{weighted}}}{{{\rm{CRPS}}}_{\mathrm{unweighted}}}\times 100 \% .\end{eqnarray} \tag{ 2 }$$

We first calculate the optimized shape parameters (σ_D and σ_S) for all regions and periods. They deviate only slightly between mid-century and end-of-century and we therefore use the average over both periods in the computation of weights (table 1). Across all three SREX regions, the sigma values for temperature average to about σ_D = 0.6 and σ_S = 0.7, which is comparable to earlier work focusing on maximum summer temperature in North America (Lorenz et al 2018). In the four grid cell regions, the performance sigma values are about 50% higher than for the SREX regions due to the larger range of model-observation differences without spatial aggregation. Generally speaking, weighting based on smaller regions is less robust and, therefore, more likely to be overconfident. As a consequence the perfect model test correctly picks higher performance sigma values that lead to more evenly distributed weights and reduce overfitting.

Weighted and unweighted time-series of Mediterranean summer temperature are shown in figure 1. Models with less warming consistently get the lowest weights and, as a consequence, the weighted multi-model mean shifts upward by an additional 0.35 °C by the end of the century. The 25th percentile also shows a strong increase, moving above 5 °C warming for 2081–2100 (see also figure 2(a)). Since the increase in the 75th percentile is less distinctive, the weighting also leads to a reduction of the interquartile spread by almost 25%.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Figure 2(a) shows temperature change distributions for the combined European domain, two SREX regions, and one of the grid cell regions. A summary of all regions can be found in figure S2(a) in the supplement. The effect of the weighting is generally smaller for the mid-century period. In the full European domain there is hardly any shift in the mean change, but a considerably reduction of the interquartile spread by about one quarter for the end-of-century period. Analyzing spatial differences reveals that the European domain seems to be a too large region to average over. The weighted mean is colder in the northern part and warmer in the southern part compared to the unweighted case consistent with the SREX regions described below. Aggregating over the entire domain cancels out most of this effect in the mean. In Northern Europe the interquartile spread widens by almost a third due to the weighting. This behavior is even stronger for the Falun grid cell with an increase by over 50%. Based on our diagnostics, it seems that models on both ends of the future temperature distribution get higher weights in NEU, indicating that a simple average and standard deviation might be slightly overconfident.

In Central Europe (figure S2(a)) neither mean nor spread are strongly affected by the weighting. This somewhat differs from findings by Vogel et al (2018), who constrain summer maximum temperature in this region to well below the ensemble mean, using the correlation between maximum temperature and precipitation as diagnostic. However, work from Stegehuis et al (2013) suggests that warming is underestimated by models in Northern and Central Europe and overestimated in the Mediterranean. We argue, therefore, that our approach might give a more robust estimate of the potential to narrow the uncertainty in this region, since it is based on a range of six diagnostics rather than on a single emergent constraint. It is, hence, not surprising that the Central European temperature distribution is not strongly affected be the weighting in our more conservative framework.

For the Mediterranean the weighting suggests that the unweighted multi-model mean strongly underestimates the potential warming in both periods. According to the weighted distribution, temperature increases of more than 2 °C are very likely already by the middle of the century when following RCP8.5. For the end of the century warming of over 5 °C compared to the 1995–2014 reference becomes increasingly likely.

Figure 2(b) shows that the best models receive up to tree times the median weight, except for the small grid cell regions Falun and Madrid with a maximum of 14× and 7× higher weights, respectively (see also figure S2(b)). The worst performing models in turn can get considerably lower weights more than 20× smaller than the median in several cases. Combining the results from figures 2(a) and (b) reveals that the effect of the weighting on the temperature distribution is a combination of the weighting strength and whether weights are correlated with simulated warming. The strong shift in the temperature distribution in the Mediterranean region is mainly driven by about 20% of the models which are heavily down-weighted for being far away from the observations in the historical period. For this specific region and target the majority of these models come from the GISS model family, which projects temperature changes on the lower end of the distribution. In Central Europe the weighting is equally strong but there is no clear clustering of weights for warmer or colder models so that there is hardly any effect on the temperature distribution. A list of all models included in the study and their corresponding weights in each of the regions can be found in figure S6 in the supplement.

The area-mean observational range for the Mediterranean (shown in black in figure 1) can cover large fractions of the interquartile model spread. This highlights the importance of accounting for observational uncertainty, particularly for diagnostics where many models lie within the observational spread. To investigate the effect of using multiple observational estimates we also calculate the weights based on each of the three used observational datasets separately. In most regions using only one dataset can lead to a shift in the mean which is not in line with the combined case (figure 3). This becomes even more important when considering smaller regions such as FAL, where using different observational estimates can lead to the weighting changes having different signs. Note that combining multiple datasets does not necessarily lead to a linear combination of the weighting effect. This can best be seen in the mean of NEU or in the 95th percentile of MED. We therefore stress the importance of using as many observational datasets as possible or feasible to best capture uncertainty in the observational estimates when evaluating historical model performance.

Zoom In Zoom Out Reset image size

To investigate the robustness of our approach to combine multiple observational datasets we also look into additional methods of combining the information from the three datasets (see supplement for a detailed description). As discussed in detail in section 4 one can even argue that the observations should also be weighted, but since the results from the different approaches tested are reasonably robust (figure S3), we limit ourselves to using the center of the observational spread here.

Precipitation changes are most notably affected by the weighting in Northern Europe (figure 4(a)): future precipitation increases are strongly constrained for both periods. This drives a reduction of the interquartile range by up to 50%. Similar to temperature, the effect on the precipitation distribution is mainly driven by a strong down-weighting of about 25% of models mainly from the CSIRO and IPSL families, which show a slight precipitation increase (figures 4(b) and S6). Looking into the absolute precipitation rather than the change reveals that the lower weighted models are predominantly dryer than the mean in the historical period and subsequently show a slightly stronger increase in precipitation than the mean under RCP8.5.

Zoom In Zoom Out Reset image size

In the combined European domain as well as Central Europe, the weighting leads to a reduction in the uncertainty of the projections of about 20% in the middle of the century as the most prominent effect. Using precipitation observations as a constraint this potential of significantly reducing uncertainty in future precipitation changes was also found by, e.g. Zhang and Soden (2019). In the Mediterranean the weighted distribution indicates a tendency towards even dryer conditions than the unweighted mean. Together with the results from temperature shown in figure 2(a) this suggests that models with higher temperatures and associated stronger drying are more reliable based on their past performance.

To evaluate the skill of the weighting approach we use the CRPS in model-as-truth experiments for the end of the century (2081–2100). Figure 5 shows the skill for three different cases: (i) weighing of the full ensemble, (ii) selecting a subset of the 10 best models (based on the same weights) with each model getting equal weight, and (iii) selecting a subset of the 10 best models where each models keeps its original weight.

Zoom In Zoom Out Reset image size

For temperature in the European domain the median skill increases by almost 50% due to the weighting. In the three SREX regions the skill also increases, while there is hardly any increase in the grid cell region (figures 5(a) and S5(a)). It is noteworthy that the skill increase is also observable in regions where there is no obvious change in the temperature distribution (see, e.g. CEU in figure 2(a)). In the 'models are indistinguishable from the truth' paradigm (Sanderson and Knutti 2012) this can be interpreted as the observations representing one realization which does not lead to a strong weighting. But other realizations (represented by model 'pseudo-observations') can lead to a weighting which yields a skill increase.

For all regions there is also a chance that the skill decreases due to the weighting. This can happen if the perfect model has a very different response to future forcing compared to the other models, leading to the weighted multi-model ensemble moving further away from the 'truth'. This represents the possibility that weighting models based on the distance to the observations may have negative impact on the skill of the future projections. To minimize this risk of overconfidently weighting the projections and giving too narrow uncertainty estimates we select the performance shape parameter σ_D also based on a model-as-truth test. Therefore, the probability for a decrease in the skill is small when using the full ensemble.

Comparing this approach to picking only the 10 best models (unweighted or weighted), yields similar results for the median CRPS change and for the interquartile range. For most regions the median performance is even slightly better than for the full weighted ensemble. Crucially, however, picking only 10 members can perform considerably worse in the lower percentiles. Selecting a subset of models has a higher chance of overfitting and can lead to a reduction in the CRPS by more than 75% for temperature.

Depending on the question it might, therefore, be sufficient to only use a subset of the best models. The median skill of the 10 weighted members is better than using the full ensemble for all regions and sub-selection naturally leads to a highly reduced spread, which can be valuable for certain applications. However, to minimize the risk of making the projections worse it can be important to use the full weighted ensemble. This is particularly crucial for applications where worst-case scenarios are important, e.g. when the goal is to explore nonlinearities in impacts.

For precipitation the above considerations are even more important with the 5th percentile of the skill metric being reduced by 75% and more in many regions when picking only 10 models. For Sibiu and Madrid selecting only 10 models even leads to negative median skill (figure S5(b)). Using the full weighted ensemble does also not lead to significant skill increases, but at least limits the maximum CRPS decrease to about 50%.

In general, the model-as-truth tests show that weighting precipitation projections only leads to increased skill in reasonably large regions. In the largest region, the combined European domain hardly any increase in the median skill is found, which can be interpreted as the region being too large to apply a single weight per model. The highest skill increases are achieved in the Northern European region, with a median increase of about 40%.