|
Supplementary material to “Should We Assess Climate Model Predictions in Light of Severe Tests?”
7 June 2011
Joel Katzav, Eindhoven University of Technology, Eindhoven, Netherlands
Citation:
Katzav, J. (2011), Should we assess climate model predictions in light of severe tests?, Eos Trans. AGU, 92(23), 195, doi:10.1029/2011EO230004. [Full Article (pdf)]
1. Randall et al. do insist (2007, p. 596) that simulation accuracy should not increase confidence in models when the accuracy in question results from the direct accommodation of data, i.e. when it is the result of tuning a parameter for a certain quantity to observations of that quantity. But since accommodation is not in general direct, this insistence allows using accommodated data to support conclusions.
2. See Howson and Urbach (2006) for the standard argument for thinking that Bayes' rule does take into account test severity in assessing posterior probabilities. This argument does not address the observation that P(data) is the same for all values of F. For an extended discussion of the differences between the severe testing approach and the Bayesian approach see Mayo (2006).
3. For a case for adopting a severe testing approach in the broad context of computer simulation modeling see Parker (2008).
4. Knutti (2008) discusses some of the worries that arise about allowing confidence in CMPs to increase as a result of simulation successes that partly result from the accommodation of data.
5. Climate models predict an ongoing radiation imbalance at the top of the atmosphere, with more radiation supposedly being retained by the system than being lost. Now, it has been suggested that measurements of changes in the heat content of the upper ocean can be used to test whether the top of the atmosphere imbalance is indeed as predicted (Pielke Sr., 2008). And this test might be a severe test given the not implausible assumption that the precise changes in the ocean's heat content implied by the predicted top of the atmosphere imbalance would be unlikely just in light of theory other than that provided by the best climate models. But the proposed test would be severe only given the independent testing of auxiliary assumptions needed to derive observable implications about changes in upper ocean heat content from the prediction of the top of the atmosphere imbalance. For example, if the observable implication derived is that the heat content in the upper ocean should be increasing by a certain amount, the assumption that this additional heat content isn’t distributed in the deep ocean would have to be independently tested. If the assumption weren't independently tested, any observations of the absence of change in upper ocean heat content could easily be squared with the prediction of a top of the atmosphere energy imbalance. The extent to which we can specify severe tests for predictions such as the prediction of the energy imbalance at the top of the atmosphere, and thus the extent to which we have carried out tests of relevant auxiliary assumptions such as the one about the distribution of the ocean's heat content, reflect the maturity of the science underlying our projections about the climate system.
References
Knutti, R., (2008), "Why are climate models reproducing the observed global surface warming so well?" Geophysical Research Letters, 35, L18704.
Mayo, D. G. (2006), Error and the Growth of Experimental Knowledge, Chicago: The University of Chicago Press.
Parker, W. S., (2008), "Computer simulation through an error statistical approach", Synthese, 163, pp. 371-384.
Pielke Sr., R.A., (2008), "A broader view of the role of humans in the climate system", Physics Today, 61, 11, pp. 54-55.
Randall, D. A., et al. (2007), "Climate Models and their evaluation", in Climate Change 2007: The Physical Science Basis—Contribution of Working Group I to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change, edited by S. Solomon et al., chap. 8, pp. 589–662, Cambridge Univ. Press, New York.
Howson, C. and P. Urbach (2006), Scientific Reasoning: the Bayesian Approach, Chicago: Open Court.
—Joel Katzav, Eindhoven University of Technology, Eindhoven, Netherlands; E-mail: j.k.katzav@tue.nl