Smerdon, J.E., A. Kaplan and D. E. Amrhein, 2013 (in press): Reply to comment by Rutherford et al. on "Erroneous Model Field Representations in Multiple Pseudoproxy Studies: Corrections and Implications", Journal of Climate. (pdf)
The above Reply and associated Comment were motivated by our 2010 paper that identified and discussed errors related to the processing of model data used in pseudoproxy experiments. The erroneously processed model data were made publicly available and affected the quantitative results in at least seven published papers. Since the publication of our 2010 paper, Rutherford et al. submitted a Comment on the work that has gone through a series of iterations. Because the original comment was made publicly available and was also discussed in variouscorners of the blogosphere, we have chosen to chronicle the original Comment/Reply exchange and all of our subsequent resposes to the successive versions of the Rutherford et al. Comment. For context, the Comment/Reply exchange was rejected after two iterations and then the Comment was resubmitted. After resubmission, the exchange again went through various iterative revisions. Hence, all of our Reply versions are linked below.
One of our principal motivations for making all of the versions of our Reply public stems from the fact that the original Comment asserted that a bug was present in the GMT software package. We were not able to confirm any such bug and this assertion was removed from subsequent versions of the Rutherford et al. Comment. This assertion nevertheless was made publicly and lives on indefinitely in blog discussions. GMT is a widely used opensource software that is solely dependent on its public reputation. We therefore believe it is important to provide our rebuttal of the original assertion publicly, which is available in Reply versions 1 and 2 above. In addition to the discussion that we provide in these versions of our Reply (reproduced below from Reply v. 1), the data for the figure panels that we provide can be accessed by clicking on the panels provided at the end of this page. A tar file containing the scripts to access and process the data using GMT as described in Reply versions 1 and 2 can be downloaded here.
Excerpt From Reply version 1 (lines 63-96) on purported GMT bug:
R10 also offer an explanation for the incorrect processing of the GKSS field in the M07 paper by claiming that a "bug" exists in the Generic Mapping Tools (GMT) software (Wessel and Smith 1991). This purported bug produces erroneous fields when the GMT surface function, which fits a continuous curved surface to randomly-spaced data, is employed using its default tension setting. If this observation is correct, it would be a valuable piece of information for a wide community of GMT users. Nevertheless, we cannot confirm any signs of such a bug in our own experiments with the GMT surface function and the peculiar nature of the error affecting the M07 processed GKSS field — namely the selective smoothing of a single hemisphere — makes the claim by R10 seem untenable. In fact, our own experiments provide a simpler and more plausible explanation.
We illustrate our findings using the GKSS annual surface temperature mean from 1880–1990 C.E., in keeping with S10. Figure 1a shows this field averaged by S10 onto a 5° spatial grid, but still in its native longitude range (0°–360°). For simple illustration purposes, we apply the GMT surface function to the field shown in Figure 1a using a default tension setting (tension = 0; this setting can range from 0 to 1), which yields the correctly gridded version of the field shown in Figure 1c with the longitudinal range changed to -180°–180° (note that we do not endorse the use of the surface function for the purpose of regridding fields in general, but we consider it here because it is at the heart of the M07 regridding procedure). This result was accomplished using a flag -fg in the call of the surface function to ensure that the spatial grid was interpreted as geographic coordinates and not as regular numbers. If the latter interpretation is made due to the absence of the -fg flag, however, the surface function will regard the input points with longitudes <0° as unavailable because the input data range from 0° to 360°. Consequently, the Western Hemisphere (WH) will be interpolated with a continuous curved surface anchored only by the points on its eastern boundary. The resulting field is shown in Figure 1d and has a striking resemblance to the M07 product shown in Figure 1b (Figure 2 replots Figures 1b and 1d over the range 0°-360° and clearly illustrates, in both cases, the effect of the anchoring of the WH on its eastern boundary and the discontinuity of the boundary at 180°). Furthermore, we find no evidence to support the dependence of these results on the tension setting of the surface function. In Figures 1e and 1f, we plot correct and incorrect results for a prescribed tension setting (0.5) that do not differ from their respective counterparts using the default tension in any substantial way. These findings thus suggest a misuse of the GMT surface function by M07 as the origin of the errors in the regridded GKSS field, rather than the existence of a hypothetical bug that only occurs at the default tension setting and only affects one hemisphere.
Finally, a discussion about validation statistics provided for the Niño3 region in the Mann et al. (2007) paper was ulitimately removed from Reply version 5 to establish the final version of our Reply. We nevertheless feel it is important to draw a distinction between the Niño3 validation statistics provided for the field reconstruction in Mann et al. (2007) and the new results highlighted by Rutherford et al. in the Emile-Geay et al. (2013) papers, which specifically target the Niño3 index and not the global field as in Mann et al. (2007). The relevant discussion from Reply version 5 highlighting the important difference is reproduced below.
Excerpt from Reply version 5 (lines 49-64) on Niño3 validation statistics:
Regarding the M07 Niño3 assessment statistics, R12 point to two papers in review (Emile-Geay et al. 2012a,b) that seek to reconstruct the Niño3 index by applying RegEM-TTLS to an expanded data set tailored for tropical Pacific sea surface temperature reconstructions. These papers only reconstruct the Niño3 index; they do not perform a hemispheric or global CFR. Testing the performance of RegEM-TTLS for global CFRs was the motivation of M07, who used reconstruction skill from the Niño3 region as a spatial validation measure. Mann et al. (2009a) and Mann et al. (2009b) subsequently used RegEM-TTLS to derive real-world global CFRs, from which Niño3 indices were derived and used to infer ocean-atmosphere dynamics or to make quantitative calculations of Atlantic hurricane counts over the last millennium. More recent efforts to reconstruct the Niño3 index exclusively, without reconstructing the entire global field, are therefore not relevant to the way in which the Niño3 index was used in M07. Despite these distinctions and the importance of the Niño3 validation statistics in previous papers, no subsequent publications, including the present R12 comment, have corrected the erroneous statistics from M07. One consequence of this omission was a confusing disparity between the Niño3 reconstruction skill in the M07 CCSM1.4 and ECHO-G experiments prior to the publication of S10.
Figure 1: (a) Average of the mean annual GKSS surface temperature field for the 1880-1980 C.E. period from S10; (b) same as (a) but for the version regridded, used and archived by M07; (c) GKSS surface temperature field derived by correctly applying the GMT surface function using the default tension setting of 0 (the longitude range has been changed to -180 –180 as in panel (b)); (d) same as (c) but without the -fg flag in the call of the surface function, resulting in large-scale smoothing of the WH due to the loss of all WH data; (e) and (f ) are the same as (c) and (d) respectively, but for a tension setting of 0.5 (the tension can range between 0 and 1).
Figure 2: Panels (a) and (b) are for the same data as those in panels (b) and (d) in Figure 1, but for the longitudinal range 0–360 to show the anchoring of the smoothed WH on its eastern boundary and the discontinuity of the field at 180: (a) M07 processing; (b) resulting field after application of the surface function to the field in Figure 1a while omitting the -fg flag.
Please note: Access to the Ingrid-based Data Catalogs is via port 81 of our data server. Some firewalls disallow access to any ports other than the default (port 80). If this is the case, you will see these links hanging forever upon clicking the link or will get an error message. Possible solutions are to change your firewall settings to allow access to port 81, or access the links from outside of your firewall.
This material is based upon work supported by the National Science Foundation (Grant: ATM-0902436) and the National Oceanographic and Atmospheric Administration (Grant: NA07OAR4310060).