Richard Tol Versus Richard Tol On The 97% Scientific Consensus
By Collin Maessen on comment
Last year Cook et al. released a paper in which they analysed the scientific consensus on anthropogenic global warming based via the peer-reviewed scientific literature.
What they did in that study was examine 11,944 abstracts from 1991 to 2011 that included the words “global climate change” or “global warming” in their abstract. What they found after analysing these abstracts is that among those that expressed a position on global warming, 97% endorsed the consensus position that humans are causing global warming. They also contacted 8,547 authors to ask if they could rate their own papers and received 1,200 responses. The results for this again found that 97% of the selected papers stated that humans are causing global warming.
For anyone aware about similar research this was not a surprising result as in 2004 Oreskes did a similar literature search – although it included ‘only’ 928 abstracts – which already found this scientific consensus. A 2009 survey of Earth scientists found that among climate scientists actively publishing climate research, 97% agreed that humans were significantly raising global temperature. A 2011 analysis of scientists’ public statements about climate change found that among those who had published peer-reviewed climate research, 97% accepted anthropogenic global warming. When you take a look at how this consensus evolved from 1996 to 2009 you see a steady increase in the agreement among scientists (Bray 2010).
The 97% consensus was found in Doran & Zimmermann 2009, Anderegg et al 2011 and Cook et al 2013.
This remarkable agreement exists because a scientific consensus is reached on the weight and amount of research that is available in the literature. It’s also this scientific evidence that led to the scientific consensus on for example evolution, plate tectonics, the big bang, germ theory, and so on. Such a consensus only arises through meticulous study and hard work by scientists.
This is also what makes it such a powerful tool to communicate science to the public. There’s a huge gap between what the experts say based on the scientific literature and the public awareness of this.
Which is also the reason the Cook et al. study is so relentlessly attacked by science deniers and pseudo-sceptics. It’s the only tactic they really have as they can’t base their case on scientific research, they just don’t have the supporting evidence to show that they are right. Most of the time they can only allude to nefarious going ons that prevent such evidence from getting into the literature. But that ignores that proving a well established scientific idea wrong advances the career and reputation of a scientist more than providing ground breaking supporting evidence.
The lack of evidence science deniers and pseudo-sceptics have shows via the methods they use for attacking the cook et al. study. Often they claim that not all needed data for verifying the results are available and/or that the study is deeply flawed. But both aren’t true.
How the abstracts were rated and how results were analysed are available in the paper. All the data needed to verify the results can be downloaded from the very same page this paper is hosted on (it’s linked under supplementary data). It contains everything you need to see if problems in the used methodology exist or that for example abstracts were incorrectly rated. Any serious problems in the Cook et al. paper will be detected by using that data and trying to replicate the results. But I already mentioned that to my knowledge science deniers and pseudo-sceptics aren’t doing that.
Even if they found problems in the abstract ratings in the Cook et al. paper it doesn’t invalidate the consensus they found. Simply because they also asked the authors of the selected abstracts to rate their own papers. The same consensus in the scientific literature was found via this method and confirmed the results the Cook et al. team found via the abstract ratings. But I rarely see any of the ‘critics’ even acknowledge this, when they do it’s often because they are dismissing it.
The growth of the scientific consensus on human-caused global warming in the peer-reviewed literature from 1991 to 2011, from Cook et al. (2013)
I expect this kind of behaviour from science deniers and pseudo-sceptics. I didn’t expect it from an economist like Richard Tol. Since the release of the Cook et al. paper Tol has made several statements about this paper being flawed. And has since been working on proving that this initial statement is correct. No matter how unwarranted or irrelevant his criticisms are towards the results found in the Cook et al. paper.
One of the more repeated ones is that data is hidden so he can’t check everything. But many, including me, have pointed out to Tol that he has everything he needs to check if the results are valid. And that so far there isn’t any real indication in the Cook et al. paper that it’s fundamentally flawed. As indicated by the agreement between the abstract ratings and the ratings done by the authors of the papers.
He also often speculates about a bias towards rating abstracts as endorsing the scientific consensus on anthropogenic global warming when they didn’t. However, one of the biggest hints that this didn’t happen are again the author ratings as they show that the raters were conservative with their ratings. Meaning that the ratings for the abstracts tended to er towards being neutral or rejecting that humans were the cause of global warming.
Another criticism Tol often talks about is fatigue influencing how abstracts were rated. Which is a real phenomenon in surveys, but the Cook et al. paper isn’t a survey. It’s not a paper based on people filling in a questionnaire, it’s about categorizing abstracts of papers. A tedious job, but that’s the reason they worked on this for months. They took their time to do this right.
One of the methods that was used is that every abstract was rated by at least two people. This was done as a person is fallible, they can make mistakes. So having at least two people do this reduces any issues that might arrise from that. And this is what the Cook et al. paper says about that particular process:
Initially, 27% of category ratings and 33% of endorsement ratings disagreed. Raters were then allowed to compare and justify or update their rating through the web system, while maintaining anonymity. Following this, 11% of category ratings and 16% of endorsement ratings disagreed; these were then resolved by a third party.
That’s a lot of work to make sure that they got their ratings right. As previously mentioned this shows through in the agreement between the abstract ratings and the author ratings, and in the detail that the abstract ratings tended to er towards being conservative. So any issues found in the ratings done on the abstracts will be minor and it won’t negate in any way the agreement found through the author ratings.
The reason all this so puzzles me is that Tol has said that “It is well-known that most papers and most authors in the climate literature support the hypothesis of anthropogenic climate change. It does not matter whether the exact number is 90% or 99.9%.” And that there’s no doubt in his mind “that the literature on climate change overwhelmingly supports the hypothesis that climate change is caused by humans. I have very little reason to doubt that the consensus is indeed correct.”
Yet he has said on twitter, and several other venues, that the found scientific consensus of “97% is a load of nonsense“. I know he says this because he thinks there’s a slight error in the found consensus percentage, but statements like this make it look like he’s saying that the entire paper is fundamentally flawed. It also leaves out the detail that he agrees with the general findings of the Cook et al. paper.
So why go after the paper in this way when he has all the data needed to verify the results? Fortunately he has explained why he’s doing it like this:
I have three choices:
a. shut up
b. destructive comment
c. constructive commenta. is wrong
c. is not an option. I don’t have the resources to redo what they did, and I think it is silly to search a large number of papers that are off-topic; there are a number of excellent surveys of the relevant literature already, so there is no point in me replicating that.that leaves b
When I read this explanation last year it left me dumbfounded, because option c is the one that you choose. Also rating all the abstracts again doesn’t have to be necessary, with a large enough sampling any big issues with the abstract ratings would be noticed. Everything Tol needs for a constructive criticism is already available to him.
In this case, considering he isn’t choosing option c, I think that option a was the correct one.
Featured comment
-
Here is another instance of Tol agreeing with the basic conclusions of Cook:
“Published papers that seek to test what caused the climate change over the last century and half, almost unanimously find that humans played a dominant role.”
Collin Maessen is the founder and editor of Real Skeptic and a proponent of scientific skepticism. For his content he uses the most up to date and best research as possible. Where necessary consulting or collaborating with scientists.
Yes, this is probably also why I find myself somewhat fascinated by this. I don’t understand why someone who you would expect to behave like a serious academic seems to be behaving like a typical science denier/pseudo-sceptic. Of course, as I learn more about this fascinating and contentious topic, there’s less and less that surprises me.
What confounds me the most is that he doesn’t disagree with what was found. He has said that on multiple occasions. He says that this is (mostly) about issues with the methodologies used for the paper. But if you look at the data this paper is based on and how the results were derived from it shows that the basic findings are solid.
Maybe more subtlety; nothing that I’ve seen him do actually shows that the ratings are wrong and that the consensus is incorrect. Everything I’ve seen is discussed in the original paper anyway. Inter-rate differences, other search terms, differences between the paper ratings and the abstract ratings. All I’ve seen Tol do so far is point out what was already known and then draw an entirely different set of conclusions from that.
I’m not sure if subtle is the right word for it. But yes, at the moment that’s also my impression from what Tol has done so far.
Don’t be fooled, it’s the Pielke two step. Step one attack climate scientists (never denialists), step two claim you agree with the climate scientists. When someone points out you are attacking the climate scientists say, well I agree with the science.
Here is another instance of Tol agreeing with the basic conclusions of Cook:
“Published papers that seek to test what caused the climate change over the last century and half, almost unanimously find that humans played a dominant role.”
http://andthentheresphysics.wordpress.com/2014/05/10/richard-tol-and-the-97-consensus-again/#comment-21162
I believe Tol’s sole purpose is to create the appearance of disagreement with the consensus.
The contrarian (ah, c’mon, plain vanilla denier) threads are full, even today, of “why won’t Cook show his data to Tol?” To amuse myself, I post the SkS link. For my troubles I am greeted with “obviously that’s not right or Tol wouldn’t be asking”. I then quote Tol stating “overwhelming consensus”. That usually results in a new denier meme being presented.
As a ruse, to create noise, it works. He must be worth what the GWPF pays to create froth from so little raw material.
Yes, climate science deniers indeed have been quite uncritical in accepting what Tol says, without even properly assessing if the data Tol is asking for is even necessary or relevant. Not that strange as this can help them maintain their perception that the consensus paper is fundamentally flawed.
However, please do not speculate what motivates Tol. We can only go on what he has said and done on this particular subject. I think it does reflect badly on Tol to be associated with the GWPF. But without anything indicating an involvement of the GWPF it’s unfair towards Tol to say that he’s doing this because of his link with the GWPF.
Science is not a set of results. It is a set of methods. If the methods are wrong, the results are irrelevant. Cook sought to estimate the degree of agreement on anthropogenic climate change in the literature, and he failed.
Not all data are available. In fact, less than half are. [snip]
I am not re-estimating the level of consensus because there are excellent, recent survey papers already.
And again you’re just repeating criticisms without properly explaining why you’re saying them or even acknowledging points raised by me.
One detail is that Cook didn’t fail with determining the consensus in the scientific literature. He used two different methods to quality check themselves and verify the results. The above article that I wrote explains this in detail and points to the indicators that they got this process right (which means they got their methodology right, which is also confirmed by the other excellent papers you mentioned). Ignoring that and not addressing it while repeating the same claims isn’t a valid response.
Also the only data that’s not available are which rater rated which abstract. This is completely irrelevant in determining if papers were correctly rated. This was withheld because it could be use to identify people and go after them personally. All the rest of the data is available and the total of what was released is more than the “less than half” that you claim.
“Cook sought to estimate the degree of agreement on anthropogenic climate change in the literature, and he failed.”
Er…. Cook’s conclusions and values matched all of the other known results that have been derived by independent surveys, using independent methodologies— within about 2%. If that is a failure, I cannot imagine what a success would have looked like.
Another way of looking at this is to see it in the context of the political amplifiers all around someone like Richard Tol at the moment.
Imagine I said to you “One of my academic colleagues can’t let go of a minor criticism of colleague X’s methodology, even though he agrees with the broad findings of the paper.” No doubt any of us in the social or natural sciences could easily think of other similar experiences. I have no doubt that there are hundreds of people in the world at any given moment who are critical of minor details of somebody’s paper, even though they agree with its conclusions.
What of course is different about this case, is that even the most pedestrian and peripheral criticism of a politically sensitive paper such as Cook et al, is immediately amplified and distorted. Even putting aside the obvious possibility that Tol is deliberately cultivating controversy, this means:
1. Tol’s comments are instantly exaggerated by the ‘anything-but-anthropogenic-climate-change’ (AACC) activist community;
2. Because of that politicisation, Tol himself is subject to the attacks of the pro-climate science (ACC) activists.
Tol thereby finds himself between one group of political supporters and one of political opponents. His public behaviour so far suggests he will be defiant towards those who he feels threatened by, which in turn simply increases the chances that his comments about Cook will become more contradictory, as the personal cost of his backing down increases.
The question is flawed. Few deny (3%?) that greenhouse gas emissions can contribute to global warming. What is in question is the degree [snip]
What you raise is directly addressed in the Cook et al. paper:
So everything that was marked as belonging in category 1 states this.
“The question is flawed. Few deny (3%?) that greenhouse gas emissions can contribute to global warming. What is in question is the degree [snip]….”
Yes, we know. The subject is Cook’s survey of the scientific consensus regarding human-caused climate change.
Richard Tol said “Science is not a set of results. It is a set of methods. If the methods are wrong, the results are irrelevant.”
Am I mad or does this not make sense? Science without results is meaningless just as a soccer match with no result is meangless. kicking the ball around but not counting the goals is pointless. Perhaps the professor meant “Science is not only a set of results. It is also a set of methods.”
«… among those that expressed a position on global warming …»
I polled 1,000,000 people. The 2 people who expressing a position, 100% agreed, “… among those that expressed a position on global warming …” is a [snip] way of polling.
What you’re proposing isn’t even relevant to the paper.The paper was a survey of the scientific literature, not a survey of opinions.
It also misrepresents why papers that didn’t state a position were left out, I’ll quote myself from a previous article that I wrote:
That’s why you only include papers that actually say something about what you’re trying to measure.
It appears to be that “MIB” is complaining about the survey only counting papers that mentioned climate change— even though the survey was specifically about climate change. I therefore cannot imagine what the complaint is about. Would “MIB” be happier if papers discussing automobile repair were also included?
“I polled 1,000,000 people. The 2 people who expressing a position, 100% agreed….”
As soon as any scientist does that, do be sure to complain, m’kay?