How can you tell when a scientific claim is bad?
Look at the results. Compare the results from the models with what happened in real life.
An August 2010 study published in Science claimed that drought induced a decline in global plant productivity during the past decade, posing a threat to global food security. Zhao and Running, the authors of that study, set up their model based on their expectations that global plant productivity would continue to increase, as it had in the 1980s and 1990s.
A new study has found that Zhao and Running’s 2010 model was flawed.
… According to the new study, their model failed miserably when tested against comparable ground measurements collected in these forests. “The large (28%) disagreement between the model’s predictions and ground truth imbues very little confidence in Zhao and Running’s results,” said Marcos Costa, coauthor, Professor of Agricultural Engineering at the Federal University of Viçosa and Coordinator of Global Change Research at the Ministry of Science and Technology, Brazil.
What went wrong?
The authors of the original study included poor quality data and did not test trends for statistical significance. They also didn’t test their assumptions against real-life. There was a 28% disagreement between the model’s results and real-life results–far too much to make for a useful model!
So what’s the lesson from all this? Don’t trust scientists? Don’t trust models?
No. The lesson is that scientific progress is made when scientists question their own and each others’ assumptions about what they think should happen.
Could all of this have been avoided? Yes, if Zhao and Running had better tested their model against real-life to remove, as much as possible, their biases from their work.
Scientists, like all other humans, make errors. Question the basic assumptions of each claim, and see how the models hold up to a real-life test. That’s how you’ll know when you’re dealing with good science.
Learn some good science in the High-Adventure Science investigations on climate, water, and space.
http://www.sciencedaily.com/releases/2011/08/110825141621.htm