I am on a bit of a writing break at the moment – but felt I should very quickly respond to this report of a recently published scientific article. The Guardian’s headline is “Religious children are meaner than their secular counterparts, study finds” and it’s been published elsewhere, including the Daily Mail, Time and other esteemed outlets.

This is rubbish, and worthy of The Guardian’s excellent former ‘Bad Science’ columnist Ben Goldacre. Why?

1. Social science experiments such as this have limited application to the wider population, for a number of reasons. Psychological experiments often test issues in artificial ways that can’t be extrapolated to real life.
2. This particular study seems to have a serious design flaw: the use of the ‘dictator game’ as a measure of altruism, when it has been critiqued to be more a measure of susceptibility to peer pressure. See this post from John Baskette for details. Even if it was a good measure of altruism, the other problems with such experiments still apply, and experimenter demand effects are common confounding variables.
3. The fact that this study is in a biology journal, when it is a social science study, makes me wonder if it was rejected from journals where a rigorous and learned peer review would have taken place. Biology and psychology are very different disciplines, with very different experimental designs.
4. The children being tested were from USA, Canada, China, Jordan, Turkey, and South Africa. Are these countries representative of people as a whole? How did they select the religious children, from which kind of religious communities were they? As anyone who has faith will know, even within one particular religion, there is wide diversity of practice and belief. There is very little information on sampling – how the researchers chose the schools and children involved, which seems as though it would have a crucial effect. For example, if their USA school was one run by the Westboro Baptist Church, it’s likely that Christian children would not be representative of Christian children as a whole!
5. The researchers appear to have a particular axe to grind. They say at the end: “More generally, they call into question whether religion is vital for moral development, supporting the idea that the secularization of moral discourse will not reduce human kindness—in fact, it will do just the opposite.” To make such a politically loaded statement in the conclusion of a research article indicates a degree of experimenter bias.
6. The fact that the experimenters claim their findings ‘robustly demonstrate’ less altruism in religious children, and omit all the usual qualifications about generalising from experiments that are in good quality social science papers, again suggests experimenter bias and poor interpretation.
7. There are other experiments that could suggest that religious people are more altruistic. These are barely mentioned in the paper. Of course, such experiments also have limitations in how they can be generalised and should have lots of qualifications too. But they should surely be given more weight and discussion in a report that finds the opposite?

[Update] Since I wrote this, more learned souls have been pointing out other problems. George Yancey points out the lack of control variables, along with other similar criticisms; and William M Briggs points out problems with the statistical analysis. And atheists who have understanding of the social sciences are pointing out the problems with this study.

This article is so poor, that I’m sure I could find more flaws if I spent more time on it, but I’ve just whizzed through. It’s a shame that it is being taken seriously by media and public alike. Please add any other flaws you see in the experiment in the comments below.