At the international congress of religious scientists, a specialist boasts of having cured his last five believers, who were suffering from an uncomfortable ailment, through prayers. His colleagues want to replicate these results in their own faithful in order to improve the future for all. Their interest in replicating the results justifies their label: religious scientists. 

The most critical congressmen ask their colleague: are you sure they had that disease? Are you really showing us all the cases or just the ones that suit you? Did no one do anything else to them? What was their natural, usual, expected evolution? Let's address these four interesting questions. To interpret results with certainty we need to: ensure that they met the initial characteristics; collect all cases; avoid confounding effects; consider whether good luck, chance, can explain the results.

1. Ensuring that the initial characteristics were met

It will not be reproducible, nor will it be of any practical use, if our religious person does not explain the initial variables of these five cases, the ones they had when he or she considered intervening to change their future. We must collect the information accurately, ensuring that each of them has the condition whose evolution we wish to change. It is these selection or eligibility criteria that will define the target population.

2. Collect all cases

It will not be reproducible, nor will it be of any practical use, if our religion only explains the five cases that have finally evolved well. Choosing cases on the basis of their evolution is called selection bias, of which there are many variants. Perhaps the clearest is when it is done intentionally, to induce an opinion; this is called selective information bias.

3. Avoid confusion of effects

It will not be reproducible, nor will it be of any practical use, if our religious man, in addition to praying, sends his patients to the Great Hospital. Well, yes, we will be able to predict that their evolution will be good, but not because of prayer. 

We will avoid confounding the effects by making the only difference between the two groups under comparison the cause under study. In this example, prayer.

4. Consider chance

It will not be reproducible, nor will it be of any practical use, if our religious man was just lucky. We have to talk about probabilities: "If it were true that the probability of cure without treatment was 50%, what is the probability that all patients would be cured? It is small, exactly 0.0315. This religious man has either been very lucky, or he has really improved that previous probability of 0.5.  

Now, if this congress has gathered a hundred specialists and all of them have seen five believers with this disease, it is to be expected that in total three colleagues have had the same results. There is nothing miraculous, not even extraordinary. 

We have seen four phenomena that could lead us to believe that prayer works. Actually, they are four reasons to study statistics. Or meta-research, which looks at the characteristics of studies that are most repeatable... and also those that are not likely to be replicated. 

Of course, another interesting question is: "Why do we want to believe? But that is a question for other experts.