Confirmation: an error of reasoning.

John Jackson — Sun, 01 Feb 2009 10:06:36 +0000

John Jackson © UK-Skeptics

C onfirmation bias is where we look for reasons, or supporting evidence, that matches our belief or thesis whilst disregarding or placing less weight on disconfirming reasons or evidence against it.

As a hypothetical example, let’s look at someone’s belief that hypnotherapy helps people to stop smoking. People do go to hypnotherapists and subsequently give up smoking and there are many people who will anecdotally state that hypnotherapy worked for them. It seems convincing, but is this proof that hypnotherapy really helps people to give up smoking?

If we only consider positive outcomes for our hypothesis, what we’re doing is introducing biases known as selective attention (seeing only what we want to see) and suppressed evidence (avoiding what we don’t want to see). Of course the problem is that using a biased data sample will most likely result in a false conclusion.

Statistics need context to be meaningful. Positive outcomes need to be compared to negative outcomes to give a success rate, and in turn, that success rate needs to be compared to something else to provide context. That could be a competing hypothesis or compared to doing nothing (as a control comparison).

Many problems can be analysed using a simple table like this one:

	(A) Gave up smoking	(B) Failed to give up smoking
(1) Used hypnotherapy	30	70
(2) Did not use hypnotherapy	45	105

Here we’re counting the number of people who used hypnotherapy and gave up smoking but also the number who used hypnotherapy and failed to give up smoking. Then we compare that result to a group of people who did not use hypnotherapy.

Even when faced with data in this form, people who are asked, “does hypnotherapy help people to give up smoking?” will look at the A1 Square and conclude that it does. This happens even with hypotheses we have no interest in because we have a natural tendency to look for confirmatory evidence. Seeking out and being influenced by confirmatory evidence is a human predisposition.

When comparing different sample sizes, we can find the percentage of success to failure in both instances and then compare the results. This can be done in the following way:

Used hypnotherapy :	A1 ————- [A1+B1]	30 ———— [30+70]	= 0.3 (30%)
Did not use hypnotherapy :	A2 ————- [A2+B2]	45 ————- [45+105]	= 0.3 (30%)

As can be seen from the figures, 30% of people who go to a hypnotherapist manage to give up smoking; however, when we give that figure context by comparing it to those who did not go to a hypnotherapist, we find that they too were successful 30% of the time. There is no difference between the two groups; the net benefit from using hypnotherapy is zero.

This example is hypothetical but the model is what is important. Whether looking at psychic “hits”, alternative remedies, whether couples are more likely to get pregnant once they give up on the idea, or that bad things happen in threes, looking only at confirmatory evidence will lead to false conclusions. It’s not unless disconfirming evidence is considered and the hypothesis under consideration is compared to something else can we state whether it is true or not.

Conclusion.

Seeking out and being influenced by confirmatory evidence is something we do naturally. This leads to what I call the “A1 effect”: where people can be influenced, often quite strongly, by information that they already believe is true or would like to be true (information that sits in square A1 in the table).

This is one reason why anecdotal evidence can be so influential. A person who is considering using hypnotherapy to give up smoking, for example, may find a lot of evidence against the method, but if even one single person says, “I tried it and it worked for me” then that can be evidence enough: the A1 effect.

We’re all prone to confirmation bias. Understanding the fact however, can help us reach conclusions that are true by analysing issues properly and not simply seeing what we want to see.

UK-Skeptics articles and commentary » confirmation

Confirmation: an error of reasoning.