Scientific arguments are probabilistic.

[Dr B]

There are one or two debates going on around here (an elsewhere) about how ideas in science come into fruition, get accepted, and how it is difficult for new ideas to over throw existing ones. I thought I would make a small, important and tangential point here.

I think some people have odd ideas about science. What often amazes me is how many people (with no training / experience / or knowledge of science) seem to have quite firm views about it because they have read the firm views of someone else :-\

The arguments from creationism against science are excellent examples of how people totally misunderstand what science does and how it provides knowledge.

One thing I have not heard much of in these debates is that in science theories vary in scope. Evolution is a grand meta-theory. Others are less so. So when people talk about how science rejects new ideas - this is not true. To overthow a meta-theory one needs a good meta-argument. The evidence needs to be sufficient to support the claims (i.e., their scope) and then a viable challenge can take place. This takes time and science acknowledges that. The absence of evidence against God creating the universe is not sufficient to establish he / she did create the universe (as well as being a fallacy).

Another issue in science is that of arriving at the wrong conclusion. There are two main types of error in science. A type I error (saying something is significant when it is not - a kind of false positive), and a Type II error (missing a significant effect - saying something is not true, when it is - a false negative). There are also additional errors (Type III errors - but for clarity I am ignoring these here).

The important issue is that these errors can NEVER be fully eliminated. However, the likelihood of the error occurring is the crucial bit. Although these errors are typical talked about in relation to statistics and probabilities - they are of course relevant to the prevalence and support of theories and knowledge. Therefore, it is a probabilistic process in reducing these errors in thinking.

Finally, many scientists argue that they are more happy to make one type of error than another as it is less damaging in an ultimate sense. For example, finding a null effect (which may be a false negative) will often result in more tests and experiments where, if the effect is real, one would expect it to emerge under the right circumstances (thus showing that earlier research was a false negative). But the initial negative result is not as misleading as throwing your theoretical weight behind a false positive (Type I error) - which may lead to all sorts of spin off ideas, tests, etc that end up being a time-consuming, costly irrelevant tangent.

So although neither error is satisfactory - they do occur and some scientists (not all of course) have argued that one is more damaging than the other. This is why science is very cautious when new ideas seek to challenge existing knowledge. It is resisting making a Type I error (at least initially) because it is so damaging. However, if the new ideas truly are sufficient to over throw the existing models - then the likelihood of a Type I error is greatly reduced and science will freely move. Thus - it is a strength and not a weakness of science that any new idea meets some resistance on the way to becoming a provisional truth. O0

Apologies for the ramble :D
Comments welcome

[DaveWood]

When analysing investigations one if very frequently make Type 1 or 2 errors.

I'm always up front in favouring Type 2.

You say science can take time, yes it can. I guess people are concerned that if science is not sufficiently advanced to accept a possibility then it will fall be the wayside and then only re-emerge decades later. And on occasion when it re-emerges it's not scientists who are responsible.

Last night I was reading Oliver Sacks account of tourrettes syndrome. His contention was that because neurology was incapable of dealing with such conditions of excess that they ignored them and therefore the condition appeared to 'disappear'.
Decades later recognition came mainly because of a patients group championing the cause?

[Dr B]

There was once a time when consciousness was a dirty word in psychology. No one denied it existed and no one denied its importance. The problem was, no one knew how to study it in a reliable, acceptable and valid way. Science aired on the side of caution instead of plodding ahead with irrelevant tangents.

Today consciousness is a hot topic and heavily researched around the world. O0

Note - science did not deny the existence of consciousness - just its role in primary function. However, while the behaviourists were claiming to be the dominant psychological force (everything is just an S-R relationship - consciousness is kind of irrelevant) others (like Piaget) had being doing cognitive science for decades. So the idea that 'the bit in the middle' i.e., conscious thought and reasoning was ignored by science is not strictly true. A dominant paradigm in a science does not mean other paradigms do not exist in their own right and may well emerge in the future - based on evidential merit O0

[Admin]

This topic can be a hard one to explain. It’s often difficult to explain types of error, p values, the null hypothesis, etc.

A great analogy, however, is to compare the testing of say drug X against a placebo or an ESP test to expected chance values to an accused in a court of law.

Stay with me…. ;)

1) In law, an accused is assumed to be innocent until proved guilty.

In statistics, when comparing for a difference, the assumption made is that there is no difference between the values being tested. This is known as the Null Hypothesis. The null hypothesis will be held as true unless there is good reason to reject it, just as an accused will be considered innocent unless there’s good reason not to.

2) In law, guilt has to be proved beyond reasonable doubt.

In statistics the same holds true. A level of significance will be set and only if results go beyond this critical value will they be deemed significant. A common level is 5% (this is where the p=0.05 comes from). Basically this figure means that the odds of the result being a false positive (i.e. occurring by chance) are 1 in 20.

This is similar to setting a reasonable doubt level. We can never be 100% certain in statistics (or law I suppose) so we need to set a level that is robust enough for us to reject the null hypothesis (or presumed innocence) and favour the alternative hypothesis (or guilt).

The significance level can be chosen by the researcher so it’s tempting to make the significance level as low as possible. If a test is positive at p=0.01 then it is less likely to have occurred by chance than at p=0.05 (1% versus 5%). So why not use p=0.00001 or less?

There are other problems in choosing significance levels.

3) Finding an innocent party guilty.

This is a Type I error. If we’re too keen to accept evidence and find someone guilty we’re in danger of convicting an innocent person. In statistics the equivalent is setting the p value too high. If we choose a significance level of say p=0.2 then there’s a 1 in 5 chance that the positive result is a fluke. This means that we’ll end up accepting meaningless results as significant and reaching false conclusions.

4) Setting a guilty person free.

The answer to avoiding Type I errors is to set the p value as low as possible. This means that the chances of accepting a chance result as significant are minimised. A p=0.0001 means that there’s only a 1 in 10,000 chance of making a Type I error. Unfortunately this introduces the possibility of another error: Type II.

A Type II error is where a significant result is wrongly rejected. This can occur by setting the p value too low. In court, it may be that the jury know that eye-witness testimony is not very reliable and are not prepared to convict on eye-witness testimony alone; but, if there are 12 independent eye-witnesses that identified the accused and the jury still refuse to convict then they are in danger of a Type II error and setting a guilty party free.

_________________________________________________

I hope that analogy gives a reasonably good layman’s terms overview of the subject. The bottom line is that setting a significance level is arbitrary and the level at which it is set determines whether we accept things as significant or not.

It’s a balancing act. If we’re too keen to accept alternative (to the null) hypotheses then we’ll draw a lot of false conclusions; if we too keen to reject alternative hypotheses, we’re in danger of missing real phenomena should they occur.

Scientists tend to err on the side of caution and would prefer to make Type II errors rather than Type I.

[Dr B]

Nice examples John.

The main thing i wanted to highlight here is that, in contrast to popular perception, science acknowledges sources of error and the fact that they can never be truly and fully ruled out. Therefore, science is probabilistic - it is about the most likely explanations.

Also - in your examples dont forget effect-sizes!!!! You can have a significant effect with a p-value yet only be explaining a small amount of the effect!!!!!! Very important with complex effects.

Or and Type III errors....arriving at the right conclusions for all the wrong reasons....

The bias in the types of errors scientists prefer i think underlies some of the publics perception in science rejecting ideas in a 'closed-minded' ;D fashion. Often, the public do not understand that the quality of evidence needs to be good (and not just the quantity of it). O0

I like a boxing metaphor. To be world champion you have to beat the current champion. In the event of a draw the current champion remains in place.

For a new theory to beat a current theory it has to do a better job than the current one. Being just as good is not, on its own, sufficient. When a new theory beats a current one - it will generally replace it. O0