Originally Posted by Harryprice
Thinking about this I'm not sure this is that different to an assessment of a diagnostic test. Presumably with psi etc, one predicts the liklihood of a definite future event. Thus for example what will the next picture on a screen be. The chance event is therefore controlled and predictible (statistically). Yes you are assessing the divergence from chance, but there is an objective outcome to be measured at the end. This is identical to having a gold standard as measured.
To have a comparable situation to most drug trials, the outcome would have to have a subjective component. Thus for example if the parapsychologist were to try to influence an observers interpretation of an undefined shape, and one was then to assess by closeness to the intended interpretation or some other non binary outcome whether the intervention was better than a placebo intervention.
The first situation is one where rigour is possible and insisted upon in any field, parapsychology is no different to any other scientific field provided one only considers the lab condition experiments. Where it falls down is by incorporating data from uncontrolled experiments to support its case. This would be a bit like physicists having Cern at one end and guys in a shed at the other end both contributing to our theory of the universe.
Firstly, you can only say at the end of a parapsychology experiment that there is, say, only a 1 in 20 chance that the result was pure luck. Thus, you cannot say it is definitely paranormal, only likely to be. So it is not 'definitely is/definitely is not' binary situation. No solution changes colour, no lesion appears in a scan, as it might in a medical diagnostic test. You're still not definitely sure. Secondly, when it comes to picture tests in parapsychology you move away from forced choice into the realm of judgment. This is where things get very sticky and it is an obvious source of the experimenter effect.
That would never happen in a parapsychology test as it is full of obvious sensory leakage problems.To have a comparable situation to most drug trials, the outcome would have to have a subjective component. Thus for example if the parapsychologist were to try to influence an observers interpretation of an undefined shape, and one was then to assess by closeness to the intended interpretation or some other non binary outcome whether the intervention was better than a placebo intervention.
I disagree. If a psychologist did an experiment and the result was statistically significant, few other scientists would question it. They might try to replicate it but using the same experimental design. By contrast, if a parapsychologist got a similar significant result, there would be scientists crawling all over the design criticising it. I have no problem with this but it does mean that parapsychologists need to demonstrate a higher level of experimental controls than in comparable, uncontroversial fields.The first situation is one where rigour is possible and insisted upon in any field, parapsychology is no different to any other scientific field provided one only considers the lab condition experiments.
Firstly, amateur scientists DO contribute usefully to hard science. Until recently, astronomers relied on amateurs to spot novae, comets, new spots on gas giants and so on. Amateur naturalists contribute hugely to natural history, reporting unusual animal behaviour, changes in species distribution and so on. Their results are not generally challenged. Amateur science is alive and well and valued (and I think some amateurs are better than certain professionals, having read their papers!).Where it falls down is by incorporating data from uncontrolled experiments to support its case. This would be a bit like physicists having Cern at one end and guys in a shed at the other end both contributing to our theory of the universe.
Secondly, nobody equates strictly controlled lab parapsychology with ghost hunting! That may be the impression those outside the field might form but it is wrong. When parapsychologists talk science, they don't invoke poltergeists, just lab studies.
You will have to give an example before I could decide. My thinking would be that if someone claims to be able to predict the future, you show them random pictures of say happy or sad perople, they need to predict in advance which pciture is coming up. In advance you agree the parameters of what a happy or sad picture is. Each prediction therefore has a concrete outcome - but chance only comes in to the reliability of predicting these outcomes, no fuzziness about the outcomes them selves, though those who fail could after wards choose to disagree.
Fine, but in real medicine there are many outcomes with a subjective component - hence the importance of blinding.
Interpretations of the implications of the findings are disagreed with all the time, especially in something a fluffy as clinical psychology. The data quality tends to be analysed more if independent repitition is not possible or the claims go beyond that which the data support.
An amateur finding something that can be confirmed by others readily is no problem - an eyewitness account from an amateur with no corroberation will get no where. The interpretation of the implications is for the experts to agree or disagree with.
Not clear that such a distinction gets beyond the inner circle, bit like psychology then.
In your test, every trial has a definite yes/no answer but the final outcome of all the trials is a statistical probability. The results of just one trial are meaningless. There is always a chance, which gets smaller with increasing numbers of trials, that the result came about by pure luck. So the overall result is never yes or no but simply the odds that it was not luck.
I'm interested to know how you would agree in advance which pictures were sad and which happy. Would you show all the targets in advance to the participant or just a judge?
Parapsychologists, unlike skeptics, rarely test people who 'claim' anything. They usually use randomly selected test subjects. The idea is not to test an individual but whether the paranormal exists within the general population.
When only a few trials are possible, in the Ganzfeld method for instance, the subject may be asked to choose which randomly chosen 'emotionally evocative' (an attempt at amplifying the 'effect', as is Ganzfeld itself) picture was selected. If there were 10 pictures, it might appear they have a 1 in 10 chance of being correct. However, what if the pictures contain similiar or overlapping content (meaning the subject could select the wrong picture despite having really paranormally 'sensed' part of the right one)? What if some pictures contain things like houses or boats or similar subjects? These are objects that people often draw if asked to just produce a random picture, so there is obviously a psychological bias towards them. It is clear that the real odds are not 1 in 10 at all, so any statistical analysis based on them is meaningless. Indeed, it is almost impossible to work out what the real odds are.
Would you agree that parapsychology papers attract more criticism, in general, than similar ones on other fields? Look at the fuss caused by Daryl Bem's precognition paper, for instance. Would a similar paper in psychology, for instance, have attracted the number of critics that it has? And the reason parapsychology papers attract a lot of criticism? Bluntly, many scientists simply think any positive result is almost certainly a mistake!Interpretations of the implications of the findings are disagreed with all the time, especially in something a fluffy as clinical psychology. The data quality tends to be analysed more if independent repitition is not possible or the claims go beyond that which the data support.
An eye witness account from anyone, including scientists, must always to treated with caution. However, single witness accounts of unusual animal behaviour, for instance, which is difficult to reproduce, are definitely accepted from known reliable amateurs. For instance, the highly respected journal British Birds (generally considered the journal of record for UK ornithology) accepts such articles and is read by both amateurs and professional ornithologists. Reports of TLPs would also be readily accepted by professionals from known, reliable amateurs. TLPs are, of course, controversial but no one disputes they are seen, just what they are.An amateur finding something that can be confirmed by others readily is no problem - an eyewitness account from an amateur with no corroberation will get no where.
Last edited by Harryprice; 17th December 2010 at 12:52 PM.
I have stated this several times now. A trial with a gold standard outcome is necessarily more robust. This is distinctly different to most clinical trials where interpretation is required in respect of individual outcomes.
The conventional method would be to agree the characteristics in advance with the investigators.
Acceptable, helpful since the random person is less likely to be biased.
If you cannot decide apriori what a valid response is, then this is worse than clinical trials. There at least all agree on a valid outcome, just need to exclude bias in stating whether the conditions (previously agreed) have been met in this particular case. Since for your picture example the investigators are in a position to choose the test materials, it would be unacceptable not to agree what category of outcome they represent. This is not a surprise finding during a study it is part of the study material.
As I recollect this paper, there were a group of studies that suggested but by no means proved a significant association between predicted outcomes and results. Much of the furore was about the interpretation. Such studies are not capable of resolving issues of mechanism of effect, only presence or absence of effect. Because of the absence of power calculations, lack of fully transparent controls, single site nature of much of the study etc - they did not even reach the basic requirements in this regard, despite ostensible significance.
The problem here is going beyond the data and failure to recognise study limitations. If this was reported as suggestive of an unexplained effect, with proposals for more rigorous validation studies, and mechanistic studies, I think few would have got bothered.
Not exactly real science.
I suspect we're talking at cross purposes here. I don't see how your reply relates to what I said.
Again, cross purposes, I think. Just because a procedure is agreed between all involved it doesn't necessarily mean it is scientifically sound. See my comments below about odds.The conventional method would be to agree the characteristics in advance with the investigators.
There IS a predefined valid outcome - it is getting the right randomly selected picture from a set. The problem is in assuming the odds are really 1 in 10 when they are not. The actual odds depend on the psychology of the person selecting the target material and the psychology of the subjects. To get true odds of 1 in 10 you need 10 mutually exclusive targets, readily understood by the participant such as, choose a number between 1 and 10! There is a subjective element here, just like in drug trials. And these sort of problems calculating odds are not restricted to parapsychology.If you cannot decide apriori what a valid response is, then this is worse than clinical trials. There at least all agree on a valid outcome, just need to exclude bias in stating whether the conditions (previously agreed) have been met in this particular case. Since for your picture example the investigators are in a position to choose the test materials, it would be unacceptable not to agree what category of outcome they represent. This is not a surprise finding during a study it is part of the study material.
Interesting. So, do you think parapsychology papers receive less, more or about the same level of criticism that equivalent papers in another field of science do? And do you think there are scientists who think any positive result from a parapsychology study must be a mistake?As I recollect this paper, there were a group of studies that suggested but by no means proved a significant association between predicted outcomes and results. Much of the furore was about the interpretation. Such studies are not capable of resolving issues of mechanism of effect, only presence or absence of effect. Because of the absence of power calculations, lack of fully transparent controls, single site nature of much of the study etc - they did not even reach the basic requirements in this regard, despite ostensible significance.
The problem here is going beyond the data and failure to recognise study limitations. If this was reported as suggestive of an unexplained effect, with proposals for more rigorous validation studies, and mechanistic studies, I think few would have got bothered.
Astronomers and naturalists think it is.Not exactly real science.
Last edited by Harryprice; 17th December 2010 at 03:54 PM.
Not that I can see. A predefined outcome would be where you have decided in advance that only card x is a valid response. No further interpretation allowed thereafter, one either spots this or not.
The parallel in clinical medicine would be using myocardial infarction as an endpoint. An MI is defined as having occurred if two of the following three are observed: typical chest pain; typical pattern of enzyme rise or typical ECG pattern. The subjective issue arises where someone presents a slightly atypical pattern - this must be adjudicated before the trial is unblinded by independent judges that do not know the patient or their treatment regime. Once the treatment group is known no interpretation is allowed.
In your example you have decided that x is an acceptable answer, but that y or z might be an acceptable answer depending on the psychology of the individual being studied - however, once the card has been revealed the blind is effectively broken - so no further analysis should take place. What would be required is defining before the test was taken what would be acceptable answers or choices for that particular individual to have made and structuring the test such that with these modifications they would have the same chance as any other participant of choosing the correct card.
This is conflating two issues. No one has demonstrated that psi related phenomena are possible - so papers that do not address this issue cannot then claim to provide evidence for psi. Any such claims will of course lead to ridicule. It would be like doing a clinical study of high dose meditation versus placebo finding a significant result and claiming that this was evidence for the intercession of Zeus.
On the other hand if unexplained findings occur which appear to support prediction by as yet unknown mechanisms then while unlikely, this should only fuel research. Lacking a plausible mechanism will of course lead to skepticism, but reprodicibility of findings by independent investigators with reasonably transparent freedom of bias will lead to interest rather than cynicism. For example studies on bismuth for treating peptic ulcers, accepted before there was any plausible mechanism.
But your example was bird watching and postulated transient undocumented changes in moon colour - not mainstream astronomy to my mind.
Sorry but I can't think of any better way of explaining the odds problem. I guess I'm just crap at it, for which I apologise.
In science you don't have to explain a phenomenon before demonstrating it exists. You don't need to explain that the moon is 'possible' before publishing photos of it. Once it has been established that something exists only then is it science's job to understand why. It's the classic scientific cycle of observe then theorise then experiment and repeat. We're still at the 'observe' stage here!This is conflating two issues. No one has demonstrated that psi related phenomena are possible - so papers that do not address this issue cannot then claim to provide evidence for psi. Any such claims will of course lead to ridicule.
If a study shows that someone can predict a random outcome at a statistically significant level above chance then you have something to explain. The idea of a 'plausible mechanism' is completely irrelevant. All that matters in the first instance is showing that the effect is real. If it is, then it follows there must be a mechanism, whether anyone considers it plausible or not.On the other hand if unexplained findings occur which appear to support prediction by as yet unknown mechanisms then while unlikely, this should only fuel research. Lacking a plausible mechanism will of course lead to skepticism, but reprodicibility of findings by independent investigators with reasonably transparent freedom of bias will lead to interest rather than cynicism. For example studies on bismuth for treating peptic ulcers, accepted before there was any plausible mechanism.
It is perfectly correct for scientists to question studies that appear to show psi. They can criticise the experimental design, its implementation, the data analysis and so on. But saying there is no 'plausible mechanism' is not a scientific objection. If science only considered 'plausible mechanisms' then fundamentally new ideas would be automatically rejected.
Animal behaviour is central to natural history as is population dynamics. You can call it 'birdwatching' if you like but what do you imagine field naturalists do? And who do you imagine helps field naturalists in their studies? TLPs are not 'postulated', they have been documented by 'mainstream' astronomers. There are even 'plausible mechanisms', (not that it would matter if there weren't) such as meteor strike or outgassing.But your example was bird watching and postulated transient undocumented changes in moon colour - not mainstream astronomy to my mind.
Last edited by Harryprice; 17th December 2010 at 05:29 PM.
Looking back at the sentance, it appears as though the 'spot's it or not' is being directed at you. The intention was to suggest that the participant can find card x or not.
Agreed. But presenting a picture of the moon and claiming that this was evidence of orbiting cheese, would cause problems. There is considerable data to suggest that predictions of the future are most accurate when based on previous behaviour and optimal current knowledge. It is also known that betting on the sex of a future child was a money making undertaking for betting shops until the advent of ultrasound etc. So there is a lot of data available to suggest that there has to date been only false dawns in terms of precognition. So it is reasonable to first ask for high quality observational data before taking the next step of theorising.
Agreed up to a point. Statistical significance is firstly just a statment of improbability, not proof. Second the plausibility is not irrelevant, but relatively unimportant. If the effect can be shown to be real, then plausibility is irrelevant - but shown is a strong word. Somethings are so obviously so that no study is required - e.g. no study is required to demonstrate the benefit of parachutes. Some are evident and simply require basic confirmation - relationship between vehicular speed and injuries sustained on impact. Others seem likely or possible but need good evidence. Finally somethings having been investigated or postulated for years with only negative evidence to date - there overturning previous data requires not just evidence of association, but good reasons to reject all competing explanations.
Sort of - see above
Conceded.
I was only suggesting it was evidence for the existence of the moon.
It depends what you're talking about. It's true for a ball falling under gravity. It is questionable when predicting the future of a country's economy and it is completely wrong when predicting when an atom of a radioactive element will decay.There is considerable data to suggest that predictions of the future are most accurate when based on previous behaviour and optimal current knowledge.
I agree. What annoys me about parapsychological papers is that so many insist on theorizing about possible causes for their observations that usually mention quantum physics when a closer examination of more everyday causes would be more realistic.So it is reasonable to first ask for high quality observational data before taking the next step of theorising.
Didn't I say that say that earlier? There is no such concept as 'proof' in science (except in certain mathematical theorems) only provisional theories.Agreed up to a point. Statistical significance is firstly just a statment of improbability, not proof.
I said that the results of a study showing statistical significance required explanation. I didn't say that explanation had to be novel, though it could be.Second the plausibility is not irrelevant, but relatively unimportant. If the effect can be shown to be real, then plausibility is irrelevant - but shown is a strong word. Somethings are so obviously so that no study is required - e.g. no study is required to demonstrate the benefit of parachutes. Some are evident and simply require basic confirmation - relationship between vehicular speed and injuries sustained on impact. Others seem likely or possible but need good evidence. Finally somethings having been investigated or postulated for years with only negative evidence to date - there overturning previous data requires not just evidence of association, but good reasons to reject all competing explanations.
I would argue that nothing is entirely 'evident' in science. I think you should test everything, within reason, because you might be in for a surprise. Everyday common occurences can be baffling when closely examined. You can see a whirpool form in your bath when you pull the plug but the mechanics of this simple, common phenomenon are still not yet well understood.
I think I've already explained the problem parapsychology finds itself in. To show compelling evidence it needs to 'amplify' the effect, so that it is bigger and readily reproducible, which is what many studies attempt to do. But without any current theory to explain how to do such amplification, it's all guesswork, like testing compounds randomly to see if any can be useful drugs. And, of course, if psi doesn't exist then no such amplification is possible. This 'try it and see' approach often leads to the introduction of new variables in experiments which make data analysis more problematic. I think a whole new approach is needed but I'm not a parapsychologist so it's not my problem.
I don't know if psi exists or not. I think the current overall slight evidence in its favour is what you would expect if it was a data artefact produced by things like unintentional biases in experimental design. But it is also what you'd expect if there was a very tiny real effect. My biggest problem is I'm not convinced psi, even if it exists, produces reported experiences of the paranormal.
No argument there, same problem with homeopathy. I think people underestimate the problem with trial design. Even where the endpoint is indisputable, chance plays a role. In most trials interpretation of events also plays a role, so the impact of unconcious bias nudges the outcome closer to significance. Result - metanalaysis or pooling of studies overestimates the probability of a positive association supposedly mandating explanation.
Psi has been examined by rigorous research (and deeply flawed research) for ca. 150 years. In the absence of robust support for it, one must conclude it does not exist. You might like to read Susan Blackmore's book titled something like: "In Search of the Light." I admire her because she believed in psi and studied it for 20 years; but she demanded scientific rigor and finally concluded there was no good evidence for it.
Continued research in psi reminds me of a joke. An inveterate gambler goes to the movies every night even though his town only has one cinema that shows the same movie for a week. One week, the movie has a horse race in which the odds-on favorite (Cheesecake) loses. Late in the week, the guy sitting next to our hero leans over and says "I bet you $100 that Cheesecake wins" and our guy accepts the bet. After the race, our guy tries to hand-over the money; but the other guy declines it, saying he had already seen the movie and knew the result. Our guy replies "I have seen it five times myself; but I thought this time Cheesecake really had a chance."
That sums-up what continued psi research is about.
Posting Permissions
Reply With Quote
Bookmarks