Mod+ Johann and Maaneli's recommendations for parapsychology research

#1
This is my first test of the idea of having mod+ threads in CD. For an explanation of what this means, see here:
http://www.skeptiko-forum.com/threads/does-it-matter.1240/page-5#post-33913

Johann and Maaneli (Baptista and Derakhshani) wrote a paper which will be published in the Journal of Parapsychology and was presented at a recent parapsychology conference and is available here.
http://www.psy.unipd.it/~tressold/cmssimple/uploads/Meta_Baptista14.pdf

They make the following recommendations in order to necessitate analysis and replication from the mainstream...
Produce replicable experiments.
Increase the power of the ganzfeld experiments by increasing the effect size, rather than sample size.
Reinvest in forced-choice experiments and increase the power by increasing sample size.
Pre-register studies in a venue such as the KPU Registry.
Follow the methodological and statistical guidelines suggested by Utts and Tressoldi (http://www.parapsych.org/blogs/patrizio/entry/49/2013/2/methodological_and_statistical.aspx#comments).
Implementing two of Kennedy's recommendations from http://jeksite.org/psi/misconduct.htm -
'registering a multiple-experimenter protocol with independent copies of study outcomes, so as to prevent tampering; and providing the raw data for analysis by others after a study is completed'.

What strikes me about these recommendations is that they have been made previously, and some are well over ten years old, and I wanted to get a sense of how likely they are to change the field.

Kennedy wrote an article in 2004 called "A proposal and challenge for proponents and skeptics of psi", which Caroline Watt picked up on in her presidential address at the 2005 PA convention.
http://jeksite.org/psi/jp04.htm
http://www.koestler-parapsychology.psy.ed.ac.uk/Documents/RevisedAddress.pdf

In his article he brings up the idea of performing pivotal experiments which are well-designed and sufficiently powered to draw conclusions without the need for post-hoc meta-analysis. And as Watt points out, he is not the first to do so. Similarly the idea of having studies pre-registered goes back for over 20 years, yet we are just now seeing the creation and use of the KPU registry. And as far as I know, none of the journals have yet tied pre-registration to publication.

So my question to those people who are familiar with the parapsychology community is, given that the suggestions made by Johann and Maaneli have already been made for years, is there hope that the field will make use of these recommendations? Johann, what sort of feedback have you received in this regard?

Linda
 
#3
Question: ¿why is it so vitally important to pre-register the studies?, ¿can a registration or a non-registration change the force in which the evidence is moving (from an effect to a non-effect)? if yes, ¿how?.

If I understand correctly, Linda works within the medical academic circles, so I would highly appreciate a deep and specific answer in case she wants to provide me with one, but I'm open to anyone who knows about this stuff replying.
 
Last edited:
#4
The KPU registry has been available since October 2012. There are nine studies registered.

~~ Paul
I'm not particuarly interested in a big chunck of the Ganzfeld studies since I found out they are almost invariably with a low power (40 trials when you need at least 100 ). I'm however more interested with the Ganzfelds that have selected subjects. I've read some that are quite big in size and also give a highly statistical desviation (40%). However, last time I asked Kennedy, he told me just 1 of this studies ever hit a pre-registration at KPU, which is sad since it seems to be (for reasons I don't fully understand currently) like the maximum ultra-mega-hyper riguruous stuff possible, which is also used in pharmacology (that seems to take their protocols to the next level by having a highly strict protocol before accepting any data as "correct").

Psychology, which is (as I was told by Kennedy; and I trust his word given his track in the field) where parapsychology draws it's methodologies and protocols is somehow deemed as having inferior protocols, for some reason, and they usually dont' engange in pre-registration.
 
#5
Question: ¿why is it so vitally important to pre-register the studies?, ¿can a registration or a non-registration change the force in which the evidence is moving (from an effect to a non-effect)? if yes, ¿how?.
Yes. Lack of pre-registration allows you to pick what combination of factors and subjects you are going to use to identify an effect after the fact, so you can pick the combination which happens to show an effect due to chance. For example, let's say you did 45 ganzfeld trials with various students and you noticed after the fact that the students who came from one particular major (or the female students, or students with prior psi experience, or students who drove red cars) had a higher hit rate. Without pre-registration, you could report it as an experiment done on 20 students from a particular major with a statistically significant hit rate, rather than an experiment done on 45 students without a statistically significant bit rate.

Linda
 
#6
I'm not particuarly interested in a big chunck of the Ganzfeld studies since I found out they are almost invariably with a low power (40 trials when you need at least 100 ). I'm however more interested with the Ganzfelds that have selected subjects. I've read some that are quite big in size and also give a highly statistical desviation (40%). However, last time I asked Kennedy, he told me just 1 of this studies ever hit a pre-registration at KPU, which is sad since it seems to be (for reasons I don't fully understand currently) like the maximum ultra-mega-hyper riguruous stuff possible, which is also used in pharmacology (that seems to take their protocols to the next level by having a highly strict protocol before accepting any data as "correct").

Psychology, which is (as I was told by Kennedy; and I trust his word given his track in the field) where parapsychology draws it's methodologies and protocols is somehow deemed as having inferior protocols, for some reason, and they usually dont' engange in pre-registration.
I think you're referring mostly to the Dalton trial. Since the Storm et al. MA, parapsychologists have slightly lost interest in using Ganzfeld results to support any psi hypothesis. The new flavor of the month is presentiment.

It's very hard, with the Ganzfeld, to pin down what you're actually studying. Is it telepathy? Is it precognition? Is it presentiment? It just seems aimed generally at the hypothesis that humans can access memory unknown to them by their current senses. Presentiment studies actually tests for a specific hypothesis; can humans use future knowledge to dictate current choices. Because such an experimental set up reduces potential bias far in excess of Ganzfeld protocol, parapsychologists feel they'll get farther.
 
#7
Yes. Lack of pre-registration allows you to pick what combination of factors and subjects you are going to use to identify an effect after the fact, so you can pick the combination which happens to show an effect due to chance. For example, let's say you did 45 ganzfeld trials with various students and you noticed after the fact that the students who came from one particular major (or the female students, or students with prior psi experience, or students who drove red cars) had a higher hit rate. Without pre-registration, you could report it as an experiment done on 20 students from a particular major with a statistically significant hit rate, rather than an experiment done on 45 students without a statistically significant bit rate.

Linda
Wouldn't that amount to say parapsychologists lie?
 
#8
I think you're referring mostly to the Dalton trial. Since the Storm et al. MA, parapsychologists have slightly lost interest in using Ganzfeld results to support any psi hypothesis. The new flavor of the month is presentiment.
Well, to the Dalton indeed, and others. What I find problematic is that I've never seen a metaanalysis focused only on selected subjects, and it seems many studies are usually left away in Ganzfeld MA in general, so I'm not sure how to approach the subject.

It's very hard, with the Ganzfeld, to pin down what you're actually studying. Is it telepathy? Is it precognition? Is it presentiment? It just seems aimed generally at the hypothesis that humans can access memory unknown to them by their current senses. Presentiment studies actually tests for a specific hypothesis; can humans use future knowledge to dictate current choices. Because such an experimental set up reduces potential bias far in excess of Ganzfeld protocol, parapsychologists feel they'll get farther.
¿Whats your opinion on the quality of the Ganzfeld, appart from the problem of pinning down what form of psi alledgedly shows, and do you think pre-registration is needed to conclude anything from them?
 
#9
Well, to the Dalton indeed, and others. What I find problematic is that I've never seen a metaanalysis focused only on selected subjects, and it seems many studies are usually left away in Ganzfeld MA in general, so I'm not sure how to approach the subject.
We had a thread in the last forum where Erbsy, Johann, Maaneli and I went through and added some other studies that were left out of the recent Storm et al MA.


¿Whats your opinion on the quality of the Ganzfeld, appart from the problem of pinning down what form of psi alledgedly shows, and do you think pre-registration is needed to conclude anything from them?
Pre-registration doesn't really hurt. I'd like to see more sessions run per experiment to increase power, with selected individuals. This seems to be the suggestion from Maaneli and Johann as well. I'm not sure a lack of pre-registration gives us any cause to doubt the post communique findings. I also don't think the Ganzfeld experiment by itself gives us enough reasonable confidence in psi. However, there has seemed to be some subtle suggestions that treat Ganzfeld as the best research paradigm out there. Post presentiment MA, I'm not sure that's the case.

But like you said, unless you're accusing parapsychologists of lying...

EDIT: Also note that parapsychology is pretty unique in being one of the only fields that publishes negative findings.
 
Last edited:
#10
Yes. Lack of pre-registration allows you to pick what combination of factors and subjects you are going to use to identify an effect after the fact, so you can pick the combination which happens to show an effect due to chance. For example, let's say you did 45 ganzfeld trials with various students and you noticed after the fact that the students who came from one particular major (or the female students, or students with prior psi experience, or students who drove red cars) had a higher hit rate. Without pre-registration, you could report it as an experiment done on 20 students from a particular major with a statistically significant hit rate, rather than an experiment done on 45 students without a statistically significant bit rate.

Linda
I would also like to note that we have our first violation of the Mod+ rules on this forum, as listed by Linda:

Lying
Implying disingenuousness (e.g., liar, dishonest, not trustworthy)
How does one go about reporting it?
 
#12
Wouldn't that amount to say parapsychologists lie?
I think you will find that the researchers wouldn't regard that as lying. From their perspective (and this applies to any researchers, I'm not just talking about parapsychologists), there is a real effect which they are trying to elicit. So it is regarded as reasonable to carefully go over the data from your study and try to find where and how the effect was elicited. The 20 students would be the group in which the experiment "worked" and the other 25 the group in which it didn't. And scientists in the field (especially a field which is trying to break new ground) will be more inclined to focus on when their experiment worked. And usually the changes or choices made to follow an inclination are more subtle than the more dramatic example I gave (the paper on flexibility in outcomes and the chapter on "bias" in the Cochrane Handbook describe the various ways in which researchers, thinking that they are acting honestly, can change the results from "no effect" to "effect").

Now, it's true that these kinds of activities cause problems, because they end up increasing the numbers of false-positives. But the researchers aren't trying to create false-positives, they think they are creating true-positives.

However, this raises an interesting question for anybody/everybody to answer. Why did Johann and Maaneli (and a long list of other parapsychologists) recommend using a registry? What is it for?

Linda
 

Paul C. Anagnostopoulos

Nap, interrupted.
Member
#13
However, this raises an interesting question for anybody/everybody to answer. Why did Johann and Maaneli (and a long list of other parapsychologists) recommend using a registry? What is it for?
If everyone registered every study, it would eliminate the file drawer. The problem is knowing whether everyone did, indeed, register every study.

~~ Paul
 
#16
I would also like to note that we have our first violation of the Mod+ rules on this forum, as listed by Linda:



How does one go about reporting it?
After reading Linda's reply to my question, I don't think she is saying they are lying. It seems more to be a sort of confirmation bias, which is different from someone lying, since in confirmation bias you actually think you are unto something, when you are actually just playing tricks upon yourself. Lying implies conscious knowledge of something that isn't true, but still trying to pass it as true.
 
#17
I think you will find that the researchers wouldn't regard that as lying. From their perspective (and this applies to any researchers, I'm not just talking about parapsychologists), there is a real effect which they are trying to elicit. So it is regarded as reasonable to carefully go over the data from your study and try to find where and how the effect was elicited. The 20 students would be the group in which the experiment "worked" and the other 25 the group in which it didn't. And scientists in the field (especially a field which is trying to break new ground) will be more inclined to focus on when their experiment worked. And usually the changes or choices made to follow an inclination are more subtle than the more dramatic example I gave (the paper on flexibility in outcomes and the chapter on "bias" in the Cochrane Handbook describe the various ways in which researchers, thinking that they are acting honestly, can change the results from "no effect" to "effect").

Now, it's true that these kinds of activities cause problems, because they end up increasing the numbers of false-positives. But the researchers aren't trying to create false-positives, they think they are creating true-positives.

However, this raises an interesting question for anybody/everybody to answer. Why did Johann and Maaneli (and a long list of other parapsychologists) recommend using a registry? What is it for?

Linda
Do you think this reasoning would apply to studies like Dalton? Seems unlikely that, given the low budget in parapsychology, they would make a 200+ set of trials and then just mine the data by selectively choosing participants, reaching a 100+ number.
 
#18
We had a thread in the last forum where Erbsy, Johann, Maaneli and I went through and added some other studies that were left out of the recent Storm et al MA.



Pre-registration doesn't really hurt. I'd like to see more sessions run per experiment to increase power, with selected individuals. This seems to be the suggestion from Maaneli and Johann as well. I'm not sure a lack of pre-registration gives us any cause to doubt the post communique findings. I also don't think the Ganzfeld experiment by itself gives us enough reasonable confidence in psi. However, there has seemed to be some subtle suggestions that treat Ganzfeld as the best research paradigm out there. Post presentiment MA, I'm not sure that's the case.

But like you said, unless you're accusing parapsychologists of lying...

EDIT: Also note that parapsychology is pretty unique in being one of the only fields that publishes negative findings.
What problems do you think exist within Ganzfelds that make you unsure about it's confindence in proving any form of psi?
 
#19
Do you think this reasoning would apply to studies like Dalton? Seems unlikely that, given the low budget in parapsychology, they would make a 200+ set of trials and then just mine the data by selectively choosing participants, reaching a 100+ number.
I have no idea if it would apply. Very little information is available on the Dalton study. But it seems unlikely, if I were to guess.

Linda
 
#20
What problems do you think exist within Ganzfelds that make you unsure about it's confindence in proving any form of psi?
The protocol is not simple enough to be able to attain proper power with limited funding. That's why Max and Johann ' s suggestions are so crucial. There is also the innate risk of bias when dealing with multiple experimenters have access to the participants. Such as in the mentation phase.

When you have a lot of moving parts, there's always some risk for contamination.

Presentiment experiments just seem much more easy to control for various influencing factors. It's one participant interacting with one computer.
 
Top