Was Bem's "Feeling the Future" paper exploratory?

Your solution looks correct.
Actually, it's not. And neither is yours. We've both made the very same error. Probably Linda has too.

The error is here:

2-set-combos, complete (individual sets of three-and-one, two-and-two):

4C3 + 4C2
That 4C2 should be (and originally was; I second-guessed myself) 4C2/2. This is a decrease of 3 (from 6). i.e. 4C2 = 6; 4C2/2 = 3.

Code:
  {{A, B}, {C, D}}          6
This should be:
Code:
  {{A, B}, {C, D}}          3
So, the total is actually 51, not 54.
 
...no one seems to be suggesting that there was any splitting or suppression of any of these results.
WTF!? Everyone but you is suggesting that there was splitting, combining, or suppression of results!

If that's what is being suggested, then the people need (or perhaps the person needs) to have the honesty to admit that what's being alleged is definite scientific fraud.
And yet again, the only person talking about fraud is you. Learn to read:

http://www.skeptiko-forum.com/threa...ure-paper-exploratory.1561/page-12#post-76032
 
Actually, it's not. And neither is yours. We've both made the very same error. Probably Linda has too.

The error is here:



That 4C2 should be (and originally was; I second-guessed myself) 4C2/2. This is a decrease of 3 (from 6). i.e. 4C2 = 6; 4C2/2 = 3.



This should be:
Code:
  {{A, B}, {C, D}}          3
So, the total is actually 51, not 54.
Another good catch!
 
And yet again, the only person talking about fraud is you.
Actually, I agree with Chris. That Bem perpetrated a fraud with this paper is implicit in your position. I still need to devote some time to looking carefully into this all, but from casually following the thread, I haven't seen any plausible case that Bem chose any other hypotheses than the blindingly obvious ones, which in my view is not at all consistent with "choosing those hypotheses which gave the best results", which seems to be the case you're trying to make for his actions. In any case, getting to the point, clearly this does imply fraud, because, as Chris points out, Bem himself has said that he chose his hypotheses in advance. For you to state otherwise is to implicitly accuse him of deception.
 
Actually, I agree with Chris. That Bem perpetrated a fraud with this paper is implicit in your position. I still need to devote some time to looking carefully into this all, but from casually following the thread, I haven't seen any plausible case that Bem chose any other hypotheses than the blindingly obvious ones, which in my view is not at all consistent with "choosing those hypotheses which gave the best results", which seems to be the case you're trying to make for his actions. In any case, getting to the point, clearly this does imply fraud, because, as Chris points out, Bem himself has said that he chose his hypotheses in advance. For you to state otherwise is to implicitly accuse him of deception.
I'll explain this one last time. I think Bem actually believes what he said. Therefore, he is not lying. Gelman wrote a whole paper on how researchers can go on a massive fishing expedition and not realize it. I think this is largely what Bem has done.
 
hat Bem perpetrated a fraud with this paper is implicit in your position.
No.
The techniques that Bem used are not regarded as fraud. I know that seems odd when a statisticians explains what is wrong with them. It is true that one might as well be making up the data for all the scientific good it does but you must remember that social scientists aren't always good at statistics...
Jay posted an excerpt from an article Bem wrote. This article recommends using exactly those tricks. Bem wasn't excoriated for that. His article was (and still is) widely recommended to students. It was, of course, also criticized because of the bad statistics implied.

I don't know why Chris is talking about fraud. Maybe he is trying to be sneaky? Think of publishing an article boldly defending Donald Trump from charges of being a child molester, saying that all the very many witnesses are totally unreliable...
It doesn't make perfect sense, I'll admit but then what does?

In any case, getting to the point, clearly this does imply fraud, because, as Chris points out, Bem himself has said that he chose his hypotheses in advance. For you to state otherwise is to implicitly accuse him of deception.
I'm not going to accuse Bem of lying without knowing his exact words. But even so, that he did engage in deception to some degree is established. When I read the 2011 paper I was lead to believe that experiments 5 and 6 were actually two experiments. I would never have guessed that they were cobbled together from, at least, two series of multiple experiments. I don't think anyone would have guessed.
 
I'll explain this one last time. I think Bem actually believes what he said. Therefore, he is not lying. Gelman wrote a whole paper on how researchers can go on a massive fishing expedition and not realize it. I think this is largely what Bem has done.
I'm not buying it, but as I said, I still need to look closely into the details of this thread which I've been following only casually. Maybe after that my mind will change.
 
When I read the 2011 paper I was lead to believe that experiments 5 and 6 were actually two experiments. I would never have guessed that they were cobbled together from, at least, two series of multiple experiments. I don't think anyone would have guessed.
I guessed, and I suspect that Experiments 1 and 2 in Feeling the Future were similarly cobbled together. These experiments employed different stimuli in earlier and later sessions. I suspect that sessions using different stimuli were originally separate experiments, but (giving him the benefit of the doubt) Bem judged them similar enough to be presented together.
 
I'm not buying it, but as I said, I still need to look closely into the details of this thread which I've been following only casually. Maybe after that my mind will change.
If I thought he was lying, I would simply say so. It's really weird that you and Chris are so invested in believing that I think Bem lied. Like Diatom, I can't get my head around what's driving this obsession.
 
Last edited:
Actually, I agree with Chris. That Bem perpetrated a fraud with this paper is implicit in your position.
Except that its not. All that is implicit is that Bem is subject to bias as much as anyone and that without taking specific steps to protect oneself from one's own bias one risks biasing one's results.

That's the thing with researcher degrees of freedom - scientists talk themselves into decisions that increase the risk. Each decision on its own seems justified. Cognitive biases are incredibly powerful, and even those who are aware of them will regularly find themselves falling victim to them.

This is what the research shows (research that is largely done outside of parapsychology, for that matter).

Outside of not being true, insisting that to point out methodological issues like this entails fraud has the effect of discouraging open discussion of these issues - and does a disservice to all.
 
Bem himself has said that he chose his hypotheses in advance. For you to state otherwise is to implicitly accuse him of deception.
You say that Bem said that he chose his hypotheses in advance. Can you present a quote where he said that, because according to Linda, "Bem stated 'in fact, there was no data exploration that required adjustment for multiple analyses in this or any other experiment' in response to Wagenmakers' question about whether the data was exploratory. [emphasis added by Jay]" That's not the same as saying that he chose his hypotheses in advance, especially when you take into account Bem's opinion on correction for multiplicity. As he stated in his textbook chapter, "Spurious findings can emerge by chance, and we need to be cautious about anything we discover in this way. In limited cases, there are statistical techniques that correct for this danger [emphasis added by Jay]."

So he seems to believe (and to an extent he is correct) that usually there is no good statistical technique to correct for multiple analyses. Now go back to the previous paragraph and reread the claim that Bem actually made.

Furthermore, as I tried in vain to explain to Chris, whether the hypotheses were stated "in advance" is not the issue. Unadjusted p-values from testing multiple hypotheses that were stated in advance usually are not interpretable, and should either be adjusted formally or taken with a grain of salt.

And finally, I don't think that adjusting Bem's p-values is even the issue, because, for one thing, I agree with him, that if he did the sort of significance seeking that he appears to have done, that there is no good way to adjust the p-values (except, perhaps making them all p=1.0, since that's what the real significance level is). Rather, I think researchers need to avoid significance-seeking analyses in the first place.
 
Last edited:
C

Chris

Jay

You suggest that Bem may have searched through thousands of hypotheses for each experiment, that he may have tried dozens of different combinations of sub-experiments to find a significant one, and that he may have suppressed results at will to boost the p values. And then you say that wouldn't amount to scientific fraud. It is simply ludicrous.

Obviously any further discussion of this topic does need to be Mod+.
 
Jay

You suggest that Bem may have searched through thousands of hypotheses for each experiment, that he may have tried dozens of different combinations of sub-experiments to find a significant one, and that he may have suppressed results at will to boost the p values. And then you say that wouldn't amount to scientific fraud. It is simply ludicrous.
No, I did not say that Bem conducted thousands of tests; I said that he made choices among thousands of possible analyses. I think those choices were subtly data driven, and I think that Bem could convince himself that the choices he ultimately made were the choices that he would have made all along.

But regardless, as has been explained to you multiple times by Diatom and me, this exploratory sort of analysis is what Bem recommends in a textbook chapter he wrote that is well regarded in his field. As bad as it is, it is the accepted way that data analysis is performed in experimental psychology (although efforts are under way to change that). THUS, IT IS NOT CONSIDERED FRAUD IN HIS FIELD.

Obviously any further discussion of this topic does need to be Mod+.
Whatever that even means.
 
Come on, Jay. Be serious. Just how many of the "thousands of possible analyses" do you think Bem would have needed to have tried to have finally chosen, ever so "subtly" based on his actual data, the most successful one(s), and after how many trials of possible analyses based on his actual data would it have become utterly impossible to have even pretended to himself to be doing anything other than blatantly, deliberately p-hacking?
 
C

Chris

But regardless, as has been explained to you multiple times by Diatom and me, this exploratory sort of analysis is what Bem recommends in a textbook chapter he wrote that is well regarded in his field.
Ludicrous. No one in their right mind could honestly think that Bem believes it would be acceptable to search through thousands of hypotheses in order to find a significant result, and then pretend that the experiment had been designed to test that hypothesis.

Again, here is a link to the full paper, so that people can see how ridiculous Jay's misrepresentation is:
http://pages.stern.nyu.edu/~wstarbuc/Writing/Bem-Emp.htm
 
I think Chris is making a point here. I have no idea how we can calculate the probability, but we cannot completely dismiss the possibility of fraud.
 
Again, here is a link to the full paper, so that people can see how ridiculous Jay's misrepresentation is:
http://pages.stern.nyu.edu/~wstarbuc/Writing/Bem-Emp.htm
Key quote (emphasis mine):

Daryl Bem said:
No, this is not immoral. The rules of scientific and statistical inference that we overlearn in graduate school apply to the "Context of Justification." They tell us what we can conclude in the articles we write for public consumption, and they give our readers criteria for deciding whether or not to believe us. But in the "Context of Discovery," there are no formal rules, only heuristics or strategies. How does one discover a new phenomenon? Smell a good idea? Have a brilliant insight into behavior? Create a new theory? In the confining context of an empirical study, there is only one strategy for discovery: exploring the data.
In other words, he clearly distinguishes between exploratory and confirmatory techniques, and the advice he gave which Jay finds so questionable was applicable to the former, not the latter. He is not suggesting that in the context of a confirmatory study, one could retrofit one's hypothesis.
 
You say that Bem said that he chose his hypotheses in advance.
Put it this way: if he didn't, then he got really, really lucky - they are the exact hypotheses which are obvious to test, and, happily (for him), his (supposed) retrofitting didn't require him to have to explain or justify any weird choices of apparently-arbitrary, subset-specific hypotheses (or worse yet, subset-specific hypotheses which were specific to different subsets across different experiments); nope, he got handed to him on a platter the generic hypotheses most widely applicable to the data of each experiment that one could hope for.
 
C

Chris

Put it this way: if he didn't, then he got really, really lucky - they are the exact hypotheses which are obvious to test, and, happily (for him), his (supposed) retrofitting didn't require him to have to explain or justify any weird choices of apparently-arbitrary, subset-specific hypotheses (or worse yet, subset-specific hypotheses which were specific to different subsets across different experiments); nope, he got handed to him on a platter the generic hypotheses most widely applicable to the data of each experiment that one could hope for.
That is essentially my feeling, except that it requires some qualification for the experiments where several types of images were used, because in those cases there are some alternative generic hypotheses (and experiment 101, whose results may have formed part of Experiment 5 of "Feeling the Future", with several hypotheses in mind).
 
Top