Was Bem's "Feeling the Future" paper exploratory?

Diatom

Sorry, but I find all the purely speculative suggestions about how Bem could have cheated a bit boring, unless they can be supported by some evidence that he did actually cheat.

Of course, I realise some people are approaching this from the point of view that he must have cheated, and working from there. But in that case, what's the point of all these elaborate scenarios? If he wanted to cheat, he could just edit the files containing the results, and save himself all the trouble. It would have been a lot simpler and a lot safer. And quite frankly, if he had been cheating, I think he'd have been a damn sight less sloppy about what he published!
Earlier you asked this question regarding "degrees of freedom":
Most importantly, is it plausible that such factors are sufficient to explain the results he found?
Since you are now asking for evidence that this actually took place, I assume that you agree that it is a possible and sufficient explanation. Am I right?
 
Sorry, but I find all the purely speculative suggestions about how Bem could have cheated a bit boring, unless they can be supported by some evidence that he did actually cheat.
Remember that all things being equal, biases caused by researcher degrees of freedom are more likely to be unintended then deliberate.

Frankly, focussing on whether Bem cheated or not (ie: was deliberately deceptive) is going to inevitably increase our own bias in assessing the paper as it places the emphasis on defending or attacking Bem the person.

Do you see now the point I was trying to get at earlier? You are seeing the folly in trying to assess research on the basis of trying to figure out exactly what happened. What you get is loads of speculation, without any practical way of resolving it one way or the other. One thing I think we can say with regard to Bem is that he left out a lot of information that would have been helpful to readers in terms of assessing his methods and results.

It is a mistake to conclude that Bem cheated, or inadvertently biased the results, based on what we know. It is also a mistake to conclude that Bem did not purposefully or inadvertently bias the results, based on what we know. What we do have are serious concerns, based on what has been reported, that the results may have been biased in certain ways. We know this because of other research that demonstrates over large numbers of studies that certain practices increase or decrease the risk of bias.

For the record, it is not a negative thing that these experiments are considered exploratory. We need exploratory studies. There's nothing wrong with it. The story that I see in these papers is the gradual development of Bem's general protocol. Feeling the future seems to tell a pretty clear tale of evolving protocols. He very clearly tinkers with it throughout the experiments - again, absolutely nothing wrong with this. The areas he reports are likely the hypotheses that he thinks merits going on to further studies. There's probably more tinkering that needs to be done (for example, in the timed reaction experiments, am I correct that he does the retro-trials first, then the forward priming trials second? Might the reaction time on the latter trials be simply that people are getting bored after the first 32?).

It's only a problem if the results are considered confirmatory. Note, the meta-analysis does the analysis both with and without Bem's studies included.
 
C

Chris

Since you are now asking for evidence that this actually took place, I assume that you agree that it is a possible and sufficient explanation. Am I right?
Anything's possible. For example, it's possible that those p values could have arisen purely through chance, without any need for psychical phenomena on the one hand or multiple hypotheses, file drawers, or any thing of that sort on the other. But that's not a very useful statement.

My question was whether the explanation was plausible. That's where some evidence would come in handy.
 
C

Chris

Do you see now the point I was trying to get at earlier?
I sometimes wish that just once in my life someone would flounce off and tell me they're never going to speak to me again because I'm such a jerk ... and actually keep their word. :(
 
Anything's possible. For example, it's possible that those p values could have arisen purely through chance, without any need for psychical phenomena on the one hand or multiple hypotheses, file drawers, or any thing of that sort on the other. But that's not a very useful statement.
Please clarify your position. Suddenly you want to change the subject. That's fine. But I don't see where we are now.

My question was whether the explanation was plausible. That's where some evidence would come in handy.
I don't see any point in discussing evidence for your accusation of cheating before we haven't cleared up where you are coming from. I am calling it your accusation because you are the only one talking about cheating.
 
I sometimes wish that just once in my life someone would flounce off and tell me they're never going to speak to me again because I'm such a jerk ... and actually keep their word. :(
You had provided an explanation that you had thought I was being deliberately provocative to you. I thought that was a reasonable excuse, and that you deserved a second chance.

Unlike some others, you actually do seem interested in discussing substance, not just throwing around personal insults. Guess I was just hopeful that we could actually have some good discussion, and your recent posts gave me some suggestion that you were coming around to what I had been advocating.
 
C

Chris

I don't see any point in discussing evidence for your accusation of cheating before we haven't cleared up where you are coming from. I am calling it your accusation because you are the only one talking about cheating.
My "accusation of cheating"? You're crazy.
 
Are you counting not using any of the studies (i.e., the empty set), as a possibility? I didn't count that, but I think it is reasonable to do so.
I didn't, but I agree that that is also a possibility (given that's what he has done in some cases (e.g. the 300 series doesn't seem to have made an appearance, although it's possible that it got rolled into experiment 7 in some way)).

I counted all 4 twice - once as one big study, and once as 4 separate studies. Is that the difference?

Linda
 
OK, so, given your more rigorous wording, here's my amended working:

[SNIP!]
That's a bit hard to follow, but this jumped out at me:

4-set-combos, complete (one-and-one-and-one-and-one):

4C1*3C1*2C1*1C1

Now combining into equivalent sets of combinations, and then dividing by the number of possible permutations of each set of combinations:

4C4 + 4C3 + 4C2 + 4C1 + (4C3*1C1 + 4C1*3C1) / 2 + 4C2*2C2 + (4C2*2C1 + 4C1*3C2) / 2 + 4C1*3C1 + (4C2*2C1*1C1 + 4C1*3C2*1C1 + 4C1*3C1*2C2) / (3*2) + 4C1*3C1*2C1 + 4C1*3C1*2C1*1C1

= 1 + 4*3*2/(3*2*1) + 4*3/(2*1) + 4 + (4*3*2/(3*2*1)*1 + 4*3) / 2 + 4*3/(2*1)*1 + (4*3/(2)*1 + 4*3*2/(2*1)) / 2 + 4*3 + (4*3/(2*1)*2*1 + 4*(3*2)/(2*1)*1 + 4*3*1) / (3*2) + 4*3*2 + 4*3*2*1
There is only one 4-set combination, {{A}, {B}, {C}, {D}}, but somehow you get 24 of them. Perhaps you're still misunderstanding my description of the problem. Keep in mind that this is not an abstract mathematical puzzle; we just want to count up the ways that the experiments can be combined and presented in the paper.

I suggest dividing the problem into four parts: (1) number of ways of using all 4 experiments, (2) number of ways of using exactly 3 experiments, (3) number of ways of using exactly 2 experiments, and (4) number of ways of using exactly 1 experiment. Or, I can post my solution.
 
I counted all 4 twice - once as one big study, and once as 4 separate studies. Is that the difference?
Good call! I forgot to count the case of presenting the four studies separately. So, I agree with you: 54 ways in total.
 
Last edited:
Diatom

Sorry, but I find all the purely speculative suggestions about how Bem could have cheated a bit boring, unless they can be supported by some evidence that he did actually cheat.

Of course, I realise some people are approaching this from the point of view that he must have cheated, and working from there. But in that case, what's the point of all these elaborate scenarios? If he wanted to cheat, he could just edit the files containing the results, and save himself all the trouble. It would have been a lot simpler and a lot safer. And quite frankly, if he had been cheating, I think he'd have been a damn sight less sloppy about what he published!
As usual, Chris, no one but you is talking about anybody lying or cheating. We are not saying that Bem cheated; we're saying that he presented a selection of analyses from a much larger pool of possibilities. And I think he rationalized that those were the choices he would have made all along.
 
That's a bit hard to follow, but this jumped out at me:
Thanks, my approach was, unfortunately, massively confused - you can't just multiply the individual combinatoric solutions of each subset together. [Edit: I was a little too hard on myself here - my approach was after all fundamentally sound, and you can "just" multiply the individual combinatoric solutions together as I did, I simply missed the need to divide by the factorial of each number of subsets which shared a common size, as noted in post #238]

Here's my modified working, by which I get the same solution as Linda. I couldn't figure out a generic way to calculate each of the combinatorics, so I pieced together each using ad-hoc logic:

[Edit: there is a mistake in the below, which I correct in post #241. I've highlighted where the error occurs in bold red]

1-set-combos, both complete and incomplete (individual sets of four, three, two and one):

4C1 + 4C2 + 4C3 + 4C4

2-set-combos, complete (individual sets of three-and-one, two-and-two):

4C3 + 4C2

2-set-combos, incomplete (individual sets of two-and-one, one-and-one):

4C2*2 + 4C2

3-set-combos, complete (individual sets of two-and-one-and-one):

4C2

3-set-combos, incomplete (one-and-one-and-one):

4C3

4-set-combos, complete (one-and-one-and-one-and-one):

4C4


Now, combining together:

= 4C1 + 4C2 + 4C3 + 4C4 + 4C3 + 4C2 + 4C2*2 + 4C2 + 4C2 + 4C3 + 4C4

= 2*4C4 + 3*4C3 + 6*4C2 + 4C1

= 2 + 3*4*3*2/(3*2*1) + 6*4*3/(2*1) + 4

= 54
Perhaps you're still misunderstanding my description of the problem.
Nope, just using a horribly misguided approach. [Note to self: be a little kinder to yourself. Sure, you overlooked something important, but your approach wasn't "horribly misguided"]

Or, I can post my solution.
Please do that regardless. :)
 
Last edited:
I couldn't figure out a generic way to calculate each of the combinatorics
I think I've figured it out. It's just the factorial of the total number of elements divided by the factorials of each of the numbers of elements of each subset, and further divided by the factorials of each of the numbers of any subsets which share a number of members - and if no subsets share a number of members, then there's no need to divide.

So, for the example of a subsetting of five-and-five-and-five-and-four-and-four-and-three, the calculation would be:

26!/(3*5!)/(2*4!)/3!/3!/2! ~= 3.241475864×10²⁰

(More on the actual substance of this thread when I've devoted some time to looking carefully at it)

Edit: actually, as stated, this formula only works when all members of the superset are used in subsets. When there are any unused elements, then the result needs to further be divided by the factorial of the number of unused elements.
 
Last edited:
Here's my modified working, by which I get the same solution as Linda. I couldn't figure out a generic way to calculate each of the combinatorics, so I pieced together each using ad-hoc logic:
Or, I can post my solution.
Please do that regardless. :)
Your solution looks correct. I just divided up the problem differently:
Code:
  Prototype               Ways
------------------------------
Use all 4 experiments:
  {{A, B, C, D}}            1
  {{A, B, C}, {D}}          4
  {{A, B}, {C, D}}          3*
  {{A, B}, {C}, {D}}        6
  {{A}, {B}, {C}, {D}}      1

Use exactly 3 experiments:
  {{A, B, C}}               4
  {{A, B}, {C}}            12
  {{A}, {B}, {C}}           4

Use exactly 2 experiments:
  {{A, B}}                  6
  {{A}, {B}}                6

Use only 1 experiment:
  {{A}}                     4
------------------------------
Total                      51*
==============================
As corrected by Laird.
 
Last edited:
C

Chris

Once we have an agreed value for that number, I look forward with interest to the calculation of the correction to allow for the lack of statistical independence of all these hypothetical combinations of the four sub-experiments. (That sounds like a really hard calculation to me - I'm not sure I'd know where to begin - but perhaps I'll be pleasantly surprised, and entertained. When one gets to a certain age, mathematics becomes more attractive as a spectator sport than a competitive activity.) And then - though perhaps it's hoping for too much - an explanation of why we're talking about it in the first place, given that no one seems to be suggesting that there was any splitting or suppression of any of these results.

Given the antics of some contributors (or perhaps one contributor), I'm not sure whether there's much chance of getting this discussion back on to a sensible track. I had considered starting a Mod+ thread, but unfortunately it seems to be mainly the disruptive elements who are interested. It's a shame really, because I feel an instructive discussion could be had with them (or perhaps him) if only they could curb their animus and remember what planet they're on. It's not as though they're going to make any converts here by posting their more fantastic suggestions.

Anyhow, if we are going to try to continue, the nettle needs to be grasped. Or the elephant in the room needs to be acknowledged. Or whatever. Considering that Daryl Bem is well aware of the issues involved, and has asserted very clearly that there's no question of multiple hypotheses or multiple analyses, it just won't do for people to claim on the one hand that he was choosing from several thousand (!) hypotheses per experiment, and yet to insist on the other that no one is accusing him of lying. If that's what is being suggested, then the people need (or perhaps the person needs) to have the honesty to admit that what's being alleged is definite scientific fraud.

If people really think it was deliberate scientific fraud, then they should have no hesitation in making the accusation (and realistically they need have no fear that he's going to pursue an anonymous Internet forum poster through the courts). But I think they should have the honesty to say that's what they mean. If, on the other hand, they aren't willing to allege any deception on the part of Bem, they have a much more challenging task on their hands. They have to explain how Daryl Bem could have obtained these results, while perfectly aware of all the issues involved, and without any intent to deceive. That's the challenge for the sceptics (if they insist they're not disputing Bem's veracity).
 
Top