Diatom

Sorry, but I find all the purely speculative suggestions about how Bem**could** have cheated a bit boring, unless they can be supported by some evidence that he **did** actually cheat.

Of course, I realise some people are approaching this from the point of view that he**must** have cheated, and working from there. But in that case, what's the point of all these elaborate scenarios? If he wanted to cheat, he could just edit the files containing the results, and save himself all the trouble. It would have been a lot simpler and a lot safer. And quite frankly, if he had been cheating, I think he'd have been a damn sight less sloppy about what he published!

Sorry, but I find all the purely speculative suggestions about how Bem

Of course, I realise some people are approaching this from the point of view that he

Most importantly, is it plausible that such factors are sufficient to explain the results he found?

Sorry, but I find all the purely speculative suggestions about how Bem **could** have cheated a bit boring, unless they can be supported by some evidence that he **did** actually cheat.

Frankly, focussing on whether Bem cheated or not (ie: was deliberately deceptive) is going to inevitably increase our own bias in assessing the paper as it places the emphasis on defending or attacking Bem the person.

Do you see now the point I was trying to get at earlier? You are seeing the folly in trying to assess research on the basis of trying to figure out exactly what happened. What you get is loads of speculation, without any practical way of resolving it one way or the other. One thing I think we can say with regard to Bem is that he left out a lot of information that would have been helpful to readers in terms of assessing his methods and results.

It is a mistake to conclude that Bem cheated, or inadvertently biased the results, based on what we know. It is also a mistake to conclude that Bem did not purposefully or inadvertently bias the results, based on what we know. What we do have are serious concerns, based on what has been reported, that the results may have been biased in certain ways. We know this because of other research that demonstrates over large numbers of studies that certain practices increase or decrease the risk of bias.

For the record, it is not a negative thing that these experiments are considered exploratory. We need exploratory studies. There's nothing wrong with it. The story that I see in these papers is the gradual development of Bem's general protocol. Feeling the future seems to tell a pretty clear tale of evolving protocols. He very clearly tinkers with it throughout the experiments - again, absolutely nothing wrong with this. The areas he reports are likely the hypotheses that he thinks merits going on to further studies. There's probably more tinkering that needs to be done (for example, in the timed reaction experiments, am I correct that he does the retro-trials first, then the forward priming trials second? Might the reaction time on the latter trials be simply that people are getting bored after the first 32?).

It's only a problem if the results are considered confirmatory. Note, the meta-analysis does the analysis both with and without Bem's studies included.

C

Since you are now asking for evidence that this actually took place, I assume that you agree that it is a possible and sufficient explanation. Am I right?

My question was whether the explanation was

Anything's **possible**. For example, it's **possible **that those p values could have arisen purely through chance, without any need for psychical phenomena on the one hand or multiple hypotheses, file drawers, or any thing of that sort on the other. But that's not a very useful statement.

My question was whether the explanation was **plausible**. That's where some evidence would come in handy.

I sometimes wish that just once in my life someone would flounce off and tell me they're never going to speak to me again because I'm such a jerk ... and actually keep their word.

Unlike some others, you actually do seem interested in discussing substance, not just throwing around personal insults. Guess I was just hopeful that we could actually have some good discussion, and your recent posts gave me some suggestion that you were coming around to what I had been advocating.

Are you counting not using any of the studies (i.e., the empty set), as a possibility? I didn't count that, but I think it is reasonable to do so.

I counted all 4 twice - once as one big study, and once as 4 separate studies. Is that the difference?

Linda

OK, so, given your more rigorous wording, here's my amended working:

[SNIP!]

[SNIP!]

4-set-combos, complete (one-and-one-and-one-and-one):

**4C1*3C1*2C1*1C1**

Now combining into equivalent sets of combinations, and then dividing by the number of possible permutations of each set of combinations:

4C4 + 4C3 + 4C2 + 4C1 + (4C3*1C1 + 4C1*3C1) / 2 + 4C2*2C2 + (4C2*2C1 + 4C1*3C2) / 2 + 4C1*3C1 + (4C2*2C1*1C1 + 4C1*3C2*1C1 + 4C1*3C1*2C2) / (3*2) + 4C1*3C1*2C1 +**4C1*3C1*2C1*1C1**

= 1 + 4*3*2/(3*2*1) + 4*3/(2*1) + 4 + (4*3*2/(3*2*1)*1 + 4*3) / 2 + 4*3/(2*1)*1 + (4*3/(2)*1 + 4*3*2/(2*1)) / 2 + 4*3 + (4*3/(2*1)*2*1 + 4*(3*2)/(2*1)*1 + 4*3*1) / (3*2) + 4*3*2 +**4*3*2*1**

Now combining into equivalent sets of combinations, and then dividing by the number of possible permutations of each set of combinations:

4C4 + 4C3 + 4C2 + 4C1 + (4C3*1C1 + 4C1*3C1) / 2 + 4C2*2C2 + (4C2*2C1 + 4C1*3C2) / 2 + 4C1*3C1 + (4C2*2C1*1C1 + 4C1*3C2*1C1 + 4C1*3C1*2C2) / (3*2) + 4C1*3C1*2C1 +

= 1 + 4*3*2/(3*2*1) + 4*3/(2*1) + 4 + (4*3*2/(3*2*1)*1 + 4*3) / 2 + 4*3/(2*1)*1 + (4*3/(2)*1 + 4*3*2/(2*1)) / 2 + 4*3 + (4*3/(2*1)*2*1 + 4*(3*2)/(2*1)*1 + 4*3*1) / (3*2) + 4*3*2 +

I suggest dividing the problem into four parts: (1) number of ways of using all 4 experiments, (2) number of ways of using exactly 3 experiments, (3) number of ways of using exactly 2 experiments, and (4) number of ways of using exactly 1 experiment. Or, I can post my solution.

Sorry, but I find all the purely speculative suggestions about how Bem

Of course, I realise some people are approaching this from the point of view that he

That's a bit hard to follow, but this jumped out at me:

Here's my modified working, by which I get the same solution as Linda. I couldn't figure out a generic way to calculate each of the combinatorics, so I pieced together each using ad-hoc logic:

[Edit: there is a mistake in the below, which I correct in post #241. I've highlighted where the error occurs in bold red]

1-set-combos, both complete and incomplete (individual sets of four, three, two and one):

4C1 + 4C2 + 4C3 + 4C4

2-set-combos, complete (individual sets of three-and-one, two-and-two):

4C3 +**4C2**

2-set-combos, incomplete (individual sets of two-and-one, one-and-one):

4C2*2 + 4C2

3-set-combos, complete (individual sets of two-and-one-and-one):

4C2

3-set-combos, incomplete (one-and-one-and-one):

4C3

4-set-combos, complete (one-and-one-and-one-and-one):

4C4

Now, combining together:

= 4C1 + 4C2 + 4C3 + 4C4 + 4C3 +**4C2** + 4C2*2 + 4C2 + 4C2 + 4C3 + 4C4

= 2*4C4 + 3*4C3 +**6***4C2 + 4C1

= 2 + 3*4*3*2/(3*2*1) +**6***4*3/(2*1) + 4

=**54**

4C1 + 4C2 + 4C3 + 4C4

2-set-combos, complete (individual sets of three-and-one, two-and-two):

4C3 +

2-set-combos, incomplete (individual sets of two-and-one, one-and-one):

4C2*2 + 4C2

3-set-combos, complete (individual sets of two-and-one-and-one):

4C2

3-set-combos, incomplete (one-and-one-and-one):

4C3

4-set-combos, complete (one-and-one-and-one-and-one):

4C4

Now, combining together:

= 4C1 + 4C2 + 4C3 + 4C4 + 4C3 +

= 2*4C4 + 3*4C3 +

= 2 + 3*4*3*2/(3*2*1) +

=

Perhaps you're still misunderstanding my description of the problem.

Or, I can post my solution.

Last edited:

I couldn't figure out a generic way to calculate each of the combinatorics

So, for the example of a subsetting of five-and-five-and-five-and-four-and-four-and-three, the calculation would be:

26!/(3*5!)/(2*4!)/3!/3!/2! ~= 3.241475864×10²⁰

(More on the actual substance of this thread when I've devoted some time to looking carefully at it)

Edit: actually, as stated, this formula only works when all members of the superset are used in subsets. When there are any unused elements, then the result needs to further be divided by the factorial of the number of unused elements.

Last edited:

Here's my modified working, by which I get the same solution as Linda. I couldn't figure out a generic way to calculate each of the combinatorics, so I pieced together each using ad-hoc logic:

Please do that regardless.

Or, I can post my solution.

Code:

```
Prototype Ways
------------------------------
Use all 4 experiments:
{{A, B, C, D}} 1
{{A, B, C}, {D}} 4
{{A, B}, {C, D}} 3*
{{A, B}, {C}, {D}} 6
{{A}, {B}, {C}, {D}} 1
Use exactly 3 experiments:
{{A, B, C}} 4
{{A, B}, {C}} 12
{{A}, {B}, {C}} 4
Use exactly 2 experiments:
{{A, B}} 6
{{A}, {B}} 6
Use only 1 experiment:
{{A}} 4
------------------------------
Total 51*
==============================
```

Last edited:

C

Given the antics of some contributors (or perhaps one contributor), I'm not sure whether there's much chance of getting this discussion back on to a sensible track. I had considered starting a Mod+ thread, but unfortunately it seems to be mainly the disruptive elements who are interested. It's a shame really, because I feel an instructive discussion could be had with them (or perhaps him) if only they could curb their animus and remember what planet they're on. It's not as though they're going to make any converts here by posting their more fantastic suggestions.

Anyhow, if we are going to try to continue, the nettle needs to be grasped. Or the elephant in the room needs to be acknowledged. Or whatever. Considering that Daryl Bem is well aware of the issues involved, and has asserted very clearly that there's no question of multiple hypotheses or multiple analyses, it just won't do for people to claim on the one hand that he was choosing from several thousand (!) hypotheses per experiment, and yet to insist on the other that no one is accusing him of lying. If that's what is being suggested, then the people need (or perhaps the person needs) to have the honesty to admit that what's being alleged is definite scientific fraud.

If people really think it was deliberate scientific fraud, then they should have no hesitation in making the accusation (and realistically they need have no fear that he's going to pursue an anonymous Internet forum poster through the courts). But I think they should have the honesty to say that's what they mean. If, on the other hand, they aren't willing to allege any deception on the part of Bem, they have a much more challenging task on their hands. They have to explain how Daryl Bem could have obtained these results, while perfectly aware of all the issues involved, and without any intent to deceive. That's the challenge for the sceptics (if they insist they're not disputing Bem's veracity).