Critiques of Science as Currently Praticed

  • Thread starter Sciborg_S_Patel
  • Start date

It's interesting that skeptics, as self appointed guardians of science, missed so many of the other major flaws in varied fields but apparently - or so they claim - managed to find flaws in a study that would seriously challenge the materialist paradigm.

"A man who wants to beat a dog always finds his stick."
-Serbian Proverb


Am I wrong that Novella mentions other skeptics? - Their own biases make their conclusions questionable to me but perhaps someone can walk us through the math?
 
Has Physics Gotten Something Really Important Really Wrong?

Adam Frank is a co-founder of the 13.7 blog, an astrophysics professor at the University of Rochester, a book author and a self-described "evangelist of science."

Some researchers now see popular ideas like string theory and the multiverse as highly suspect. These physicists feel our study of the cosmos has been taken too far from what data can constrain with the extra "hidden" dimensions of string theory and the unobservable other universes of the multiverse. Of course, there are many scientists who continue to see great promise in string theory and the multiverse. But, as Marcelo and I wrote in The New York Times last year, it all adds up to muddied waters and something some researchers see as a "crisis in physics."

A couple of months ago, I wrote about the great debate between the philosopher Henri Bergson and Albert Einstein over the role of philosophy in discussions of time. That contentious historical echo makes it all the more remarkable to see Bergson (and Indian classical world views) making an appearance in a book about re-envisioning cosmology.

There is a lot in Unger and Smolin's book that I find both bracing and exciting as I work my way through the text. It must be remembered that the program they advocate constitutes just an outline — a beginning — for a new approach to physics. It's not at all clear that something truly useful can be made of it. But by taking on the task in its full measure, they offer us a view beyond the status quo of the last few decades in cosmology and foundational physics.

While it is entirely possible that string theory or the multiverse may yet find strong links with data, what Smolin and Unger offer is a view of what a truly different philosophical approach would look like.

That, in itself, is worthwhile.

More critiques of the multiverse can be found here.
 
.

"A man who wants to beat a dog always finds his stick."
-Serbian Proverb

Heh, I agree with that sentiment (one can find abuses in any profession, and highlight them in order to paint the sky as falling). But this article wasn't a science-bashing rant. The criticism is constructive and he makes a point to note that science has got a lot right as well.

It's interesting, the articles highlighting the retractions due to fraud jump to the conclusion that it is because more scientists are being fraudulent. That may be the case, but it is also worth considering whether it is because there is more attention being placed on identifying fraud and more people are being caught.

And take the replication crisis. Yes it seems that in the past some people put too much reliance on studies that were at high risk of methodological bias and that weren't confirmed by adequate replication. But one reason we know about his is because some scientists considered it important enough to look into it.

Sure there have been some findings that have shocked some people. But it is because over the last few years there has been an increased focus on methodological issues. We see scientists in diverse fields turning their attention to it. The upshot is we're going to discover a lot of sub-optimal work. It is messy and no doubt we will discover even more issues as the work continues. But it doesn't mean the sky is falling- because it is this work that is going to help improve research practices going forward.

now, while I don't think the sky is falling, I don't want to whitewash the problems- just put them in context. while I think some people are too harsh or doomy and gloomy, I do think it is good that for many people the rose coloured glasses have come off. Many people have far too unrealistic expectations of science and too readily accept as true the results of scientific experiments. Some people do put science on too high a pedestal. It is good that scientists are seen as they they are: imperfect and flawed human beings undertaking very difficult tasks.
 



Fixing published research mistakes not easy; fixing the publishing system may be harder

The process for fixing mistakes in peer-reviewed research articles is flawed, a new article suggests. It points out that journals are slow to respond and even slower to take action when questions regarding the accuracy of a published research paper are raised.

"These errors involved factual mistakes or practices which veered substantially from clearly accepted procedures in ways that, if corrected, might alter a paper's conclusions," said Andrew Brown, Ph.D., a scientist in the UAB School of Public Health and co-author of the commentary. "In several cases, our noting these errors led to retractions of the papers containing them."

=-=-=

.

"A man who wants to beat a dog always finds his stick."
-Serbian Proverb

Amusing, but I did say if someone can explain the math I'm willing to listen. Given all the questionable tactics of (pseudo?)skeptics in the past I think my wariness is, IMO, reasonable. [And anyone is free to present empirical evidence that proves a multiverse exists, if the comment is directed at Unger/Smolin/Frank.]

Additionally I don't think the situations are comparable between me (or Smolin/Unger, or Frank) & Novella - I mean assume any Psi effect was proved to be real. Suddenly it calls into question if any other Psi effects are real, including the kind of effects utilized in psychic healing. This would mean people whose lives might've been saved are dead - as a doctor Novella likely realizes the blood on his hands could far exceed anyone he's ever saved.

That's a strong psychological motivation to discount any Psi effect, not just for Novella but for any skeptic really. Perhaps it's what motivated what the Nobel physicist Josephson sees as biased examination of "the girl with x-ray eyes":

Scientists' unethical use of media for propaganda purposes

Many viewers of a recent Discovery Channel programme, previewed in a Guardian article, where the claims of Natasha Demkina, a 17-year-old Russian girl who says she is able to "look deep inside people's bodies, watch their organs at work and spot when things are going wrong", were investigated, ended up with a strong impression that the main test in the investigation had been deliberately set up with a view to ensuring that she would fail it. The test involved her being given a set of seven cards, with a medical condition indicated on each. Medical subjects with these seven conditions (one of which was 'no condition'), each bearing an identifying number, stood in a row and Natasha had to mark each card with the number of the person whom she thought had the condition indicated on the card. Despite the difficulties associated with the rigorous and unfamiliar conditions imposed by the experimenters, she identified four of the seven correctly. A fairly straightforward calculation shows that the odds of getting 4 hits or more out of 7 by chance are more than 50 to 1 against. Surely a case for celebrating Natasha's success?

Well, no, actually. The experimental protocol, to which Natasha and her agent had been asked to agree, rather curiously states:

"If Natasha correctly matches fewer than 5 target medical conditions, then the Test Proctor will declare that results are more consistent with chance guessing and does not support any belief in her claimed abilities."

Accordingly, it was announced that Natasha had 'failed the test'. In the article about the programme in the Guardian, Richard Wiseman, one of the investigators, emphasised this conclusion, declaring "a failure is a failure".

(added November 11th., 2004)The investigators' own account is now available on the web: observe that the 50 to 1 statistic does not feature anywhere in it. The fact that "everyone [had] agreed to the written protocols" (including the above italicised condition) is given as sufficient justification for asserting "[the] test, as preliminary as it was, will likely close the chapter in this case". I think not: real science does not work on a basis of getting someone to sign their agreement to a long list of conditions, then later coming back saying "this is what you signed; the challenge goes to us!".
 
It's interesting, the articles highlighting the retractions due to fraud jump to the conclusion that it is because more scientists are being fraudulent. That may be the case, but it is also worth considering whether it is because there is more attention being placed on identifying fraud and more people are being caught.

Actually, just found an article in Nature that suggests exactly this: Science publishing: The trouble with retractions.
 
Actually, just found an article in Nature that suggests exactly this: Science publishing: The trouble with retractions.

The article states the reason for the rise is unclear:

The reasons behind the rise in retractions are still unclear.

After that it gets into guesswork but without massive replications of all scientific fields - perhaps excluding work that is purely speculative (string theory mathematics for example) & that which has been used in reliable tech/medicine - I don't see how anyone can say for sure where the rise comes from or evaluate just how broken (or not) Science-as-practiced is.

In fact the article suggests this as well:

In surveys, around 1–2% of scientists admit to having fabricated, falsified or modified data or results at least once (D. Fanelli PLoS ONE4, e5738; 2009). But over the past decade, retraction notices for published papers have increased from 0.001% of the total to only about 0.02%. And, Ioannidis says, that subset of papers is "the tip of the iceberg" — too small and fragmentary for any useful conclusions to be drawn about the overall rates of sloppiness or misconduct.

Admittedly the article was also published in 2011, so perhaps there's new information that exonerates much of science. OTOH, it could be the reality is the other direction - as potentially suggested by this previously posted article published this year:

Cancer Research Is Broken: There’s a replication crisis in biomedicine—and no one even knows how deep it runs.
 
Admittedly the article was also published in 2011, so perhaps there's new information that exonerates much of science. OTOH, it could be the reality is the other direction - as potentially suggested by this previously posted article published this year:

Cancer Research Is Broken: There’s a replication crisis in biomedicine—and no one even knows how deep it runs.

Putting aside the title, this article does not point in the opposite direction of the arguments I've been making. It's all about the increasing focus on the importance of methodology and of scientists going back and taking a closer look at previous findings. It also highlights some of the practical difficulties that must be overcome in order to do it properly. The problems that this article highlights are very real problems, and I submit, much more important than the problem of fraud in terms of reducing the reliability of scientific findings.

Again, this story is an example of how our increasing understanding of methodology is allowing us to discover cases where best practices were not used. Scientists are realising that many experiments have not been replicated (nor attempted to be replicated) and are going back and revisiting old findings. Findings that should not have been relied on in the first place.

If we look only backward it is easy to paint the picture as gloomy. However, this ignores that these problems are being discovered precisely because there is an increasing focus on research methodology and the factors that increase reliability of findings. It is precisely because problems that have been previously ignored are no longer being ignored that all these poor practices are coming to light. Sure we can wish that these issues had been discovered earlier - but what we're seeing, slowly but surely over the past two decades, is positive change. From what I can tell, the last 20 years (and in particular, the last 10 years since Ioannidis' 2005 paper) has seen a leap forward in methodology development. What we're seeing now is the fruits of a tremendous amount of work. Normally such scientific leaps are celebrated. In this case, however, it leap comes along with a realisation that a lot of science was not as solid as many people thought.

We see many projects like that described in this article, or efforts such as the Opentrials link I posted above. I recall seeing a list of similar efforts somewhere but I can't remember where it was. At some point maybe I'll try and find it or compile my own list.I agree that sunlight should be shone on those issues. In fact it is extremely important that we continue to do so. But to have a fair assessment of science as a whole we need to look at the good as well as the bad. We should take off the rose-coloured glasses, but not replace them with shades.
 
Putting aside the title, this article does not point in the opposite direction of the arguments I've been making...

We see many projects like that described in this article, or efforts such as the Opentrials link I posted above. I recall seeing a list of similar efforts somewhere but I can't remember where it was. At some point maybe I'll try and find it or compile my own list.I agree that sunlight should be shone on those issues. In fact it is extremely important that we continue to do so. But to have a fair assessment of science as a whole we need to look at the good as well as the bad. We should take off the rose-coloured glasses, but not replace them with shades.

The article notes that there's a crisis, and nobody knows how deep the rot runs.

Let's say I find a dead mouse in my burger when eating at "Burger Place". Now that could be a fluke, maybe even a prank by a disgruntled employee. But it could be that there are some major health issues with Burger Place. The only way to know for sure is to have someone do a deep inspection of Burger Place.

The case with science and all its problems is comparable, though I suppose we could make it a food court with dead mice only found in, say, 2 out of 10 food outlets.
 
The article notes that there's a crisis, and nobody knows how deep the rot runs.

I read the article.

Let's say I find a dead mouse in my burger when eating at "Burger Place". Now that could be a fluke, maybe even a prank by a disgruntled employee. But it could be that there are some major health issues with Burger Place. The only way to know for sure is to have someone do a deep inspection of Burger Place.

Yes, one of my points above is that we're finding a lot more mice now because those inspections are taking place, and with a better understanding of mice habits, and better mouse tracking methods. Also, there are a bunch of policies being put into place to reduce future mouse infestations.
 
Yes, one of my points above is that we're finding a lot more mice now because those inspections are taking place, and with a better understanding of mice habits, and better mouse tracking methods. Also, there are a bunch of policies being put into place to reduce future mouse infestations.

We still don't know how many people ate mice, which is where I guess the analogy fails.

Perhaps a better way to think of it is product recalls, where mass replication is the recall for all these varied bodies of work from the different fields. Or maybe a test for a disease is found to be faulty, and everyone who was believed to be infected needs to be retested?

Although this talk of mice reminds me of the potential issue with a variety of past studies due to mouse fear (IIRC Iyace mentioned this awhile back):

Lab mice fear men but not women, and that's a big problem for science: A new study casts doubt on decades of research

The history of science is one chock-full of mice and men. Historically, biological and medical research has largely depended on rodents, which provide scientists with everything from cells and organs to behavioral data. That's why a new study in which researchers found that mice actually fear men, but not women, has the potential to be so disruptive. It might mean that a number of researchers have published mouse studies in which their results reflect this male-induced stress effect — and they know nothing about it.

Perhaps there's been a review of all those studies since this was published, and it was all found to be okay? Or this study itself was fundamentally flawed, and there's no problem?
 
The article notes that there's a crisis, and nobody knows how deep the rot runs.

Let's say I find a dead mouse in my burger when eating at "Burger Place". Now that could be a fluke, maybe even a prank by a disgruntled employee. But it could be that there are some major health issues with Burger Place. The only way to know for sure is to have someone do a deep inspection of Burger Place.

The case with science and all its problems is comparable, though I suppose we could make it a food court with dead mice only found in, say, 2 out of 10 food outlets.


Practically, how does one "inspect" science? As far as I can see, science has it's own built in correction - better science. What, in addition to what Arouet has already suggested, would you suggest is needed?

If one wants to overlook the incredible feats of science, it is easy to see science as broken.
 
Practically, how does one "inspect" science? As far as I can see, science has it's own built in correction - better science. What, in addition to what Arouet has already suggested, would you suggest is needed?

If one wants to overlook the incredible feats of science, it is easy to see science as broken.

The problem is that that built in correction can become rather weak, submerged under other interests such as the need to be seen to be making good progress so as to get the next grant, etc etc.

Keeping science honest and careful is a huge problem. One thing is certain - we should take a lot more interest in informed whistle blowers than we tend to do at present. As it is, people who were respected members of the science community get shunted aside as soon as they start to make awkward claims - Halton Arp, Henry Bauer, Rupert Sheldrake, etc etc.

As for those incredible feats - yes there are such feats - putting a satellite into orbit round Jupiter today is one example - but those are the areas of science that get tested in a very obvious and public way. The US had a period where a succession of Mars probes failed. I am sure that was an excruciating period for those involved, and if they could have buried the outcome in some way, they would. In so many areas of science, it is possible to bury failure, or even to maintain a hypothesis against available evidence.

David
 
Yes, one of my points above is that we're finding a lot more mice now because those inspections are taking place, and with a better understanding of mice habits, and better mouse tracking methods. Also, there are a bunch of policies being put into place to reduce future mouse infestations.

I guess the difference between you and me, is that you seem to see this crisis as a little bit of grit in the well oiled wheels of science, that can be flushed out with some lubrication. I see the crisis as something that has festered for decades now, and become a pervasive flaw in science, that will probably only get corrected after some major failures become public. Medical science looks likely to be the first to experience this, but there seem to be a range of contenders.

If you really want to understand a bit of what is wrong with science, read this book:
http://www.doctoringdata.co.uk/

It is written by a practising UK doctor, and obviously it is concerned with medical science. It is fairly accessible, and it illustrates the layered institutional way in which inconvenient data is lost, or distorted over decades.

I mean Ben Goldacre's desire to have an open database of the details of medical trials, might reduce the problem, but how do you deal with the fact that most of those trials are sponsored by drug companies? Also, some scientific abuses are not visible in the data - that is just one reason not to get obsessed by statistical issues.

David
 
Last edited:
We still don't know how many people ate mice, which is where I guess the analogy fails.

I think you underestimate how much we do know, or at least have a good idea about.

Even if we haven't seen any actual mice stutter across the floor of the restaurant, if we know what to look for we can find all sorts of evidence of mice infestation: for example droppings, scratches, holes, smells. By studying the signs at a large enough sample of restaurants we can develop a reasonable estimate of how many restaurants have mice living in them, how likely a particular restaurant has a big problem or a little problem.

If you read papers such as the Why most published research is false that I posted above, plus other papers that have been published in the 10 years since we see a growing understanding of the methodological markers to watch out for that indicate risk of bias/error/reduction in reliability.

If you read that literature it becomes less surprising that many experiments fail to replicate. Frankly, that's how it should be, even with the system working optimally. Don't mistake me for encouraging sloppiness, we don't want that, but it is a mistake to attribute the problem primarily to sloppiness. These articles also discuss the difficulties in performing these experiments and the financial challenges involved. It simply does not make sense to invest the resources necessary for the highest quality experiments right off the bat. You start with less resource intensive, and less reliable experiments, and the ones that show promise should advance to more extensive study. This will naturally produce a lot of results that don't hold up- and not because of negligence.

Other experiments will not replicate because it is damn hard to come up with every factor that must be taken into account on the first go. Inevitably some variables will be missed in the early experiments. Even without negligence.

In my opinion as an interested layperson observer, it is not the fact that so many experiments failed to replicate that is unexpected. Rather, it is the apparent tendency of at least some scientists build on previous research that had not been suitably replicated.

One reason for this is that it is very expensive to do replications. this is where David's point that research needs to proceed more slowly is particularly important. Before reaearchers attempt to build on previous studies, they need to spend more time confirming the initial studies.

A number of reform projects currently going on aim at making this process easier and more efficient for scientists. Such as it public registries or increased online acess to source material.

Perhaps a better way to think of it is product recalls, where mass replication is the recall for all these varied bodies of work from the different fields. Or maybe a test for a disease is found to be faulty, and everyone who was believed to be infected needs to be retested?
.

That may very well be the case in some situations. And it always will be so long as we are dealing with fallible researchers with limited resources. It's unfortunate, but it's the nature of the game. We do the best that we can given practical realities. And we need to keep in mind that real research will rarely be theoretically perfect. There is no certainty and we should not expect it. We should expect that as science continues to move forward future results improve on past results, that errors are identified and steps taken to rectify.

Although this talk of mice reminds me of the potential issue with a variety of past studies due to mouse fear (IIRC Iyace mentioned this awhile back):

Lab mice fear men but not women, and that's a big problem for science: A new study casts doubt on decades of research



Perhaps there's been a review of all those studies since this was published, and it was all found to be okay? Or this study itself was fundamentally flawed, and there's no problem?

Thanks for linking to this. I've looked at it a bit but need to dig in a bit more, I'm going to get back on this but I suspect it will prove to be a really good example to illustrate what I'm trying to get at here.
 
If you read papers such as the Why most published research is false that I posted above, plus other papers that have been published in the 10 years since we see a growing understanding of the methodological markers to watch out for that indicate risk of bias/error/reduction in reliability.

I've gone over the Ioannidis paper multiple times (IIRC Linda even had a thread talking through it? Maybe that was another site though...). Admittedly I'd be happy if someone can talk us through some of the math as it's been some time since my studying of statistics.

In any case perhaps you can tell me where this idea that we can accurately [assess] the extent of the rot comes from? Here's a few excerpts from the paper pointing to specific problems - it doesn't seem to say anything about the problem being under control or not? -->

The smaller the effect sizes in a scientific field, the less likely the research findings are to be true. Power is also related to the effect size. Thus research findings are more likely true in scientific fields with large effects, such as the impact of smoking on cancer or cardiovascular disease (relative risks 3–20), than in scientific fields where postulated effects are small, such as genetic risk factors for multigenetic diseases (relative risks 1.1–1.5) [7]. Modern epidemiology is increasingly obliged to target smaller effect sizes [16]. Consequently, the proportion of true research findings is expected to decrease.

Corollary 5: The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true. Conflicts of interest and prejudice may increase bias, u. Conflicts of interest are very common in biomedical research [26], and typically they are inadequately and sparsely reported [26,27]. Prejudice may not necessarily have financial roots. Scientists in a given field may be prejudiced purely because of their belief in a scientific theory or commitment to their own findings. Many otherwise seemingly independent, university-based studies may be conducted for no other reason than to give physicians and researchers qualifications for promotion or tenure. Such nonfinancial conflicts may also lead to distorted reported results and interpretations. Prestigious investigators may suppress via the peer review process the appearance and dissemination of findings that refute their findings, thus condemning their field to perpetuate false dogma. Empirical evidence on expert opinion shows that it is extremely unreliable [28].

Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true. This seemingly paradoxical corollary follows because, as stated above, the PPV of isolated findings decreases when many teams of investigators are involved in the same field. This may explain why we occasionally see major excitement followed rapidly by severe disappointments in fields that draw wide attention. With many teams working on the same field and with massive experimental data being produced, timing is of the essence in beating competition. Thus, each team may prioritize on pursuing and disseminating its most impressive “positive” results. “Negative” results may become attractive for dissemination only if some other team has found a “positive” association on the same question. In that case, it may be attractive to refute a claim made in some prestigious journal. The term Proteus phenomenon has been coined to describe this phenomenon of rapidly alternating extreme research claims and extremely opposite refutations [29]. Empirical evidence suggests that this sequence of extreme opposites is very common in molecular genetics

Even the Nature article you linked mentions Ioannidis view on the problem of retractions:

In surveys, around 1–2% of scientists admit to having fabricated, falsified or modified data or results at least once (D. Fanelli PLoS ONE4,e5738; 2009). But over the past decade, retraction notices for published papers have increased from 0.001% of the total to only about 0.02%.And, Ioannidis says, that subset of papers is "the tip of the iceberg" — too small and fragmentary for any useful conclusions to be drawn about the overall rates of sloppiness or misconduct.

If you read that literature it becomes less surprising that many experiments fail to replicate. Frankly, that's how it should be, even with the system working optimally. Don't mistake me for encouraging sloppiness, we don't want that, but it is a mistake to attribute the problem primarily to sloppiness. These articles also discuss the difficulties in performing these experiments and the financial challenges involved. It simply does not make sense to invest the resources necessary for the highest quality experiments right off the bat. You start with less resource intensive, and less reliable experiments, and the ones that show promise should advance to more extensive study. This will naturally produce a lot of results that don't hold up- and not because of negligence.

Well the problems mentioned in this thread go beyond failed replications. There's also the question of how often experiments have been accepted as truth despite later failure to replicate.

Also these studies are getting published in journals and it's noted as being a problem.. From Cancer Research Is Broken: There’s a replication crisis in biomedicine—and no one even knows how deep it runs:

Many science funders share Parker’s antsiness over all the waste of time and money. In February, the White House announced its plan to put $1 billion toward a similar objective—a “Cancer Moonshot” aimed at making research more techy and efficient. But recent studies of the research enterprise reveal a more confounding issue, and one that won’t be solved with bigger grants and increasingly disruptive attitudes. The deeper problem is that much of cancer research in the lab—maybe even most of it—simply can’t be trusted. The data are corrupt. The findings are unstable. The science doesn’t work.

In other words, we face a replication crisis in the field of biomedicine, not unlike the one we’ve seen in psychology but with far more dire implications. Sloppy data analysis, contaminated lab materials, and poor experimental design all contribute to the problem. Last summer, Leonard P. Freedman, a scientist who worked for years in both academia and big pharma, published a paper with two colleagues on “the economics of reproducibility in preclinical research.” After reviewing the estimated prevalence of each of these flaws and fault-lines in biomedical literature, Freedman and his co-authors guessed that fully half of all results rest on shaky ground, and might not be replicable in other labs. These cancer studies don’t merely fail to find a cure; they might not offer any useful data whatsoever. Given current U.S. spending habits, the resulting waste amounts to more than $28 billion.

Look at the dismal replication rate [in psychology] that, to some degree, sparked the realization of the replication crisis in science:

Technical discussion aside, I want to make two points here. First, the Reproducibility Project is far from the only line of evidence for psychology’s problems. There’s the growing list of failures to replicate textbook phenomena. There’s publication bias—the tendency to only publish studies with positive results, while dismissing those with negative ones. There’s evidence ofquestionable research practices that are widespread and condoned.

The article even notes attempts to downplay the problem are, arguably, part of the problem:

“What is not helping is a reluctance to dig into our past and ask what needs revisiting. Time is nigh to reckon with our past.”


There was a Symposium on the reproducibility and reliably of biomedical research held in 2015 in the UK. Look at editor-in-chief Dr. Richard Horton's comments on the symposium:

“A lot of what is published is incorrect.” I’m not allowed to say who made this remark because we were asked to observe Chatham House rules. We were also asked not to take photographs of slides. Why the paranoid concern for secrecy and non-attribution? Because this symposium — on the reproducibility and reliability of biomedical research — touched on one of the most sensitive issues in science today: the idea that something has gone fundamentally wrong with one of our greatest human creations.

The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. As one participant put it, “poor methods get results”.

Can bad scientific practices be fixed? Part of the problem is that no-one is incentivised to be right. Instead, scientists are incentivised to be productive and innovative. Would a Hippocratic Oath for science help? Certainly don't add more layers of research red-tape. Instead of changing incentives, perhaps one could remove incentives altogether. Or insist on replicability statements in grant applications and research papers. Or emphasise collaboration, not competition. Or insist on preregistration of protocols. Or reward better pre and post publication peer review. Or improve research training and mentorship. Or implement the recommendations from our Series on increasing research value, published last year. One of the most convincing proposals came from outside the biomedical community. Tony Weidberg is a Professor of Particle Physics at Oxford. Following several high-profile errors, the particle physics community now invests great effort into intensive checking and re-checking of data prior to publication. By filtering results through independent working groups, physicists are encouraged to criticise. Good criticism is rewarded. The goal is a reliable result, and the incentives for scientists are aligned around this goal.

That may very well be the case in some situations. And it always will be so long as we are dealing with fallible researchers with limited resources. It's unfortunate, but it's the nature of the game. We do the best that we can given practical realities. And we need to keep in mind that real research will rarely be theoretically perfect. There is no certainty and we should not expect it. We should expect that as science continues to move forward future results improve on past results, that errors are identified and steps taken to rectify.

Without massive replication in every field, how do we know what results from the past are valid?

Of course this doesn't necessarily address other potential problems like fraud, prevention of new ideas, resistance to retraction, etc.
 
Last edited by a moderator:
In any case perhaps you can tell me where this idea that we can accurately [assess] the extent of the rot comes from? Here's a few excerpts from the paper pointing to specific problems - it doesn't seem to say anything about the problem being under control or not? -->
I agree with everything you wrote in that post. I just want to add, that research that 'demonstrates' no ψ effects, will obviously be less scrutinised than almost all the rest of the scientific output. It is assumed to be merely proving a self-evident truth! It is almost a waste of time looking at such research, because it is so often designed to get the 'right' result (science seems awfully good at that nowadays).

Ultimately, perhaps the biggest failure of current science may be be the failure to realise that we need to go beyond materialism.

David
 
I've gone over the Ioannidis paper multiple times (IIRC Linda even had a thread talking through it? Maybe that was another site though...). Admittedly I'd be happy if someone can talk us through some of the math as it's been some time since my studying of statistics.

There were a few discussions on it in the old forum, as I recall. Not sure if about on this one.

In any case perhaps you can tell me where this idea that we can accurately [assess] the extent of the rot comes from? Here's a few excerpts from the paper pointing to specific problems - it doesn't seem to say anything about the problem being under control or not? -->

Well, first of all, I said it was expected, not under control- if by under control you mean "not a problem". The entire point of Iaonnidis' paper is that there are problems that increase the bias (ie: error) of papers. IIRC Ioannidis put it around 50% and that is consistent with other evaluations I've seen.

For example, from the article by Richard Horton, editor in chief of the Lancet that you cite, in that statement, he notes that these problems are now getting due attention, however there has not been enough action.
Dr. Horton bemoans that "Those who have the power to act seem to think somebody else should act first". While I think he overlooks the efforts that are going on, I think he's probably right to an extent - there certainly needs to be more action, much more in fact, and especially when it comes to institutional change it takes courage to be the frontrunner - and no-one wants to make the situation worse!

The article highlights difficulties in coming up with what those institutional changes should be. The competing priorities. The pros and cons of each proposed solution. Yes change is urgently needed. But the reality is that it takes time, especially at the institutional level.

There is a LOT of work being done studying these issues. The Ioannidis paper has been cited almost 3800 times! I read somewhere that it is the most cited paper on PLOS. These replication studies are important in assessing the problem and in identifying solutions. There are many papers across diverse fields making recommendations to improve research methods.

Even the Nature article you linked mentions Ioannidis view on the problem of retractions:

Yes, the iceberg that Ioannidis is referring to is the 50%, only a small percentage of which is fraud. My point was not that fraud isn't a problem, but that in terms of focusing resources and making real improvements, the focus should be on the balance of the 50% (for example, that are discussed in those corrollaries you cited).

Well the problems mentioned in this thread go beyond failed replications. There's also the question of how often experiments have been accepted as truth despite later failure to replicate.

I highlighted my surprise that so many scientists seemed to be accepting as reliable studies that they really should not have been. This, in my opinion, is one of the most important discoveries of these replication studies.

Well yes! Haven't we been talking about this the whole discussions?

The article even notes attempts to downplay the problem are, arguably, part of the problem:

I don't think we should be downplaying it, I've stated so explicitly. And I agree that some degree of rhetoric is often necessary to galvanize action. But when trying to assess the situation accurately, or when discussing what should be done to address the issue, I think we need to take back and take a more sober assessment of the situation.

“What is not helping is a reluctance to dig into our past and ask what needs revisiting. Time is nigh to reckon with our past.”

There was a Symposium on the reproducibility and reliably of biomedical research held in 2015 in the UK. Look at editor-in-chief Dr. Richard Horton's comments on the symposium:

I've discussed that above. And I agree that we need to dig in to the past. My point was that this process is being done. It is not being ignored. There are thousands of papers on it!

Without massive replication in every field, how do we know what results from the past are valid?

There is going to have to be a lot of going back and reviewing old results. Which ones will have to be decided on by the researchers. Again, the history of science is that of gradually gaining better understanding of these issues, and correcting wrong ideas. It's a continuing process. There will have to be a balance between going back and moving forward, hopefully using methodologies based on our current understanding of best practices.
Of course this doesn't necessarily address other potential problems like fraud, prevention of new ideas, resistance to retraction, etc.
There are lots of issues to address!

I think there has been a nuance to my point of view that I may not have adequately expressed. I am not saying there aren't problems, there are, big ones. What I'm saying is that there have always been problems! The history of science is one of slowly, often very slowly, figuring out better ways to do science. And there will always be problems. No matter how good our understanding of best practices becomes, we have to expect a certain amount of impefection in the system. We can express shock everytime we learn that someone has not met this or that standard, but those situations are always going to happen. In any field, in any industry, in any activity for that matter.

For all the problems that have been identified over the last ten years, we're still in a better situation than we were - science wise - ten years before that, and ten years before that, etc.etc. etc. We may wish that science plugged all its holes earlier, but is that realistic? Given the nature of the task?
 
Retraction Action: Science Fraud Is Up, but More Retractions Could Be a Good Thing

Scientific retractions are on the rise. In 2001 there were 40 incidents in which published results of scientific research were retracted, but in less than a decade that number had ballooned to 400. And yes, the publication rate had also increased during that time, but by only 44 percent—not nearly enough to explain away a tenfold jump in retractions.

So why is this happening?

“Some people come out and say, ‘Well, clearly fraud is on the rise.’ And theoretically, that’s true…. A lot of these retractions are due to fraud,” says medical journalist Ivan Oransky, whose talk last night was co-sponsored by the Berkeley School of Public Health, the Graduate School of Journalism and Kaiser Permanente. Oransky runs Retraction Watch, a nonprofit that he cofounded with medical news editor Adam Marcus, which blows the whistle on scientific fraud and calls attention to retractions.

One study found that two-thirds of retractions in a sampling of biomedical and life-sciences research were due to misconduct—fraud, suspected fraud, plagiarism, or duplicate publication. Only about one in five retractions were attributable to mere error.

Even though the data suggests that retraction rates have been increasing since the 1950s, Oransky notes that he believes research today is more scrutinized. In other words, it’s possible that there was just as much need for retractions in the 1950s, but we weren’t looking so diligently to spot problems. And it’s important that we do so, because scientific “mistakes” can yield life-threatening results.

Among the most infamous examples is Andrew Wakefield, the anti-vaccine proponent who authored a fraudulent study that suggested measles, mumps and rubella vaccine can cause autism—resulting in plummeting vaccination rates and increases in measles in the United Kingdom and perhaps elsewhere. Another case involves Dong-Pyou Han, a former Iowa State University biochemist who falsified results of HIV vaccine trials by spiking rabbit blood samples with human antibodies to make it appear as if they developed HIV immunity. His deception could have had devastating public health consequences, and ultimately he was sentenced to five years in prison for research misconduct.

...often when publications are forced to say oopsies, the retractions themselves are vague.

For instance, a Retraction Watch post cites a retraction notice in Computational and Mathematical Methods in Medicine that reads: “This article has been retracted upon the authors request as it was found to include unreliable interpretation due to insufficient provision of studying materials.” This explanation is, as Oransky and Marcus note, “completely inscrutable.”

After all: What’s the point of a retraction when no one knows what the mistake was? It’s hard for scientists to monitor each other’s work without transparency.
 
Back
Top