Post: #1

Best article I have read this year. It was published three years ago in The New Yorker magazine.

Really interesting look at publication bias in science. And the subtle human biases which contribute to it.

Really makes you question alot of the 'facts' you have learned from medicine, psychology and biology.

Quote:One of the classic examples of selective reporting concerns the testing of acupuncture in different countries. While acupuncture is widely accepted as a medical treatment in various Asian countries, its use is much more contested in the West. These cultural differences have profoundly influenced the results of clinical trials.

Between 1966 and 1995, there were forty-seven studies of acupuncture in China, Taiwan, and Japan, and every single trial concluded that acupuncture was an effective treatment. During the same period, there were ninety-four clinical trials of acupuncture in the United States, Sweden, and the U.K., and only fifty-six per cent of these studies found any therapeutic benefits.

As Palmer notes, this wide discrepancy suggests that scientists find ways to confirm their preferred hypothesis, disregarding what they don’t want to see. Our beliefs are a form of blindness.

http://www.newyorker.com/reporting/2010/...ntPage=all

Post: #2

The whole article is great. It really adds detail to alot of the doubts I have about alot of science:

Quote:Although such reforms would mitigate the dangers of publication bias and selective reporting, they still wouldn’t erase the decline effect. This is largely because scientific research will always be shadowed by a force that can’t be curbed, only contained: sheer randomness. Although little research has been done on the experimental dangers of chance and happenstance, the research that exists isn’t encouraging.

In the late nineteen-nineties, John Crabbe, a neuroscientist at the Oregon Health and Science University, conducted an experiment that showed how unknowable chance events can skew tests of replicability. He performed a series of experiments on mouse behavior in three different science labs: in Albany, New York; Edmonton, Alberta; and Portland, Oregon. Before he conducted the experiments, he tried to standardize every variable he could think of. The same strains of mice were used in each lab, shipped on the same day from the same supplier. The animals were raised in the same kind of enclosure, with the same brand of sawdust bedding. They had been exposed to the same amount of incandescent light, were living with the same number of littermates, and were fed the exact same type of chow pellets. When the mice were handled, it was with the same kind of surgical glove, and when they were tested it was on the same equipment, at the same time in the morning.

The premise of this test of replicability, of course, is that each of the labs should have generated the same pattern of results. “If any set of experiments should have passed the test, it should have been ours,” Crabbe says. “But that’s not the way it turned out.” In one experiment, Crabbe injected a particular strain of mouse with cocaine. In Portland the mice given the drug moved, on average, six hundred centimetres more than they normally did; in Albany they moved seven hundred and one additional centimetres. But in the Edmonton lab they moved more than five thousand additional centimetres. Similar deviations were observed in a test of anxiety. Furthermore, these inconsistencies didn’t follow any detectable pattern. In Portland one strain of mouse proved most anxious, while in Albany another strain won that distinction.

The disturbing implication of the Crabbe study is that a lot of extraordinary scientific data are nothing but noise. The hyperactivity of those coked-up Edmonton mice wasn’t an interesting new fact—it was a meaningless outlier, a by-product of invisible variables we don’t understand. The problem, of course, is that such dramatic findings are also the most likely to get published in prestigious journals, since the data are both statistically significant and entirely unexpected. Grants get written, follow-up studies are conducted. The end result is a scientific accident that can take years to unravel.

Also - something not mentioned in the article - but worth thinking about - is pessimistic meta-induction:

http://en.wikipedia.org/wiki/Pessimistic_induction

Post: #3

I wouldn't call it a 'subtle' bias.

I'd rather call it 'blatant' bias.

80-90% of the world's research output is junk.

80-90% of the relevant research is being done by 10-20% of the output.

Although with the arrival of big data, there is now so much noise that I think we'll end up in a 1%:99% ratio.

No one publishes science for the sake of science.

Science is published if the sake of being published.

Very important difference to grasp.

It's important to be able to actually read the full paper and have a grasp of statistics and research methodology, otherwise it's easy to be fooled.

Post: #4

As a practicing scientist the largest bias is not publishing negative results. All scientists are guilty of this. There's only so much time in the day and we'd much rather be publishing shit that we find interesting than the stuff that didn't work. There's a huge amount of work required to publish a research result and it's just not worth putting in the effort for negative results even though the field as a whole would probably benefit.

Post: #5

Yeah, there was an issue of the Economist written about this a few months ago. http://www.economist.com/news/leaders/21...goes-wrong

Post: #6

It's all about churning out papers to get the next grant. Like my former PI once said "It's a business."