Is Peer Review broken?

[rbohm]

Originally Posted by Fiddlehead

As an Associate Editor for a journal, I had done a lot of reviewing, and I have had many of my papers reviewed. I tend to be pretty successful, but I have been excoriated many times. It almost always improves my work, so I appreciate it. I don't tend to publish speculative stuff, so the reviews keep me from shooting myself in the foot. They are very valuable.

Peer review is not perfect. But generally, it is self-correcting. When I review a paper, I tend to look at whether the arguments are sound, the methods sound and sufficiently explained as to be repeatable, and that the conclusions match with the results of the analysis. The papers I reject tend to be where the questions are trivial or poorly stated, the methods are bogus or unclear, and especially, when one makes grand conclusions that are not supported by the data. By nature, most scientists assume you are an idiot and talking out your azz unless you make it clear you are not. So, my impression after many years is that most scientists err on the side of being too intellectually conservative rather than suporting grand theories that might possibly be supported by the facts. If you want that you go to the blogosphere.

Do bad papers get published? Yes, sometimes.

Do good papers get rejected? Yes, often (we tend to be conservative and dismissive).

Does politics enter into the peer review process? Rarely. It happens, but most of us are focused on the paper in front of us, and its merits or lack thereof.

One worrying trend that I see is an increase in online journals that actually invite papers. If they are looking to fill pages, I don't trust the rigor of their peer review process. For most of the high stature journals (Science, Nature, Global Change Biology, New England Journal of Medicine,etc.) the peer review process is very rigorous.

Originally Posted by Glitch

I think it depends on the topic. The more controversial the topic, the less likely there will be good reviews. I read dozens of astrophysics papers annually, which tend not to be very controversial. The biggest problem I am finding in those papers is a tendency for hyperbole. Apparently there is a lot of competition in the astronomy profession to find the furthest object, or the exoplanet that is closest in size to Earth, or to find the first forms of alien life, etc., etc. As a result many of these papers make assertions that are not supported by observations or the facts. It would seem that fame is a powerful motivator in the scientific community, almost as powerful as money.

In the case of topics like global warming, you are talking about an industry that is competing for hundreds of billions of dollars in government funding. Funding that is only issued if their findings support the predetermined government agenda.

Politics and science is just as bad of a combination as religion and science. Both have a predetermined conclusion and they are attempting to use science to prove their point of view. Nobody ever sat down and looked at all the data first then concluded that the planet is getting warmer. They first concluded that the planet is getting warmer, then they set out to prove it. That is not science, that is politics.

Where politics is often about concealing the truth, science is about understanding the truth. It does not matter if the evidence and observations prove one right or wrong, it is exciting just knowing the truth either way.

there is a lot of truth in these posts about peer review. and i agree the system is fractured, and needs some reforming, but it shouldnt be dismissed entirely. if the politics can be eliminated as completely as possible from the process, there will always be some politics in the process, then it can get back to what it was designed to ultimately be.