Saturday, November 29, 2014

Peer review review




Peer review review

Practically within living memory peer review has passed from amounting to intelligent and practical consultation and evaluation of one's work, through a phase of helpful, if not necessarily standardised criticism, to being an obstacle course best traversed by specialists in its traversal, rather than by specialists in scientific research. In many fields it has degenerated into a ritual that commonly is meaningful mainly in offering establishment conspiracies the opportunity to exclude inconvenient publications. For some polemicists peer review is a mantra, a talisman with which they cow inconsiderate journalists and paralyse embarrassing public discussion. Though it once had a well-defined function, peer review now has largely lost that function so untraceably that few practitioners in relevant fields could even give a coherent description of what that function should be, let alone what it now is.
This question forced itself on my simultaneously fascinated and disgusted attention years ago, when I still was interested in the climate controversy, and until the Climategate storm broke. This essay is an edited version of what I had to say then, though I don't expect it to gain much more attention now than it did then. But I post it here because it deals with a point far more important than the question of anthropogenic global warming (good ol' AGW!)

Incidentally, in case anyone is about to wave the denialist stick at me, don't bother; you are barking up the wrong tree. My attitude is neither for nor against AGW nor any other climate change; I am mainly interested in the ethics, practice and application of science and its publication, and that is what this essay is about. If AGW or any other form of climate change (such as The Sixth Winter)  begins to bite, the question of Humanity's continued survival increasingly will assume a fascination all its own, but for the present there are more immediate concerns, in particular the question of the survival of integrity and sense in science, and the associated role of peer review.

Consider: "Michael Mann of Pennsylvania State University said that if Mr. McIntyre wants to be taken seriously he has to move more from blogging to publishing in the refereed literature. . ."  and "'Skepticism is essential for the functioning of science,' Dr. Mann said. 'It yields an erratic path towards eventual truth. But legitimate scientific skepticism is exercised through formal scientific circles, in particular the peer review process. . . Those such as McIntyre who operate almost entirely outside of this system are not to be trusted." (Quoted in i.a. http://dotearth.blogs.nytimes.com/2009/10/05/climate-auditor-challenged-to-do-climate-science/)

Such stern sententiousness, irrespective of its source, should shame the most unregenerate cavillers to kennel, except those who pause slitty-eyed, to reflect on what he actually meant.  "Peer review process", hmmm. . .?  Those operating almost entirely inside of this system are to be trusted, are they? The same system that passed all sorts of publications of the most assorted standards during the last century or so, not to mention certain particularly embarrassing examples very recently?  Publications that led to blushes inversely proportional to how effectively and for how long the parties concerned could distract attention from them? The same peer review process that has served as the most powerful tool for intimidating, quashing, and crippling the slightest dissent from the approved line?  For punishing anyone who breaks the ranks of the favoured? For protecting the reputations of friends whose earlier work was being discredited by the work tendered for publication? For emasculating or deferring publication of the research of upstarts? The most powerful weapon for delaying outsiders' discoveries to the point of loss of priority of publication, or even to fatal obscurity?

Surely not! Which is fortunate, because that is not the point that I had referred to. Plenty of abler critics have raised similar objections more bitingly than ever I could.

No, the peer review that I write to praise and not to bury is the peer review that for generations of scientists has been the sentinel and shield against erosion of standards. It has been a sheet anchor both of the elite and the merely workmanlike journal, the means of assuring the editorial staff that the work they publish is sound, non-trivial, constructive, an advance on preceding work, a stone in the edifice of growing human knowledge. It has been an aid to efficiency, speeding the selection and augmenting the quality of the product of the researchers' labour and ingenuity; and of course (though perish the thought of any such sordid considerations crossing the mind of the authors) enhancing the kudos appertaining to the publication of the item.

Good Stuff. Very good indeed.

And yet I cannot rid my mind of a framed engineering degree on the wall of the office of an erstwhile young colleague of mine. It was in a large company, employing many graduates, and yet he was the only one that I remember nailing his colours to the er, wall in such a way. Any time the standard of his work or his good sense got challenged, he would point at his degree in rebuttal.  Unanswerable of course.

And yet he did not last long, strangely.

Am I the only one to see this anecdote as relevant? Sorry. . .

Peer review as it should be used in a perfect world should not be a major concern of the author except when a generous reviewer offers assistance or admonition, typically anonymous.

Peer review also should not be a major concern of the reader; if I read material dealing with a field I am so unfamiliar with that I cannot even follow the train of logic, then I act in bad faith and bad sense if I accept or condemn it on the grounds that it was or was not peer-reviewed.  If however I can follow the logic, but without being able to challenge actual facts or observations, then I am able, with appropriate reservations, to accept, challenge, or reject the logic in good faith, but I still cannot justify my opinion by reliance on any peer review process. If I can claim to be fully conversant with the field, then I can accept, challenge, or reject any part, context or aspect of the work. If in doing so I need to defer to the dread dignity of peer reviewers, than how can I claim competence in the field at all? If I need to ask how it was reviewed before I consent to trust the work, then why am I reading such stuff, when there are plenty of Mills & Boone books to challenge my intellect?

Peer review or no review, it is for all readers to accept or reject research results according to what they find personally convincing. Readers who are reduced to trusting work on the basis that it has been peer reviewed, could about equally well accept its accuracy, validity, importance and good faith on the grounds that they had read about it in the Sun or Daily Mirror. These are respected publications of course, but many members of the research community have reservations on their like as sources of evaluation and explication of leading-edge scientific work. On the other hand, if peer review is intended to rival run-of-the-mill tabloid journalism in its value to the proverbial man on the Clapham omnibus, it might thereby perform a useful function, but a function that strikes me as ignominious if at all effective.

In any case, in good sense or in good faith no research worker can justify a decision to accept or challenge work according to whether it had been peer reviewed.


That is not what peer review is for.


Learned publications, whether for the benefit of the learned or anyone else, hardly benefit anyone at all if they depend on referees for validation. In good faith or good sense referees can no more believe or disbelieve factual claims or scientific arguments on behalf of readers, than prescribe moral views or artistic merit. Peer review has nothing to do with validation. It is a recourse for the editor rather than the reader, and accordingly (mercifully) is rarely published. The ideal editor of a learned journal in the ideal world would have the time and specialist knowledge to evaluate every item submitted for publication without peer assistance, but perfection is not in us, not even editors, and even the perfect editor has limited time. Understandably, as he riffles through submissions, he discards many out of hand, but occasionally: "Could be... Mike and Maurice have made the running there lately; let them referee it." 

He is not bound by the reactions of M&M; as presumed experts -- competent, diligent, prompt, and fair, irrespective of their reputations, or whether they concur -- they provide their perspectives as resources, not mandates. Instead, truly independent evaluation would be nice, but hardly possible; independence of judgement in abstruse matters almost guarantees ignorance: expert judgement requires engagement, personal work, thought, and wide-ranging study, not intellectual isolation. The best one can hope for is balance, flexibility, and perspective rather than independence. Apart from criticism, such a paragon might offer helpful comments to pass on to the authors, but that is a luxury. 


Consider what an unusually conscientious and competent referee might say: "Section two is too trivial for inclusion, not so? Section three conflicts with the widely accepted results of Blank (reference X), presumably because of the error in deriving equation 5 (e.g. examine the case for B=0). Section six uses inappropriate statistical methods (reference Y). The authors fail to clarify how they selected the data in the main graph from the voluminous raw data they very properly supplied. As it stands, it might attract accusations of bias. The materials and methods section needs drastic revision; it is almost incomprehensible.  I cannot recommend publication before these issues are dealt with."

And so on. Such contributions amount to specialised editorial assistance. They mitigate neither the responsibilities of authors nor the challenge to readers who must understand enough to trust or reject the material accordingly. Readers owe neither belief nor trust, either to authors or editors, and they cannot in good faith or sense elect to accept or reject the conclusion in the title or abstract simply because the journal routinely refers articles to referees. Such referral may establish respect, but not indisputability; it does not even guarantee competence. Even elite journals require untiring and responsible enforcement of standards to maintain their stature; lip-service won't do.  Rarely an editor might protest: "It wasn't my fault, but the referees'." Rarely understanding readers might respond: "it happens in the best of families."

In the best of families it happens more rarely and less ignominiously.

To criticise or praise a journal because of its eschewal or quality of peer review could be reasonable in suitable contexts; even if one were to assume that the editor were omniscient, it might be comforting to reflect that independent review guarantees lack of bias. However, to acclaim or condemn the work of an author because it had or had not been favourably peer reviewed, is the most breathtakingly abject tactic I have seen, short of running crying to mummy because these nasty people had been disagreeing with ums. The more I contemplate it, the less it makes sense.

Consider what such justification for rejection amounts to: non-peer-reviewed material is work that some third parties somewhere, who hadn't been asked to vet the work, but who might or might not have approved it if they had been asked, had not actually said anything about the work because they had not been asked. Right? So because the work was not considered by those third parties, it thereby is errr... to be neglected without rebuttal by those in response to whose work it had been presented? Irrespective of logic or available evidence?

When or why should we respect or trust authors whose work had not been peer reviewed? How about because they pointed out facts concerning calculations and behaviour in work that has been published in the sanctum sanctorum of modern science, top international journals that not only apply peer review rigorously, but insist on receiving and keeping full copies of supporting data when it would be too cumbersome to publish it? Especially when it turns out that those journals had neglected their self-avowed duty and practice? In the examples under consideration, the criticism after all, did not involve novel work or novel techniques, but a critique of (peer reviewed) work in terms of the very claims and arguments in that selfsame work. What role is peer review of the critique to play in such a case? What sort of peril to author, journal or public would such peer review be intended to avert? Given that the criticism is public and couched in the very idiom and context of the work, what would be the point of peer review of the criticism? It would be open to rebuttal by everyone who read and understood it anyway.

For precisely such reasons, even in top scientific journals, letters to the editor in response to peer reviewed articles are not in general required to be peer reviewed. Right? And if they were, what would demands for such peer review suggest? And who would review the reviewers? And in reports of primary research, what about (unusual, but not unique) instances where the editorial staff announce reasons why they have elected to publish work over the objections of one or more of the reviewers or even without review? Surely peer review is infallible, or why rely on it in the first place, or trust it as we are exhorted to do? And what do we see in rebuttal by the trustworthy supporters of the trustworthy peer-reviewed work? Cries of "total garbage", statements to the effect that a critic only would be taken "... seriously if he creates his own temperature reconstructions and submits them for peer review".  

Do your own web scanning if you don't believe me.

Note that the critic in question was not trying to establish a thing about temperature reconstructions, apart from the point that the arguments for certain reconstructions by someone else were internally invalid and incomplete.  By a similar line of reasoning, if I see a circus acrobat claiming to do a quintuple somersault, and find he is doing only a double somersault, then I am not permitted to repudiate his claim unless I can do a quintuple somersault?

Dear reader, forbear to analyse any such responses in terms of science, logic, ethics or even self respect. You will achieve nothing for your popularity in certain quarters. And that popularity is what attracts people to science isn't it?

Isn't it?

Oh.

Oh well...

Never mind! Let's get back to the real world.

This much at least should be clear: science is passing through a most painful phase. (At least I hope that "passing" is not too optimistic a word!) As scientists we have a century or so of frequently (not invariably) inappropriate reliance on a cumbersome system. We have to deal with problems of ethics, politics, information explosion, population explosion, and technology explosion. In my opinion the peer review system in its current form has outlived its usefulness, in many respects even its viability. Whether the next generation is to rely on something totally new or on an amended review system, I cannot say, but what served for say the 1950s is hardly likely to serve for the 2050s. Some developments apparently are in development within some Internet publication media, in which pre-publications are exposed to public execration or appreciation before the final editing; such publication may point the way to the future, but whatever form future publication takes, something new certainly is needed.

Whether it turns out to be in the interest of the editorial staff, the author, or the reader, the fact remains that peer review as she currently is spoke, notionally is primarily a tool of the editorial staff. It is only contingently for the benefit of the author, and usually irrelevant to the reader, whether fan, friend or foe.  But those who appeal to the process for shelter from unwelcome assessments of their work, or their duties to their readers; for some reason recall to me two lines of Burns written in a slightly different context:

From Envy and Hatred your corps is exempt,
But where is your shield from the darts of Contempt!

In case that strikes you as insulting, I invite you to consider it in the perspective of the insult to the reader at whom certain helpful remarks were directed -- remarks of the form: "Those such as McIntyre who operate almost entirely outside of this system [of the peer review process] are not to be trusted." We readers apparently are seen as stupid enough to swallow the hockey stick without choking on the mediaeval optimum or little ice age, but too stupid to gag at the implications of the physics of photon absorption, the history of volcanic influences on the climate, the principles of sample significance, or the implication of withheld data -- and far, far too stupid to read a statistical argument?

Unless it is peer reviewed?

...

...

...

Pause for applause to die down and catcalls to grind to a halt in hoarseness...

I wrote most of the foregoing some years ago, about at the time of Climategate. It occurred to me that it might be of interest here, partly because I see that the essay was quoted in one or two places, without my permission, but as far as I could tell, in good faith, and accordingly also without my objections, so here it is, slightly edited, but without any intentional material change.

One point that it should have been unnecessary to mention is that nothing in the article addresses the merits of the various ideas and accusations concerning climate change, desirable, undesirable, or indifferent. I did not assert any merit, demerit nor even agnosticism concerning AGW, neither then, nor now. I urge any reader to observe that this essay only tangentially has anything to do with climate or its study, logic or facts. It addresses issues that I consider far more important. Anyone bothering to respond please do me the courtesy of refraining from discussing AGW itself, partly because any such material is outside the scope of this essay.

But why so sensitive, you may ask? Well, in my editing of this old material in preparation for the new presentation in this blog, I did a bit of web surfing and inevitably encountered material that left me actually physically nauseated; it included explicit abuse of logic, honesty, and good faith on the part of prominent persons, some of them politicians and partisans, of whom one expects no better, but also some persons in positions of authority in research and learned publishing. Insofar as there are two sides to the question (if you can identify two sides cleanly and coherently, you are better at it than I am, and more enterprising as well) the dirt flinging and nonsense milling abounded on each side. About it and about. It read like sample lists of logical fallacies: Special pleading. Ad hominem. Red herrings that would shame a middle-school debate. Proof by assertion of absurdities and legalisms concerning what peer review or research are or should be. Some state in effect that because peer review isn't auditing, therefore matters of fact, figures, or calculation are not to be challenged by reviewers...

A great darkness of the spirit descended on me as I abandoned such threads, apparently peopled at least partly by practising research workers.

I am not without my views on climate change, but they are not denialist, not affirmative, and as I have hinted, not even agnostic except in many, many considerations where there are matters too far outside my competence for me to offer any substantial opinion. I might have been interested in discussion of technological means of climate control, energy efficiency, and associated fields, but they are not relevant to this essay anyway, and as far as I can tell the associated levels of venom and logic are not much more attractive or constructive than those concerning AGW.

This article deals with peer review, its fundamental implications and relevance.

So spare me the rest. 

...

...

...

Since writing the foregoing I have been mildly surprised and deeply disgusted to read about systematic peer review fraud -- too self-pityingly disgusted to detail it. If you care enough and your digestion will stand it, have a read at:
http://www.the-scientist.com/?articles.view/articleNo/41534/title/Peer-Review-Manipulation-/
and

http://www.the-scientist.com//?articles.view/articleNo/43130/title/Retractions-Often-Due-to-Plagiarism--Study/

How did that go again? We have an expert in the ethics of science and scientific publishing assuring us that: "...legitimate scientific skepticism is exercised through formal scientific circles, in particular the peer review process. . . Those such as McIntyre who operate almost entirely outside of this system are not to be trusted." And that from one of those who had conspired in private to control and frustrate that very process, to avoid rational assessment in public, of his own work and competence, both ethically and functionally.

And last I heard, this person was still practising!

What can I say? As I type this I feel actually dizzy with revulsion.

Best say nothing I suppose; this is an unreviewed blog entry -- not to be trusted.  



2 comments:

  1. What a drivel. What's the alternative to peer review then?

    ReplyDelete
    Replies
    1. Read the article. If you still can't understand, get mummy to explain the haaard parts.

      I wouldn't like to burden you with anything as difficult as THINKING about it, or reading some of the developments in the field, so never mind.

      Delete