Peer review review

Peer review review

Practically within living memory peer review has passed from amounting to intelligent and practical consultation and evaluation of one's work, through a phase of helpful, if not necessarily standardised criticism, to being an obstacle course best traversed by specialists in its traversal, rather than by specialists in scientific research. In many fields it has degenerated into a ritual that commonly is meaningful mainly in offering establishment conspiracies the opportunity to exclude inconvenient publications. For some polemicists peer review is a mantra, a talisman with which they cow inconsiderate journalists and paralyse embarrassing public discussion. Though it once had a well-defined function, peer review now has largely lost that function so untraceably that few practitioners in relevant fields could even give a coherent description of what that function should be, let alone what it now is.

This question forced itself on my simultaneously fascinated and disgusted attention years ago, when I still was interested in the climate controversy, and until the Climategate storm broke. This essay is an edited version of what I had to say then, though I don't expect it to gain much more attention now than it did then. But I post it here because it deals with a point far more important than the question of anthropogenic global warming (good ol' AGW!)

Incidentally, in case anyone is about to wave the denialist stick at me, don't bother; you are barking up the wrong tree. My attitude is neither for nor against AGW nor any other climate change; I am mainly interested in the ethics, practice and application of science and its publication, and that is what this essay is about. If AGW or any other form of climate change (such as The Sixth Winter)  begins to bite, the question of Humanity's continued survival increasingly will assume a fascination all its own, but for the present there are more immediate concerns, in particular the question of the survival of integrity and sense in science, and the associated role of peer review.

Consider: "Michael Mann of Pennsylvania State University said that if Mr. McIntyre wants to be taken seriously he has to move more from blogging to publishing in the refereed literature. . ."  and "'Skepticism is essential for the functioning of science,' Dr. Mann said. 'It yields an erratic path towards eventual truth. But legitimate scientific skepticism is exercised through formal scientific circles, in particular the peer review process. . . Those such as McIntyre who operate almost entirely outside of this system are not to be trusted." (Quoted in i.a. http://dotearth.blogs.nytimes.com/2009/10/05/climate-auditor-challenged-to-do-climate-science/)

Such stern sententiousness, irrespective of its source, should shame the most unregenerate cavillers to kennel, except those who pause slitty-eyed, to reflect on what he actually meant.  "Peer review process", hmmm. . .?  Those operating almost entirely inside of this system are to be trusted, are they? The same system that passed all sorts of publications of the most assorted standards during the last century or so, not to mention certain particularly embarrassing examples very recently?  Publications that led to blushes inversely proportional to how effectively and for how long the parties concerned could distract attention from them? The same peer review process that has served as the most powerful tool for intimidating, quashing, and crippling the slightest dissent from the approved line?  For punishing anyone who breaks the ranks of the favoured? For protecting the reputations of friends whose earlier work was being discredited by the work tendered for publication? For emasculating or deferring publication of the research of upstarts? The most powerful weapon for delaying outsiders' discoveries to the point of loss of priority of publication, or even to fatal obscurity?

Surely not! Which is fortunate, because that is not the point that I had referred to. Plenty of abler critics have raised similar objections more bitingly than ever I could.

No, the peer review that I write to praise and not to bury is the peer review that for generations of scientists has been the sentinel and shield against erosion of standards. It has been a sheet anchor both of the elite and the merely workmanlike journal, the means of assuring the editorial staff that the work they publish is sound, non-trivial, constructive, an advance on preceding work, a stone in the edifice of growing human knowledge. It has been an aid to efficiency, speeding the selection and augmenting the quality of the product of the researchers' labour and ingenuity; and of course (though perish the thought of any such sordid considerations crossing the mind of the authors) enhancing the kudos appertaining to the publication of the item.

Good Stuff. Very good indeed.

And yet I cannot rid my mind of a framed engineering degree on the wall of the office of an erstwhile young colleague of mine. It was in a large company, employing many graduates, and yet he was the only one that I remember nailing his colours to the er, wall in such a way. Any time the standard of his work or his good sense got challenged, he would point at his degree in rebuttal.  Unanswerable of course.

And yet he did not last long, strangely.

Am I the only one to see this anecdote as relevant? Sorry. . .

Peer review as it should be used in a perfect world should not be a major concern of the author except when a generous reviewer offers assistance or admonition, typically anonymous.

Peer review also should not be a major concern of the reader; if I read material dealing with a field I am so unfamiliar with that I cannot even follow the train of logic, then I act in bad faith and bad sense if I accept or condemn it on the grounds that it was or was not peer-reviewed.  If however I can follow the logic, but without being able to challenge actual facts or observations, then I am able, with appropriate reservations, to accept, challenge, or reject the logic in good faith, but I still cannot justify my opinion by reliance on any peer review process. If I can claim to be fully conversant with the field, then I can accept, challenge, or reject any part, context or aspect of the work. If in doing so I need to defer to the dread dignity of peer reviewers, than how can I claim competence in the field at all? If I need to ask how it was reviewed before I consent to trust the work, then why am I reading such stuff, when there are plenty of Mills & Boone books to challenge my intellect?

Peer review or no review, it is for all readers to accept or reject research results according to what they find personally convincing. Readers who are reduced to trusting work on the basis that it has been peer reviewed, could about equally well accept its accuracy, validity, importance and good faith on the grounds that they had read about it in the Sun or Daily Mirror. These are respected publications of course, but many members of the research community have reservations on their like as sources of evaluation and explication of leading-edge scientific work. On the other hand, if peer review is intended to rival run-of-the-mill tabloid journalism in its value to the proverbial man on the Clapham omnibus, it might thereby perform a useful function, but a function that strikes me as ignominious if at all effective.

In any case, in good sense or in good faith no research worker can justify a decision to accept or challenge work according to whether it had been peer reviewed.

That is not what peer review is for.

Learned publications, whether for the benefit of the learned or anyone else, hardly benefit anyone at all if they depend on referees for validation. In good faith or good sense referees can no more believe or disbelieve factual claims or scientific arguments on behalf of readers, than prescribe moral views or artistic merit. Peer review has nothing to do with validation. It is a recourse for the editor rather than the reader, and accordingly (mercifully) is rarely published. The ideal editor of a learned journal in the ideal world would have the time and specialist knowledge to evaluate every item submitted for publication without peer assistance, but perfection is not in us, not even editors, and even the perfect editor has limited time. Understandably, as he riffles through submissions, he discards many out of hand, but occasionally: "Could be... Mike and Maurice have made the running there lately; let them referee it." 

He is not bound by the reactions of M&M; as presumed experts -- competent, diligent, prompt, and fair, irrespective of their reputations, or whether they concur -- they provide their perspectives as resources, not mandates. Instead, truly independent evaluation would be nice, but hardly possible; independence of judgement in abstruse matters almost guarantees ignorance: expert judgement requires engagement, personal work, thought, and wide-ranging study, not intellectual isolation. The best one can hope for is balance, flexibility, and perspective rather than independence. Apart from criticism, such a paragon might offer helpful comments to pass on to the authors, but that is a luxury. 

Consider what an unusually conscientious and competent referee might say: "Section two is too trivial for inclusion, not so? Section three conflicts with the widely accepted results of Blank (reference X), presumably because of the error in deriving equation 5 (e.g. examine the case for B=0). Section six uses inappropriate statistical methods (reference Y). The authors fail to clarify how they selected the data in the main graph from the voluminous raw data they very properly supplied. As it stands, it might attract accusations of bias. The materials and methods section needs drastic revision; it is almost incomprehensible.  I cannot recommend publication before these issues are dealt with."

And so on. Such contributions amount to specialised editorial assistance. They mitigate neither the responsibilities of authors nor the challenge to readers who must understand enough to trust or reject the material accordingly. Readers owe neither belief nor trust, either to authors or editors, and they cannot in good faith or sense elect to accept or reject the conclusion in the title or abstract simply because the journal routinely refers articles to referees. Such referral may establish respect, but not indisputability; it does not even guarantee competence. Even elite journals require untiring and responsible enforcement of standards to maintain their stature; lip-service won't do.  Rarely an editor might protest: "It wasn't my fault, but the referees'." Rarely understanding readers might respond: "it happens in the best of families."

In the best of families it happens more rarely and less ignominiously.

To criticise or praise a journal because of its eschewal or quality of peer review could be reasonable in suitable contexts; even if one were to assume that the editor were omniscient, it might be comforting to reflect that independent review guarantees lack of bias. However, to acclaim or condemn the work of an author because it had or had not been favourably peer reviewed, is the most breathtakingly abject tactic I have seen, short of running crying to mummy because these nasty people had been disagreeing with ums. The more I contemplate it, the less it makes sense.

Consider what such justification for rejection amounts to: non-peer-reviewed material is work that some third parties somewhere, who hadn't been asked to vet the work, but who might or might not have approved it if they had been asked, had not actually said anything about the work because they had not been asked. Right? So because the work was not considered by those third parties, it thereby is errr... to be neglected without rebuttal by those in response to whose work it had been presented? Irrespective of logic or available evidence?

When or why should we respect or trust authors whose work had not been peer reviewed? How about because they pointed out facts concerning calculations and behaviour in work that has been published in the sanctum sanctorum of modern science, top international journals that not only apply peer review rigorously, but insist on receiving and keeping full copies of supporting data when it would be too cumbersome to publish it? Especially when it turns out that those journals had neglected their self-avowed duty and practice? In the examples under consideration, the criticism after all, did not involve novel work or novel techniques, but a critique of (peer reviewed) work in terms of the very claims and arguments in that selfsame work. What role is peer review of the critique to play in such a case? What sort of peril to author, journal or public would such peer review be intended to avert? Given that the criticism is public and couched in the very idiom and context of the work, what would be the point of peer review of the criticism? It would be open to rebuttal by everyone who read and understood it anyway.

For precisely such reasons, even in top scientific journals, letters to the editor in response to peer reviewed articles are not in general required to be peer reviewed. Right? And if they were, what would demands for such peer review suggest? And who would review the reviewers? And in reports of primary research, what about (unusual, but not unique) instances where the editorial staff announce reasons why they have elected to publish work over the objections of one or more of the reviewers or even without review? Surely peer review is infallible, or why rely on it in the first place, or trust it as we are exhorted to do? And what do we see in rebuttal by the trustworthy supporters of the trustworthy peer-reviewed work? Cries of "total garbage", statements to the effect that a critic only would be taken "... seriously if he creates his own temperature reconstructions and submits them for peer review".  

Do your own web scanning if you don't believe me.

Note that the critic in question was not trying to establish a thing about temperature reconstructions, apart from the point that the arguments for certain reconstructions by someone else were internally invalid and incomplete.  By a similar line of reasoning, if I see a circus acrobat claiming to do a quintuple somersault, and find he is doing only a double somersault, then I am not permitted to repudiate his claim unless I can do a quintuple somersault?

Dear reader, forbear to analyse any such responses in terms of science, logic, ethics or even self respect. You will achieve nothing for your popularity in certain quarters. And that popularity is what attracts people to science isn't it?

Isn't it?

Oh.

Oh well...

Never mind! Let's get back to the real world.

This much at least should be clear: science is passing through a most painful phase. (At least I hope that "passing" is not too optimistic a word!) As scientists we have a century or so of frequently (not invariably) inappropriate reliance on a cumbersome system. We have to deal with problems of ethics, politics, information explosion, population explosion, and technology explosion. In my opinion the peer review system in its current form has outlived its usefulness, in many respects even its viability. Whether the next generation is to rely on something totally new or on an amended review system, I cannot say, but what served for say the 1950s is hardly likely to serve for the 2050s. Some developments apparently are in development within some Internet publication media, in which pre-publications are exposed to public execration or appreciation before the final editing; such publication may point the way to the future, but whatever form future publication takes, something new certainly is needed.

Whether it turns out to be in the interest of the editorial staff, the author, or the reader, the fact remains that peer review as she currently is spoke, notionally is primarily a tool of the editorial staff. It is only contingently for the benefit of the author, and usually irrelevant to the reader, whether fan, friend or foe.  But those who appeal to the process for shelter from unwelcome assessments of their work, or their duties to their readers; for some reason recall to me two lines of Burns written in a slightly different context:

From Envy and Hatred your corps is exempt,
But where is your shield from the darts of Contempt!

In case that strikes you as insulting, I invite you to consider it in the perspective of the insult to the reader at whom certain helpful remarks were directed -- remarks of the form: "Those such as McIntyre who operate almost entirely outside of this system [of the peer review process] are not to be trusted." We readers apparently are seen as stupid enough to swallow the hockey stick without choking on the mediaeval optimum or little ice age, but too stupid to gag at the implications of the physics of photon absorption, the history of volcanic influences on the climate, the principles of sample significance, or the implication of withheld data -- and far, far too stupid to read a statistical argument?

Unless it is peer reviewed?

...

...

...

Pause for applause to die down and catcalls to grind to a halt in hoarseness...

I wrote most of the foregoing some years ago, about at the time of Climategate. It occurred to me that it might be of interest here, partly because I see that the essay was quoted in one or two places, without my permission, but as far as I could tell, in good faith, and accordingly also without my objections, so here it is, slightly edited, but without any intentional material change.

One point that it should have been unnecessary to mention is that nothing in the article addresses the merits of the various ideas and accusations concerning climate change, desirable, undesirable, or indifferent. I did not assert any merit, demerit nor even agnosticism concerning AGW, neither then, nor now. I urge any reader to observe that this essay only tangentially has anything to do with climate or its study, logic or facts. It addresses issues that I consider far more important. Anyone bothering to respond please do me the courtesy of refraining from discussing AGW itself, partly because any such material is outside the scope of this essay.

But why so sensitive, you may ask? Well, in my editing of this old material in preparation for the new presentation in this blog, I did a bit of web surfing and inevitably encountered material that left me actually physically nauseated; it included explicit abuse of logic, honesty, and good faith on the part of prominent persons, some of them politicians and partisans, of whom one expects no better, but also some persons in positions of authority in research and learned publishing. Insofar as there are two sides to the question (if you can identify two sides cleanly and coherently, you are better at it than I am, and more enterprising as well) the dirt flinging and nonsense milling abounded on each side. About it and about. It read like sample lists of logical fallacies: Special pleading. Ad hominem. Red herrings that would shame a middle-school debate. Proof by assertion of absurdities and legalisms concerning what peer review or research are or should be. Some state in effect that because peer review isn't auditing, therefore matters of fact, figures, or calculation are not to be challenged by reviewers...

A great darkness of the spirit descended on me as I abandoned such threads, apparently peopled at least partly by practising research workers.

I am not without my views on climate change, but they are not denialist, not affirmative, and as I have hinted, not even agnostic except in many, many considerations where there are matters too far outside my competence for me to offer any substantial opinion. I might have been interested in discussion of technological means of climate control, energy efficiency, and associated fields, but they are not relevant to this essay anyway, and as far as I can tell the associated levels of venom and logic are not much more attractive or constructive than those concerning AGW.

This article deals with peer review, its fundamental implications and relevance.

So spare me the rest. 

...

...

...

Since writing the foregoing I have been mildly surprised and deeply disgusted to read about systematic peer review fraud -- too self-pityingly disgusted to detail it. If you care enough and your digestion will stand it, have a read at:
http://www.the-scientist.com/?articles.view/articleNo/41534/title/Peer-Review-Manipulation-/
and
http://www.the-scientist.com//?articles.view/articleNo/43130/title/Retractions-Often-Due-to-Plagiarism--Study/

How did that go again? We have an expert in the ethics of science and scientific publishing assuring us that: "...legitimate scientific skepticism is exercised through formal scientific circles, in particular the peer review process. . . Those such as McIntyre who operate almost entirely outside of this system are not to be trusted." And that from one of those who had conspired in private to control and frustrate that very process, to avoid rational assessment in public, of his own work and competence, both ethically and functionally.

And last I heard, this person was still practising!

What can I say? As I type this I feel actually dizzy with revulsion.

Best say nothing I suppose; this is an unreviewed blog entry -- not to be trusted.  

On Certain Recent Advances in Scientific Proof

On Certain Recent Advances in Scientific Proof.

(A lightly edited version of an essay I wrote in the days of Climategate. Think of how things have changed since then...)

Climategate has been a nearly universally nauseating experience for anyone who cares for good facts or good faith, but let us not be too negative. In its own way it has been instructive.  Even nausea can favour the prepared mind with insights, as Pasteur didn't quite say: for example, in the small hours one night, many years ago, I gained a helpful insight into the significance of motion sickness; I was bowed over a lavatory bowl in obeisance to a transient, but peremptory, attack of viral gastritis, and too sleepy to concentrate on the personal nature of a normally very personal experience. This left my mind free to concentrate on the resemblance between my sensations and those of seasickness. 

Fascinating.

And Climategate in all its unexpectedness takes me in much the same way. I have lived a largely sheltered life, most of my associates being persons of as much good sense, good will, and honesty as anyone might reasonably ask. One gets spoilt. Perhaps the occasional experience of the likes of Climategate is a necessary corrective for the poor, abused victims of sheltered lives. In my attempts to develop a proper sense of gratitude however, I could not help reflecting that this particular course of treatment seems excessively expensive, and pernicious to boot. An occasional violent two-minute submersion in sewage, followed by a couple of hours in Monty Python' s room for abuse or the one for an argument should be just as effective and far cheaper. 

(Hmmm... I feel I am onto something here; I must contact Al Gore for advice on how to apply for a peace prize.)

Now, even a superficial sampling of Internet exchanges on the subject of Climategate will yield a pretty representative list of the classical logical modes of proof. Argumentum ad hominem occurs so frequently in various forms, ranging from fairly reasonable to downright hysterical, that one ceases to notice it. Argumentum ad baculum is rife, hardly surprisingly. Argumentum ad misericordiam is popular, and those are just examples. Ticking off the lists is too easy to be interesting.

Possibly least surprising has been the ubiquitous Red Herring, the long-standing favourite of the lawyer, the politician, and Sir Humphrey Appleby in his various incarnations. In fairness however, it would be ungracious to dismiss the artless hopefulness, the sheer creativity, and the deftness, with which different, often diametrically opposing, simultaneous Red Herring trails are intermingled and combined. What starts out as kipper ends up as kedgeree. 

Such tours de force of persuasion are undeniably impressive, but hardly novel. The Internet supports expression with impunity better than the traditional gutter press, or even Hyde Park Corner, where appropriate sensitivity might forefend painful physical counter-arguments before police can step in. Immunity from punched heads and thick lips has encouraged online development of novel forms of scientific proof, and Climategate has fostered many. I lack the time, energy, and erudition to construct a diagnostic taxonomic key, but readers will recognise hopeful seedlings that never could have survived on the stony ground of traditional civilised debate. 

Consider a few examples. 

To begin with, proof by assertion is rife. Proof by stamina is common (to call it Argumentum ad Nauseam would do the Internet version scant justice). Proof by abuse nearly deafens one to read it. Proof by undirected rage practically covers the reader with spittle when the page comes up. Proofs of the form: "he was my professor and I have worked under him and he would never do such a thing..." comes in six-packs, exculpating prominent scientists who publicly have delivered themselves of the likes of: "Why should I make the data available to you, when your aim is to try and find something wrong with it". 

(No, I jest not, nor speak in metaphor, worse luck.)

Then there is Proof by Democracy, in which organisations appeal to the widely-accepted fact that thousands of "scientists" form a "consensus". This they distinguish from Argumentum ad populum by repeated double-entry accountancy which expands a quorum of arguably a few dozen to an army of thousands (still a pretty sparse consensus mind you, but undeniably impressive propagation, all the same). They sagely spare us tedious data concerning the proportion of these scientists that are climatologists supporting their climatological consensus as professionals working in the field.

 That makes good sense, because each scientist, as a professional in a relevant discipline, should no doubt contribute his or her own skills to the relevant questions, of which there are many apart from puristic climatology. Toleration in science is of course a virtue in virtuous moderation. Excessive tolerance in science being a fault however, those selfsame professional cadres of climatologists then are very properly dismissive of tactless questions. Among infidels, non-climatologists may be dismissed out of hand by condemning them as ignorant denialists; no one but a climatologist could validly base climatological doubts on their mastery as climatologists, of pure logic, or empirical knowledge of history, archaeology, palaeontology, physics, thermodynamics, astronomy, statistics, engineering or biology. And any sort of climatologist who actually expresses doubt or reservation on any such points must be in the pay of Satanic corporations.  

Denialists all! Proof positive. 

Nor should anyone harbour doubts simply because the Prominent Faithful occasionally present false or conflicting pronouncements, much less conflicting computer models. Such aberrations offer no extenuation for Denialism; after all, the science has been proved by assertion and re-affirmed by Scientific Consensus (Cast of thousands! Don't miss the show!) What does a bit of inconsistency count for in unanimity on such a scale?  

One towering figure refuses to be intimidated into revealing his data to non-climatologists? That surely is nothing but praiseworthy amour propre? Another, a veritable data-Hampden, will not yield in his protection of his data from anyone who might try to find something wrong with it? Well, naturally if it has nothing wrong with it, how could it not deserve protection? So, yet another climatologist rejected data that failed to support his thesis? Well, of course he should! What is the point of data that is not a team player? See for how many years judicious exclusion of data averted the growth of the ozone hole!

Nothing in all that invalidates their conclusions or their credibility. On the other hand, when the correspondence of such Good People is made publicly available, that patently is a criminal act, in stark contrast to responsible data protection or negation. And criminality logically demonstrates that any uncomfortable premises thus exposed should be rejected as being without merit. 

Mmm...  No doubt. If you say so. 

I must confess that those hacker fellows had me going for a while there, so the Faithful patently have a point... 

So there was I, demoralised by all the denialists, totally unable to imagine how the IPCC could retain any credence in the AGW thesis, much less in carbon reduction as a mitigation strategy; a right slough of despond. But then the Good and the Great assembled to save the world from itself, from the denialists and their paymasters. In their dread assurance of their new Forms of Proof they brooked neither doubt nor denial. Son of Kyoto will surely make good all that we got wrong the first time round, all the bickering, all the futility, all the misdirected technology, and especially all the corruption. 

After all, if it were otherwise, why should all those disinterested leaders and savants in their tens of thousands, expose their persons, their careers, their efforts, their models, their credibility, to denigration by denialists, to the exhaustion of travel and conferences, to the discomfort of foreign hotels, to the difficulties of explaining the inexplicable to press and public, or their conference expenditure to their countrymen, why, I repeat, should they commit without thought of self or of reward, to the cause?

And so reflecting I turn to the news broadcasts. And inevitably more dignified spokesmen appear to explain it all again and yet again in language more emphatic than four-letter words, loudly, reprovingly, uncompromisingly, urgently, even angrily. Just as in the days of Kyoto, there is no time to spare; doom still is imminent and no denialist should question it, nor even discuss it in terms that could give rise to pointed off-script questions. 

And anon nausea assails the prepared mind once more as it did over the lavatory bowl one night of yore. And do you know what I hear? Every time any of them opens his mouth I hear a cash register, loud and clear. 

Only, I don't know how to prove it.