To add 2 cents to the peer review discussion I think the
following is the most importanat:
(1) Peer review by a reasonably broad representation
of 'experts' may not be that bad in priniple, PROVIDED
there is a reliable mechanism of HEDGING it. By this
I mean that (in the case of grant applications) if there
is some mixed peer review assessment then there should
be some averaging of the scored points and SOME level
of funding should be awarded - low if score is low, but
zero only in case of unanimity of ALL peer reviewers
that the entire work in not worth a penny. In short,
the case should really be very weak by all parameters
in order not to get ANY funding. I expect that
only very few (if any) will reach such a bottom.
(2) Also, in addition to peer review scoring,
some OBJECTIVE scoring should be added to it. For
example, each paper published in a peer reviewed
journal should bring a point.
(3) We have advocated such hedged schemes for years
and many people (Don Forsdyke, Rostom Roy, etc) gave
rather elaborated designs how it all can be
implemented in practice. In short, such scheme mean
"don't leave everything up to the whim of GSC",
SOMETING should be a result of some (more or
less) OBJECTIVE parametization of the work in
question.
(4) Understanably, NOTHING of this kind was done
or ever will be done as long as the present
uncontrolled, unaccountable and self-serving
structure of NSERC remains in place. This is
simply a matter of power grab and has very little
(or nothing) to do with all built-in limitations
of peer review. Any progress (if any possible)
can only be achieved along this route (of questioning
power structure), not along the line what should not
and can not be perfected (peer review).
Cheers
Alex Berezin
On Mon, 22 Mar 1999, Joseph Pear wrote:
> The following is in response to Dick's correspondence regarding peer
> review:
>> It seems to me that the definition of "peer" in the term "peer review"
> needs clarification. Does it refer to the members of the Grant Selection
> Committees or to the external "referees" to whom the GSCs send the
> applications for reviews? Since the external referees are typically
> workers in the applicant's field whereas the members of the GSCs often are
> not, I think it should be the former rather than the latter. In my
> experience with all three of the major granting agencies, however, I find
> that I typically get excellent reviews from the external referees and yet
> my applications are usually NILed by the GSCs. Moreover, the stated
> reasons for disapproving my applications are usually unrelated to the
> external referees comments and appear to make little sense from the
> perspective of workers in my field. Thus, I think that perhaps before
> considering the merits and demerits of peer review we might ask whether,
> in fact, the frequent invocation of the term "peer review" and the great
> display made of obtaining the opinions of external referees are simply
> being used as a smoke screen. In other words there may not be a problem
> with peer review per se, but with the fact that peer reviews are not taken
> seriously resulting in little correlation between peer review and the
> awarding of grants.
>> - Joe Pear
>> P.S. Note that peer review of journal articles is quite different from
> peer review of grant applications, and overall seems to work fairly well.
> The reason I would say is that in the processing of journal articles peer
> review is not just a matter of show but is actually used.
>>
