Highly read article on the European Research Council by Dr. Lucas Brunet and Prof. Dr. Ruth Müller

The new article by Dr. Lucas Brunet and Prof. Dr. Ruth Müller published in the journal Research Evaluation has been highly shared and downloaded, reaching almost 1500 views in less than one month after its publication. Since two weeks, the article is in the second position of the Most Read Articles in the journal.

The article examines the evaluation practices of panel reviewers in the European Research Council. To the article: https://academic.oup.com/rev/advance-article/doi/10.1093/reseval/rvac040/6845514?searchresult=1

Abstract:

The European Research Council (ERC) receives many high-quality applications, but funds only a few. We analyze how members of ERC review panels assess applications in the first, highly competitive step of evaluations for ERC Starting and Consolidator Grants. Drawing on interviews with ERC panel members in different fields, we show that they adopt a set of evaluation devices that offer pragmatic and standardized ways of evaluating in a time-constrained and highly competitive setting. Through the use of evaluation devices, panel reviewers enact and generate a distinct reviewing expertise that encompasses subject-specific knowledge and knowledge about how to accomplish evaluation within a situated setting. We find that ERC panel reviewers employ four evaluation devices during the first step of ERC reviews: first, reviewers base judgments on applicants’ prior achievements (delegation devices); second, they adjust their evaluations of individual applications to the quality of a given set of applications (calibration devices); third, they combine multiple elements to assess the feasibility of proposals (articulation devices); and finally, they consider the impact of the proposed research on science and society (contribution devices). We show that the current use of these devices generates what we have termed evaluative pragmatism: a mode of reviewing that is shaped by and accommodated to the need to review many high-quality proposals in a short time period with possibly limited expert knowledge. In conclusion, we discuss how the prevalence of evaluative pragmatism in the first step of ERC panel reviews shapes candidate selection, particularly regarding human and epistemic diversity in European research.

Reference: Brunet, L., & Müller, R. (2022). Making the cut: How panel reviewers use evaluation devices to select applications at the European Research Council. Research Evaluation. https://academic.oup.com/rev/advance-article/doi/10.1093/reseval/rvac040/6845514?searchresult=1