Identifying outstanding research proposals: an examination of reviewers’ judgement processes

Background

Understanding proposal evaluation helps explain the science of peer review. This study aimed to identify how experienced reviewers use application information when making funding recommendations.

Methods

Social judgement theory methods were used to examine experienced peer reviewers’ funding recommendations. The research had three phases. Phases: I) ‘think aloud’ interviews was undertaken by 7 reviewers on a grant proposal they reviewed for the Wellcome Trust’s ‘Research Fellowship in Humanities and Social Sciences’ scheme, to uncover quality factors used: II) 28 expert reviewers identified and ranked the 10 most important quality factors, which were then used to generate a set of 60 fellowship application vignettes using fractional factorial design III) 24 reviewers made funding recommendations individually, on the vignettes. Multiple regression analysis was used to model the weighting given to the factors.

Results

Fifty-seven quality factors were initially uncovered through the content analysis of the think aloud protocols, and the 10 factors deemed most important in assessing the quality of fellowship applications were identified. Accounting for 92% of the variance, the 24 expert reviewers used all 10 factors in their decision making; the applicant’s publication record was the most significant in terms of funding recommendation, followed by the applicant’s experience and expertise. Reliability of the mean ratings across 22 reviewers’ recommendations was very high: ICC(2,22) = 0.94 (95% CI, 0.915 to 0.96).

Conclusions

There was high interrater reliability of reviews in this funding scheme. The focus on the candidate’s publication record and experience is likely to be due to the funding scheme being for ‘fellowships’, where the potential development of the individual is central to the award. This information may be used differently when considering ‘project’ funding applications. The Narrative CV has helped widen recognition of research contributions beyond traditional metrics. Findings will support the ongoing development of peer reviewing approaches.

Plain Language summary

Peer reviewers assess the quality of funding applications and advise funders whether proposals should be supported. Little is known about how peer reviewers use information to make their recommendations. This study examined how experienced reviewers evaluate fellowship applications.

Seven reviewers were invited to participate in a “think aloud” interview while assessing a real fellowship application, allowing us to identify 57 factors considered when judging quality. A separate group of 28 experienced reviewers then selected and ranked the 10 most important factors. These factors were used to create 60 hypothetical fellowship applications, presented online as vignettes. Finally, 24 experienced reviewers assessed the vignettes and made funding recommendations.

Reviewers used all 10 factors when making decisions and were very similar in the way they used information. An applicant's publication record had the strongest influence on recommendations. Fellowships aim to support the development of applicants as well as their research. Since this study, narrative CVs have been introduced to broaden the applicants' information considered in funding applications.