The objectivity of national research foundation peer review in South Africa assessed against bibliometric indexes

Research output: Contribution to journalReview article

6 Citations (Scopus)

Abstract

This paper examines the strength of association between the outcomes of National Research Foundation (NRF) peer review based rating mechanisms, and a range of objective measures of performance of researchers. The analysis is conducted on 1932 scholars that have received an NRF rating or an NRF research chair. We find that on average scholars with higher NRF ratings record higher performance against research output and impact metrics. However, we also record anomalies in the probabilities of different NRF ratings when assessed against bibliometric performance measures, and record a disproportionately large incidence of scholars with high peer-review based ratings with low levels of recorded research output and impact. Moreover, we find strong cross-disciplinary differences in terms of the impact that objective levels of performance have on the probability of achieving different NRF ratings. Finally, we report evidence that NRF peer review is less likely to reward multi-authored research output than single-authored output. Claims of a lack of bias in NRF peer review are thus difficult to sustain.

Original languageEnglish (US)
Pages (from-to)177-206
Number of pages30
JournalScientometrics
Volume97
Issue number2
DOIs
StatePublished - Nov 2013

Fingerprint

peer review
objectivity
rating
performance
reward
incidence

All Science Journal Classification (ASJC) codes

  • Social Sciences(all)
  • Computer Science Applications
  • Library and Information Sciences

Cite this

@article{735c4cfb2a5a42c384de19b874f8950d,
title = "The objectivity of national research foundation peer review in South Africa assessed against bibliometric indexes",
abstract = "This paper examines the strength of association between the outcomes of National Research Foundation (NRF) peer review based rating mechanisms, and a range of objective measures of performance of researchers. The analysis is conducted on 1932 scholars that have received an NRF rating or an NRF research chair. We find that on average scholars with higher NRF ratings record higher performance against research output and impact metrics. However, we also record anomalies in the probabilities of different NRF ratings when assessed against bibliometric performance measures, and record a disproportionately large incidence of scholars with high peer-review based ratings with low levels of recorded research output and impact. Moreover, we find strong cross-disciplinary differences in terms of the impact that objective levels of performance have on the probability of achieving different NRF ratings. Finally, we report evidence that NRF peer review is less likely to reward multi-authored research output than single-authored output. Claims of a lack of bias in NRF peer review are thus difficult to sustain.",
author = "Fedderke, {J. W.}",
year = "2013",
month = "11",
doi = "10.1007/s11192-013-0981-0",
language = "English (US)",
volume = "97",
pages = "177--206",
journal = "Scientometrics",
issn = "0138-9130",
publisher = "Springer Netherlands",
number = "2",

}

The objectivity of national research foundation peer review in South Africa assessed against bibliometric indexes. / Fedderke, J. W.

In: Scientometrics, Vol. 97, No. 2, 11.2013, p. 177-206.

Research output: Contribution to journalReview article

TY - JOUR

T1 - The objectivity of national research foundation peer review in South Africa assessed against bibliometric indexes

AU - Fedderke, J. W.

PY - 2013/11

Y1 - 2013/11

N2 - This paper examines the strength of association between the outcomes of National Research Foundation (NRF) peer review based rating mechanisms, and a range of objective measures of performance of researchers. The analysis is conducted on 1932 scholars that have received an NRF rating or an NRF research chair. We find that on average scholars with higher NRF ratings record higher performance against research output and impact metrics. However, we also record anomalies in the probabilities of different NRF ratings when assessed against bibliometric performance measures, and record a disproportionately large incidence of scholars with high peer-review based ratings with low levels of recorded research output and impact. Moreover, we find strong cross-disciplinary differences in terms of the impact that objective levels of performance have on the probability of achieving different NRF ratings. Finally, we report evidence that NRF peer review is less likely to reward multi-authored research output than single-authored output. Claims of a lack of bias in NRF peer review are thus difficult to sustain.

AB - This paper examines the strength of association between the outcomes of National Research Foundation (NRF) peer review based rating mechanisms, and a range of objective measures of performance of researchers. The analysis is conducted on 1932 scholars that have received an NRF rating or an NRF research chair. We find that on average scholars with higher NRF ratings record higher performance against research output and impact metrics. However, we also record anomalies in the probabilities of different NRF ratings when assessed against bibliometric performance measures, and record a disproportionately large incidence of scholars with high peer-review based ratings with low levels of recorded research output and impact. Moreover, we find strong cross-disciplinary differences in terms of the impact that objective levels of performance have on the probability of achieving different NRF ratings. Finally, we report evidence that NRF peer review is less likely to reward multi-authored research output than single-authored output. Claims of a lack of bias in NRF peer review are thus difficult to sustain.

UR - http://www.scopus.com/inward/record.url?scp=84885856782&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84885856782&partnerID=8YFLogxK

U2 - 10.1007/s11192-013-0981-0

DO - 10.1007/s11192-013-0981-0

M3 - Review article

AN - SCOPUS:84885856782

VL - 97

SP - 177

EP - 206

JO - Scientometrics

JF - Scientometrics

SN - 0138-9130

IS - 2

ER -