The importance of individualized article-specific metrics for evaluating research productivity
Retrovirology volume 6, Article number: 82 (2009)
This editorial discusses the rationale for using article-specific rather than journal-specific metrics for evaluating highly published authors.
Mark Patterson of PLoS (Public Library of Science) recently wrote an online piece http://www.plos.org/cms/node/478 on how to measure impact where it matters. Patterson makes an important point that one should focus on article specific metrics when evaluating a published paper rather than relying "on the name and the impact factor (IF) of the journal in which the work is published". In the past, it was not always easy to assess quickly and accurately the citations to individually published articles. Today, many electronic tools (e.g. ISI Web of Science, Scopus, Google Scholar) exist that can accomplish this task facilely and reliably. Because there are inherent shortcomings to how a journal's IF is calculated and because of the rather poor representativeness of the IF for the citations to individual articles [1, 2], institutions and peer-review bodies should be encouraged strongly to employ article-specific measures in preference to journal IFs in evaluations.
Article-specific citations are often not used properly in evaluating published authors. For example, in some circles, it has become fashionable to create lists of "highly cited" scientists in various fields (e.g. http://isihighlycited.com/; highly cited in immunology, highly cited in microbiology, highly cited in molecular biology and genetics etc...). In some respects, these lists could be useful conveniences, provided that the users understand clearly how they are generated and what they mean (and do not mean). One could assume that "highly cited in microbiology" is based on article specific-citations. In fact, this would be a mistaken assumption because the listing is actually based on journal-specific data. What does this mean? By way of explanation, let's consider a hypothetical illustration. If John Smith were an author of 10 papers on HIV-1 published in Cell or the Journal of Biological Chemistry (which are not counted by ISIHighlyCited as microbiology journals) and if these 10 papers were cited cumulatively 1,000 times over a specified duration, then Smith's citation counts based on these papers for purposes of "highly cited in microbiology" would be 0. On the other hand, if the exactly same 10 Smith papers on HIV were unsuccessful in initial submissions to Cell or the Journal of Biological Chemistry, but were subsequently successfully published in the Journal of Virology, Retrovirology, or Virology (all counted as microbiology journals), then the 1,000 citations to these papers would add 1,000 counts to Smith's ranking for purposes of "highly cited in microbiology". So, here is an example where journal-specific metrics trump article-specific measures. In order to be "highly cited in microbiology", what one publishes (i.e. article-specific content on HIV) counts not unless it is published in a journal deemed as "microbiology" (i.e. a journal-specific metric). Thus, this illustration shows that ratings based on journal-specific data that do not properly integrate article-specific measures can be misleading when used to rate scientists. For retrovirologists, Retrovirology has emphasized consistently the use of person-specific measures of H-index  and total citations. Indeed, annually for the past three years, these data have been presented, using the Scopus data base http://www.scopus.com, for selected Retrovirology editorial board members (see Table 1) [1, 2].
Finally, one should not overlook the merits of awards and prizes in evaluating highly accomplished colleagues. Awards/prizes can come in two flavors; one as "leading" and the other as "lagging" indicators of scientific potential/productivity. For example, "life-time achievement" awards would be a "lagging" measure of one's achievements, while a "young" investigator prize might be a "leading" indicator of future potential. Retrovirology annually awards a "Retrovirology Prize" to a mid-career scientist [4, 5]. The Prize aims to recognize "lagging" and "leading" benchmarks. It rewards the past achievements of a scientist who is in his/her mid-career and who still has substantial lead-time to accomplish future breakthrough research in retrovirology [5–8]. With this editorial, this year's nomination period for the 2009 Retrovirology Prize to recognize a retrovirologist for non-HIV-retrovirology research is open. The nomination period will close on October 31, 2009. The rules for nomination and the selection procedures remain the same as in past years [9, 10]. Interested individuals can direct email inquiries to email@example.com.
Jeang KT: Impact factor, H index, peer comparisons, and Retrovirology: is it time to individualize citation metrics?. Retrovirology. 2007, 4: 42-10.1186/1742-4690-4-42.
Jeang KT: H-index, mentoring-index, highly-cited and highly-accessed: how to evaluate scientists?. Retrovirology. 2008, 5: 106-10.1186/1742-4690-5-106.
Hirsch JE: Does the H index have predictive power?. Proc Natl Acad Sci USA. 2007, 104: 19193-19198. 10.1073/pnas.0707962104.
Jeang KT: Life after 45 and before 60: the Retrovirology Prize. Retrovirology. 2005, 2: 26-10.1186/1742-4690-2-26.
Jeang KT: The 2008 Retrovirology Prize: Ben Berkhout and his RNA world. Retrovirology. 2008, 5: 113-10.1186/1742-4690-5-113.
Jeang KT: Small philanthropy and big science: the RETROVIROLOGY prize and Stephen P. Goff. Retrovirology. 2005, 2: 43-10.1186/1742-4690-2-43.
Lever AM: Science--a life fully lived: Joe Sodroski wins the 2006 Retrovirology Prize. Retrovirology. 2006, 3: 45-10.1186/1742-4690-3-45.
Boris-Lawrie K: Bridging fundamental RNA biology, retroviral replication, and oncogenesis: Karen Beemon wins the 2007 Retrovirology Prize. Retrovirology. 2007, 4: 88-10.1186/1742-4690-4-88.
Jeang KT: The young, not-so-young, and the 2007 Retrovirology Prize: call for nominations. Retrovirology. 2007, 4: 64-10.1186/1742-4690-4-64.
Jeang KT: Recognizing mid-career productivity: the 2008 Retrovirology Prize, call for nomination. Retrovirology. 2008, 5: 80-10.1186/1742-4690-5-80.
I thank Andrew Dayton and Mark Wainberg for critically reading this editorial and Daniel Schmidt for assistance in preparing Table 1.
KTJ wrote this editorial.
About this article
Cite this article
Jeang, KT. The importance of individualized article-specific metrics for evaluating research productivity. Retrovirology 6, 82 (2009). https://0-doi-org.brum.beds.ac.uk/10.1186/1742-4690-6-82