Member Login

Lost your password?

Registration is closed

Sorry, you are not allowed to register by yourself on this site!


Scholarship 2.0: Rethinking “impact factor”

[NOTE 1: this is a follow-up post to something I wrote that was published in the UCEA Review (warning: pdf) and posted to the UCEA website. That was about how technology affords scholars the opportunity to be public with their work and to disseminate their knowledge more widely. Here, I write about how technology affords us new ways of thinking about and “knowing” how impactful our work is, especially when we use the open web for knowledge dissemination.]
[NOTE 2: There are many scholars who have studied/thought/written about impact factor for much longer than I have. I write this as an early synthesis of what I’ve learned and specifically for my colleagues in the field of education.]
[NOTE 3: I’m writing this here, now, but I’m also thinking that I can combine the earlier piece, this piece, and maybe a bit on modern forms of peer-review to create a more comprehensive “article” on modern scholarship in education to be submitted to an open access, peer-reviewed journal.]

Impact factor is a metric used to quantify the quality of a journal. It is basically a measure reflecting the relative frequency with which articles in a journal are cited in science and social science journals. There are a few different such measures, but the one most commonly referred to is the impact factor computed by Thompson Reuters ISI, which houses a massive index of academic journals (the “Web of Science”). This particular impact factor is computed as a fraction: the numerator is the number of citations a journal received in a given year to articles published in the previous two years1; the denominator is the total number of articles published in that journal over those two previous years. As an example, to compute a 2010 impact factor for a journal (call it Journal J), let’s say that Journal J published 50 total articles in 2008 and 2009. If 25 articles published in 2010 in other journals (or in Journal J) cited articles published in Journal J during 2008 or 2009, Journal J would have an impact factor of 0.5 in 2010. In other words, on average, each of the articles published in Journal J in 2008 and 2009 was cited 0.5 times in 2010.

Those of you in the professoriate are likely in the same position as me, where tenure and/or promotion are tied to scholarly output, which is largely judged by the impact factors of the journals in which we have published. That is, we largely judge each other’s scholarship based on impact factor.

Numerous critiques of this “quantification of quality” have been made. In a recent article in the Journal of Philosophy of Education, Smeyers & Burbules (2011) offer a particularly compelling critique. They offer a number of problems with the way impact factors are computed and used. Consider just the following four concerns:

  1. First and foremost, the numerator does not include citations/references in books, book chapters, newspapers, or web-based publications (which is particularly problematic in the age of digital information).
  2. The impact factor is a journal-level metric and not necessarily an indicator of article quality. They cite an example offered by Monastersky (2005): “…a quarter of the articles in Nature last year drew 89 percent of the citations to that journal, so a vast majority of the articles received far fewer than the average of 32 citations reflected in the most recent impact factor.”
  3. Furthermore, among the 25% of articles that drew 89% of the citations, one article might have been cited frequently (thus driving up the impact factor of the journal that it is in) when it is actually being “…cited critically, as a bad example or as a foil for another argument.” In this way, impact factor may be more a measure of publicity than quality.
  4. Finally, there is the self-fulfilling nature of the impact factor. We might call this the social construction of journal quality. For example, if you were to ask any professor of educational leadership what the “top” journal in the field is, you would more often than not hear “Educational Administration Quarterly (EAQ).” Why? Because we are socialized within the discipline to believe that. Therefore, scholars within the discipline read that journal most frequently and look to it most frequently for warrants for their knowledge claims. Lazy Busy scholars of educational leadership will not necessarily find the time to search for the best warrants. So, of course EAQ articles will be cited more frequently, thus boosting its impact factor.

Smeyers & Burbules are most convincing when writing about the privileged place of quantification in academia.

There is something intrinsically reifying about measures like impact factors: they are a god-send for overworked and distracted officials, because they serve as a handy shortcut; they obviate the need for more difficult and potentially controversial judgments, such as actually reading and arguing over the value of scholarship; their pseudo-scientific aura seems to protect review processes from certain kinds of challenge,including legal challenge, because everyone is treated ‘the same’ procedurally and held to ‘the same’ criteria.

Smeyers & Burbules mention alternate methods of computing impact factors (see e.g. the Australian government’s Australian Research Council,  HERE and HERE). They also address the “future of the impact factor” and throw out possibilities such as “counting downloads, or hyperlinks…,” but, ultimately, come up short in this part of the article.

For “overworked and distracted officials,” short of expecting them to actually read and make independent judgments of a body of scholarship, the question, then, is how might we “judge” impact in the future. Or, stated differently, if we take seriously the idea of scholars as public intellectuals embracing the affordances of the open web, it behooves us to consider how to rethink “impact factor.”

Fortunately, there are pieces of a puzzle that are starting to come together. There are certainly other ideas out there, but consider these four possibilities:

  • Rick Hess’s public presence ranking system is far from perfect. He uses a combination of Google scholar, Amazon.com, and mentions in widely-disseminated education publications (namely Education Week and the Chronicle of Higher Education), blogs, newspapers, and the Congressional Record.
  • Hess’ use of use of the Google Scholar score preceded Google’s announcement of Google Scholar Citations which is, “a statistical model based on author names, bibliographic data, and article content to group articles likely written by the same author.” Once an author identifies her articles, Google collects citations to them, computes three different dynamic citation metrics (they are automatically updated as new citations to articles are found) and graphs the metrics over time.

Those two approaches allow us to think about two things: the use of multiple measures to get at “impact,” and how one might start to scrape data from the open web. Getting a bit more technologically sophisticated, however, we start to see some greater possibilities.

  • brainSCANr (The Brain Systems, Connections, Associations, and Network Relationships) is a budding/growing system that explores the relationships between terms (in their case, for now, neuroscience) in peer-reviewed publications. As is written on their homepage, the system “…assumes that somewhere in all the chaos and noise of the more than 20 million papers…there must be some order and rationality.” By clicking here, you see a dynamic sociograph of the term “happiness” is related to other terms within the articles in the PubMed database. The image below is a static version of the sociograph pulled on August 3, 2011.

While BrainSCANr is a technology used to examine relationships between terms, it does not take a huge stretch to imagine a similar system that examines relationships between articles, authors, etc. BrainSCANr uses social network analysis (SNA) which can be used to generate statistics related to the density of a network or the centrality of a node (which could be an individual, an idea, an article, etc.). Density of a network around an article is not unlike current impact factor metrics, but centrality metrics might add to our thinking about impact.

  • In addition to centrality within a network of articles/ideas, we might also consider adding “engagement” metrics to a multi-factored look at impact. PostRank “monitors and collects social engagement events correlated with online content in real-time across the web. PostRank gathers where and when stories generate comments, bookmarks, tweets, and other forms of interaction from a host of social hubs.” PostRank has been acquired by Google. Recently, the folks at PostRank ran a “quick experiment” to see if they could determine what the most engaging TED talks have been. The initial result was a spreadsheet of 766 TED talks, which has become a dynamic site where you can view TED talks ranked by PostRank’s engagement score. Engagement is measured by the number of times a given talk has been tweeted, bookmarked (on Delicious), posted (to Facebook), and voted (on Reddit). Engagement is essentially a measure of virality.

Public presence, density, centrality, engagement…these are concepts that we need to embrace and harness as we re-think what “impact factor” might look like as scholars disseminate knowledge more widely on the open web. Modern archiving and computing technologies allow us to do SO much better than relying on outdated, limited, self-fulfilling metrics of impact factor.

  1. It is important to note that for the Thompson Reuters ISI impact factor, only citations from and to journals that are indexed in the Web of Science get counted. []