Citation Metrics: Trust But Verify

Citation metric: a) A numerical measure of the influence of a journal or journal article based on the number of citations.
b) An simplistic way of evaluating the value of research based on flawed numbers, that fails to take into account the actual complexities involved in evaluation, but is still seen as being valuable because it provides a clear numerical way to rank things, as the people doing the evaluation very often have neither the time nor the background to evaluate the research personally.

Citation Counts

In this post, we’ll look at the most basic citation metric of all: citation counts. Simply put, a citation count is a count of citations that have been made to <something>. The <something> could be a book, a book chapter or essay, a journal article, a white paper, a technical report, a conference paper, a video, a webpage, a blog, a newspaper article, a magazine article, a preprint, a dataset, a patent, an unpublished communication, a corporate research report, a standard, an image, a performance, and so on and so forth.

The citation could also appear in almost any one of those (though perhaps not an image!).

Where to find citation counts

It seems simple enough. We have Article A, written by one of our many notable faculty. How many times has it been cited? But of course, not all citations are equal! If we want to evaluate Professor A’s work, we only want citations made by people who are peers of Professor A; fellow researchers, academics, or professionals. We also only want citations that appear in proper research, academic, or professional publications, not popular magazines, newspapers, and the ilk, since we want to evaluate what impact Professor A’s research has had on their field.

To that end, the Library offers two commercial tools , and one free one.

Citation Tools

Web of Knowledge (also know as the Web of Science, the Science (or Social Science or Arts and Humanities) Citation Index. The oldest source of citation data (besides counting it yourself), with data going back to the 1970s in some disciplines  It has a fairly narrow focus, and only indexes citation made in what they consider the significant journals in particular fields. It doesn’t count citations made in anything that isn’t one of the significant journals. So if your article was cited in a book, a patent, a journal the WoK doesn’t consider significant, etc. the WoK doesn’t know about it.

Scopus. A newer database that has citation data for citations made in a selection of journal and conference papers published from 1996 onward. It covers quite a bit more journals than WoK, and does cover some conference proceedings, but it focuses a bit more on science, technology, and engineering disciplines. It does include some coverage of the social sciences (business and psychology predominately), and the arts and humanities, but those aren’t its strengths. Like the WoK, it doesn’t cover books, and doesn’t cover all journals.

Google Scholar. Google’s attempt to index all of the scholarly material on the Internet. It can generate citation counts, but it has a tendency to list duplicates if a particular research article is posted in multiple places, which throws off citation counts, and it can only index citations made in material that is a) on the Internet (it connects to Google Books so it’s one of the few places to find citations in books) and b) that allows Google to index the full text (or otherwise makes the list of references in an article available). The main drawback to Google Scholar is that, because it depends on publicly available data, it can be easy for the unscrupulous to manipulate.

The problems with citation counts

The core problem with citation counts is that assumption upon which it depends for value, is not entirely true. The assumption is that a work is cited because it is a worthwhile piece of research.

There are many reasons why an author may choose to cite a particular article that have nothing to do with approving of the article being cited.

Consider Fleischmann, Martin; Pons, Stanley (1989), “Electrochemically induced nuclear fusion of deuterium.” J. Electroanalytical Chemistry 261 (2A): 301–30. This article has, according to the Web of Knowledge, received over 790 citations. A naive reading of the citation count would lead one to declare that this must be a very influential article. And, in a sense, that’s correct; except for the fact that it’s one of the more famous cases of science performed badly, and that most of the citations are articles stating that they can’t duplicate the results, or that the results are wrong, or writing about it as an example of science done wrong.

Even if an article isn’t fraudulent, or wrong for more innocent reasons, there are plenty of reasons that a citing article may cite an article in order to disagree with its conclusions, particularly in the social science and humanities. These negative citations don’t necessarily diminish the worth of the article in question, but they don’t really support the idea of citation counts as a measure of worth.

There can also be political, or social reasons to cite an article, that have nothing to do with its value as a piece of research. Perhaps the author is a friend that is applying for tenure. Perhaps they did a favor for the author, and now the author is repaying them. Perhaps they want to borrow the legitimacy of a more famous author by citing them in their own work, even if it’s not particularly relevant. Perhaps they’re a grad student, and it’s been “suggested” that they cite some papers written by their advisor.

Sometimes the journals you’re publishing in can get in on the act. There are some less-than-scrupulous journals that, in order to boost their prestige, encourage authors to cite other articles within the same journal, or within another journal from the same publisher. Others  inadvertently stumble into the same patterns of distortion without intending to (particularly in very niche research areas where there may only be a few people worldwide writing on the topic).

Other issues with citation counts can be more systematic. A article written for a small, niche, audience will be cited less than one written for a broad audience, but may be (within that context) a very important, influential, article. A review article (one summing up the current state of research on a topic) will usually be cited many more times than an average original research article. Obviously, self-citations by any author on a paper should be discounted (but aren’t by any of the tools listed above). Sometimes authors will cite an article incorrectly (getting the pages wrong is common, but sometimes they get something as basic as the journal wrong!), which means that, unless you catch that, it won’t get counted.

Using citation counts

So, we’ve covered the trust part  of citation counts, which leaves us with the hard task of verifying them. Or, in other words, how do we use these flawed, easily misinterpreted numbers?

The answer is to not treat them as simple numbers with simple, obvious, meaning. There are no shortcuts when it comes to evaluating a work, whether it’s an article, or a book, or something else.

You can use the tools above to get the basic citation count for an article, but once you’ve done that, you have more work to do. You’ll need to look at the citing articles and see how they used the cited article, and then decide whether to count them. You’ll have to look at where the cited article, and the citing articles were published, their acceptance policies, and other factors. You’ll have to see what a typical citation count is for articles on the same (or similar) topic to make sure you’re correctly comparing it to its peers.

Or, you can use them without all of the supplemental analysis, but with an understanding that they are not error free, and that they have fairly large error bars. How large are those error bars? It’s difficult to estimate.

Ultimately, the most accurate way to evaluate an article, or book, or whatever, is to have someone you trust (or yourself), who understands the subject material and methodology, read the article and give their opinion. Citation counts can act as a very crude filter to find works that may be significant, but in the end, you have to dig deeper.

Finding Free Full-text Resources Online

One of the most exciting aspects of pursuing Humanities research in the digital age is the prodigious and ever-growing quantity of source material available for free online. This includes books, periodicals, broadsheets and ephemera printed before 1923, collections of medieval manuscripts and archival manuscript sources. Single leaves from these online sources can be used as illustrations in publications and presentations. Material that within living memory was once difficult to locate and required on-site visits to study, or long waits for interlibrary loans to procure, is now available in seconds with a few clicks of the mouse. The scope of this material is so vast that a single post cannot do it justice, so this will be the first of several blog posts on the subject. Below are some of the prime sources for locating and accessing such content.

Google books-many things you are looking for or will find useful will come up in a search of google books. You can customize these searches by limiting them (using the “Search tools” tab) to e-books (these will be complete), books with previews (these will not provide a complete view, but will often show tables of contents and indices, which can be used for requesting chapters to order electronically from Interlibrary Loan), and year or range of years of publication. You can also limit your search to magazines.

Hathitrust is a partnership of academic & research institutions, offering a collection of millions of titles digitized from libraries around the world. UConn is a member of Hathitrust, so our users can download items that are available “full view”, and you can limit your search to these items

Internet Archivea vast repository of print books and journals useful to scholars, from early printed books to indispensable classics such as Pastor’s History of the Popes. Its usefulness can be illustrated by a recent experience: after reading a new biography of the poet Marianne Moore, I wanted to read one of her early poems. Since Moore often revised or omitted early poems in later anthologies, I was fortunate to find the poem on Internet Archive, which had scanned the very rare first collection of her poems, published in London in 1921.

Europeanamillions of books, newspapers, letters, diaries and archival papers from a range of Europe’s leading galleries, libraries, archives and museums

Zentrales Verzeichnis Digitalisierter Drucke-For medieval manuscripts and books or journals printed in German or published in other languages in places where German was spoken, this is a marvelous resource. (Subsequent posts will include portals for digital content in other European languages)

National Libraries often offer digitized runs of historic newspapers and periodicals, sometimes in unexpected places. For example, the Austrian State Library has historic German–language periodicals from regions of the former Austro-Hungarian Monarchy that are now separate countries, such as Bohemia and Moravia (Current Czech Republic). Links to National Libraries in Europe can be found at

National libraries outside of Europe can be found with a google search.

Bear in mind that the portals and points of access that exist at this writing will very likely be transformed, replaced or augmented over time. The process of locating full-text online resources is dynamic and ever-changing. That is part of the challenge, and part of the fun.

More information about free full-text available online can be found at

If you would like assistance in finding source material online, do not hesitate to contact Michael Young,

Practicing Perfection: Memory and Music Performance

Practicing Perfection: Memory and Music Performance

Tânia Lisboa
Royal College of Music, London, UK
& Roger Chaffin
Department of Psychology, University of Connecticut

How do classical musicians perform long programs from memory with note-perfect accuracy? To find out how, we have studied concert soloists as they prepare new works. Experienced performers have a mental map of the piece in mind as they play. The map tells them where they are and what comes next. Landmarks in the map are established during practice by thinking about particular features of the music so that later, during performance, they come to mind automatically. The map allows the soloist to monitor the rapid, automatic actions of playing and adjust to the needs of the moment. We will describe our research and Dr. Lisboa will perform two short works that we are currently studying: Prelude from cello suite No. 1 & Bourree from cello suite No. 3, by J.S. Bach.

Cellist Tânia Lisboa is widely acknowledged as one of Brazil’s foremost musical personalities with an international profile and an extensive range of recordings. She holds a PhD in performance and, in parallel to her solo career, she appears in masterclasses and lectures world-wide. Tânia Lisboa is a member of staff at the Royal College of Music, Center for Performance Science, in London.

Roger Chaffin is a professor of psychology at the University of Connecticut. His longitudinal case studies of experienced musicians provide a new way of understanding the cognitive processes involved in skilled performance.

Tuesday, April 10
12:30 pm-1:30 pm

University of Connecticut, Greater Hartford Campus, Library Building, Room 404
1800 Asylum Ave., West Hartford
Free and open to the public

Co-sponsored by: University of Connecticut Research Foundation, Center for Latin American & Caribbean Studies Center for the Study of Perception and Action (CESPA), Community School of the Arts, Music Department, and Trecker Library

Old Library/New Learning Commons

The Jeremy Richard Library at the UConn Stamford campus got a massive makeover last year.  We removed books (that are online or out of date) from our main floor to make way for our new computer lab, writing/tutoring center, new study tables in various configurations, media scapes, shared monitors, a quiet reading room, and lots of new comfy reading spaces supplied with electrical outlets. Image

Our goal is to bring together in an inviting, collaborative learning space, a place where students will have the tools and academic support services that enable them to research, produce and finalize their academic work assignments.

Hispanic Heritage Month at the UCONN School of Social Work

The UCONN School of Social Work kicked off the celebration of Hispanic Heritage Month (September 15-October 15) on September 14th with a presentation at the School by students who participated in a recent travel study program to Vieques and San Juan, Puerto Rico. The course, Social Work Practice in Puerto Rico: Culture, Politics and Human Rights, included meetings with leaders of grassroots movements on the Island of Vieques; visits to social service agencies; discussions with Social Work faculty and students at the University of Puerto Rico; and tours of cultural and historical sites.

The presentation today also served as the opening event of this fall’s Latin@Educational Series, sponsored by the Puerto Rican and Latin@ Studies Project (PRLSP) and the Latin American Student Organization (LASO). The following programs will be presented: Latin@ Immigration and Migration: Realties and Myths (October 25, 2011); On Being LatinA (November 1, 2011); Health Disparities and Latin@s: Micro & Macro Perspectives (November 8, 2011); and Coming Out, The Never Ending Process: Panel Presentation by LGBT Latin@s (November 15, 2011).

The programs will be held at the School of Social Work from 6:30-8:30 p.m. on the dates noted. They are free and open to the public.

Jan Lambert, Social Work Librarian