D. Problems With Citation Analysis
1. General Problems
Citation analysis is not a perfect measure of scholarly influence; the technique has a number of problems. One objection to citation analysis is that it may be biased against scholars who work in a narrow or less-populated specialty. If there are only a few researchers working in a particular area, there will be fewer articles published in which those researchers can and will be cited. As a result, these scholars, regardless of how influential they are in their area of expertise, are less likely to be frequently cited.
Another concern is that citation analysis counts all citations to a given work equally and does not distinguish among positive, negative, and neutral citations. In other words, just because a work is highly cited does not automatically mean that it is looked upon favorably by others in the field. However, a number of researchers have considered this and have found that the vast majority of citations appear to be positive or neutral. Very few citations appear to be critical or negative.
Several researchers have suggested that “recipe” articles, such as those outlining a new personality test or explaining a statistical technique, tend to be highly cited. However, in general this does not appear to be the case. Recent research into the most-cited scholars and works in criminology and criminal justice journals (Cohn & Farrington, 2007a, 2007b, 2008) has not found methodological works or their authors to be among the most cited.
Citation counts also may be affected by a scholar’s productivity. In general, the most-cited criminologists tend to be older and more established in the field. They also tend to be fairly prolific writers and to have long publication records. However, Cohn and Farrington (2007a, 2007b) have found that a scholar’s high ranking may be either a function of the large number of different works cited (versatility) or a function of a large number of citations of one or two major works (specialization). Of course, it is possible for a scholar to be both versatile and specialized (e.g., to have a large number of works cited, with one or two of these receiving most of the citations).
Another concern is that an author’s choice of which articles and/or authors to cite may be influenced by social factors, such as personal likes and dislikes, attempts to please journal editors, a desire to inflate individual or departmental citation counts, or even a preference for citing same-sex authors. For example, a scholar may cite others in the same department to boost overall departmental citation counts or may avoid citing faculty from a rival department to reduce their citation counts. Although it is difficult to constantly cite oneself (self-citation) as a way of inflating one’s own citation count without being rather obvious about it, other more devious methods, such “You cite me, I’ll cite you,” could be used. However, for such an approach to work, the scholars would still have to have a reasonably high publication rate in good quality journals.
Chapman (1989) discussed the issue of “obliteration by incorporation”:
Certain scholars may be so eminent and prolific in their fields that, although their names appear in the body of an article, they can elude the ordinary counting process because the writer neglects to list them in the references at the end of the text. In psychology, textual mentions of Freud, for example, often are not referenced and are thus underrepresented in citation counts. (p. 341)
It has been suggested that a similar situation may occur in criminology with mentions of scholars such as Karl Marx.
However, despite these difficulties, citation analysis is generally accepted as a valid technique and is increasingly becoming more popular and more widely used in the sciences and social sciences.
2. Problems With Citation Indexes
The SSCI lists all bibliographic references made in an extremely large number of social science journals. The index is an extremely useful tool for general bibliometric research but has several problems when used for citation analysis.
First, the SSCI does not include references in all published works. According to the SSCI Web site, as of 2008, the SSCI fully indexes over 2,000 journals in 50 social science disciplines, as well as individually selected items from over 3,300 leading scientific and technical journals. However, not all criminology journals are included. For example, neither Criminology and Public Policy nor Violence and Victims currently are included in the SSCI.
In addition, although the SSCI includes citations of books and book chapters, works cited in books or book chapters are not indexed. It is possible that this may lead to a bias, because books appear to be highly significant in criminology. Cohn and Farrington (2007a, 2007b, 2008) have examined the most-cited works of the most-cited authors and found that most were books, rather than journal articles.
Another problem with the SSCI is that it lists only the initials and surnames of cited authors, making it almost impossible to distinguish between different people with the same last name and first initial. For example, Cohn and Farrington repeatedly have pointed out the difficulty of determining which of the citations of “J. Cohen” belong to Jacqueline Cohen and which refer to other individuals, such as Jacob Cohen or Joseph Cohen. Similarly, it is difficult, if not impossible, to distinguish between the various R. Berks, P. Brantinghams, and D. Smiths. This confusion is only increased if citations include or omit an author’s middle initial (e.g., making it more difficult to distinguish between E. G. Cohn and E. S. Cohn) or use a “nickname” first initial (such as R-for-Robert vs. B-for-Bobby Brame). In addition, citations of married female scholars may appear under more than one surname (e.g., Ilene Nagel, Ilene Bernstein, Ilene Nagel-Bernstein). Other scholars who hyphenate or change their surnames create similar difficulties for the citation analyst attempting to use the SSCI.
The SSCI also creates a bias against junior authors in collaborative works, because it lists citations only under the name of the first author. Therefore, an individual who is not listed as the first author of scholarly work will not be found in the SSCI. This may penalize younger scholars, scholars whose names come later in the alphabet (for articles where authors are listed alphabetically), and wives (e.g., in research by Sheldon and Eleanor Glueck, Sheldon’s name is almost invariably listed first).
Citation indexes such as the SSCI also may contain errors. First, because it assumes that all citations in the source journals are accurate, any clerical or other errors in the original reference lists, such as misspelled names, incorrect dates of publication, or incorrect initials, are transferred to the SSCI. There is considerable evidence to suggest that this assumption is, for the SSCI at the least, questionable. For example, Cohn and Farrington (1996) pointed out that “Farrington” is often misspelled as “Farringdon.” Similarly, “Hirschi” is frequently misspelled as “Hirsch,” “Hirsh,” or “Hirshi.” In addition, Hirschi’s citations have been found under “P. Hirschi” and “L. Hirschi” as well as the correct “T. Hirschi.” When dates of articles are incorrect in the original reference list, they are entered into the SSCI under the incorrect year. In general, studies of citation accuracy have found that between 25% and 30% of all citations contained errors. Of course, this does not include any errors that may be made by the staff at Institute for Scientific Information who compile the SSCI.
Finally, the SSCI includes self-citations. Self-citations occur when a scholar cites his or her own work. It is, of course, perfectly reasonable and justifiable for scholars to cite themselves, especially when their work is building on their prior research. However, self-citations do not indicate the influence of the cited work on other scholars, and a study of citations as a measure of influence on others in the field should omit self-citations. Because only the first author of a cited work is included in the SSCI, it is extremely difficult to exclude self-citations when using the SSCI to obtain citation counts.