What Is Your Research Culture? Part 3: The Web of Science

I have procrastinated writing this third and last article in my Research Culture series. (Please read the other two parts first [1] [2].) The reason for procrastination is that one cannot expect pleasant results from talking about the ISI (Thomson Reuters) Web of Science and its undue influence. Anyone competent in the evaluation of computer science research and researchers knows that the role of WoS for computer science lies  somewhere between irrelevant and harmful. Reactions to the mention of WoS in professional meetings about research policies tend to be sighs, or, in writing, their emoticon equivalents: we know it's awful, but nothing can be done.

Still, pleasant or not, the topic has to be brought up, even for the umpteenth time, because every day a new university springs up somewhere, or an old university decides to modernize and become part of of what it perceives as the modern research world, and management starts screaming: This is taxpayers' and sponsors' money you are spending! We need to assess researchers using objective metrics! From now on we will apply the Web of Science metrics!

The worst part is that when people object, pointing out that study after study has shown WoS as inadequate and misleading for computer science, it is all too easy to dismiss them as sore losers. After all, aren't physicists content enough with WoS? Well, they may be, but we are not. And yes, we are sore losers. A football team assessed according to the rules of hockey would be a sore loser too.

The problem does not arise in any siginificant way in first-class universites. The current president of Stanford University is a well-known computer scientist (more precisely, an expert in computer  architecture). I have no inside information, but I very much doubt that when he sees an application for professor of computer science he rushes to his browser to look up the person's profile on WoS. No top institution I know takes these indicators seriously. The problem is with institutions whose management does not understand the research culture of computer science.

So let us belabor the obvious once more. WoS may be relevant for other disciplines but for computer science, computer engineering and other IT disciplines it is largely worthless. The reasons include under-representation of conferences (a special version of WoS includes conferences, but not the basic indicators that most institutions use), the absence of books, and in general the opaque process applied by these tools, depriving them of a community-based process that would permit continuous improvement.

It is tedious to go again what has been explored by so many studies. So let us just pick an example or two. The quintessential computer scientist is Donald E. Knuth. Let us search for "Knuth D". (The links given here and below are to the search results I get, so that if I messed up you can provide a correction. This one seems to work whether or not your institution has a WoS subscription.) A few papers come up, but not what makes Knuth an icon in our field: his universally acclaimed multi-volume The Art of Computer Programming book series. ("Dear Dr. Knuth: thank you for Associate Professor application at Nouveau-Riche University. I regret to inform you that your Web of Science h-index of 34 puts you far behind other candidates, some of whom even have publications in the South Kentucky Journal of Information Technology for Marine Entomology. Yours sincerely, …") On Google Scholar, The Art of Computer Programming gets over thirty-seven thousand citations, and two of his other books get around 1600  and 1200. The Microsoft Academic Search similarly recognizes the value of the books.

Whatever the reasons besides not counting books, many WoS results make little sense. The Thomson-Reuters "list of Highly Cited Researchers" includes 117 people under "Computer Science". A quick look at it reveals that only a few of the field's leaders, the ones whose names would come up immediately if you asked a typical computer scientist, appear in the list. I did not spot a single recipient of the discipline's top prize, the Turing Award. (What would you think of a list of top physicists that includes no Nobel prize?) Now there is every reason to believe that the people listed are excellent scientists; they are just not, in most cases, computer scientists. They often belong to departments having nothing to do with CS, such as the Institute of Enzymology of a country's Academy of Sciences.

One cannot just retort that any such compendium will be by nature arbitrary. Not true. Look at the corresponding list in the Microsoft Academic Search. If you are a computer scientist, a few of the names might surprise you, but most will be familiar. I checked around: other computer scientists, like me,  know almost every name among the first 100 in the Microsoft list, and fewer than a tenth of those in the Thomson-Reuters list. Why WoS classifies them under computer science is somewhat mystifying, but a clue may be that many seem to use computational techniques in other disciplines such as chemistry or biology. WoS is comfortable with these disciplines (if not with computer science) and so captures these authors. Work applying computational techniques to high-particle physics may be brilliant, but it is not necessarily relevant as computer science work.

An accumulation of such examples of inadequacy does not constitute a scientific survey. But the scientific surveys have been made; see the references in the Informatics Europe article [3] and the associated report [4] on evaluation of computer science researchers. Although the statistics there are from some five years ago (I'd be glad to talk to anyone who has the time to update them), the general picture has not changed. If someone in your institution brings up WoS for evaluating CS people, please point to these articles. Any institution who applies Web of Science for computer science is behaving improperly. (Scopus, by the way, is just marginally better.)

Highlighting this inadequacy does not mean rejecting bibliometric indicators. The role of these indicators is the source of considerable discussion; but that is a separate discussion. Quantitative indicators, when and if applied, should be meaningful; not hockey rules for football games. Good computer scientists apply rational principles in their work, and expect those who evaluate them to do the same.

People will, of course, say that they need measurements, and ask which ones they can use. almost anything is better than WoS. "Try shoe size", said a colleague dejectedly, but those who ask the question seriously deserve a serious answer.

First — sorry again for belaboring what to experienced readers will be obvious — you should remind your colleagues that if they count anything about publications it should be citations, not the number of publications. Many institutions, believe it or not, still rely on  publication counts, which do not reward quality but mere prolificacy. Citation counts too are only an imperfect proxy for quality, but much less imperfect than publication counts.

For citations, good institutions typically rely on Google Scholar or Microsoft Academic Search. Both have their limits (see note [7] about the latter); they can be gamed a bit, for example through self-citations, but the effect is marginal. Some authors have also succeeded in demonstrating the possibility of outright fraud with Google Scholar, but the techniques are so extreme that it is hard to imagine a serious institution falling for them. (Any organization that blindly relies on automated metrics as the basic evaluation technique, without human filtering, is begging for disaster anyway.) In fact these sources, based on the simple and objective criterion of counting references on the Web, are more robust than the competition. It is no accident that tools like Anne Wil-Harzing's Publish or Perish [5] and Jens Palsberg's "h Index for Computer Science" at UCLA [6] rely on Google Scholar, complemented in the first case by Microsoft Academic Search. An imperfect tool is better, especially when its imperfections are clear and can be factored into the discussion, than a bogus indicator dressed up in the trappings of respectability.

In the years since we published the Informatics Europe article and report, we received countless notes of thanks from academics who successfully used them to dissuade their naïve management from applying inadequate criteria. (Naïve but honest: many administrators, or colleagues from other disciplines, simply assume that what works for other fields works for computer science, and are open to being enlightened, especially if you present them with credible alternative solutions.) As new institutions get into IT and new people take on leadership roles, the battle must, whether we like it or not, recommence again and again.

References

[1] Bertrand Meyer: What is Your Research Culture, Part 1: The Questionnaire, in this blog, 2 November 2014, see here.

[2] Bertrand Meyer: What is Your Research Culture, Part 2: Background, in this blog, 2 November 2014, see here. 

[3] Bertrand Meyer, Christine Choppy, Jørgen Staunstrup and Jan van Leeuwen: Research Evaluation for Computer Science, in Communications of the ACM, vol. 52, no. 4, April 2009, pages 31-34, text available here on the ACM site.

[4] Same authors as [3], same title, extended version (full report) on the Informatics Europe site, available here.

[5] Publish or Perish site, see here.

[6] Jens Palsberg, The h Index for Computer Science, see here.

[7] Microsoft Academic Search seems to index about half as many articles as Google Search, and a blog article in Nature last year questioned its future.