The Role of Conference Publications in CS

The role of conference publications in computer science is controversial. Conferences have the undeniable advantages of providing fast and regular publication of papers and of bringing researchers together by offering the opportunity to present and discuss the paper with peers. These peculiar features of conferences are particularly important because computer science is a relatively young and fast-evolving discipline. The fundamental role of conferences in computer science is underlined with strength in the best-practices memo for evaluating computer scientists and engineers for promotion and tenure published in 1999 by the U.S. Computing Research Association^a (CRA) and, more recently, in a study of the Informatics Europe, whose preliminary results are summarized in Choppy et al.²

Recently, Communications published a series of thought-provoking Viewpoint columns and letters that swim against the tide.^1,3,7,9 These contributions highlight many flaws of the conference system, in particular when compared to archival journals, and also suggest a game-based solution to scale the academic publication process to Internet scale.⁷ Some of the mentioned flaws are: short time for referees to review the papers, limited number of pages for publication, limited time for authors to polish the paper after receiving comments from reviewers, and overload of the best researchers as reviewers in conference program committees. The result is a deadline-driven publication system, in which “we submit a paper when we reach an appropriate conference deadline instead of when the research has been properly fleshed out,”³ that “encourages and rewards production of publishing quarks—units of intellectual endeavor that can be generated, summarized, and reviewed in a calendar year“⁸ (interestingly, the author of the latter claim is CRA Board Chair Dan Reed). Furthermore, the current conference system “leads to an emphasis on safe papers (incremental and technical) versus those that explore new models and research directions outside the established core areas of the conferences.”³ “And arguably it is the more innovative papers that suffer, because they are time consuming to read and understand, so they are the most likely to be either completely misunderstood or underappreciated by an increasingly error-prone process.”¹ Are we driving on the wrong side of the publication road? The question is raised by Moshe Vardi in a May 2009 Communications editor’s letter.⁹

This article gives an alternative view on this hot issue: the bibliometric perspective. Bibliometrics has become a standard tool of science policy and research management in the last decades. In particular, academic institutions increasingly rely on bibliometric analysis for making decisions regarding hiring, promotion, tenure, and funding of scholars. I investigate the frequency and impact of conference publications in computer science, comparing with journal articles. I stratify the set of computer science publications by author, topic, and nation; in particular, I analyze publications of the most prolific, most popular, and most prestigious scholars in computer science.

Back to Top

The Frequency of Conference Publications in Computer Science

I use the DBLP computer science bibliography (faceted search) to retrieve journal and conference publication counts for scholars. Unfortunately, DBLP does not record citations. For the analysis of impact based on citations and on the h index,⁵ I take advantage of Google Scholar and of Thomson-Reuters Web of Science enhanced with the brand-new Conference Proceedings index.

I start with the analysis of the publications of the most prolific computer science authors according to DBLP (see Table 1). Only two scholars, Chin-Chen Chang and Grzegorz Rozenberg, published more journal papers than conference papers. On average, two-thirds of the author publications are conference papers and one-third are journal articles.

Moreover, I analyzed the publications of the most popular computer science authors according to the author h index computed on Google Scholar. The h index of a scholar is the highest number h of papers published by the scholar that have each received at least h citations.⁵ I took advantage of the h index compilation maintained by Jens Palsberg^b; Table 2 illustrates the outcomes of the analysis. The frequency of conference papers is higher than the frequency of journal papers: on average, 59% of the author publications are in conference proceedings, and 40% are in journals. Only one author, Robert Tarjan, published more journal articles than conference papers. Notice, however, that the average share of journal articles is higher for popular scholars (40%) than for prolific scholars (34%). Furthermore, the average number of publications of prolific scholars (464.5) is two times higher than the mean number of publications of popular authors (230.1). Hence, high-impact scholars publish significantly less than prolific ones, and more frequently in journals.

Finally, I analyzed the publications of prestigious computer science scholars. I identified prestigious scholars as the winners of the ACM A.M. Turing Award (see Table 3). Once again, the share of conference publications (on average 65%) dominates that of journals articles (on average 33%). Only two authors, Peter Naur and Robert E. Kahn, published more in journals than in conference proceedings (notice, however, that I found only five publications for Robert E. Kahn in DBLP). The average share of journal publications for prestigious scholars (33%) is close to that of prolific authors (34%), but lower than the one for popular authors (40%). Furthermore, prestigious scholars published on average approximately one-third of the papers published by popular authors, and one-sixth of the articles published by prolific authors.

I offer two additional observations to conclude this section. The first one is well known to the computer science community: books do not represent a frequent publication in computer science (from 1% to 3% in the analyzed samples). This is a difference with respect to the humanities and to (some of) the social sciences. The second observation is more intriguing: the concepts of productivity, popularity, and prestige are well separate in computer science: the pairwise intersections of the corresponding top-10 compilations are always empty. The divergence between the bibliometric concepts of popularity and prestige is a known phenomenon for the hard sciences, but it is less evident for biology-medicine disciplines and for the social sciences.⁴

Back to Top

The Impact of Conference Publications in Computer Science

I analyzed the frequency and impact of conference and journal publications that contain the 10 most popular computer science topics (see Table 4). As found earlier, conferences are more popular than journals: for each topic in the list, the number of conference papers containing the topic phrase in title, keywords, or abstract is significantly greater than the number of journal papers containing the same topic. On average, 78% of the publications containing some of the hot topics are conference papers, and 22% of them are journal articles. Nevertheless, journal papers collect more citations (67%) than conference papers (33%). This means that journal papers have a much higher impact (on average, 5.41 citations per paper) than conference papers (on average, only 0.71 citations per paper). The higher impact of journals with respect to conferences is confirmed when inspecting the h index column: on average, at most 16 conference papers are cited at least 16 times, while at most 27 journals papers are cited at least 27 times. The topic with the highest conference impact is security, and that with the highest journal impact is data mining. Notice that both the number of citations and the h index for topic security are higher for conference papers than for journal papers, and this is the only exception among all topics. On the other hand, the topic with the lowest conference impact and that with the lowest journal impact is QoS.

---

Academic institutions increasingly rely on bibliometric analysis for making decisions regarding hiring, promotion, tenure, and funding of scholars.

---

I also analyzed the citation patterns of conference and journal publications. Conference papers are strongly cited by conference papers (67%), while journal papers are cited more by journal papers (56%). However, journal papers are also significantly cited by conference papers (44%).

Finally, I analyzed the frequency and impact of conference and journal publications per country. I assigned a publication to a country if at least one author is affiliated with the country. I restricted the investigation to the 10 nations with the highest scientific impact according to the share of top 1% cited papers.⁶ Table 5 lists the statistics I computed. Once again, the percentage of conference papers (on average 76%) dominates that of journal papers (on average 24%), but journal articles harvest more citations (57%) than conference articles (43%): the average impact of journal articles (7.73) is more than four times higher than the impact of conference articles (1.77). Moreover, the average journal h index is 27 and dominates the average conference h index, which is 16. Conference papers are mostly cited by conference papers (67%), while journals are cited by both publication sources, but more by journals (56%). Notice that Japan represents an exception: conference papers collect more citations than journal papers and have the same h index. Moreover, journal papers are cited more by conference papers.

---

Although it is more difficult to get published in journals, the effort is ultimately rewarded with a higher impact.

---

The nation with the highest productivity (number of papers) and also that with the highest scientific impact (number of citations or h index) is, not surprisingly, the U.S. The top-three in number of journal papers are the U.S., England, and Canada; the top-three in number of conference papers is the U.S., Germany, and Japan. The U.S., England, and Germany are the countries that receive most of the journal citations and that have the highest journal h number, while the U.S., Germany, and France are the nations with the highest number of conference citations and with the largest conference h index. Interestingly, Switzerland is the nation with the highest journal impact, followed by The Netherlands and England (the U.S. in only 7th here), while the top-three with respect to conference impact is Switzerland, The Netherlands, and France (the U.S. is 4th).

Back to Top

Conclusion

The main conclusions are:

• computer scientists publish more in conference proceedings than in archival journals;
• the impact of journal publications is significantly higher than the impact of conference papers.

The fundamental message for the computer science community is: although it is more difficult to get published in journals, the effort is ultimately rewarded with a higher impact. From a bibliometric perspective, the best strategy to gain impact seems to be that of publishing few, final, and well-polished contributions in archival journals instead of many premature “publishing quarks” in conference proceedings.

Back to Top

Back to Top

Back to Top

Back to Top

Back to top