How Productivity and Impact Differ Across Computer Science Subareas

Some computer science researchers believe different subareas within CS follow different publishing practices, so applying a single production criterion would be unfair to some areas. It is reasonable to believe the subarea of, say, theory follows different publishing practices from subareas like software engineering and image processing. Scientific advances in theory are often bounded by the time needed to prove theorems. Moreover, at most institutions, CS faculty whose work involves theory advise fewer students and are thus likely to produce fewer publishable results per year.

Back to Top

It is also reasonable to believe CS subareas (we call them “CS areas” from here on) that deal mainly with data (such as image processing) are more likely to have greater productivity than areas in which evaluation procedures require users (such as human-computer interaction), programmers (such as software engineering), and organizations (such as management information systems). Researcher productivity in these human- and organization-based areas is bounded by the difficulty of carrying out the empirical evaluations the fields require. Though these beliefs are all reasonable, it is all they are, as they are as yet unproved.

Along with expected differences in productivity, we also often hear that different CS areas prefer and value conferences and journal publications in different ways; for example, bioinformatics seems more journal-oriented, while computer architecture seems more conference-oriented.

If there are indeed significant differences in publishing practices among the various CS areas, then a single production-based evaluation criterion for all CS researchers would favor some areas and disfavor others. A probable consequence, beyond possible unfairness to the disfavored areas, is that researchers would tend to avoid those areas in the future. Barbosa and Souza¹ discussed this problem with respect to a uniform publication evaluation standard in Brazil and its negative impact on human-computer interaction among Brazilian researchers.

Beyond publication practices, citation practices might also differ among areas. Areas with fewer researchers probably reflect fewer citations of papers published in these areas; a uniform evaluation-criteria impact of one’s research across different CS areas would favor some areas while disfavoring others.

How to evaluate CS researchers has been discussed elsewhere, including Meyer et al.,⁹ Patterson et al.,¹⁰ and Wainer et al.,¹⁴ emphasizing the differences in scientific publication culture between CS and other scientific domains; for example, Meyer et al.⁹ discussed the importance of conference publication in CS, saying, a conference publication is, in some cases, more prestigious than a journal publication. The same general guideline of attributing importance to conferences is included in the Computing Research Association (CRA) guideline to faculty promotion in CS.¹⁰ Wainer et al.¹⁴ showed that a typical CS researcher’s work is not represented in the standard citation services (such as Scopus and Thomson Reuters) compared to, say, mathematics and physics; thus, when using metrics based on these services, a CS researcher or university department could be unfairly evaluated, especially when competing against other disciplines. The role of conferences in CS has also been discussed by others; Grudin⁶ collected many of the relevant articles and discussions, especially those published in Communications.

We are not aware of research that discusses the problems of uniform evaluation criteria across different CS areas, except for Barbosa and de Souza.¹ In other scientific disciplines (such as economics), some discussion focuses on the negative impact of uniform evaluation metrics on the different subareas of the discipline.⁸

Back to Top

General Description

Our methodology, as described here, relied on a sampling approach to evaluate the productivity and impact metrics of researchers and papers in different CS areas; we considered productivity as the number of articles published (in English) per year in journals, conferences, and workshops. Other than the distinction between journals and conferences (including workshops), we did not take into account any measures of venue quality (such as impact factor for journals and acceptance rate or scientific society sponsorships of conferences). The first step was to define the CS areas; the second to define the set of researchers working in each area, along with the set of conferences and journals associated with each area; the third to sample the set of researchers working in an area and collect from their own webpages the number of papers published from 2006 to 2010; and, finally, from the set of conferences and journals associated with a particular area, we sampled a set of papers and collected information about their citation counts. We briefly expand on each step; for a more detailed explanation of our methods, see Wainer et al.¹⁵

When deciding how to define and select a CS area, we were guided by some of the existing classification schemes. For example, ACM and IEEE each divides CS into different areas—ACM, through special interest groups, or SIGs, and IEEE, althrough technical committees, or TCs—though some of these divisions reflect historical decisions that may be less relevant today. DBLP, Microsoft Academic Search, and Scopus each classify different CS areas, though none describes how it arrived at its classifications.

We wanted our set of areas to include both new and more traditional CS areas, to evaluate whether or not the traditional areas follow publication practices that differ from the newer ones. Finally, we also wanted to include some areas on the “fringe” of CS that are not always present in university CS departments in different countries; Table 1 lists the areas we chose, the abbreviations we use for them, and the seed venues (using their usual abbreviations) for each area. We do not claim they are the only, or most important, areas of CS.

Bioinformatics and security are newer areas. Communications and networking, programming languages, databases, computer architecture, distributed computing, and software engineering are more traditional areas. Operations research and management information systems are the two “fringe” areas. Our choice of areas is compatible with, but not the same as, those of other research (such as Biryukov and Dong² and Laender et al.⁷) that also subdivides CS into areas.

For the second step, defining the population of researchers in each area and associated conferences and journals, we used DBLP data (as of August 2011) as our universe of interest. DBLP is a bibliographic server with a focus on CS that indexes more than 1.8 million CS articles.

Here, we use the words “publication venue” or just “venue” as a generic name for conferences or journals. We started by defining a set of venues clearly representative of each area, or “seed venues.” The idea is that researchers in each area clearly recognize the seed venues as “central” and “important” to their area. We asked colleagues in each area and used information regarding citations received per published paper available by, say, Microsoft Academic Search and Thompson Reuters Journal of Citation Reports.

Using the set of venues associated with a particular area, we defined an iterative process that computes the set of researchers working in the area, then recomputes the set of venues in the area, and so on, until convergence. At the first iteration, the set of venues associated with an area is its seed venues, and the researchers working in the area are co-authors of at least two papers published in any seed venue from 2006 to 2010.

A new venue v is associated with area A if a clear majority of co-authors of papers in v (for the period 2006 to 2010) of researchers work in area A; for a formal definition of “clear majority,” see Wainer et al.¹⁵ Finally, all researchers publishing at least one paper in a seed venue and at least one in the newly added venue v are also considered researchers in area A. Note that a researcher may work in more than one area, but a venue is associated with at most one area. This method is a contribution of this research because it simultaneously classifies publications and authors by means of a semi-supervised algorithm, based on co-publication. Most publication-classification systems (such as Chen³) are based on unsupervised algorithms based on different citation links between journals and conferences. Other research (such as Rosen-Zvi et al.¹²) also uses unsupervised algorithms based on co-authorship and topic detection on the documents themselves.

At the end of the iterative process our algorithm revealed a universe of 56,589 researchers, of whom 4,827 work in more than one area. The algorithm also identified a set of 612 venues (247 journals and 365 conference proceedings); see Wainer et al.¹⁵ for a list of all venues associated with each area.

The third step in our method was the sampling of researchers and papers. We randomly ordered the set of researchers in each area and sequentially searched each one until we found a set of 30 researchers with a personal or institutional webpage listing all their publications. We did not consider webpages that explicitly mentioned “selected” or “partial publication list”; 30 researchers per area was our attempt to balance the need to collect enough information about each area with the cost of finding researchers with a current list of publications.

For each researcher, we collected the number of conference papers (including workshops but excluding posters) and journal papers listed on the researcher’s page for the period 2006 to 2010 (inclusive) in English. We also collected the researchers’ first and last publications years as listed on their webpages. The intersection of this interval with the interval 2006 to 2010 is a researcher’s windowed publication interval. We defined the researcher’s productivity as number of journal and conference papers published during that time divided by windowed publication interval. We also collected information on whether researchers were students (at the end of their windowed publication interval) and whether they were faculty in a non-CS university department. We considered non-CS any department that did not include the words “comput*” or “information” in its name.

For citation analysis, we randomly selected 100 papers for each area from all CS papers published in 2006 from all venues associated with a particular area (not just the seeds). In November 2011, we collected the number of citations received by each paper as compiled by Google Scholar. Given that citation counts are susceptible to outliers, we used the median number of citations per paper to perform the statistical calculations.

Finally, we used a 95% confidence level to make claims of statistical significance; see Wainer et al.¹⁵ for the details of the statistical analysis.

Back to Top

Results

Table 2 lists some of the characteristics of the sample of 30 researchers per area. The column labeled “Stud.” lists the number of students in the sample. The column labeled “Non-CS” lists how many researchers in the sample are faculty in non-CS departments. The column labeled “Resea.” lists the total number of researchers in that area (based on the DBLP data), including those also working in other areas. The column labeled “Papers” refers to the total number of papers in all conferences and journals associated with the area published from 2006 to 2010.

The areas BIO, IPCV, MIS, and OR are the most “non-central” of the areas, based on the number of non-CS faculty in the sample. We were expecting non-centrality for BIO, MIS, and OR included in our set of areas specifically because they are not always represented in university CS departments; for BIO, some non-CS faculty were hosted in biology-related departments, for MIS, in business and marketing departments, and for OR in applied math and engineering departments. However, surprisingly, this pattern also holds for IPCV researchers, with about one-third affiliated with radiology and medical departments. The quantity of researchers in those areas not in CS departments may indicate those areas are more “interdisciplinary.”¹³

The number of students in the sample may indicate, in a first approximation, the number of students working in each area. That number divided by the number of CS faculty in the sample may likewise be an indicator of the availability of students per CS researcher. Thus, MM and ARCH had the most students per CS researcher, while MIS, DC, and TH had the fewest students per CS researcher.

Productivity measures. Figure 1 outlines the mean conference and journal productivity of the sampled researchers in each area, in papers per year, ordered by total productivity (the sum of conference and journal productivity).

Table 3 lists the significant differences in total and journal productivity as a “compact letter display,” a visualization tool showing nonsignificant differences between two areas; that is, the difference between the average total productivity of two areas is not statistically significant if the areas share a letter in common. When comparing any two areas, two or more letters in common means the same as one letter in common; that is, the areas are not significantly different. Only when two areas have no letters in common is the difference between them statistically significant. Each letter is an indicator of a maximal subset of areas in which differences are not statistically significant; for example, total production of DC (“d”) is significantly different from that of DB (“ab”), since they have no common letter. However, DC (“d”) is not significantly different from COMM (“cd”) because they have the letter “d” in common. The second column lists the nonsignificant differences for journal productivity; for example, the OR in the journal productivity column includes all four letters (“abcd”), meaning OR is not significantly different from any other area, because OR has at least one letter in common with each other area.

Regarding total productivity, although the data seems to show three groups—higher productivity (ARCH, COMM, DC, and IPCV); middle; and lower productivity (MIS and OR)—almost all differences are not significant at 95% confidence level, as in Table 3. The only significant differences are, in general, between MIS and OR and the higher-productivity group. There are also significant differences between DB and TH and some of the higher-productivity areas but not all. Thus, one cannot claim that in general there are total productivity differences among the CS areas except for a few cases covered earlier.

For journals, BIO has significantly higher productivity (3.44 papers per year) than all other areas, except the next four higher—COMM, MIS, ML, and OR; COMM is significantly different from the lower journal productivity areas—AI, C+PL, and DB—as in Table 3.

As for conferences, the highest two conference productivity areas—DC and IPCV—are significantly different from the bottom third—MIS, OR, BIO, TH, ML, and DB—whereas the lowest two—MIS and OR—are significantly different from the top third—DC, IPCV, ARCH, COMM, HCI, and SE, information not in Table 3.

If we consider the ratio of journal productivity to total productivity, there are basically two groups: BIO, MIS, and, OR prefer journal publications to conferences, with about 70% of their production published in journals; the differences from all other areas are significant. ML and TH represent an intermediary group that publishes almost half its production in journals; the difference is statistically significant compared to most other areas.

Impact measures. Figure 2 includes the mean and median citations rates (citations per paper per year) for our sample of randomly selected 100 papers (from 2006) from each area (in order of median citations rate); the third column of Table 3 lists the compact letter display of the median citation rates for each area.

MIS citations rates are not significantly different from the next four higher rates—GRAPH, DB, BIO, and HCI—in decreasing order. The two lower-rate areas—ARCH and MM—are significantly different from the third lower-rate area—DC; the other areas are in the same group, with no significant differences among them.

The citation numbers reflect an interesting relation with productivity. The higher-productivity areas also have lower median citation rates. The correlation is moderately high and significant (Spearman rho = −0.63, p-value = 0.007). We use the Spearman rank correlation (rho) to detect any monotonic correlation between the variables, not just linear correlation. The correlation is even higher for conference productivity (rho = −0.71, p-value = 0.001). Thus, on the surface, in areas like ARCH and MM, researchers write many papers per year, especially conference papers, but few other researchers cite them. One notable aspect of the high-productivity/low-citation pattern is that the high-productivity areas tend to focus on conferences that, given the usual restrictions on number of pages in the publications, force authors to cite only a few relevant papers. However, a regression of the citation rates with both total productivity and proportion of journal publication reveals that only the negative coefficient of the total production is significant.

A reasonable hypothesis is thus that the median citation rate is correlated with the size of the area; that is, if an area includes few researchers, few potential readers are available to read the papers, inevitably yielding a low citation rate. However, the correlation between median citation rate and number of researchers in an area is not significant (rho = −0.26, p-value = 0.32) nor is the correlation with number of papers published in the area (rho = −0.25, p-value = 0.33). The size of a research area does not explain the different median citation rates.

The greater difference between mean and median citation rates for BIO, COMM, C+PL, and SE seem to indicate, at least in these areas, there is an even higher than usual concentration of citations on only a few papers, possibly increasing the mean but not the median. The two papers with the highest citation counts in our sample are from BIO.

Back to Top

Discussion

Our productivity analysis found that although the productivity of the CS areas range from 2.5 (MIS) to 7.8 (DC) papers per year, the only significant differences are between the extremes of the spectrum. The total productivity of researchers in ARCH, COMM, DC, and IPCV is significantly higher than those for researchers in MIS and OR. The total productivity of the other areas does not differ significantly. Thus CS departments and evaluation bodies should be mindful when comparing researchers in MIS and OR to researchers in ARCH, COMM, DC, and IPCV.

Some evaluation criteria, especially those that apply to disciplines other than CS, put more emphasis on journal publications. CS departments that emphasize journal publications must be mindful that BIO in one group, and all marked areas without a “d” in the second column of Table 3 in the other, have significantly different journal productivity. However, BIO journal publication practices are not significantly different from those of COMM, MIS, ML, and OR.

There are more pronounced differences regarding whether the areas are conference- or journal-oriented in their publication practices. BIO, MIS, and OR are clearly journal-oriented and significantly different from the other areas. ML and TH are also significantly different from the most conference-oriented areas.

Regarding citations, there are significant differences among MIS (by itself), BIO, DB, HCI, and GRAPH (in another group), and finally, ARCH and MM. There is also an interesting negative correlation between productivity and citation rates beyond the influence of one area’s emphasis on conference or journal publications.

Consider, too, these other interesting findings:

• We included BIO and SEC as examples of new CS areas. BIO indeed reflects very different publication and citation patterns from most other CS areas. SEC publication and citation patterns are not different from the majority;
• BIO, MIS, and OR are less-central CS areas, in the sense that a larger proportion of researchers in them are not in CS departments though, to our surprise, likewise IPCV. In some sense this non-centrality might indicate these areas are more interdisciplinary or multidisciplinary. In terms of publication and citation practices they differ somewhat from the bulk of CS, as discussed earlier, probably due to CS researchers adapting their practices to that of their research colleagues in other disciplines; and
• As far as our sampling was able to identify student availability per CS researcher, MM and ARCH seem to have the most students per CS researcher, while MIS, DC, and TH have the fewest.

Our research quantifies information researchers in the various CS areas already known, as in, say, the emphasis some of them put on conference publications. Some CS researchers have intuition regarding the differences among the areas derived from their personal observations of colleagues and acquaintances in these areas. However, as discussed earlier, before we began this research, this intuition should have been viewed as unproved beliefs gathered from a limited sample of convenience. We derive our conclusion from a random sample of 30 researchers worldwide and of 100 papers in each CS area. On the other hand, our research should be viewed as only a first step toward understanding the differences among CS areas. Moreover, our conclusions are limited by some issues that need to be further discussed:

The first is that our sampling of researchers introduced some bias. We discovered it is more likely that a non-senior faculty researcher in a university in a Western country would have an up-to-date publications page than the alternatives, including, say, a researcher in an Eastern country, students, industry-based researchers, and senior faculty researchers. Given that junior faculty are the researchers most likely to be evaluated through some of the metrics covered here, this bias has a limited effect. However, faculty in non-Western universities should take care when using our results, as they may not reflect their professional experience.

The second issue is sample size. Sampling researchers is labor intensive, so the sample size is small and the standard error associated with the measures is high. Not all differences are statistically significant at the level of 95%; some of our claims of nonsignificance may be revised if a larger sample is used; for example, we were surprised to find no statistically significant difference between the group of higher-productivity areas—ARCH, COMM, DC, and IPCV—and the middle group, including all areas but MIS and OR. A larger sample size might reveal whether this difference is significant.

---

CS areas that may be limited in their citation rates may consider encouraging all papers, especially conference papers, to include more elaborate analysis of the related literature.

---

Most research on bibliometric aspects of CS, including Biryukov and Dong,² Elmacioglu and Lee,⁴ Franceschet,⁵ and Laender et al.,⁷ uses the whole DBLP as its data source. We could not follow such an approach. As pointed out by Laender et al.⁷ and Feitz and Hoffmann,¹¹ DBLP has different coverage for different CS areas. If we used DBLP as the data source we would not know if the difference in productivity was due to the different practices of researchers in different areas or the difference in the DBLP coverage of these areas. We therefore used DBLP to define the populations and the set of researchers and publications in each CS area, but the final productivity measurements were not based on DBLP but on the papers listed on researchers’ personal webpages. However, we used the DBLP data to define the size of each area in order to correlate it with the citation rates.

The procedure we describe here is repeatable. One may choose a different set of areas and initial seeds to explore more specific questions. The costly step is defining the sample, or finding which researchers have up-to-date webpages listing their publications. For help expanding on our results or exploring other issues regarding CS publication and citation practices, we provide the data used in this research at http://www.ic.unicamp.br/~wainer/datasets/CSareas/.

Our main purpose here is to provide the data needed to establish evaluation criteria for CS researchers; for example, one should not propose a single journal productivity goal for both COMM and AI researchers, as it would be unfair to AI researchers and could ultimately drive researchers away from the area. Productivity and citation rates differ between some but not all CS areas, and evaluation criteria for CS researchers must account for these differences.

This research does not answer why there are publication and citation differences between different CS areas and what might be done about them. We began by claiming there may be intrinsic differences among the areas and that doing research in one may be more difficult than in another; for instance, we mentioned that research in HCI, MIS, and SE could have lower productivity due to the difficulty of creating and conducting empirical research in these areas. However, HCI and SE have high productivity. The difference in total productivity of the three areas taken together when compared to the other areas, also taken together, is not statistically significant (t test p-value = 0.12). A second explanation for the differences in productivity is the availability of students. Again, the data we collected does not allow us to make that claim; the correlation between availability of students per CS researcher and total productivity is not significant (Spearman rho = 0.24 p-value = 0.36).

We also found another intrinsic possible explanation mentioned earlier to be false. There is no significant correlation between citation rates and the size of a CS research area. Surprisingly, all reasonable explanations for different productivity and citation rates across different areas mentioned here are not true as far as the data shows.

Back to Top

Conclusion

We are not able to claim one publication practice is “better” than another. Moreover, it may not be possible for a research community to change its publication practices without undergoing internal turmoil, though citation practices may be more amenable to change. Areas with low citation rates may look to areas like DB and GRAPH, which for most other characteristics are in the mainstream of CS practices but still have very high citation rates. It seems papers in these areas dedicate much more space to show how the research connects to previously published papers, with a corresponding increase in the references they include. CS areas that may be limited in their citation rates may consider encouraging all authors, especially of conference papers, to include more elaborate analysis and inclusion of the related literature.

Back to Top

Back to Top

Back to Top

Back to Top

Back to top