An Analysis of Research in Computing Disciplines

Many people consider the contemporary time period the “era of computing.” Indeed, technological advances in the computing field are leading the world in many exciting new directions. Research is, of course, a primary mechanism by which the computing field initiates its advances. It is our intention here to analyze the field of computing by examining computing research, in order to better understand where the field has been, and to consider where it may be going. To accomplish this, we break the computing field down into its most common academic subdivisions: computer science (CS), software engineering (SE), and information systems (IS). With those three disciplines in mind, we examine the following questions: How different are the topics upon which they do research? How similar are research approaches and research methods? Upon which reference disciplines do they draw? At what level of analysis is research typically conducted?

There are many reasons why such an analysis of research in the computing field might be of interest. In general, such analysis will help to better understand the whole of the computing field, and the interrelationships among its subdivisions. More specifically, the field of computing appears to be in a state of transition. Although historically it has evolved as several stovepipes of knowledge—predominantly, as we have said, CS, SE, and IS—there is now some impetus for amalgamation. Denning, in [1], explains that some integrated schools of computing have already been formed, citing the School of Information Technology and Engineering at George Mason (1986), the College of Computing at Georgia Tech (1991), and the Indiana University School of Informatics (1999), among others. Amalgamation is also evidenced in the fact that, in 2001, the responsibility for accrediting both CS programs, formerly accredited by the CSAB, and IS programs, which were not accredited, was assumed by ABET,¹ a federation of 31 professional engineering and technical societies [7].

The possibility of amalgamation among computing disciplines raises some potentially interesting questions. How well prepared is the computing field in general for changes of this nature? How well do the people in each of the disciplines comprising the computing field understand each other? How well prepared are they to accept each other?

There is reason to believe these issues may be problematic. With respect to both acceptance and understanding, a topic that arises regularly on computing Web sites is “Just what are the differences among these fields?” A recent dialogue on that question drew 526 comments [9]. Some answers to the question were encouraging, if somewhat simplistic; for example, “CS people are the ones who write the software that MIS people implement and use.” However some comments were truly disturbing, such as “Most CS people laugh at MIS people,” and “MIS people make more money and manage the CS folks.”

Understanding of the three disciplines, and their distinct roles, appears to be problematic in more formal circles, as well. While Freeman and Aspray [2] do an excellent job of researching data on worker population in the computing fields, some of their data is disturbingly inaccurate; for example, they show that a field called “Management Information Science” grants at most three Ph.D. degrees each year. It is unclear what that field is, because its name does not match any of the 20 names for computing fields listed in their Table 2-1 in [2]. But if it is Management Information Systems (IS), which seems likely because that is the closest name of any real computing field, then its count of graduates is incorrect. For the past decade, IS has produced between 70 and 100 Ph.D. graduates annually in North America alone [3].

If there are indeed problems with acceptance and understanding, what can be done about them? Clearly, the answer is better information about the nature of the computing field, and better dissemination of that information. It is the purpose of this article to provide some of that information, specifically information about the nature of its research.

Back to Top

Study Approach

We began our analysis of computing research by defining a comprehensive classification system to cover CS, SE, and IS. The classification system and the rationale underlying it can be found in Vessey, Ramesh, and Glass [11]. A brief overview follows:

• Topic (see Table 1) addresses the subject matter of the research. Topics covered are concepts associated with problem solving, computers, systems/software, data/information, problem domain-specific, systems/software management, organizations, and society, as well as disciplinary issues. Each of these categories is further subdivided into a number of subcategories.
• Research approach (see Table 2) addresses the general way the research is conducted. Approaches identified are descriptive, evaluative, and formulative, again with subcategories defined in each.
• Research method (see Table 2) addresses the specific methods used. Research methods examined include conceptual analysis, case study, data analysis, field experiment, laboratory experiment, and simulation. Categories are not further subdivided.
• Reference discipline (see Table 3) addresses the disciplines whose theories formed a basis for the research. Examples are cognitive psychology, social and behavioral science, economics, and management. We also included self-references, such as references to papers/theories in the discipline under examination.
• Level of analysis (see Table 3) addresses the object on which the research study focused. Research can be conducted on both the technical level (such as computing element and abstract concept), and the behavioral level (such as society, profession, organizational, project, group/team, or individual).

Following the development of the classification system, we chose a set of representative, well-recognized, journals from each of the three computing fields, and classified a selection of papers from those journals over the five-year time period 1995–1999, according to the five characteristics previously discussed. We coded 628 papers from CS journals, 369 from SE journals, and 488 from IS journals (see the sidebar, “Journals Examined,” for details).

For each paper examined, we selected a single topic, a single research approach/method, a single reference discipline, and a single level of analysis that best represented the paper. Two coders independently categorized each of the papers. Agreement ranged between 70% and 90%. Differences between coders were then resolved to form the data for this study. We then prepared three papers presenting the findings specific to each of the three computing disciplines, including journal analyses [11]; here, we compare and contrast the findings.

Back to Top

Findings

A comparative analysis of our findings about computing research follows. The findings define, for each of the disciplines, the most dominant research topics, research approaches, research methods, reference disciplines, and levels of analysis.

Topic. The findings for Topic are presented numerically in Table 1 and graphically in Figure 1. CS topics were fairly diversified, with an emphasis on Computer (29%), Problem domain (22%), and Systems/software concepts (19%); SE focused primarily on Systems/software (55%), and Systems/software management concepts (12%); IS focused heavily on Organizational concepts (66%) with Systems/software management and Systems/software concepts next at the 6%–7% level.

Interesting distinctions also appear within categories. The major CS subcategories within the Computer category were Intercomputer communication (18%) and Hardware principles/architecture (10%), while Problem domain was almost entirely about Computer graphics/pattern analysis (20%). The major subcategories in the Systems/software category were Tools (5%), Programming languages (4%), and Methods/techniques (4%). SE subcategories within Systems/software were Methods/techniques (18%) and Tools (12%), while Systems/software management was largely about Measurement/metrics (6%). IS subcategories within Organizational concepts were Usage/operation (24%) and Technology transfer (19%). IS also focused on the Information Systems Problem domain (for example, decision support or group support systems) within the category of Problem domain-specific concepts.

Overall, we see that there was minimal topic overlap among the three disciplines. The primary overlap was in the Systems/software category, which appears to be the common link among the three fields.

Research Approach. The findings for Research Approach are presented numerically in Table 2 and graphically in Figure 2. CS research approaches were overwhelmingly Formulative in nature (for example, Formulate an algorithm), at 79%; SE also used Formulative approaches, but less so than CS, at 55%; IS used predominantly Evaluative research approaches (for example, evaluate the use of an Enterprise Resource Planning system), at 67%.

For CS, the dominant subcategory was Formulate a process, method, or algorithm (53%). Formulate a concept followed, at 17%. Few papers used Evaluative (11%) or Descriptive (10%) research approaches. For SE, the dominant Formulative subcategory was also process, method, or algorithm (36%). Some studies used Descriptive (28%) research approaches, while a few were Evaluative in nature (14%). For IS, most of the Evaluative studies were deductive in nature (47%). Some studies used Formulative approaches (24%), primarily formulating models (13%), while a few were Descriptive in nature (9%).

Overall, we see that CS and SE emphasized Formulative research approaches, with IS also using them, but to a much lesser extent. Given the fact that CS and SE have been criticized for underutilizing evaluative research approaches [10], it is interesting to note the predominance of Evaluative research in IS.

Research Method. The findings for Research Method are presented numerically in Table 2 and graphically in Figure 2. CS research methods consisted predominantly of mathematically based Conceptual Analysis (73%). SE used Conceptual Analysis that is not mathematically based (44%) with Concept Implementation also representing a significant research method at 17%. IS research used predominantly five types of research methods, the most notable being Field Study (27%), Laboratory Experiment (Human) (16%), Conceptual Analysis (15%), and Case Study (13%).

Reference Discipline. The findings for Reference Discipline are presented numerically in Table 3 and graphically in Figure 3. Neither CS nor SE relied much on outside reference disciplines for their work. CS (89%) and SE (98%) used primarily self-references. IS also relied on its own discipline (27%), but also used theories from several other disciplines, for example, Management (18%), Economics (11%), Cognitive Psychology (11%), and the Social and Behavioral sciences (9%). These figures clearly support the notion that IS is an applied discipline, applying the concepts of other disciplines, most notably derived from the field of management.

Level of Analysis. The findings for Level of Analysis are presented numerically in Table 3 and graphically in Figure 3. Nearly all CS and SE work was conducted at the technical level, examining artifacts or entities. CS research focused on the Computing Element (53%) and Abstract Concept (39%) categories. SE also focused on the Abstract Concept (50%) and Computing Element (28%) categories. Behavioral levels of analysis were present in approximately 2% of CS and 8% of SE research.

A majority of the IS work focused on the behavioral levels: Organizational (26%), Individual (24%), and Group/team (11%). The technical levels of Abstract Concept, System, and Computing Element were represented at 9%, 7%, and 5%, respectively.

Back to Top

Discussion

It is interesting to contrast the findings for the three disciplines. CS examines topics related to computer concepts at technical levels of analysis by formulating processes/methods/algorithms largely using mathematically-based conceptual analysis; further, it does not rely on reference disciplines. SE is somewhat similar, but quite distinguishable from CS. It examines topics related to systems/software concepts at technical levels of analysis by formulating processes/methods/algorithms using non-mathematically-based conceptual analysis; like CS, it does not rely on reference disciplines. IS, by contrast, is quite different. It examines topics related largely to organizational concepts, especially usage/operation and technology transfer, although it also explores systems/software topics, all primarily at a behavioral level of analysis. It uses evaluative research approaches, using field studies, laboratory experiments, case studies, as well as several other research methods. The IS discipline also draws from and relies on a variety of reference disciplines, some of which are located in schools of business.

It is particularly interesting to consider these differences from the perspective of an amalgamation of the three disciplines. The topic differences are fairly obvious, and as long as each field respects the topic goals of the other, an amalgamation could effectively occur. The remaining differences may be more problematic, however. Researchers in CS, and to some extent, SE, primarily expect to produce new things—processes, methods, algorithms, products. IS researchers, on the other hand, expect to explore things—theories, concepts, techniques, projects. The things CS and SE produce are almost entirely technical. The explorations of IS are usually performed in an organizational and therefore behavioral context. CS and SE research are, for the most part, not based on theories from other disciplines, but the work is often performed within the rules and practices of mathematics. IS researchers emphasize their work is theory-based, and, as well as using theories based in IS, researchers in this realm also explore the relevance of theories extracted from other disciplines. CS and SE research are often funded externally: seeking grants is one of the tasks of the CS/SE researcher. IS research, on the other hand, has historically most often been funded internally.

All of these differences have resulted in significant problems in the past. Clearly, such differences are at the root of the establishment of a discipline of IS distinct from that of CS. And similar problems may arise again in the future. Each field tends to view the research approaches and contributions of the other field negatively. If you value formulating things, then evaluating things may seem like a lesser pursuit. The opposite is true, as well: for each of the differences noted in the previous paragraph, it is all too easy for one group to think its work is superior to that of another. It is no accident, for example, that CS and SE tend to avoid doing evaluative research, or that IS tends to avoid deeply technical studies.

There is an old saying, in academia and elsewhere, to the effect that the hard drives out the soft—for example, that research using deep technical, and perhaps mathematically based, approaches tends to overwhelm research that uses behaviorally based approaches. If that tendency holds true in any amalgamation of the computing disciplines, then CS and SE would tend to dominate the amalgamation, and the work of IS would be pushed aside. Should that situation occur, it would be to the detriment of the computing field.

Back to Top

Conclusion

Our primary intent in performing this research is to increase our understanding of the computing field from the viewpoint of the research conducted in its three major disciplines. In particular, it is interesting to examine the research similarities and differences across the three fields. (Note that a previous study examining the pedagogy of the three fields by doing a comparative analysis of their curriculum topics found the fields to be satisfyingly distinct [5].)

Regarding any potential amalgamation of the fields, whether at the level of the discipline or of the institution, it is important that it be based on both mutual understanding and mutual acceptance. Our research findings provide information to address such issues directly. Especially important is the fact that each of the fields has singled out a set of topics on which to focus its research, topic areas that have little overlap. The most significant problem area appears to be that each of the fields has its own set of preferred research approaches and research methods, which do not necessarily command the respect of the other disciplines.

We include a personal remark based on the fact that each of the authors of this article in some sense represents one of these disciplines. There are problems on the amalgamation horizon, emerging from the history of these fields. They have not, in the past, communicated well with each other. Their journals tend to be unknown outside disciplinary borders. Terminology differs, sometimes in important ways.² In what may be the biggest problem of all, there is a tendency for each of the fields to disdain the work of the others. Faculty advancement and tenure have, in the past, been problematic when some of these fields have combined (for example, software engineering faculty being denied tenure by computer scientists). These problems must be addressed before any amalgamation could possibly be effective.

Back to Top

Back to Top

Back to Top

Back to Top

Back to Top

Back to Top