Structural Challenges and the Need to Adapt

At key junctures in the course of a field’s evolution adjustments may be needed to stimulate and sustain rich, vital scholarship. In this Viewpoint, I will argue that the field of computing and information research is at just such a juncture and that structural changes are needed to ensure the field’s ongoing health. Recently, Communications contributors and others have engaged in a discussion around issues related to the publication culture in computing research and its effects on the field.^1,3,6 That discussion responds in part to a shift in the late 1990s within the main computing archival publication format away from journal publications with variable-page lengths, rolling submissions, and multiple review cycles to conference proceedings with typically 10–15 page limits, set deadlines, and minimal review cycles.⁴ Here, I seek to broaden the conversation to one that foregrounds the question: How do we—as a field and as individual researchers—create robust new scientific and engineering knowledge? Related conversations that concern depth and rigor of scholarship, individual career trajectories, publication, authorship norms, reporting of primary data and results, creation and deployment of artifacts, evaluation criteria, and others, follow from this central question.

To place my comments in perspective, step back to consider a few general observations about the development of any young field in relation to the development of the intellectual lifespans of individual scholars within that field. A new field by definition emerges and takes shape out of one or more existing fields.^a At the onset, there are likely a host of new research questions and opportunities. Early in a field’s history, there may be relatively little prior work to build directly upon; and more work is likely to be the first of its kind. The first scholars in an emerging field, even as they seek to create the early canonical work, often bring an interdisciplinary orientation to their thought. In such cases, they came from and were trained in other fields and bring diverse ways of asking scientific questions and doing scientific work. Over time the infusion of interdisciplinary perspectives can dissipate as those who created the field build the first departments, train the next generation of researchers, and award Ph.D.’s to inddividuals who, in turn, train the second generation of young researchers, and so on. The movement often tends away from an interdisciplinary orientation and toward developing the new field’s distinctive culture and norms.

As a field continues to mature and research accumulates, the need for synthetic, integrative activities emerges. Substantive contributions that are truly novel may be less frequent and may require even greater ingenuity. The sequence described here is not uncommon for young fields and computing and information research is no exception. These and other factors converge in important and complex ways in our field such that the time is right to revisit some of the processes and norms that have evolved and consider adjustments. Such adjustments shape and enable continued strong growth.

---

As a field continues to mature and research accumulates, the need for synthetic, integrative activities emerges.

---

Toward that end, in this Viewpoint I articulate seven structural challenges to the field’s vitality and capacity for knowledge creation, and point briefly to the potential for practices and incentives within the field to act as constructive forces while simultaneously carefully attending to managing the transition, particularly where the careers of young scholars could be at risk.

Back to Top

Seven Structural Challenges to Vibrant Scholarship

Each of the structural challenges hypothesized and discussed here bear in important ways on the kinds of scientific questions that we ask as well as on the kind of research we conduct and report in response to those questions.

1. Building on Prior Work. Knowledge advances in part by building upon, extending, or reacting against earlier ideas. The field profits when researchers acknowledge and make explicit how their new ideas and findings stand in relation to what previously was understood. Doing so typically requires time and space. Time to think through those connections and space in publication venues to report on them. A significant majority of subfields in computing and information science and engineering currently place an emphasis on conference proceedings with comparatively short papers as the predominant publication venue. One significant consequence of this emphasis: there simply may not be enough space to report substantively on how one built on prior work. Given the constraints of page length, priority reasonably must be for reporting the contributions and results of the new work. Furthermore, and in part because research results typically are reported in small(er) pieces, finding let alone exploring and developing the linkages across problem domains and subfields may be difficult if not impossible. At the individual level, researchers are no longer as readily accountable for the hard work of making and elaborating on explicit connections between prior scholarship and the ways in which their new contributions complement, extend, differ from, or challenge it. At the field level, continuity and coherence of research suffers. Over time, the larger development of knowledge may slip from view.
2. Methodological Robustness. Research findings and outcomes have meaning in context: when we know how and under what conditions they were generated. Only then are we positioned as individual scholars and as a field to judge their novelty, rigor, and, of equal importance, their limitations. Here, too, shorter publication lengths may have taken their toll. The robust reporting of method requires space to substantively convey enough details so that reviewers can evaluate the work’s appropriateness and quality; near-term readers of publications can understand the results in context in order to use, apply, replicate, extend, or refute them; and readers in a farther future can understand the historical development of methods and ideas within the field. All of the above are at risk. Moreover, and perhaps most troubling, as researchers have implicitly become less accountable for reporting method robustly, there can be a corresponding tendency for methodological approaches to become less rigorous.
3. Scope and Depth of Research Questions. A field defines itself by the questions that it asks. Here, again, the current publication culture can be felt. Whereas the relatively fast conference publication cycle of shorter papers could lead to more concise reporting of results, it may also lead researchers to pose questions that can be answered more readily within 15 pages or less.^b Such papers may also fare better in the review process as they can be contained and dealt with more fully within the page limits. Granted, some large(r) meaningful questions and results can be expressed and defended concisely, particularly in those subfields that express ideas in precise mathematical notation. Still other meaningful questions are of smaller scope. These should be pursued. However, the undue broader influence of the current conference publication cycle on the scope and depth of research questions needs to be investigated and addressed.
4. Synthesis and Theory Building. Synthesis and theory building^c are two tools for making sense of vast amounts of individual units of knowledge and, reciprocally, providing direction for subsequent investigations. They provide a means for taking up big(ger) ideas, working them out in large(r) ways, and sustained intellectual dialogue. While valuable at any point in a field’s development, these tools become essential as a field matures, subdivides, and accumulates large numbers of seemingly unconnected research findings. For example, both in human-computer interaction and in computer security there are literally hundreds of studies that engage privacy on some dimension; yet we see few analyses that bring these literatures together and offer overarching interpretations and synthesis of the results. In some subfields of computing and information research there are few outlets for such work. Moreover and again depending on subfield, such contributions tend to carry limited weight in the tenure and promotion process. The skills to produce these sorts of quality components take sustained effort to develop, are rarely taught explicitly to younger scholars, and currently often go largely unrewarded.
5. Interdisciplinary Work: Authorship Norms and Reporting of Primary Results. Mature and maturing fields require a steady influx of new ideas to sustain innovation and vitality. Interdisciplinary work is one such well-spring. At intellectual boundaries, established constructs encounter alternative paradigms and familiar problems, new methods, and tools, from which, in turn, new problems (and solutions) emerge. More generally, diverse ways of thinking inspire creativity and imagination. That said, conducting interdisciplinary work is not without its challenges. Two, among the many, pertain to publishing interdisciplinary work and are notable for the ethical quandaries into which they can inadvertently thrust researchers: authorship norms and publication of primary results. With regard to authorship, diverse fields (and even subfields) may have distinct and at times conflicting norms for assigning credit for intellectual contribution.^d Given such conflicts, restructuring authorship norms for interdisciplinary work is critical if interdisciplinary work and the scholars who undertake it are to be positioned to thrive.
     A similar challenge concerns the publication of primary results. Current ethical conventions around scientific publishing prohibit publishing primary results more than once. Those engaged in interdisciplinary work must choose either between violating that ethical norm or inadequately publishing to all the relevant research communities the primary results. Moreover, the latter choice means that over time, inadvertent and unintentional biases could emerge such that some fields are not directly made aware or given access to primary results within the expected communication (for example, publication) venues of their field. From the perspective of those research communities, it can be as if the research was not conducted at all. There, too, lies a certain sort of ethical failing. Underlying both of these challenges are our understandings of ethical scientific practice and the fact that with interdisciplinary work done well there is no meaningful way to speak of a primary discipline. To be sure, solutions to these challenges must reach beyond the computing and information research community into the other sciences and potentially further.
6. Solid Scholarship and Rarity of Innovation. By and large, most research in mature fields contributes incremental new knowledge that fills in gaps and tests the boundaries of established ideas. What we might call interesting solid work. Every now and then, truly new ideas are advanced that result in paradigm shifts, question previously accepted foundational knowledge, or provide radically new ways of thinking. This work is truly novel and innovative. And, if we are honest, quite rare. Both types of activities are important and complementary. That said, hiring and tenure committees at first tier universities by and large privilege novelty. In effect, these committees ask young scholars: "What new thing have you done? What new field (or subfield or sub-subfield) have you created?" Such criteria have at least two unintended consequences: they both encourage or even reward researchers who take familiar ideas adjusted slightly and rename them as new ideas, and they undervalue or even penalize researchers who build in deep ways on prior work and appropriately acknowledge that intellectual legacy. Occasionally, of course, there will be entirely new inventions and significant breakthroughs, and these will need to be recognized as such.
7. Growth of a Scholar. Not only does a field mature but so, too, do the individual scholars whose efforts lead to that field’s ongoing vitality and development. Moreover, experience matters. Yet, I observe that—through complex forces and pressures within the field’s ecosystem—less experienced researchers often prematurely take on key intellectual leadership roles (for example, graduate students now typically present the majority of papers at conferences and in some subfields act as reviewers for major conferences and journals), and increasingly frame research questions and even major research programs. My point is not to suggest disallowing any of the activities mentioned here for less-experienced researchers but that currently the balance is skewed. In short, we fail to adequately nurture the less-experienced researcher or to provide time and space for intellectual maturation; and correspondingly we erode opportunities for the type of substantive intellectual contributions that can come from more seasoned researchers.

Back to Top

Cultivating Adaptations: Toward a (More) Vibrant Scholarly Ecosystem

Exactly how to adapt current practices and incentives in the field to address the structural issues identified in this Viewpoint remains an open question. At a minimum, we can expect to engage the norms and expectations that underlie research and scholarship. For example, the current trend to produce many small(er) publications could be reversed with policies and incentives that reward a smaller number of stronger publications. Indeed, and in part in response to these and related issues, in the U.S. we have seen a recent policy change at the National Science Foundation that now limits the number of proposal submissions for some programs to two per year for each Principal Investigator. This policy and others like it could serve to incentivize writing a smaller number of stronger proposals. Other practices and incentives will need to value building substantively on prior work; recognize solid intellectual development (without requiring all or even most researchers to invent new subfields or coin new terms); and reward synthesis and theory building. For interdisciplinary work, we will need to rethink and clarify norms for crediting intellectual contribution and authorship as well as for the primary publication of results. Critical to all of this is the need to revisit the balance and distribution of activities among more and less experienced researchers. Of course, each of these will need to be carefully thought through and debated within the community (and for those aspects tied to interdisciplinary research within the broader scientific community).

---

As with any dynamic ecosystem, the computing and information research field does not have the possibility of remaining static.

---

Biologists warn us that in shifting ecosystems, those who were well adapted to one environment may be at risk in another.² So, too, when there are significant shifts in a social ecosystem, such as those of the scope advocated for here. Even as we adapt norms, expectations, and practices to sustain and continue to evolve the field, we will need to attend to the careers of talented young researchers who will need to navigate that transition. In particular, the kind of changes discussed here will have teeth when hiring and promotion and tenure committees correspondingly shift their evaluation criteria to, for example, emphasize a smaller coherent body of more substantive publications. Thus, faculty mentoring Ph.D. students and new Ph.D.’s as well as hiring committees and tenure and promotion committees will need to be alert to the transition and its implications for the scholars they are mentoring and evaluating.

As with any dynamic ecosystem, the computing and information research field does not have the possibility of remaining static. In the normal course of events, as some aspects of the field change—with the accumulation of new knowledge, training of younger researchers, and shifts in publication modes—others will need to be adjusted in response. Moreover, at key junctures dynamic systems may require critical, intentional adjustments to ensure their ongoing viability and vibrancy. I have argued that this is just such a moment for the field, if we are to ensure the field’s ongoing ability to generate new, transformative knowledge; ensure deep scholarship; and sustain impact. That said, perturbing any functioning ecosystem is risky business. A small adjustment in one area may have far reaching effects, some or many of which may be unanticipated. Thus, changes will need to be considered carefully, managed over time, and refined (readjusted) as they unfold. As a community, we must proceed both boldly to ensure great scholarship and continuing impact and with alertness so as to minimize harm to the next generation that will carry the field forward.

Admittedly, this Viewpoint is just that. Each of the challenges I have articulated could be the subject of a serious, deep analysis. That would be an excellent next step. Moreover, I have refrained from suggesting specific solutions, as I believe those need to come from the community as a result of thoughtful process and debate. It is my hope this Viewpoint continues and deepens the conversation about these and related issues.

Back to Top

Back to Top

Back to Top