A Wiki For Discussing and Promoting Best Practices in Research

In late 2005, ACM President David Patterson charged the Health of Conferences Committee with the following task:

[C]ollect the best practices onto a Web page so that conference organizers can see innovative ways to cope with the demands of paper submissions, refereeing, and presentations, as the number of papers increase. The hope is that organizers will either try good new ideas or at least avoid the mistakes of others.

The result of our work—a combined effort with many other ACM and IEEE leaders—is a Wiki (wiki.acm.org/healthcc) for presenting and discussing ideas and responses to this growing challenge.

The Wiki offers our process, including the questionnaire we sent to many ACM and IEEE leaders, as well as all responses we received. The responses have been divided among groups that run small (<100 attendees), medium (>100 and <1,000) and large conferences (>1,000). We do not produce summary statistics because we expect our audience is more interested in groups facing situations similar to their own than in statistical averages.

At the heart of the Wiki is "Let’s Talk About Selected Ideas," which serves as a forum for you to participate in a discussion of selected actionable and failed ideas. There are also several categories within the Wiki that focus on specific conference situations. These include:

Accepting More Papers. Some participants argue that too-high acceptance rates (for example, 40%) don’t challenge the field enough; while too-low acceptance rates (<15%) encourage too much conservatism in program committees. Thus, as a field grows, some respondents feel the paper publishing opportunities should also grow to keep acceptance reasonable (for example, 20%–30%).

Visionary Venues. Many groups discuss ways of showcasing papers that present more farsighted or creative ideas. SIGMOD’s CIDR conference was explicitly organized to help disseminate "sketchy" big ideas, rather than fully formed (big or small) ideas. SIGMETRICS includes five-minute lightning talks. SIGAda, however, abandoned a conference of invited papers because non-invited people felt snubbed.

Author Responses (Rebuttals). The idea is to allow authors to provide the program committee a (short) response to reviewer concerns. This practice has been used by such SIGs as SIGARCH, SIGCHI, SIGGRAPH, SIGMICRO, and SIGPLAN. On the plus side, rebuttals avoid the compounding of small misunderstandings, let authors feel they had a chance to make their case, and pressure reviewers to be on time. Others, including ICML, SIGCOMM, SIGKDD, SIGMOBILE, SIGMOD, are more skeptical that rebuttals are worth the effort.

Competitions. Many groups talked about competitions. The committee thought it was important that the competition be decided by an objective measure, to avoid leaving many authors feeling irritated. The computer architecture community, for example, recently had a branch prediction competition decided by testing branch predictors on traces not provided while competitors were developing their entries. SIGecom sponsors an open-invitation trading agent competition in which researchers test ideas about trading strategy in several market games.

The comments noted here scratch the surface of the ideas found on the Wiki today.

Tracking Reviews. The most direct approach to tracking reviews has been undertaken by SIGMOD, where they have created a pipeline with another major database conference—VLDB—where it is common for some papers rejected at one conference to be sent to the next with their reviews carried over. This practice is being tried on a limited basis, only for borderline papers where reviewers feel a round of author revision could lead to a solid contribution.

Two-phase Reviewing. Some conferences (for example, SIGCOMM, SIGMOBILE, SIGMOD) have introduced a two-phase review process where papers with a critical flaw, such as those considered out of scope or clearly non-novel, are rejected with a less rigorous review than those that are competitive. This is a compromise that allows the maximum reviewing resources to be devoted to those papers in serious contention.

Double-blind Submissions. Blind submissions hide the referee name from the authors, while double-blind submissions also hide the author name from the referees (however, author names are typically known at the program committee meeting). Some SIGs (for example, SIGMIS) believe this practice mitigates the perceived bias toward large research institutions, while others SIGs (for example, SIGGRAPH) note that it complicates managing conflicts of interest.

Hierarchical Program Committees. As some conferences receive more papers, they add more members to the program committee. At some point this scaling does not work and other options should be considered. SIGART’s AAMAS conference, for example, has a senior program committee manage the work of a larger program rather than review papers directly.

Co-located Workshops. Many groups had workshops co-located with conferences. SIGIR has 10 workshops on the day preceding its main conference. OOSPLA’s workshops, in fact, have greatly increased conference attendance.

Catch All. This category allows a potpourri of comments. SIGPLAN touted the value of supported conference management software. SIGARCH noted the value of selected shepherding for papers with good ideas, but non-ideal presentations. SIGMICRO would like to see better ways of managing conflicts of interest.

In summary, the comments noted here scratch the surface of the ideas found on the Wiki today. Moreover, we encourage readers to add your ideas. Let the conversation begin!

Back to Top

Back to Top