For the 2023 Individualized Cybersecurity Research Mentoring Workshop (iMentor),1 I gave a 45-minute presentation on writing conference rebuttals. The insights would benefit others beyond the cybersecurity research community, so I organized my thoughts and shared them with our broader computing community. Submitting rebuttals, also known as the author response period, is a common practice in computing conferences. After reading the reviews, the authors can submit a succinct—for example approximately 700 words—itemized response, within several days. The program committee then discusses the work further to reach the final verdict for the manuscript.
Writing rebuttals energizes me, even though most of my rebuttals do not change the rejection fate of my papers. I genuinely enjoy the opportunity of communicating with experts in the field. This process of intensely reading others’ opinions about my work also helps solidify my own research style. Discussing rebuttal plans is a wonderful team bonding activity, so be sure to analyze reviews together with coauthors. Unconstructive reviews, however, induce impostor syndrome.13 Therefore, plan rebuttals after you have overcome the initial shock.
However, for junior researchers experiencing job and graduation pressure, this rebuttal process can be rather stressful and confusing. I hope this discussion on rebuttal strategies, tactics, and the big picture of research principles mentioned here are useful for writing itemized author responses for journals, too.
Joan of Arc said, “All battles are first won or lost, in the mind.” Have faith in your work, regardless of the outcome. Authors must think beyond the outcomes of a particular paper. I have also shared some of my rebuttals to help others get familiar with this type of writing.14
Why Bother Submitting a Rebuttal?
Researchers show 1%–4.4% of papers were positively impacted by rebuttals in five recent conferences.4 The low statistical impact on papers’ outcomes is also observed by others.7 I briefly describe three scenarios where writing rebuttals could be immensely useful: major revision competency; refutation to set the record straight; and training critical thinking and question answering.
For conferences that have a major revision (for example, USENIX Security Symposium) or conditional accept option, authors should submit author responses. For USENIX Security, the acceptance rates of major revised manuscripts are very high: 85.7% in 20203 and 86% in 2021.5 However, before handing out the precious major revision verdict, reviewers examine the rebuttal for indicators showing the team’s revision competency. Are the authors willing to conduct the necessary new experiments? Can authors adequately address the requested revision items in a few months? Poorly written rebuttals may indicate low-quality revision down the road. Thus, the rebuttal needs to show strong revision competency and commitment.
I sometimes submit rebuttals even when my paper receives entirely hopeless ratings. The purpose is to refute. This refutation scenario is where authors need to clarify factual errors and serious misunderstandings about their work, for example, regarding novelty, significance, or correctness. You may need to strongly refute reviewers to set the record straight, preparing for future submissions. The paper review circle is small—resubmissions may be assigned to the same, possibly biased, reviewers at later conferences. I experienced this situation in our CryptoGuard work,11 with reoccurring negative talking points incorrectly insisting that earlier prototypes had already solved the problem of deployment-grade cryptographic API misuse detection. It is unnatural not to feel upset and frustrated—the work is your brainchild, but some experts think it is unworthy. Tap into that energy when writing rebuttals. Such situations also indicate that your research vision and style differ from others—an advantage in the long run.
Lastly, writing rebuttals improves one’s ability to brainstorm and answer questions in a straightforward manner. Mastering these essential skills requires practice. Therefore, even with heartbreaking ratings, completing the rebuttal process has training values.
Dynamics of the Paper Review Process
Strategically, the key mission of rebuttals is twofold: to solidify the champion’s support and to help the champion defend your work or “embolden” the champion.6 The latter is also known as “arm the champion,”15 that is, equipping the champion with additional evidence, so they can argue for your acceptance. Champions are reviewers who vocally advocate for your work.9 Some conference assigns a champion to each paper.16
The intuitive idea that a rebuttal does not need to carefully address champions’ comments15 is overly optimistic. Upon seeing weak rebuttals, a champion may declare ending their championing position and lower their rating, which was also reported by other researchers (for example, Morris8). After all, most champions do see your work’s flaws and want to hear your defense and justification. Thus, seriously addressing champions’ comments and concerns is absolutely necessary.
Social psychology experiments have repetitively shown that people have a tendency to conform.a Herd mentality also continues in online spaces.b Most reviewers tend to converge to the majority opinion, as it is the safest. Consider this scenario. There are four initial reviews, including one reject, one weak reject, one weak accept, and one accept. However, the weak accept reviewer — upon seeing the assertive negative reviews — immediately lowers their rating to weak reject. At this point, what does it take for the only positive reviewer to champion this submission? Championing takes energy. It also takes courage, patience, and persistence. The positive reviewer needs to read all reviews, re-read parts of the paper, and initiate the discussion to admit the current deficiencies in the paper, but continue to explain why these deficiencies are somewhat fixable or tolerable. In addition, the champion would need to ask strong detractors whether or not the rebuttal alleviates their concerns. Occasionally, this negotiation may also give the superficial impression that champions may have a lower scientific standard, which is not true at all. Being a champion is exhausting. Therefore, if the champion’s concerns about your work are inadequately resolved, it would be difficult for them to support your work wholeheartedly.
The preceding example involves mixed diverging reviews. If planned well, such controversial cases may present promising outcomes. Another common situation is where all reviewers sound lukewarm and ratings are mostly borderline-level, for example, two weak rejects and one weak accept. The excitement is clearly low. This scenario is more thorny, as the paper is likely to be rejected if there is no further discussion. As authors, you still have hope, as you can organize and interpret the reviews as part of your rebuttal, possibly delivering some degree of enthusiasm via your words. The rebuttal is your last resort to energize reviewers. For most conferences, PC chairs or area chairs also read your rebuttals and attempt to energize the discussion, as others also pointed out.10 Oftentimes, I found reviewers give low ratings without any serious issues or have unrealistic expectations for research prototypes. For cybersecurity conferences, occasionally my rebuttals had to remind reviewers that security is relative and it is impossible to achieve absolute security.
Here, I highlight some key rebuttal writing tactics with a simple question.
Some Rebuttal-Writing Tactics
When addressing critical comments, the response needs to be thoughtful and have depth. Brainstorm a little. Share your expert opinions. Let me illustrate using one seemingly innocuous comment: “Will your solution handle situation A?”
This is not a simple yes-or-no question, if A is a complex scenario and you have not thoroughly discussed it. Trivializing reviewers’ critical comments is a common rebuttal pitfall. Recognize key concerns and address them with thoughtfulness. For cybersecurity work, typically there are several possible options.
To defend your work: Simply stating yes is insufficient. One needs to provide evidence, reasoning, or justification, as well as possible challenges associated with porting your solution to handle A.
To point out that the concern is universal: A is a common challenge (to point out, lack of security ground truth in real-world code), thus not a dealbreaker. However, do thoughtfully discuss how A impacts your work, for example, on precision and recall values and security guarantees.
To point out it is an open problem: No one knows how to handle A well, but you plan to offer good insight in the revision.
To admit true limitations: This response still needs to be thoughtfully worded to provide good insights. For example, A is out of our threat model, but other solutions can be borrowed to handle A. However, the integration may present complications in our setting. After all, it is unreasonable to expect one paper to solve everything everywhere all at once.
A combination of all: For example, A may represent an open problem that is universal to some approaches that your work can potentially handle, but with limitations. Capturing these subtleties helps convey your technical depth.
Paraphrase the reviews and reorder them.
You have full control over what goes into the rebuttal. You can reorder comments by importance. You can paraphrase the comments. Describe their concern in your own words. This process also shows your deep understanding and appreciation of their comments. Most importantly, using consistent terminology in both the question and the answer makes it easy to follow.
Avoid excessively referencing existing sections.
What was already said in the paper clearly is unconvincing; repeating it would be unwise. The rebuttal is a precious space to provide new information, new perspective, new content, and new results (if permitted). Virtually all my rebuttals include new numbers and new experimental data. Running experiments takes time. Thus, the author team needs to meet as soon as possible to devise a rebuttal plan.
Convey your strong willingness to revise and do new work.
A common pitfall in writing rebuttals is the lack of commitment to revision. However, depending on whether the conference offers a major revision or conditional acceptance option, you would write differently. Most reviewers do not feel comfortable conditionally accepting a paper, if the new version will likely look drastically different. Several blog articles also give other great suggestions.6,8,10,12,15
Appeal a Decision
My recent Communications Medicine work on AI digital health fairness2 was rejected in the initial round of review. Luckily, Nature journals have a streamlined appeal process. Eventually, the editor decided to reverse their decision and send my revised version out again to reviewers. Journal reviewers operate in isolation and thus do not influence each other’s opinion, at least in the initial round of review.
In contrast, appealing is less useful in our conference review systems based on my experience, as outlier reviewers tend to quickly give up and conform to the majority verdict. Thus, even if the PC chair agrees to add a couple more reviewers who like your work, the outcome rarely changes. Here is why—the minute new reviewers see the existing negative ratings, they will likely swiftly reduce their ratings and declare alignment with the existing votes. This behavior is understandable, as championing such a paper is clearly an uphill battle. The authors can always improve their work and resubmit later, so why rush? Regardless of the outcomes, PC chairs and area chairs are instrumental in creating thorough post-response discussion, by asking “Reviewers, does the rebuttal address your concerns? Why not?”
The Big Picture in Research
In The Art of War, Sun Tzu wrote “Victorious warriors win first and then go to war, while defeated warriors go to war first and then seek to win.” For researchers, doing good work in the first place is the ultimate key. The most brilliant rebuttal could not rescue ill-formed research.
The peer review system has randomness due to the unpredictable nature of the discussion (for example, your champion may be unavailable for discussion due to a medical emergency). The peer review system also has implicit biases. For example, the importance of a research direction is heavily influenced by the reviewers’ own research interests. In my experience with cybersecurity conferences, papers in hot areas tend to receive longer reviews with more excitement, as they address “important and timely problems.” In contrast, work addressing older-but-still-unsolved problems (for example, false positives in deploying anomaly-based intrusion detection) is likely met with dampened enthusiasm and perfunctory lackluster reviews. Non-expert reviewers may also have the incorrect impression that the problem has long been solved. Continuing to diversify program committees, as many conferences are doing, would help reduce such implicit biases.
For researchers, understanding the review mechanism and the dynamics among reviewers would help them navigate the publication process. Keep submitting. Win the battle in your mind.
Join the Discussion (0)
Become a Member or Sign In to Post a Comment