ACM Multimedia 2013 – Best Papers and the Revision Process

This year, ACM Multimedia 2013 brought several changes to its review process for the Technical Program and introduced a new organizational role called the “Author’s Advocate.”

Since this process is all new, and to get a better understanding of everything involved, I sat down with General Co-Chair Nicu Sebe, from University of Trento, Program Co-chair David A. Shamma (Ayman), from Yahoo Labs San Francisco, and Author’s Advocate Pablo Cesar, from CWI Amsterdam, at this year’s conference last week.

Marco: I understand that this year saw many changes to the Technical Program Committee and Review Process for ACM MM 2013. What prompted the changes to the review process?

Ayman: When my program co-chairs (Daniel Gatica-Perez, Marcel Worring, and Roger Zimmerman) and I accepted our role, we began walking through the whole process to get a clear understanding of how it’s structured. However, ultimately, we felt it needed some refactoring. First, we started with feedback we’ve informally gathered over the years, for me this came from my involvement in SIGMM’s technical area on Arts and Digital Culture. We also did a fair bit of data analysis from the last 3 years of MM conferences to concretely measure some basic questions like how many reviewers per area were there, what was the distribution of number of papers per reviewer and how long were the reviews. From there, we designed several changes.

Marco: How did you come up with these changes?

Ayman: Some of the changes we adopted from other conferences. Most notably, in past years authors would write a rebuttal before the AC’s meta-review. This year a first set of meta reviews were written by the ACs before we opened the rebuttal period, which is how CHI does it. But the new thing is really the “Author’s Advocate.”

Marco: I’ve never heard of an “Author’s Advocate” before. Where did it come about?

Pablo: It is a new role driven by this year’s general chairs Alejandro (Alex) Jaimes, Nicu Sebe, and Nozha Boujemaa. They were concerned about the lack of protection authors typically have and wanted to create an independent body, who would listen to their concerns. This new role would mediate when low-quality reviews are reported. At first I was reluctant, as I thought that it would be a controversial and undefined role that will make everyone in the community angry: the authors, the reviewers, and the TPC. After reconsideration, I accepted, as I believed it was a healthy position for the overall review process.

Ayman: We were handed the Advocate early on, but not at the start of the process. Since we were already amidst changing things around, it was a great time to introduce a new role.

Marco: Had you heard about other conferences with such a role?

Pablo: Never heard of it before. We had to figure out how to all work together, the responsibilities, the level of access to information, what “detected” actually means.

Ayman: Not from my side either.

Marco: What exactly does the Advocate do?

Pablo: The advocate is the voice for the authors, auditing and ensuring that the review process works as it should. He or she is appointed by the General Chairs to advocate the author’s concerns to the Program Chairs. As an independent body, the advocate somehow acts as the “defensor del pueblo” or ombudsman. In particular, this year we implemented it in the following manner. After the meta reviews were sent out, authors could raise an exception and request the mediation from me during the rebuttal period. The authors were instructed that a request for mediation was different from a rebuttal as the rebuttal is intended as a discussion between the authors and the reviewers where as an advocate request is a red flag about the quality of a given review. After an exception was received, the Program Chairs would grant me access to everything related to the paper: reviews, meta-review, and its discussion thread. From there I could inform the ACs and PCs about further attention and action being needed or not for the paper.

Marco: Sounds like everyone would just request advocation regardless.

Ayman: That was our running joke with Pablo.

Pablo: In the beginning I was quite concerned about the expected number of cases. This lead to a number of heated discussions with the Program Chairs about the way raising exceptions should be implemented. Given that the TPC handled about 250 long papers, and with around a 20% acceptance rate, I expected to receive 200 red flags! In reality, the number was much lower, probably because authors are not yet familiar with this role and because the PCs did an excellent job reformulating the review process. In total, I received 16 cases, out of a total of 235 long papers. From those only 5 cases were accepted, leading to new reviews. Still, I believe the impact of the advocacy cannot be measured by the number of cases, but by the impact such a role has on the TPC and on the process.

Ayman: As Program Chairs, we took it (the Advocate Request) as a challenge. Could we improve the review process to minimize the chance of someone invoking the Advocate. In that spirit, we gave the ACs a fair bit of autonomy and accountability over their reviewers and reviews. We’re quite happy with the extra effort from the ACs as we did see reviews were more thorough this year than in the past.

Pablo: I have to add that ACs perfectly reacted in all the cases, requesting extra reviews when needed and re-starting discussions after those reviews were in. They did as well a good job in updating their own rebuttals, reflecting the review process in a transparent manner.

Marco: Do you know what prompted advocation requests in general?

Pablo: In most cases authors used it as an extra rebuttal channel, indicating how they did not agree with one of the reviewers that scored low the paper. These cases were not accepted for mediation, since the discussion about opinions should be handled by the TPC. In the five accepted cases, the reviews were indeed well below the high expected standards.

Ayman: For each of those papers we added Pablo as a Senior PC Member to see the reviews and the private discussions. Once his decision was made, he would leave a Advocate’s Response as a note for the authors and if that response required action, he told us and we connected him directly with the AC(s) responsible for that paper. From there extra reviews were requested and he participated in the confidential discussions.

Marco: You said you only “accepted” 5 of 16 requests for advocation. Wouldn’t an advocate accept all cases?

Pablo: Well, it is important to differentiate between cases where the quality of the reviews is below the expectation from those where authors disagree with well argumented opinions of the reviewers. In all the cases, the authors received one extra meta review from the advocate summarizing the actions taken because of the mediation request.

Marco: In the end, what happened to the five papers? Did they get accepted?

Pablo: In the end the five papers did not make it to the proceedings

Marco: If nothing advocated got accepted in the end, did the introduction of the authors advocate role improve the review process?

Ayman: The point here is about delivering quality reviews for the authors for feedback and for the Program Chairs to make decisions. If you look at the reviewer load and distribution of review lengths it all looks like an improvement. But to better get a gauge here, again borrowing from SIGCHI’s example, we issued two follow up surveys, one to the Authors of both Accepted and Rejected papers and a second one to everyone on the Technical Program Committee. About a third of all the submitted authors felt the advocate improved the process, about 8% felt it didn’t.

Marco: Will the Advocate Role continue and will there be improvements?

Pablo: Yes. Next year there will be an Author’s Advocate again. In my opinion, the aim should be to ensure continuity, to better defining the boundaries, responsibilities, and the authority. The next goal is to export this new role to other conferences and SIGs!

Ayman: I’d like to see short papers included in the process in the coming years. This year, the advocate was only for full papers.

Marco: Did the advocate play a role during the PC Meeting or in the Best Paper selection?

Pablo: All the advocacy work happened before the TPC meeting, so high-quality reviews were in place before the final discussion about the papers. As the advocate, I participated in the PC meeting as an observer and auditor. It was a great opportunity as well to discuss with the ACs about the exceptions, and to make sure they understood the new role and the author’s raised concerns. Even though it is not necessary, it is healthy to have in the room an independent body who has not been selected by the Program Chairs. Typically, authors are not represented specifically during the TPC meetings. I was not involved in selecting the best paper candidates as it’s outside of the Advocate’s role. The advocate should not care about the quality of the papers, but only about the quality of the reviews.

Ayman: I’d say the Advocate had a positive indirect influence here. In the PC meeting each area got to nominate one paper for candidacy. From there we brought the number of papers down to the final four via consensus amongst the PCs and ACs. The final selection was made by a non TPC-committee of junior and senior researchers at the conference. Having stronger reviews overall made the Best Paper selection process easier for sure. I’d have to say the Advocate and our ACs and Reviewers did a fantastic job along side us…much of the credit goes to them.

Marco: Nicu, do you think the introduction of the authors’ advocate improved the review process? Has the conference program benefitted from it (or simply the whole process has become more “accountable” and fair)?

Nicu: Yes, definitely. Every single conference has particular cases in which the reviewers do not perform their job as expected or the area chair is very superficial in taking a decision. One step to alleviate the former is the introduction of rebuttals and these were introduced some three years ago at my initiative. However, the latter problem is not solved by this as the area chairs can simply (and has happened in many cases) ignore the response received from the authors. In the past the only solution was to ask the program or general chairs for help but this process is cumbersone simply because even if an action should be taken is after the notification and everything get complicated to manage. The idea of the “advocate” came precisely from this. Not only the process became more accountable and fair but we were also sure that such inherent complaints (“I didn’t get the reviews I deserve”, “I wrote such a brilliant paper and the reviewers simply didn’t want to acknowledge this”, “two reviewers really liked the paper but the third one didn’t understand anything so he suggested rejection”, etc.) are treated in an independet way but an independent person that was not part of the actual review process. The idea of the advocate was not to replace the program chairs but rather to signal out cases in which there was a clear misconduct from the reviewers, area chairs, or even the authors.

Marco: How hard was the selection of best papers this year ? There were some research topic more represented than others in the short list?

Nicu: As hard as always! Since the area system was introduced to replace the old three/four tracks structure of the conference, we have always had the problem of unbalance between several areas. Of course, such unbalance existed before at the track level but with predefined areas the problem was regarded as simpler. The main issue was that some areas decided by the organization committee for a particular year were not as popular as others (or the authors did not have the same understanding of the scope of the area) and this had as a consequence the fact that more areas received far more papers than the others. We have designed two solutions for this: (1) allocated more area chairs for some of the areas to take in the extra load that was expected (e.g., the ocntent related areas always had this problem; (2) to choose area chairs that are able to cover more than a single area. In this way, this year we have tried to have a balanced distribution of the workload per area chair (was not always possible but at least we have tried it).

ACM Multimedia Best Papers

Finally, let’s review the best paper awards. In a previous blog post I mentioned the one that can recommend make-ups: it was the winner of this year. Here are the full details. Everybody can access it for three more weeks, even if they do not have paid access to ACM Digital Library:

“Wow! You Are So Beautiful Today!“

As I’ve written their demo was impressive and during the demo session the authors were very busy “beautifying” all the people that tried the system.

The best student paper went to a paper that was already in the best paper session:

“Attributes-augmented Semantic Hierarchy for Image Retrieval“.

Despite the fact that the first paper is a very specialized recommender system and the second one addresses the problem of content based image retrieval, these two works share a common base: they heavily rely on attributes modelling. The first learns them from web images of actresses and top-models, while the second uses attributes classifiers to enrich the outcome of visual conept classifiers in retrieval tasks.

David A. Shamma is senior scientist at Yahoo Labs in San Francisco where he runs the HCI Research Group, and he is Program co-chair for ACM Multimedia 2013.