POPL

It has been my great pleasure and privilege to serve as the Program Chair of the 39th POPL. This section of the proceedings gives me the opportunity to tell you about the process we used to select this year's excellent program, and to thank those who played an important role in that process.

Following the recent history of SIGPLAN conferences such as PLDI and ASPLOS, we used both a program committee (PC) and extended review committee (ERC) to assess submitted papers. The PC of 26 researchers reviewed 20--22 papers each, while the ERC of 60 members reviewed an average of 3--4 papers each, with a maximum of 6. Both PC and ERC members could submit papers, but only the ERC reviewed PC papers, thus avoiding potential conflicts that could otherwise arise at the in-person meeting. At the end of the main review period (10 weeks total), authors were given four days to respond to their reviews, and the reviewers then discussed papers electronically for two weeks. The PC met in person over two days at the University of Maryland, College Park, USA, to decide the majority of the program. Out of 205 submissions, 44 were accepted; three of seventeen PC submissions were accepted following on-line discussion among the ERC.

This year we employed a form of double-blind reviewing (DBR), a first for POPL. In a double-blind process, the reviewers do not learn the identity of the authors (nor do authors learn reviewer identities, as usual). The intention of DBR is to increase the fairness of reviewing and the quality of the accepted papers by avoiding initial, perhaps unconscious bias for or against a paper based on its authorship. A reviewer who picks up a paper known to be written by a famous author and/or institution may grant more to it ("Joe wrote it so I'm sure it must be good") than to a paper by unknown authors from an obscure institution ("I've never heard of this place -- do they even have reasonable PL researchers there?"). If this bias is strong enough, lesser papers may be accepted over ones of higher quality and certain population segments may be underrepresented. After consulting with the POPL Steering Committee and past Chairs of other conferences, and reading some relevant literature, I decided that DBR could have a positive effect and thus it was worth trying for POPL.

While DBR aims to improve fairness and quality, it complicates the process of writing and reviewing papers. To mitigate its disadvantages, we employed a "light" form of DBR. Authors needed to make only a few identity-masking changes to their paper: they redacted their names from the front page and cited their own work in the third person. More draconian changes to obscure likely authorship, such as altering the names of well-known systems, were not required, and most forms of post-submission dissemination were permitted. A paper's authors were hidden from a reviewer only until his or her review was submitted to the conference management system. The idea was that by the time a review is submitted, initial bias has been mitigated, so revealing author identity should not negatively impact fairness. This relaxed identity rule allowed me to call upon the PC to help find additional expert reviewers when needed without worry of unintended conflicts. I did this primarily by assigning a guardian to each paper. The guardian was responsible for submitting a review mid-way through the review period and if his or her own review was not expert-level (most were), he or she could request an additional review, either from the ERC or PC, or from an outside reviewer. In total, we received reviews or feedback from 50 outside reviewers, and we occasionally shuffled ERC and PC assignments to better ntake advantage of available expertise.

I believe this review process worked well. In a survey conducted after paper decisions were announced, a significant majority of authors and reviewers favored the process we used, with most reviewers very strongly in favor of using an ERC and using guardians (even if DBR is not employed). I plan to publish an analysis of the survey data in a forthcoming essay. Most compelling to me were anecdotes from PC members who confessed that in several cases they spent more time reviewing a paper than they would have if they had known the authors, and sometimes were very surprised when the actual authorship contradicted their expectations ("If I had known the paper was by Joe I would have reviewed it differently").

On the other hand, even the diminished costs of light DBR were troublesome to some authors and reviewers, and its benefits are not so easy to quantitatively assess, so we are not at the end of the story. Therefore I hope the community will continue to carefully consider the review processes we use. Peer review is the foundation of the scientific process -- it is a gateway for new ideas and the foundation of our trust in published results. It is essential that we have the most effective process possible.