Nothing Special   »   [go: up one dir, main page]
skip to main content
Advanced Search

Registered Papers


Guest Editor in Chief

  • Marcel Böhme

Guest Editors

  • Abhik Roychoudhury
  • Burak Turhan

We invite contributions for Registered Papers in ACM Transactions on Software Engineering and Methodology (TOSEM).  

The publication model revolves around two stages:

  • In Stage 1, authors submit a registered report, which is essentially a full paper minus the results of the proposed study. The report shall include a section called "Results" with the following text: "This is a Stage 1 submission and results are not included. Please see https://dl.acm.org/journal/tosem/registered-papers for details." Importantly, the report shall also include a detailed description of the methodology to be used to obtain the study results, as well as preliminary results demonstrating the feasibility of the approach.  Reviewers assess the registered reports in terms of novelty and significance, and the study design in terms of soundness and reproducibility.  In-principle acceptance is decided at this stage, after potential revisions. 
  • In Stage 2, authors submit the full paper, including the results of their study.  In this stage, reviewers only assess whether the full paper followed the agreed methodology and the results are correctly interpreted.  Stage 2 may also involve revisions. Stage 2 authors are expected to submit the full paper within 10 weeks and have a maximum of six months to submit it; after this period, the in-principle acceptance expires and, unless a request for extension is granted, the paper is considered rejected.

This model offers several advantages. First, it encourages the exploration of risky ideas, which might not be pursued otherwise.  Reviewers are forced to focus on the novelty and significance of the idea, rather than the results.   Therefore, this publication model also facilitates the publication of valuable negative results, which might otherwise be rejected, by eliminating several sources of bias in the reviewing process, such as publication bias (where authors’ are inclined to selectively publish only positive results), confirmation bias (where reviewers might give more credence to results that support their own views), and impact bias (where reviewers might give novel results more consideration).  Furthermore, this publication model allows early feedback and revisions of the proposed methodology, at a point in time when it is still cheap to do so (i.e. only preliminary results have been obtained).

The TOSEM Registered Paper track aims to improve the reproducibility of study results and adds to the trustability of the evidence in the articles published in TOSEM. To this end, we intend to require the final manuscripts submitted in Stage 2 to be accompanied by the digital artifacts that could be reasonably expected to be provided to someone seeking to reproduce the study. The digital artifacts will be evaluated as part of the Replicating Computational Result (RCR) initiative. We follow the ACM Artifact Review and Badging guidelines and evaluate the degree to which the artifacts are “documented, consistent, complete, exercisable, and include appropriate evidence of verification and validation”.

Please submit your papers at https://mc.manuscriptcentral.com/tosem and choose the Registered Paper as the submission type. Papers submitted to this track are handled as fast-impact track papers. Hence, they must comply with the constraints of such track:

  1. They must qualify as journal-first papers;
  2. Their length can be at most 45 pages (excluding bibliography).
 

Reviewer Guidelines

We expect the same reviewers to handle both stages.

Stage 1

A submission of a registered report should be evaluated based on the following criteria:

    (i) the significance and novelty of the hypotheses or techniques, and

    (ii) the soundness and reproducibility of the methodology specified to validate the claims or hypotheses.

The strength of the (preliminary) results is explicitly not a reviewing criterion. The study design is a critical component of a registered report. Hence, the soundness of the study design should be evaluated very carefully. The registered report is meant to act as an agreement between authors and reviewers: in particular, if the experimental methodology outlined in the report is followed and deviations are explained, the paper will be accepted, irrespective of the study results. Unlike for full papers, for registered reports it is reasonable to request Minor/Major revisions of the study design. Every request for a revision should be accompanied by concrete action items for the authors.

Stage 2

A submission of the full paper should be evaluated based on the following criteria:

   (i) the degree to which the study design in the accepted registered report has been followed, and

   (ii) the degree to which any deviations have been explained.

The strength of the final results is explicitly not a reviewing criterion, but the reviewers should check that the results have been properly interpreted and explained.

Stage 2 can also go through minor/major revisions, but these can not be requested for the study design, which should be agreed at Stage 1.

During Stage 2, reviewers also have the opportunity to raise questions about the implementation of the study design for consideration during the artifact evaluation within the Replicated Computational Results (RCR) track.