ACM SIGMOD City, Country, Year
sigmod pods logo

SIGMOD 2012: Experimental Reproducibility

The goal of establishing reproducibility is to ensure your SIGMOD 2012 research paper stands as reliable work that can be referenced by future research. The premise is that experimental papers will be most useful when their results have been tested and generalized by objective third parties.

The Review Process

The committee contacts the authors of accepted papers, who can submit experiments for review --on a voluntary basis-- from April to September 2012. Details about the submission process will be communicated directly to authors. The committee makes the decision to award or not the following labels:

How does the committee assess whether the experiments reproduced by the committee support the central results reported in the paper? To get a reproducible label, a submission must fulfill the following three criteria:

  1. Depth: Each submitted experiment contains:
    • A prototype system provided as a white box (source, configuration files, build environment) or a commercial system fully specified
    • The set of experiments (system configuration and initialization, scripts, workload, measurement protocol) used to produce the raw experimental data
    • The scripts needed to transform the raw data into the graphs included in the paper
  2. Portability: The results can be reproduced on a different environment (i.e., on a different OS or machine) than the original development environment.
  3. Coverage: Central results and claims from the paper are supported by the submitted experiments.

Some Guidelines

Authors should make it easy for reviewers (and the community at large) to reproduce the central experimental results reported in a paper. Here are some guidelines for authors based on the experience from previous years.

We distinguish two phases in any experimentation effort, namely, primary data acquisition and data derivation:

The experiments published by Jens Teubner and Rene Mueller, from ETH Zurich, together with their SIGMOD 2011 article titled "How Soccer Players Would Do Stream Joins," are an excellent illustration of these guidelines.

Reproducibility Committee

Philippe Bonnet, ITU, Denmark, chair
Juliana Freire, NYU, USA, chair
Matias Bjorling, ITU, Denmark
Wei Cao, Renmin University, China
Eli Cortez, Universidade Federal do Amazonas, Brazil
Stratos Idreos, CWI, Netherlands
Ryan Johnson, University of Toronto, Canada
Martin Kaufmann, ETHZ, Switzerland
David Koop, University of Utah, USA
Lucja Kot, Cornell University, USA
Willis Lang, University of Wisconsin-Madison, USA
Mian Lu, HKUST, China
Dan Olteanu, Oxford University, UK
Paolo Papotti, Qatar Computing Research Institute (QCRI), Qatar
Ben Sowell, Cornell University, USA
Radu Stoica, EPFL, Switzerland
Dimitris Tsirogiannis, Microsoft, USA

Follow our progress: Facebook