Trustworthy Systems

Benchmarking flaws undermine security research

Authors

Erik van der Kouwe, Gernot Heiser, Dennis Andriesse, Herbert Bos and Cristiano Giuffrida

DATA61

Leiden University

Vrije Universiteit

UNSW Sydney

Abstract

Benchmarking systems is difficult. Mistakes can compromise guarantees and threaten reproducibility and comparability. We conduct a study to show that benchmarking flaws are widespread in systems security defense papers, even at tier-1 venues. We aim to raise awareness and provide recommendations for safeguarding the scientific process in our community.

BibTeX Entry

  @article{vanderKouwe_HABG_20,
    author           = {van der Kouwe, Erik and Heiser, Gernot and Andriesse, Dennis and Bos, Herbert and Giuffrida,
                        Cristiano},
    date             = {2020-5-11},
    doi              = {https://doi.org/10.1109/MSEC.2020.2969862},
    issue            = {3},
    journal          = {IEEE Security and Privacy Magazine},
    keywords         = {benchmarking, computer systems, performance evaluation, reproducibility of results, security,
                        standardization guidelines},
    month            = may,
    numpages         = {10},
    paperurl         = {https://trustworthy.systems/publications/full_text/vanderKouwe_HABG_20.pdf},
    publisher        = {IEEE},
    title            = {Benchmarking Flaws Undermine Security Research},
    volume           = {18},
    year             = {2020}
  }

Download