Trustworthy Systems

Benchmarking flaws undermine security research


Erik van der Kouwe, Gernot Heiser, Dennis Andriesse, Herbert Bos and Cristiano Giuffrida


Leiden University

Vrije Universiteit

UNSW Sydney


Benchmarking systems is difficult. Mistakes can compromise guarantees and threaten reproducibility and comparability. We conduct a study to show that benchmarking flaws are widespread in systems security defense papers, even at tier-1 venues. We aim to raise awareness and provide recommendations for safeguarding the scientific process in our community.

BibTeX Entry

    author           = {van der Kouwe, Erik and Heiser, Gernot and Andriesse, Dennis and Bos, Herbert and Giuffrida,
    date             = {2020-5-11},
    doi              = {},
    issue            = {3},
    journal          = {IEEE Security and Privacy},
    keywords         = {benchmarking, computer systems, performance evaluation, reproducibility of results, security,
                        standardization guidelines},
    month            = may,
    numpages         = {10},
    paperurl         = {},
    publisher        = {IEEE},
    title            = {Benchmarking Flaws Undermine Security Research},
    volume           = {18},
    year             = {2020}