DATA61
Leiden University
Vrije Universiteit
UNSW Sydney
Benchmarking systems is difficult. Mistakes can compromise guarantees and threaten reproducibility and comparability. We conduct a study to show that benchmarking flaws are widespread in systems security defense papers, even at tier-1 venues. We aim to raise awareness and provide recommendations for safeguarding the scientific process in our community.
@article{vanderKouwe_HABG_20, doi = {https://doi.org/10.1109/MSEC.2020.2969862}, publisher = {IEEE}, month = may, journal = {IEEE Security and Privacy Magazine}, paperurl = {https://trustworthy.systems/publications/csiro_full_text/vanderKouwe_HABG_20.pdf}, year = {2020}, issue = {3}, keywords = {benchmarking, computer systems, performance evaluation, reproducibility of results, security, standardization guidelines}, volume = {18}, numpages = {10}, title = {Benchmarking Flaws Undermine Security Research}, author = {van der Kouwe, Erik and Heiser, Gernot and Andriesse, Dennis and Bos, Herbert and Giuffrida, Cristiano}, date = {2020-5-11} }