Trustworthy Systems

The truth, the whole truth, and nothing but the truth: A pragmatic guide to assessing empirical evaluations

Authors

Stephen Blackburn, Amer Diwan, Matthias Hauswirth, Peter Sweeney, José Nelson Amaral, Tim Brecht, Lubom\'\ir Bulej, Cliff Click, Lieven Eeckhout, Sebastian Fischmeister, Daniel Frampton, Laurie Hendren, Michael Hind, Tony Hosking, Richard Jones, Tomas Kalibera, Nathan Keynes, Nathaniel Nystrom and Andreas Zeller

Azul
H2O
Neurensic

Google

Charles University

DATA61

University of Alberta

McGill University

Australian National University

Ghent University

University of Kent

IBM Research

University of Waterloo

Saarland University

Università della Svizzera italiana

Microsoft

Oracle

Abstract

An unsound claim can misdirect a field, encouraging the pursuit of unworthy ideas and the abandonment of promising ideas. An inadequate description of a claim can make it difficult to reason about the claim, for example, to determine whether the claim is sound. Many practitioners will acknowledge the threat of unsound claims or inadequate descriptions of claims to their field. We believe that this situation is exacerbated, and even encouraged, by the lack of a systematic approach to exploring, exposing, and addressing the source of unsound claims and poor exposition. This article proposes a framework that identifies three sins of reasoning that lead to unsound claims and two sins of exposition that lead to poorly described claims and evaluations. Sins of exposition obfuscate the objective of determining whether or not a claim is sound, while sins of reasoning lead directly to unsound claims. Our framework provides practitioners with a principled way of critiquing the integrity of their own work and the work of others. We hope that this will help individuals conduct better science and encourage a cultural shift in our research community to identify and promulgate sound claims.

BibTeX Entry

  @article{Blackburn_DHSABBCEFFHHHJKKNZ_16_2,
    publisher        = {ACM},
    doi              = {https://doi.org/10.1145/2983574},
    month            = oct,
    journal          = {ACM Transactions on Programming Languages and Systems},
    year             = {2016},
    issue            = {4},
    volume           = {38},
    title            = {The Truth, The Whole Truth, and Nothing But the Truth: {A} Pragmatic Guide to Assessing Empirical
                        Evaluations},
    author           = {Blackburn, Stephen and Diwan, Amer and Hauswirth, Matthias and Sweeney, Peter and Amaral, Jos\'{e}
                        Nelson and Brecht, Tim and Bulej, Lubom\'{\i}r and Click, Cliff and Eeckhout, Lieven and
                        Fischmeister, Sebastian and Frampton, Daniel and Hendren, Laurie and Hind, Michael and Hosking, Tony
                        and Jones, Richard and Kalibera, Tomas and Keynes, Nathan and Nystrom, Nathaniel and Zeller, Andreas},
    date             = {2016-10-1},
    pages            = {15:1-20}
  }

Download