What We Talk About When We Talk About Artifact Evaluation

Le résumé fourni par la source

Creating quality research artifacts and conducting artifact evaluations are both considered time consuming and poorly rewarded activities. Toward simplifying these tasks, we conducted a mixed-methods study of artifact evaluation at a recent conference. Through analysis of reviewers' comments and other data, we recognized common issues reviewers faced and the features of high quality artifacts. To lessen the time and difficulty of artifact creation and evaluation, we identify design implications for infrastructures like the testbeds used to execute research artifacts, the repositories used to share those artifacts, and the review platforms used by conference committees. By applying the knowledge gleaned through our study, we hope to improve the usability of research infrastructure and, consequently, the reproducibility of research artifacts. In the supplement, we include our diary study questions, interview protocol, and analysis scripts.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.