Reproducibility Debt: Challenges and Future Pathways
Authors: 
Zara Hassan, 
Christoph Treude, 
Michael Norrish, Graham Williams, 
Alex PotaninAuthors Info & Claims
Zara Hassan, Christoph Treude, Michael Norrish, Graham Williams, Alex PotaninAuthors Info & ClaimsPages 462 - 466
Abstract
Reproducibility of scientific computation is a critical factor in validating its underlying process, but it is often elusive. Complexity and continuous evolution in software systems have introduced new challenges for reproducibility across a myriad of computational sciences, resulting in growing debt. This requires a comprehensive domain-agnostic study to define and asses Reproducibility Debt (RpD) in scientific software, thus uncovering and classifying all underlying factors attributed towards its emergence and identification i.e., causes and effects. Moreover, an organised map of prevention strategies is imperative to guide researchers for its proactive management. This vision paper highlights the challenges that hinder effective management of RpD in scientific software, with preliminary results from our ongoing work and an agenda for future research.
References
[1]
Mohammad Akhlaghi, Raul Infante-Sainz, Boudewijn F. Roukema, Mohammadreza Khellat, David Valls-Gabaud, and Roberto Baena-Galle. 2021. Toward Long-Term and Archivable Reproducibility. Computing in Science & Engineering, 23, 3 (2021), 82–91. issn:1521-9615 https://doi.org/10.1109/mcse.2021.3072860
[2]
Nicolli S.R. Alves, Thiago S. Mendes, Manoel G. de Mendonça, Rodrigo O. Spínola, Forrest Shull, and Carolyn Seaman. 2016. Identification and Management of Technical Debt: A Systematic Mapping Study. Information and Software Technology, 70 (2016), 100–121. issn:0950-5849 https://doi.org/10.1016/j.infsof.2015.10.008
[3]
Monya Baker. 2016. 1,500 Scientists Lift the Lid on Reproducibility. Nature, 533, 7604 (2016), 452–454. issn:1476-4687 https://doi.org/10.1038/533452a
[4]
Lorena A. Barba. 2018. Terminologies for Reproducible Research. https://doi.org/10.48550/ARXIV.1802.03311
[5]
Adam Brinckman, Kyle Chard, Niall Gaffney, Mihael Hategan, Matthew B Jones, Kacper Kowalik, Sivakumar Kulasekaran, Bertram Ludäscher, Bryce D Mecum, and Jarek Nabrzyski. 2019. Computing Environments for Reproducibility: Capturing the “Whole Tale”. Future Generation Computer Systems, 94 (2019), 854–867. issn:0167-739X
[6]
R. Shane Canon. 2020. The Role of Containers in Reproducibility. In 2020 2nd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC). 19–25. https://doi.org/10.1109/CANOPIEHPC51917.2020.00008
[7]
L. Fernando Capretz. 2014. Bringing the Human Factor to Software Engineering. IEEE Software, 31, 02 (2014), mar, 104–104. issn:1937-4194 https://doi.org/10.1109/MS.2014.30
[8]
Nigel Cook, Dejan Milojicic, Richard Kaufmann, and Joel Sevinsky. 2012. N3phele: Open Science-as-a-Service Workbench for Cloud-based Scientific Computing. In 2012 7th Open Cirrus Summit. 1–5. https://doi.org/10.1109/OCS.2012.30
[9]
Chris Drummond. 2009. Replicability is not Reproducibility: Nor is it Good Science. In Evaluation Methods for Machine Learning Workshop at the 26th ICML. 1, National Research Council of Canada Montreal, Canada.
[10]
Bakinam T. Essawy, Jonathan L. Goodall, Hao Xu, and Yolanda Gil. 2017. Evaluation of the OntoSoft Ontology for Describing Metadata for Legacy Hydrologic Modeling Software. Environmental Modelling & Software, 92 (2017), 317–329. issn:1364-8152 https://doi.org/10.1016/j.envsoft.2017.01.024
[11]
FASEB. 2019. National Academies Releases Report on Reproducibility and Replicability in Science. https://www.faseb.org/journals-and-news/washington-update/national-academies-releases-report-on-reproducibility-and-replicability-in-science
[12]
Carlos Fernández-Sánchez, Juan Garbajosa, Agustín Yagüe, and Jennifer Perez. 2017. Identification and analysis of the elements required to manage technical debt by means of a systematic mapping study. Journal of Systems and Software, 124 (2017), 22–38. issn:0164-1212 https://doi.org/10.1016/j.jss.2016.10.018
[13]
Jesús M. González-Barahona and Gregorio Robles. 2012. On the Reproducibility of Empirical Software Engineering Studies Based on Data Retrieved From Development Repositories. Empirical Software Engineering, 17, 1 (2012), 75–89. issn:1573-7616 https://doi.org/10.1007/s10664-011-9181-9
[14]
Dustin Heaton and Jeffrey C. Carver. 2015. Claims about the use of software engineering practices in science: A systematic literature review. Information and Software Technology, 67 (2015), 207–219. issn:0950-5849 https://doi.org/10.1016/j.infsof.2015.07.011
[15]
Peter Ivie and Douglas Thain. 2019. Reproducibility in Scientific Computing. Comput. Surveys, 51, 3 (2019), 1–36. issn:0360-0300 https://doi.org/10.1145/3186266
[16]
Clemente Izurieta, Ipek Ozkaya, Carolyn Budinger Seaman, Philippe B Kruchten, Robert L. Nord, Will Snipes, and Paris Avgeriou. 2016. Perspectives on Managing Technical Debt: A Transition Point and Roadmap from Dagstuhl. In QuASoQ/TDA@APSEC.
[17]
Ivo Jimenez, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau, Jay Lofstead, Carlos Maltzahn, Kathryn Mohror, and Robert Ricci. 2017. PopperCI: Automated reproducibility validation. In 2017 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS). 450–455. https://doi.org/10.1109/INFCOMW.2017.8116418
[18]
Arne N. Johanson and Wilhelm Hasselbring. 2018. Software Engineering for Computational Science: Past, Present, Future. Computing in Science & Engineering, 20 (2018), 90–109. https://doi.org/10.1109/MCSE.2018.021651343
[19]
Upulee Kanewala and James M. Bieman. 2014. Testing scientific software: A systematic literature review. Information and Software Technology, 56, 10 (2014), 1219–1232. issn:0950-5849 https://doi.org/10.1016/j.infsof.2014.05.006
[20]
Barbara Ann Kitchenham and Stuart Charters. 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering. Keele University and Durham University Joint Report.
[21]
Valentina Lenarduzzi, Francesco Lomio, Sergio Moreschini, Davide Taibi, and Damian Andrew Tamburri. 2021. Software Quality for AI: Where We Are Now? In Software Quality: Future Perspectives on Software Engineering Quality, Dietmar Winkler, Stefan Biffl, Daniel Mendez, Manuel Wimmer, and Johannes Bergsmann (Eds.). Springer, Cham. 43–53.
[22]
Ben Marwick. 2017. Computational Reproducibility in Archaeological Research: Basic Principles and a Case Study of Their Implementation. Journal of Archaeological Method and Theory, 24, 2 (2017), 424–450. issn:1072-5369 https://doi.org/10.1007/s10816-015-9272-9
[23]
Haiyan Meng, Rupa Kommineni, Quan Pham, Robert Gardner, Tanu Malik, and Douglas Thain. 2015. An invariant framework for conducting reproducible computational science. Journal of Computational Science, 9 (2015), 137–142. issn:1877-7503 https://doi.org/10.1016/j.jocs.2015.04.012 Computational Science at the Gates of Nature
[24]
Suchita Mukherjee, Abigail Almanza, and Cindy Rubio-González. 2021. Fixing dependency errors for Python build reproducibility. https://doi.org/10.1145/3460319.3464797
[25]
NASEM. 2019. Reproducibility and Replicability in Science. The National Academies Press, Washington, DC. isbn:978-0-309-48616-3 https://doi.org/10.17226/25303
[26]
Roger D. Peng. 2011. Reproducible Research in Computational Science. Science, 334, 6060 (2011), 1226–1227. issn:0036-8075 https://doi.org/10.1126/science.1213847
[27]
G. Pinto, I. Wiese, and L. F. Dias. 2018. How Do Scientists Develop Scientific Software? An External Replication. In IEEE 25th International Conference on Software Analysis, Evolution and Reengineering. IEEE, Campobasso, Italy. 582–591.
[28]
Edward Raff and Andrew L. Farris. 2022. A Siren Song of Open Source Reproducibility. https://doi.org/10.48550/ARXIV.2204.04372
[29]
Nicolli Rios, Manoel Gomes de Mendonça Neto, and Rodrigo Oliveira Spínola. 2018. A Tertiary Study on Technical Debt: Types, Management Strategies, Research Trends, and Base Information for Practitioners. Information and Software Technology, 102 (2018), 117–145. issn:0950-5849 https://doi.org/10.1016/j.infsof.2018.05.010
[30]
Nicolli Rios, Leonardo Mendes, Cristina Cerdeiral, Ana Patrícia F. Magalhães, Boris Perez, Darío Correal, Hernán Astudillo, Carolyn Seaman, Clemente Izurieta, Gleison Santos, and Rodrigo Oliveira Spínola. 2020. Hearing the Voice of Software Practitioners on Causes, Effects, and Practices to Deal with Documentation Debt. In Requirements Engineering: Foundation for Software Quality: 26th International Working Conference, REFSQ 2020, Pisa, Italy, March 24–27, 2020, Proceedings. Springer-Verlag, Berlin, Heidelberg. 55–70. isbn:978-3-030-44428-0 https://doi.org/10.1007/978-3-030-44429-7_4
[31]
Junior Cesar Rocha, Vanius Zapalowski, and Ingrid Nunes. 2017. Understanding Technical Debt at the Code Level from the Perspective of Software Developers. Association for Computing Machinery, New York, NY, USA. 64–73. isbn:9781450353267 https://doi.org/10.1145/3131151.3131164
[32]
D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. 2015. Hidden technical debt in Machine learning systems. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2 (NIPS’15). MIT Press, Cambridge, MA, USA. 2503–2511.
[33]
Victoria Stodden. 2010. The Scientific Method in Practice: Reproducibility in the Computational Sciences. SSRN Electronic Journal, 4773–10. issn:1556-5068 https://doi.org/10.2139/ssrn.1550193
[34]
Victoria Stodden and Sheila Miguez. 2014. Best Practices for Computational Science: Software Infrastructure and Environments for Reproducible and Extensible Research. Journal of Open Research Software, 2, 1 (2014), e21. issn:2049-9647 https://doi.org/10.5334/jors.ay
[35]
Edith Tom, Aybüke Aurum, and Richard Vidgen. 2013. An exploration of Technical Debt. Journal of Systems and Software, 86, 6 (2013), 1498–1516. issn:0164-1212 https://doi.org/10.1016/j.jss.2012.12.052
[36]
Melina Vidoni. 2021. Evaluating Unit Testing Practices in R Packages. In Proceedings of the 43rd International Conference on Software Engineering. IEEE Press, USA. 1523–1534. isbn:9781450390859 https://doi.org/10.1109/ICSE43902.2021.00136
[37]
M. Vidoni. 2021. Self-Admitted Technical Debt in R Packages: An Exploratory Study. In IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE Computer Society, Los Alamitos, CA, USA. 179–189. https://doi.org/10.1109/MSR52588.2021.00030
[38]
Jiawei Wang, Tzu-yang Kuo, Li Li, and Andreas Zeller. 2020. Restoring reproducibility of Jupyter notebooks. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings (ICSE ’20). Association for Computing Machinery, New York, NY, USA. 288–289. isbn:9781450371223 https://doi.org/10.1145/3377812.3390803
[39]
Claes Wohlin. 2014. Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (EASE ’14). ACM, New York, NY, USA. Article 38, 10 pages. https://doi.org/10.1145/2601248.2601268
Index Terms
- Reproducibility Debt: Challenges and Future Pathways
Recommendations
Towards Evidence-Based Software Quality Practices for Reproducibility: Preliminary Results and Research Directions
ACM REP '23: Proceedings of the 2023 ACM Conference on Reproducibility and ReplicabilityIn the computational science and engineering (CSE) community, there is a prevailing belief that adopting better software development practices and investing in software quality will directly lead to more robust, reproducible software. There is, however, ...
Disciplinary differences of software use and impact in scientific literature
AbstractSoftware plays an important role in the advancement of science. Software developers, users, and funding agencies have deep interests in the impact of software on science. This study investigates the use and impact of software by examining how ...
Changes and challenges of technical debt and its management during ongoing digital transformation
XP '17: Proceedings of the XP2017 Scientific WorkshopsIn this paper, we look how changes and challenges in technical debt and its management that could emerge from the ongoing digital transformation. The digital transformation ensues from companies' opportunities to join various digital ecosystems and use ...
Comments
Information & Contributors
Information
Published In
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 10 July 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
FSE '24
Sponsor:
FSE '24: 32nd ACM International Conference on the Foundations of Software Engineering
July 15 - 19, 2024
Porto de Galinhas, Brazil
Acceptance Rates
Overall Acceptance Rate 112 of 543 submissions, 21%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 189Total Downloads
- Downloads (Last 12 months)189
- Downloads (Last 6 weeks)42
Reflects downloads up to 06 Nov 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in