skip to main content
research-article

Are You Still Working on This? An Empirical Study on Pull Request Abandonment

Published: 01 June 2022 Publication History

Abstract

The great success of numerous community-based open source software (OSS) is based on volunteers continuously submitting contributions, but ensuring sustainability is a persistent challenge in OSS communities. Although the motivations behind and barriers to OSS contributors’ joining and retention have been extensively studied, the impacts of, reasons for and solutions to contribution abandonment at the individual level have not been well studied, especially for pull-based development. To bridge this gap, we present an empirical study on pull request abandonment based on a sizable dataset. We manually examine 321 abandoned pull requests on GitHub and then quantify the manual observations by surveying 710 OSS developers. We find that while the lack of integrators’ responsiveness and the lack of contributors’ time and interest remain the main reasons that deter contributors from participation, limitations during the processes of patch updating and consensus reaching can also cause abandonment. We also show the significant impacts of pull request abandonment on project management and maintenance. Moreover, we elucidate the strategies used by project integrators to cope with abandoned pull requests and highlight the need for a practical handover mechanism. We discuss the actionable suggestions and implications for OSS practitioners and tool builders, which can help to upgrade the infrastructure and optimize the mechanisms of OSS communities.

References

[1]
About checkstyle, 2020. Accessed: Jul.6, 2020. [Online]. Available: https://github.com/checkstyle/checkstyle
[2]
About github API, 2020. Accessed: Jul.6, 2020. [Online]. Available: http://developer.github.com/v3/
[3]
About slack, 2020. Accessed: Dec.6, 2020. [Online]. Available: https://www.slack.com/
[4]
About surveymonkey, 2020. Accessed: Jul.6, 2020. [Online]. Available: https://www.surveymonkey.com/mp/aboutus
[7]
A. Bacchelli and C. Bird, “Expectations, outcomes, and challenges of modern code review,” in Proc. 35th Int. Conf. Softw. Eng., 2013, pp. 712–721.
[8]
A. Begel and T. Zimmermann, “Analyze this! 145 questions for data scientists in software engineering,” in Proc. 36th Int. Conf. Softw. Eng., 2014, pp. 12–23.
[9]
A. Bonaccorsi and C. Rossi, “Comparing motivations of individual programmers and firms to take part in the open source movement: From community to business,” Knowl. Technol. Policy, vol. 18, no. 4, pp. 40–64, 2006.
[10]
H. Borges, A. Hora, and M. T. Valente, “Understanding the factors that impact the popularity of github repositories,” in Proc. IEEE Int. Conf. Softw. Maintenance Evol., 2016, pp. 334–344.
[11]
J. P. Borst, N. A. Taatgen, and H. van Rijn, “What makes interruptions disruptive?: A process-model account of the effects of the problem state bottleneck on task interruption and resumption,” in Proc. 33rd Annu. ACM Conf. Human Factors Comput. Syst., 2015, pp. 2971–2980.
[12]
A. Bosu, J. C. Carver, C. Bird, J. Orbeck, and C. Chockley, “Process aspects and social dynamics of contemporary code review: Insights from open source development and industrial practice at microsoft,” IEEE Trans. Softw. Eng., vol. 43, no. 1, pp. 56–75, Jan. 2016.
[13]
A. Bosu, J. C. Carver, C. Bird, J. Orbeck, and C. Chockley, “Process aspects and social dynamics of contemporary code review: Insights from open source development and industrial practice at microsoft,” IEEE Trans. Softw. Eng., vol. 43, no. 1, pp. 56–75, Jan. 2017.
[14]
J. L. Campbell, C. Quincy, J. Osserman, and O. K. Pedersen, “Coding in-depth semistructured interviews: Problems of unitization and intercoder reliability and agreement,” Sociol. Methods Res., vol. 42, no. 3, pp. 294–320, 2013.
[15]
G. Canfora, M. Di Penta, R. Oliveto, and S. Panichella, “Who is going to mentor newcomers in open source projects?,” in Proc. ACM SIGSOFT 20th Int. Symp. Foundations Softw. Eng., 2012, pp. 1–11.
[16]
K. Crowston and J. Howison, “The social structure of free and open source software development,” First Monday, vol. 10, no. 2, pp. 1–27, 2005.
[17]
K. Crowston, K. Wei, Q. Li, and J. Howison, “Core and periphery in free/libre and open source software team communications,” in Proc. 39th Annu. Hawaii Inte. Conf. Syst. Sci., 2006, pp. 118a–118a.
[18]
L. Dabbish, C. Stuart, J. Tsay, and J. Herbsleb, “Social coding in github: Transparency and collaboration in an open software repository,” in Proc. ACM Conf. Comput. Supported Cooperative Work, 2012, pp. 1277–1286.
[19]
F. Fagerholm, A. S. Guinea, J. Münch, and J. Borenstein, “The role of mentoring and project characteristics for onboarding in open source software projects,” in Proc. 8th ACM/IEEE Int. Symp. Empir. Softw. Eng. Meas., 2014, pp. 1–10.
[20]
M. Foucault, M. Palyart, X. Blanc, G. C. Murphy, and J.-R. Falleri, “Impact of developer turnover on quality in open-source software,” in Proc. 10th Joint Meeting Foundations Softw. Eng., 2015, pp. 829–841.
[21]
J. Gamalielsson and B. Lundell, “Sustainability of open source software communities beyond a fork: How and why has the libreoffice project evolved?,” J. Syst. Softw., vol. 89, pp. 128–145, 2014.
[22]
G. Gousios, “The ghtorent dataset and tool suite,” in Proc. 10th Working Conf. Mining Softw. Repositories, 2013, pp. 233–236.
[23]
G. Gousios, M. Pinzger, and A. Van Deursen, “An exploratory study of the pull-based software development model,” in Proc. 36th Int. Conf. Softw. Eng., 2014, pp. 345–355.
[24]
G. Gousios, M.-A. Storey, and A. Bacchelli, “Work practices and challenges in pull-based development: the contributor’s perspective,” in Proc. 38th Int. Conf. Softw. Eng., 2016, pp. 285–296.
[25]
G. Gousios, A. Zaidman, M.-A. Storey, and A. Van Deursen, “Work practices and challenges in pull-based development: the integrator’s perspective,” in Proc. 37th Int. Conf. Softw. Eng., 2015, pp. 358–368.
[26]
R. M. Groves, F. J. Fowler Jr, M. P. Couper, J. M Lepkowski, E. Singer, and R. Tourangeau, Survey Methodol., vol. 561. Hoboken, NJ, USA: Wiley2011.
[27]
D. C. Gumm, “Distribution dimensions in software development projects: A taxonomy,” IEEE Softw., vol. 23, no. 5, pp. 45–51, Sep./Oct. 2006.
[28]
A. Guzzi, A. Bacchelli, M. Lanza, M. Pinzger, and A. Van Deursen, “Communication in open source software development mailing lists,” in Proc. 10th IEEE Working Conf. Mining Softw. Repositories, 2013, pp. 277–286.
[29]
C. Hannebauer, M. Book, and V. Gruhn, “An exploratory study of contribution barriers experienced by newcomers to open source software projects,” in Proc. 1st Int. Workshop CrowdSourcing Softw. Eng., 2014, pp. 11–14.
[30]
A. Hars and S. Qu, “Working for free? motivations for participating in open-source projects,” Int. J. Electron. Commerce, vol. 6, no. 3, pp. 25–39, 2002.
[31]
M. Hilton, N. Nelson, T. Tunnell, D. Marinov, and D. Dig, “Trade-offs in continuous integration: assurance, security, and flexibility,” in Proc. 11th Joint Meeting Foundations Softw. Eng., 2017, pp. 197–207.
[32]
W. Huang, T. Lu, H. Zhu, G. Li, and N. Gu, “Effectiveness of conflict management strategies in peer review process of online collaboration projects,” in Proc. 19th ACM Conf. Comput.-Supported Cooperative Work Social Comput., 2016, pp. 717–728.
[33]
G. Iaffaldano, I. Steinmacher, F. Calefato, M. Gerosa, and F. Lanubile, “Why do developers take breaks from contributing to oss projects?: A preliminary analysis,” in Proc. 2nd Int. Workshop Softw. Health, 2019, pp. 9–16.
[34]
R. N. Iyer, S. A. Yun, M. Nagappan, and J. Hoey, “Effects of personality traits on pull request acceptance,” IEEE Trans. Softw. Eng., early access, Dec. 17, 2019.
[35]
J. L. Cánovas Izquierdo, V. Cosentino, B. Rolandi, A. Bergel, and J. Cabot, “Gila: Github label analyzer,” in Proc. 22nd Int. Conf. Softw. Anal. Evol. Reeng., 2015, pp. 479–483.
[36]
D. Izquierdo-Cortazar, G. Robles, F. Ortega, and J. M. Gonzalez-Barahona, “Using software archaeology to measure knowledge loss in software projects due to developer turnover,” in Proc. 42nd Hawaii Int. Conf. Syst. Sci., 2009, pp. 1–10.
[37]
C. Jergensen, A. Sarma, and P. Wagstrom, “The onion patch: Migration in open source ecosystems,” in Proc. 19th ACM SIGSOFT Symp., 13th Eur. Conf. Foundations Softw. Eng., 2011, pp. 70–80.
[38]
D. Kavaler, P. Devanbu, and V. Filkov, “Whom are you going to call? determinants of @-mentions in github discussions,” Empir. Softw. Eng., vol. 24, pp. 3904–3932, 2019.
[39]
O. Kononenko, T. Rose, O. Baysal, M. Godfrey, D. Theisen, and B. De Water, “Studying pull request merges: a case study of shopify’s active merchant,” in Proc. 40th Int. Conf. Softw. Eng.: Softw. Eng. Practice, 2018, pp. 124–133.
[40]
K. R. Lakhani and R. G. Wolf, “Why hackers do what they do: Understanding motivation and effort in free/open source software projects,” Perspectives Free Open Source Softw., pp. 3–22, 2005.
[41]
A. Lee, J. C. Carver, and A. Bosu, “Understanding the impressions, motivations, and barriers of one time code contributors to floss projects: A survey,” in Proc. 39th Int. Conf. Softw. Eng., 2017, pp. 187–197.
[42]
Z. Liet al., “Redundancy, context, and preference: An empirical study of duplicate pull requests in OSS projects,” IEEE Trans. Softw. Eng., early access, Aug. 24, 2020.
[43]
B. Lin, G. Robles, and A. Serebrenik, “Developer turnover in global, industrial open source projects: Insights from applying survival analysis,” in Proc. IEEE 12th Int. Conf. Global Softw. Eng., 2017, pp. 66–75.
[44]
W. Ma, L. Chen, X. Zhang, Y. Zhou, and B. Xu, “How do developers fix cross-project correlated bugs? A case study on the github scientific python ecosystem,” in Proc. 39th Int. Conf. Softw. Eng., 2017, pp. 381–392.
[45]
I. Manotaset al., “An empirical study of practitioners’ perspectives on green software engineering,” in Proc. IEEE/ACM 38th Int. Conf. Softw. Eng., 2016, pp. 237–248.
[46]
J. Marlow, L. Dabbish, and J. Herbsleb, “Impression formation in online peer production: Activity traces and personal profiles in github,” in Proc. Conf. Comput. Supported Cooperative Work, 2013, pp. 117–128.
[47]
C. Miller, David G. Widder, C. Kästner, and B. Vasilescu, “Why do people give up flossing? A study of contributor disengagement in open source,” in Proc. Int. Conf. Open Source Syst., 2019, pp. 116–129.
[48]
C. Overney, J. Meinicke, C. Kästner, and B. Vasilescu, “How to not get rich: An empirical study of donations in open source,” in Proc. Int. Conf. Softw. Eng., 2020, pp. 1209–1221.
[49]
F. Palombaet al., “Recommending and localizing change requests for mobile apps based on user reviews,” in Proc. IEEE/ACM 39th Int. Conf. Softw. Eng., 2017, pp. 106–117.
[50]
C. N. Parkinson, Parkinson’s Law: The Pursuit of Progress. London,U.K.: Readers Union [in association with] John Murray, 1959.
[51]
G. Pinto, L. F. Dias, and I. Steinmacher, “Who gets a patch accepted first? Comparing the contributions of employees and volunteers,” in Proc. IEEE/ACM 11th Int. Workshop Cooperative Human Aspects Softw. Eng., 2018, pp. 110–113.
[52]
G. Pinto, I. Steinmacher, and M. A. Gerosa, “More common than you think: An in-depth study of casual contributors,” in Proc. 23rd Int. Conf. Softw. Anal. Evol. Reengineering, 2016, pp. 112–123.
[53]
T. Punter, M. Ciolkowski, B. Freimut, and I. John, “Conducting on-line surveys in software engineering,” in Proc. Int. Symp. Empir. Softw. Eng., 2003, pp. 80–88.
[54]
H. S. Qiu, A. Nolte, A. Brown, A. Serebrenik, and B. Vasilescu, “Going farther together: The impact of social capital on sustained participation in open source,” in Proc. IEEE/ACM 41st Int. Conf. Softw. Eng., 2019, pp. 688–699.
[55]
A. Rastogi, “Do biases related to geographical location influence work-related decisions in github?,” in Proc. IEEE/ACM 38th Int. Conf. Softw. Eng. Companion, 2016, pp. 665–667.
[56]
A. Rastogi, N. Nagappan, G. Gousios, and A. van der Hoek, “Relationship between geographical location and evaluation of developer contributions in github,” in Proc. 12th ACM/IEEE Int. Symp. Empir. Softw. Eng. Meas., 2018, pp. 1–8.
[57]
E. Raymond, “The cathedral and the bazaar,” Knowl. Technol. Policy, vol. 12, no. 3, pp. 23–49, 1999.
[58]
J. A. Roberts, I.-H. Hann, and S. A. Slaughter, “Understanding the motivations, participation, and performance of open source software developers: A longitudinal study of the apache projects,” Manage. Sci., vol. 52, no. 7, pp. 984–999, 2006.
[59]
A. Schilling, S. Laumer, and T. Weitzel, “Who will remain? an evaluation of actual person-job and person-team fit to predict developer retention in floss projects,” in Proc. 45th Hawaii Int. Conf. Syst. Sci., 2012, pp. 3446–3455.
[60]
E. Shihab, Z. M. Jiang, and A. E. Hassan, “Studying the use of developer IRC meetings in open source projects,” in Proc. IEEE Int. Conf. Softw. Maintenance, 2009, pp. 147–156.
[61]
L. Singer, F. F. Filho, B. Cleary, C. Treude, M.-A. Storey, and K. Schneider, “Mutual assessment in the social programmer ecosystem: An empirical investigation of developer profile aggregators,” in Proc. Conf. Comput. Supported Cooperative Work, 2013, pp. 103–116.
[62]
D. Spencer, Card Sorting: Designing Usable Categories. New York, NY, USA: Rosenfeld Media, 2009.
[63]
I. Steinmacher, T. Conte, M. A. Gerosa, and D. Redmiles, “Social barriers faced by newcomers placing their first contribution in open source software projects,” in Proc. 18th ACM Conf. Comput. Supported Cooperative Work Soc. Comput., 2015, pp. 1379–1392.
[64]
I. Steinmacher, T. Uchoa Conte, C. Treude, and M. A. Gerosa, “Overcoming open source project entry barriers with a portal for newcomers,” in Proc. 38th Int. Conf. Softw. Eng., 2016, pp. 273–284.
[65]
I. Steinmacher, G. Pinto, I. S. Wiese, and M. A. Gerosa, “Almost there: A study on quasi-contributors in open-source software projects,” in Proc. 40th Int. Conf. Softw. Eng., 2018, pp. 256–266.
[66]
I. Steinmacher, I. Wiese, A. P. Chaves, and M. A. Gerosa, “Why do newcomers abandon open source software projects?” in Proc. 6th Int. Workshop Cooperative Human Aspects Softw. Eng., 2013, pp. 25–32.
[67]
J. Terrellet al., “Gender differences and bias in open source: Pull request acceptance of women versus men,” PeerJ Comput. Sci., vol. 3, 2017, Art. no.
[68]
J. Tsay, L. Dabbish, and J. Herbsleb, “Influence of social and technical factors for evaluating contribution in github,” in Proc. 36th Int. Conf. Softw. Eng., 2014, pp. 356–366.
[69]
E. Van Der Veen, G. Gousios, and A. Zaidman, “Automatically prioritizing pull requests,” in Proc. 12th Working Conf. Mining Softw. Repositories, 2015, pp. 357–361.
[70]
B. Vasilescuet al., “The sky is not the limit: multitasking across github projects,” in Proc. 38th Int. Conf. Softw. Eng., 2016, pp. 994–1005.
[71]
B. Vasilescu, Y. Yu, H. Wang, P. Devanbu, and V. Filkov, “Quality and productivity outcomes relating to continuous integration in github,” in Proc. 10th Joint Meeting Foundations Softw. Eng., 2015, pp. 805–816.
[72]
M. Wesselet al., “The power of bots: Characterizing and understanding bots in OSS projects,” in Proc. ACM Human-Comput. Interaction, 2018, pp. 1–19.
[73]
M. J. Wolf, K. Bowyer, D. Gotterbarn, and K. Miller, “Open source software: intellectual challenges to the status QUO,” ACM SIGCSE Bulletin, vol. 34, no. 1, pp. 317–318, 2002.
[74]
V. Wolff-Marting, C. Hannebauer, and V. Gruhn, “Patterns for tearing down contribution barriers to floss projects,” in Proc. IEEE 12th Int. Conf. Intell. Softw. Methodologies Tools Techn., 2013, pp. 9–14.
[75]
Y. Ye and K. Kishida, “Toward an understanding of the motivation open source software developers,” in Proc. 25th Int. Conf. Softw. Eng., 2003, pp. 419–429.
[76]
Y. Yu, H. Wang, V. Filkov, P. Devanbu, and B. Vasilescu, “Wait for it: Determinants of pull request evaluation latency on github,” in Proc. IEEE/ACM 12th Working Conf. Mining Softw. Repositories, 2015, pp. 367–371.
[77]
Y. Yu, H. Wang, G. Yin, and T. Wang, “Reviewer recommendation for pull-requests in github,” Inf. Softw. Technol., vol. 74, pp. 204–218, 2016.
[78]
M. Zhou and A. Mockus, “What make long term contributors: Willingness and opportunity in OSS community,” in Proc. 34th Int. Conf. Softw. Eng., 2012, pp. 518–528.
[79]
S. Zhou, B. Vasilescu, and C. Kästner, “What the fork: A study of inefficient and efficient forking practices in social coding,” in Proc. 27th ACM Joint Meeting Eur. Softw. Eng. Conf. Symp. Foundations Softw. Eng., 2019, pp. 350–361.
[80]
J. Zhu, M. Zhou, and A. Mockus, “Effectiveness of code contribution: From patch-based to pull-request-based tools,” in Proc. 24th ACM SIGSOFT Int. Symp. Foundations Softw. Eng., 2016, pp. 871–882.
[81]
T. Zimmermann, “Card-sorting: From text to themes,” in Perspectives on Data Science for Software Engineering. Amsterdam, The Netherlands: Elsevier, 2016, pp. 137–141.
[82]
W. Zouet al., “Smart contract development: Challenges and opportunities,” IEEE Trans. Softw. Eng., early access, Sep. 24, 2019.
[83]
W. Zou, J. Xuan, X. Xie, Z. Chen, and B. Xu, “How does code style inconsistency affect pull request integration? An exploratory study on 117 github projects,” Empir. Softw. Eng., vol. 24, no. 6, pp. 3871–3903, 2019.

Cited By

View all
  • (2024)GPP: A Graph-Powered Prioritizer for Code Review RequestsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3694990(104-116)Online publication date: 27-Oct-2024
  • (2024)From the Inside Out: Organizational Impact on Open-Source Communities and Women's RepresentationProceedings of the 2024 IEEE/ACM 17th International Conference on Cooperative and Human Aspects of Software Engineering10.1145/3641822.3641875(36-50)Online publication date: 14-Apr-2024
  • (2024)Mining Pull Requests to Detect Process Anomalies in Open Source Software DevelopmentProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639196(1-13)Online publication date: 20-May-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Software Engineering
IEEE Transactions on Software Engineering  Volume 48, Issue 6
June 2022
355 pages

Publisher

IEEE Press

Publication History

Published: 01 June 2022

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 25 Dec 2024

Other Metrics

Citations

Cited By

View all

View Options

View options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media