Development in times of hype: How freelancers explore Generative AI?
Article No.: 183, Pages 1 - 13
Abstract
The rise of generative AI has led many companies to hire freelancers to harness its potential. However, this technology presents unique challenges to developers who have not previously engaged with it. Freelancers may find these challenges daunting due to the absence of organizational support and their reliance on positive client feedback. In a study involving 52 freelance developers, we identified multiple challenges associated with developing solutions based on generative AI. Freelancers often struggle with aspects they perceive as unique to generative AI such as unpredictability of its output, the occurrence of hallucinations, and the inconsistent effort required due to trial-and-error prompting cycles. Further, the limitations of specific frameworks, such as token limits and long response times, add to the complexity. Hype-related issues, such as inflated client expectations and a rapidly evolving technological ecosystem, further exacerbate the difficulties. To address these issues, we propose Software Engineering for Generative AI (SE4GenAI) and Hype-Induced Software Engineering (HypeSE) as areas where the software engineering community can provide effective guidance. This support is essential for freelancers working with generative AI and other emerging technologies.
References
[1]
Abhinav, K. and Dubey, A. 2017. Predicting budget for Crowdsourced and Freelance Software development Projects. Proc. Innovations in Software Engineering Conf. (New York, NY, USA, Feb. 2017), 165--171.
[2]
Agile Adoption Report 2022: 2022. https://certiprof.com/pages/certiprof-agile-adoption-report-2022. Accessed: 2023-07-23.
[3]
Arawjo, I., Vaithilingam, P., Wallenberg, M. and Glassman, E. 2023. ChainForge: An open-source visual programming environment for prompt engineering. Adjunct Proc. Annual ACM Symposium on User Interface Software and Technology (New York, NY, USA, Oktober 2023), 1--3.
[4]
Are You a Hype-Driven Developer? 2020. https://betterprogramming.pub/are-you-a-hype-driven-developer-368586c27051. Accessed: 2023-07-23.
[5]
Arjovsky, M., Chintala, S. and Bottou, L. 2017. Wasserstein GAN. arXiv. http://arxiv.org/abs/1701.07875
[6]
Arpteg, A., Brinne, B., Crnkovic-Friis, L. and Bosch, J. 2018. Software Engineering Challenges of Deep Learning. Proc. Euromicro Conf. on Software Engineering and Advanced Applications (Prague, Czech Republic, Aug. 2018), 50--59.
[7]
Ayers, J.W. et al. 2023. Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Internal Medicine. 183, 6 (Jun. 2023), 589--596.
[8]
Bommasani, R. et al. 2022. On the Opportunities and Risks of Foundation Models. arXiv. http://arxiv.org/abs/2108.07258
[9]
Briggs, R.O., Böhmann, T., Schwabe, G. and Tuunanen, T. 2019. Advancing Design Science Research with Solution-based Probing. Proc. Hawaii International Conference on System Sciences (Grans Wailea, Hawaii, Jan. 2019).
[10]
Cambridge Dictionary 2023. hype. Cambridge Dictionary. Cambridge UP.
[11]
Chen, B., Zhang, Z., Langrené, N. and Zhu, S. 2023. Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review. arXiv. http://arxiv.org/abs/2310.14735
[12]
Colomo-Palacios, R. 2020. Cross Fertilization in Software Engineering. Systems, Software and Services Process Improvement (Cham, 2020), 3--13.
[13]
Destefanis, G., Bartolucci, S. and Ortu, M. 2023. A Preliminary Analysis on the Code Generation Capabilities of GPT-3.5 and Bard AI Models for Java Functions. arXiv. http://arxiv.org/abs/2305.09402
[14]
Destefanis, G., Marchesi, M., Ortu, M., Tonelli, R., Bracciali, A. and Hierons, R. 2018. Smart contracts vulnerabilities: a call for blockchain software engineering? Proc. Intl. Workshop on Blockchain Oriented Software Engineering (Campobasso. Italy, Mar. 2018), 19--25.
[15]
Dolata, M. and Crowston, K. 2024. Making sense of AI systems development. IEEE Transactions on Software Engineering. 50, 1 (2024).
[16]
Dolata, M., Crowston, K. and Schwabe, G. 2022. Project Archetypes: A Blessing and a Curse for AI Development. Proc. Intl. Conf. on Information Systems (Dec. 2022).
[17]
Dolata, M., Lange, N. and Schwabe, G. 2023. Supplementary Material: Dolata et al. "Development in times of hype." OSF.
[18]
Dwivedi, Y.K. et al. 2023. Opinion Paper: "So what if ChatGPT wrote it?" Multi-disciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management. 71, (Aug. 2023), 102642.
[19]
Fritzsch, J., Wyrich, M., Bogner, J. and Wagner, S. 2023. Resist the Hype! Practical Recommendations to Cope With Résumé-Driven Development. IEEE Software. (2023), 1--8.
[20]
Fritzsch, J., Wyrich, M., Bogner, J. and Wagner, S. 2021. Résumé-Driven Development: A Definition and Empirical Characterization. Proc. Intl. Conf. on Software Engineering: Software Engineering in Society (May 2021), 19--28.
[21]
Gartner Hype Cycle Research Methodology: 2023. https://www.gartner.com/en/research/methodologies/gartner-hype-cycle. Accessed: 2023-07-23.
[22]
Gayvoronskaya, T. and Meinel, C. 2021. Blockchain: Hype or Innovation. Springer International Publishing.
[23]
Giray, G. 2021. A software engineering perspective on engineering machine learning systems: State of the art and challenges. Journal of Systems and Software. 180, (Oct. 2021), 111031.
[24]
Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S. Courville, A. and Bengio, Y. 2014. Generative Adversarial Networks. arXiv. http://arxiv.org/abs/1406.2661
[25]
Gupta, V., Fernandez-Crehuet, J.M., Gupta, C. and Hanne, T. 2020. Freelancing Models for Fostering Innovation and Problem Solving in Software Startups: An Empirical Comparative Study. Sustainability. 12, 23 (Jan. 2020), 10106.
[26]
Gupta, V., Fernandez-Crehuet, J.M. and Hanne, T. 2020. Freelancers in the Software Development Process: A Systematic Mapping Study. Processes. 8, 10 (Oct. 2020), 1215.
[27]
Gupta, V., Fernandez-Crehuet, J.M., Hanne, T. and Telesko, R. 2020. Fostering product innovations in software startups through freelancer supported requirement engineering. Results in Engineering. 8, (Dec. 2020), 100175.
[28]
Gussek, L., Grabbe, A. and Wiesche, M. 2023. Challenges of IT freelancers on digital labor platforms: A topic model approach. Electronic Markets. 33, 1 (Oct. 2023), 55.
[29]
Gussek, L. and Wiesche, M. 2023. IT Professionals in the Gig Economy. Business & Information Systems Engineering. 65, 5 (Oct. 2023), 555--575.
[30]
Hadi, M.U., Tashi, Q.A., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A. Shaikh, M.B., Akhtar, N., Wu, J. and Mirjalili, S. 2023. Large Language Models: A Comprehensive Survey of its Applications, Challenges, Limitations, and Future Prospects. TechRxiv. https://www.techrxiv.org/users/618307/articles/682263
[31]
Hesenius, M., Schwenzfeier, N., Meyer, O., Koop, W. and Gruhn, V. 2019. Towards a Software Engineering Process for Developing Data-Driven Applications. Proc. Intl. Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (Montreal, QC, Canada, May 2019), 35--41.
[32]
Huang, S., Liu, E.-H., Hui, Z.-W., Tang, S.-Q. and Zhang, and S.-J. 2018. Challenges of Testing Machine Learning Applications. International Journal of Performability Engineering. 14, 6 (Jun. 2018), 1275.
[33]
Hype Driven Development: 2017. https://blog.daftcode.pl/hype-driven-development-3469fc2e9b22. Accessed: 2023-07-23.
[34]
Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R. and McHardy, R. 2023. Challenges and Applications of Large Language Models. arXiv. http://arxiv.org/abs/2307.10169
[35]
Katsiuba, D., Kew, T., Dolata, M., Gurica, M. and Schwabe, G. 2023. Artificially Human: Examining the Potential of Text-generating Technologies in Online Customer Feedback Management. Proc. Intl. Conf. Information Systems (2023).
[36]
Katsiuba, D., Kew, T., Dolata, M. and Schwabe, G. 2022. Supporting Online Customer Feedback Management with Automatic Review Response Generation. Proc. Hawaii International Conference on System Sciences (2022).
[37]
Khomh, F., Adams, B., Cheng, J., Fokaefs, M. and Antoniol, G. 2018. Software Engineering for Machine-Learning Applications: The Road Ahead. IEEE Software. 35, 5 (Sep. 2018), 81--84.
[38]
Kim, M., Zimmermann, T., DeLine, R. and Begel, A. 2018. Data Scientists in Software Teams: State of the Art and Challenges. IEEE Transactions on Software Engineering. 44, 11 (Nov. 2018), 1024--1038.
[39]
LaToza, T.D. and van der Hoek, A. 2016. Crowdsourcing in Software Engineering: Models, Motivations, and Challenges. IEEE Software. 33, 1 (Jan. 2016), 74--80.
[40]
Lorenzoni, G., Alencar, P., Nascimento, N. and Cowan, D. 2021. Machine Learning Model Development from a Software Engineering Perspective: A Systematic Literature Review. arXiv. http://arxiv.org/abs/2102.07574
[41]
Marchesi, M. 2018. Why blockchain is important for software developers, and why software engineering is important for blockchain software (Keynote). Proc. Intl. Workshop on Blockchain Oriented Software Engineering (Campobasso, Italy. Mar. 2018), 1--1.
[42]
Martinez-Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz. A., Vollmer, A.M. and Wagner, S. 2022. Software Engineering for AI-Based Systems: A Survey. ACM Transactions on Software Engineering and Methodology. 31. 2 (Apr. 2022), 37e:1--37e:59.
[43]
Miraz, M.H. and Ali, M. 2020. Blockchain Enabled Smart Contract Based Applications: Deficiencies with the Software Development Life Cycle Models. arXiv. http://arxiv.org/abs/2001.10589
[44]
Nahar, N., Zhang, H., Lewis, G., Zhou, S. and Kästner, C. 2023. A Meta-Summary of Challenges in Building Products with ML Components - Collecting Experiences from 4758+ Practitioners. Proc. Intl. Conf. on AI Engineering - Software Engineering for AI (Melbourne, Australia, Mar. 2023).
[45]
Nascimento, E., Nguyen-Duc, A., Sundbø, I. and Conte, T. 2020. Software engineering for artificial intelligence and machine learning software: A systematic literature review. arXiv. http://arxiv.org/abs/2011.03751
[46]
Nguyen-Duc, A. et al. 2023. Generative Artificial Intelligence for Software Engineering - A Research Agenda. arXiv. http://arxiv.org/abs/2310.18648
[47]
Paleyes, A., Urma, R.-G. and Lawrence, N.D. 2022. Challenges in Deploying Machine Learning: A Survey of Case Studies. ACM Computing Surveys. 55, 6 (Dezember 2022), 114:1--114:29.
[48]
Pei, Z., Liu, L., Wang, C. and Wang, J. 2022. Requirements Engineering for Machine Learning: A Review and Reflection. Proc. Intl. Requirements Engineering Conf. Workshops (Melbourne, Australia, Aug. 2022), 166--175.
[49]
Porru, S., Pinna, A., Marchesi, M. and Tonelli, R. 2017. Blockchain-Oriented Software Engineering: Challenges and New Directions. Proc. Intl. Conf. on Software Engineering: Companion (Buenos Aires, Argentina, May 2017), 169--171.
[50]
Schwartz-Shea, P. and Yanow, D. 2012. Interpretive research design: concepts and processes. Routledge.
[51]
Simmel, G. 2020. Fashion. Fashion Theory. Routledge.
[52]
Stappen, L., Dillmann, J., Striegel, S., Vögel, H.-J., Flores-Herr, N. and Schuller. B.W. 2023. Integrating Generative Artificial Intelligence in Intelligent Vehicle Systems. arXiv. http://arxiv.org/abs/2305.17137
[53]
Steinert, M. and Leifer, L. 2010. Scrutinizing Gartner's hype cycle approach. Proc. Portland Intl. Conf. on Management of Engineering and Technology (Portland Oregon, USA, Jul. 2010), 1--13.
[54]
Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H. and Rush. A.M. 2023. Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models. IEEE Transactions on Visualization and Computer Graphics. 29, 1 (Jan. 2023), 1146--1156.
[55]
Vinsel, L. 2023. Don't Get Distracted by the Hype Around Generative AI. MIT Sloan Management Review. 64, 3 (2023), 1--3.
[56]
Walsham, G. 1995. Interpretive case studies in IS research: nature and method. European Journal of Information Systems. 4, 2 (May 1995), 74--81.
[57]
Wan, Z., Xia, X., Lo, D. and Murphy, G.C. 2021. How does Machine Learning Change Software Development Practices? IEEE Transactions on Software Engineering. 47, 9 (Sep. 2021), 1857--1871.
[58]
Witzel, A. and Reiter, H. 2012. The Problem-Centred Interview: Principles and Practice. SAGE Publications Ltd.
[59]
Wolf, C.T. and Paine, D. 2020. Sensemaking Practices in the Everyday Work of AI/ML Software Engineering. Proc. Intl. Conf. on Software Engineering: Workshops (New York, NY, USA, Sep. 2020), 86--92.
[60]
Wu, W., Heierli, J., Meisterhans, M., Moser, A., Färber, A., Dolata, M., Gavagnin, E., de Spindler, A. and Schwabe, G. 2023. PROMISE: A Framework for Model-Driven Stateful Prompt Orchestration. arXiv. http://arxiv.org/abs/2312.03699
[61]
Xu, J., Cui, Z., Zhao, Y., Zhang, X., He, S., He, P., Li, L., Kang, Y., Lin, Q., Dang, Y., Rajmohan, S. and Zhang, D. 2024. UniLog: Automatic Logging via LLM and In-Context Learning. Proc. Intl. Conf. Software Engineering (New York, NY, US. Apr. 2024), 129--140.
[62]
Yu, H., Shen, B., Ran, D., Zhang, J., Zhang, Q., Ma, Y., Liang, G., Li, Y., Wang, Q. and Xie, T. 2024. CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models. Proc. Intl. Conf. Software Engineering (New York. NY, US, Apr. 2024), 417--428.
[63]
Zamfirescu-Pereira, J.D., Wong, R.Y., Hartmann, B. and Yang, Q. 2023. Why Johnny Can't Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. Proc. Conf. on Human Factors in Computing Systems (New York, NY. USA, Apr. 2023), 1--21.
[64]
Zhang, C. et al. 2023. A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need? arXiv. http://arxiv.org/abs/2303.11717
Index Terms
- Development in times of hype: How freelancers explore Generative AI?
Recommendations
The Short-Term Effects of Generative Artificial Intelligence on Employment: Evidence from an Online Labor Market
Generative artificial intelligence (AI) holds the potential to either complement workers by enhancing their productivity or substitute them. We examine the short-term effects of the recently released generative AI models (ChatGPT, DALL-E 2, and Midjourney)...
Impact of Generative AI Technologies on Software Development Professionals' Perceptions of Job Security
SBQS '24: Proceedings of the XXIII Brazilian Symposium on Software QualityCONTEXT: The rapid evolution of generative AI technologies has sparked significant interest in the software development sector. These technologies promise to automate repetitive tasks, improve code quality, and assist in various managerial and technical ...
How Do Information Technology Professionals Use Generative Artificial Intelligence?
SBSI '24: Proceedings of the 20th Brazilian Symposium on Information SystemsContext: The emergence of generative Artificial Intelligence (AI) and, more recently, the dissemination of Copilot, ChatGPT-3 and similar tools have broadened the discussion about the possibility of using generative AI tools in many professional segments ...
Comments
Information & Contributors
Information
Published In
May 2024
2942 pages
ISBN:9798400702174
DOI:10.1145/3597503
- Co-chairs:
- Ana Paiva,
- Rui Abreu,
- Program Co-chairs:
- Abhik Roychoudhury,
- Margaret Storey
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
In-Cooperation
- Faculty of Engineering of University of Porto
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 12 April 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
ICSE '24
Sponsor:
ICSE '24: IEEE/ACM 46th International Conference on Software Engineering
April 14 - 20, 2024
Lisbon, Portugal
Acceptance Rates
Overall Acceptance Rate 276 of 1,856 submissions, 15%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 566Total Downloads
- Downloads (Last 12 months)566
- Downloads (Last 6 weeks)83
Reflects downloads up to 18 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in