research-article

Open access

A Disruptive Research Playbook for Studying Disruptive Innovations

Authors:

Margaret-Anne Storey,

Daniel Russo,

Nicole Novielli,

Takashi Kobayashi,

Dong WangAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology, Volume 33, Issue 8

Article No.: 195, Pages 1 - 29

https://doi.org/10.1145/3678172

Published: 19 November 2024 Publication History

PDF eReader

Abstract

As researchers today, we are witnessing a fundamental change in our technologically-enabled world due to the advent and diffusion of highly disruptive technologies such as generative Artificial Intelligence (AI), Augmented Reality (AR) and Virtual Reality (VR). In particular, software engineering has been profoundly affected by the transformative power of disruptive innovations for decades, with a significant impact of technical advancements on social dynamics due to its socio-technical nature. In this article, we reflect on the importance of formulating and addressing research problems in software engineering through a socio-technical lens, thus ensuring a holistic understanding of the complex phenomena in this field. We propose a research playbook with the aim of providing a guide to formulate compelling and socially relevant research questions and to identify the appropriate research strategies for empirical investigations, with an eye on the long-term implications of technologies or their use. We showcase how to apply the research playbook. Firstly, we show how it can be used retrospectively to reflect on a prior disruptive technology, Stack Overflow, and its impact on software development. Secondly, we show how it can be used to question the impact of two current disruptive technologies: AI and AR/VR. Finally, we introduce a specialized GPT model to support the researcher in framing future investigations. We conclude by discussing the broader implications of adopting the playbook for both researchers and practitioners in software engineering and beyond.

1 Introduction

“Disruptive technologies aren’t disruptive by themselves, it is how people use them that makes them disruptive.”
—Anonymous

As software engineering researchers, our passion lies in understanding and developing theories about software engineering and crafting software engineering solutions. Yet, in our pursuit of scientific and technological excellence, we occasionally lose sight of the broader purpose: the developers, the end users of software, and the societal impact our work is meant to serve. This oversight is not just a personal lapse but a professional one because our role as scientists demands that we create value for society at large. It is easy to fall into the trap of assuming that the research questions that fascinate us will inherently fascinate and serve others. However, this assumption can often lead us astray.

Recognizing this crucial gap, we have crafted a provocative playbook, which represents the main contribution of this article and is designed to offer strategic guidance to those among us who struggle to identify compelling and socially relevant research questions in software engineering. The playbook, we hope, will provoke researchers to pause and take a step back to reflect on what other questions they could be answering through their research and what the impact of those answers may be on society at large. We propose a playbook rather than a framework as it is a more actionable approach to broadening the research questions we ask. In addition to offering this research playbook to guide the formulation of compelling research questions, we also developed a specialized GPT model called “MyResearchPlaybook” to serve as an adjunct tool to assist in the brainstorming process that the playbook invokes. Details on this supporting GPT model can be found in the Appendix.

We justify the need for such a playbook by tracing the arc of human history. This arc reveals a continuum of technological innovation, from the earliest tools enhancing physical capabilities to today’s sophisticated digital systems that amplify our intellectual prowess. This relentless pursuit of advancement has been a hallmark of human civilization [51]. Each technological breakthrough has brought us forward as a society—from primitive hunting tools to the development of language, writing systems, and beyond—and has fundamentally altered how we work, communicate, and create. These leaps not only showcase our ingenuity but also challenge us to reflect on the role and impact of our creations within the larger societal context. Marshall McLuhan, a visionary scholar active in the 1960s, astutely observed that disruptive technologies follow intrinsic laws: they amplify certain human abilities, render previous technologies obsolete, and, when pushed to their extremes, create a need for new innovations to address emergent challenges [46]. McLuhan’s insights compel us to reflect not only on the immediate benefits of new technologies but also to anticipate their long-term and potentially unforeseen impacts. Consider the automobile: while it has revolutionized transportation, its excessive use has led to traffic congestion and significant environmental degradation, necessitating further innovation and intervention.

Software engineering has not been immune to the transformative power of disruptive innovations since it is fundamentally socio-technical, intertwining technical advancements and social dynamics. Despite rapid progress through innovations like high-level programming languages and social coding tools, the integration of tools such as static analysis and automated debugging remains a challenge. Parnin and Orso’s study highlighted that the efficacy of these tools is contingent not only on their technical capabilities but also (and maybe more importantly) on their adoption and use by developers in real-world scenarios [57]. This underscores the necessity for a holistic approach in software engineering, ensuring that technical innovations are seamlessly aligned with and supportive of the human elements of software development.

Generative Artificial Intelligence (AI) technology, along with Augmented Reality (AR) and Virtual Reality (VR), exemplifies the current wave of disruptive technologies, showing the potential to revolutionize the field of software engineering. These innovations are not merely incremental improvements; they are catalysts for fundamental change, challenging existing paradigms and creating new opportunities for advancement and understanding. The integration of AI is transforming various aspects of software engineering, automating routine tasks, and injecting creativity into software design and development, while simultaneously lowering the barrier to software authorship. Concurrently, AR and VR are emerging as potential disruptors, with technologies such as the Metaverse [84] also anticipated to unlock opportunities to enhance our understanding of complex dependencies. These technologies play a crucial role in improving collaborative efforts during development and bolstering educational support, as highlighted by Krause [39] and Fernandes [28].

While AI, AR and VR are the disruptors of today that shape the current and future landscape of software engineering, it is important to note that they are examples in a long line of innovations. To navigate the ever-evolving landscape of software engineering in our research, we can not overlook the long-term implications of technologies or their use. Thus, the research playbook we propose is intentionally designed to be provocative, versatile, and technology-agnostic. This ensures that it retains its relevance and applicability, providing a reliable guide through the ongoing waves of innovation and disruption in software engineering.

Given the sweeping changes brought about by disruptive technologies, it is imperative for research within this domain to adopt a socio-technical lens, as the lessons of Parnin and Orso show us, ensuring a holistic understanding that encompasses both the technological innovations and the social contexts in which they operate. This approach goes beyond merely selecting the right research methodologies or adhering to specific methods, as guided by established resources in software engineering [64, 82]. While these guides are invaluable for ensuring methodological rigor, they fall short in assisting researchers to formulate critical questions that delve into the broader and long-term impacts on both technical and socio-technical phenomena in the field.

When studying disruptive technologies, traditional empirical approaches in software engineering research often fall short. Disruptive innovations like AI, AR and VR fundamentally challenge existing norms, practices and assumptions in the field, demanding a reevaluation of established research paradigms. These technologies reshape the research landscape by introducing novel phenomena and creating new research questions that traditional methods may not address comprehensively. Our playbook aims to fill this gap by providing a systematic framework for researchers to rethink their research questions and methods when confronted with such game-changing technologies. It promotes a self-reflective mindset that is crucial for aligning research with the rapid and often unpredictable advancements brought about by these disruptive technologies.

Here, we underscore the importance of integrating a socio-technical perspective in all studies within this domain. This integration is crucial not only for harnessing the full potential of disruptive technologies but also for mitigating potential negative impacts and fostering a sustainable and inclusive technological future. The emphasis here is on encouraging researchers to adopt a holistic perspective that encompasses not just research methods but also a critical examination of the underlying questions asked and the wider phenomena impacted by disruptive technologies in software engineering. This comprehensive approach ensures a balanced and thorough understanding of the subject matter and its socio-technical implications.

Our article begins by defining what constitutes a “disruptive technology,” followed by an overview of key disruptive technologies that have significantly impacted software engineering (Section 2). Our research playbook is then presented in detail (Section 3), serving as a tool to evaluate innovative technologies in software engineering. As we present the playbook, we illustrate how the playbook can be used to retrospectively reflect on a prior disruptive technology, Stack Overflow, and its impact on software engineering. This retrospective application of the playbook revealed to us how relatively little research emphasis there has been on studying social and human aspects in the Stack Overflow related research. To demonstrate practical, forward-looking applications of the research playbook, we explore two specific disruptive technologies in software engineering that are still emerging: generative AI and AR/VR. Specific example scenarios are provided to illustrate how the playbook can be employed within these domains (Section 4). In Section 4, we also introduce “MyResearchPlaybook,” a specialized GPT model to assist in the brainstorming activity the playbook invokes, whose usage we demonstrate for the second scenario. We conclude the article by discussing the broader implications of adopting the playbook, highlighting its potential benefits for both researchers and practitioners (Section 5).

2 Disruptive Software Engineering Technologies Disrupt Research

Frequent instances of technological leaps and creative breakthroughs have a history of shattering existing stable environments, elevating performance across various dimensions, and profoundly reshaping entire societies. Christensen et al. [13] termed these profound shifts “disruptive technologies.” This concept was then expanded to encompass innovations not only in technology but also in products, processes, and business models, known as “disruptive innovation” [20].

Christensen’s original definition of disruptive innovation, which focuses on the competitive dynamics in economic activity, contains two meanings. The first is the disruption of economic market structures, and the second refers to the disruption experienced by incumbent firms. Different from sustainable innovations that are predictable in how they introduce incremental changes in existing business methods, disruptive innovations introduce significant changes in the habits of customers, thus leading to changes in processes and business models that are not predictable in their emergence and evolution. Usually, such disruptive innovations emerge from a group of customers whose needs appear not fully addressed by mainstream solutions. The quality of the innovative product or service then improves over time, thus reaching the mainstream market and introducing major changes in the daily habits of people. Christensen further observed that normally successful and stable businesses fail not at the introduction of sustainable innovations as they are well positioned to stay on top of these changes technology and market wise, but rather fail in the face of disruptive innovations that lead to cheaper new products and shifts in new emerging markets [13]. Established companies may be slow to recognize shifts in market needs, may face inertia to change and fail to recognize the potential of the new innovations, and consequently stick with formerly effective processes and tools that are no longer successful in the face of disruptive change.

The Internet can be seen as an example of a disruptive technology. Originally designed to address the needs of research and military institutions for robust networking technology for collaboration purposes [43], the Internet evolved over time and eventually had a profound impact on individual and collective aspects of our society. It dramatically reshaped the way we communicate with each other while enhancing our ability to access and share information and enabling the creation of new business models. Companies that embraced the Internet (e.g., the case of eCommerce) were more successful than those that did not.

This conceptualization of disruptive innovation, originally conceived to model market processes, also holds true for the software engineering domain. In software engineering, instances of such transformative breakthroughs are frequent. For instance, the introduction of Integrated Development Environments (IDEs) revolutionized the interaction between developer and development support tools. They reduced the friction of using different tools, and they standardized processes and tools across teams. IDE plug-ins replaced the need for standalone tools like Computer-Aided Software Engineering tools, which enabled simplified environment setup and tool integration. Social knowledge-sharing platforms like Stack Overflow disrupted the barrier of collaboration and drastically changed the use of traditional documentation, such as FAQs and manuals, and promoted collaborative learning and problem-solving.

In the contemporary context, the advances achieved by large-scale language modeling (LLM) and AR/VR through deep learning, as well as dramatic advances in computing power are undeniably remarkable. To further enhance the disruptive potential of such technologies, we note that they are accessible to a wider audience compared to technologies introduced in the past, whose benefits were thanks to interaction paradigms such as the use of natural language for LLM-based tools (e.g., Copilot).

We find ourselves in a dynamic phase of disruptive innovation that has the potential to reshape how we interact with technology and experience reality. Just as formerly successful businesses may fail in the face of disruptive rather than sustainable innovations, researchers also need to consider if they too may suffer from inertia and too much faith in existing approaches for doing research when faced with disruptive innovations. Disruptive innovations change not only the market but also the value propositions that need to be understood. In response to this observation, our article offers a comprehensive and action-inspiring research playbook that encourages researchers to reflect on the socio-technical phenomena affected by disruptions, such as AI and AR/VR adoption, and the essence of the research questions they should be asking (based on the ideas of how these phenomena may be impacted). We further leverage the disruption of generative AI on the research itself by designing a specialized GPT model, called “MyResearchPlaybook.” That is, not only do we anticipate that generative AI is disrupting software engineering, but it may also directly disrupt how we do research and disrupt the very questions that we pose in our work.

3 The Design of a Disruptive Research Playbook

We designed and developed a research playbook to guide and assist in the evaluation and prediction of how disruptive innovations may impact software engineering. Frameworks to guide software engineering research have been proposed before [24, 71, 82], but these focused on selecting and using research methods with little attention to the importance of selecting a pertinent research question. Our playbook, in contrast, emphasizes a step-by-step process, starting with the selection of research problems to be considered, followed by which socio-technical phenomena may be affected by the introduction of new technology and which ideas should be considered. That is, it emphasizes the construction and selection of research questions. Lastly, the playbook guides a reflection on the selection of research strategies to use while studying the potentially disruptive impacts of new technology, with a focus on human and social aspects. Please note that our goal is not to define constraints on what research questions are worthy of investigation. We aim to provoke and increase creativity through the playbook and the step-by-step process we propose.

The playbook is inspired by two research frameworks, McLuhan’s framework which describes four laws that all new innovations have been observed to follow [46], and McGrath’s methodological framework for conducting research in the behavioural sciences [45]. Throughout, we use a former disruptive technology in software engineering, Stack Overflow—which disrupted how developers share, curate, and learn knowledge—to illustrate how the playbook can be used not just to question the impact of new technologies but also to reflect on the impact and research that has been done on prior disruptive technologies.

3.1 Framing the Research Landscape with McLuhan’s Tetradic Dimensions

When conducting research, the importance of articulating and crafting one’s guiding research goals and questions can never be overstated [10, 24]. At the heart of any empirical research study or evaluation is the phenomenon or phenomena to be studied and the ideas we have about them, while the research questions become the navigational compass points that are considered in the research study.

Marshall McLuhan recognised some decades ago the importance of choosing probing questions to ask about new media [46] and their impact on humans and society. By media, McLuhan refers to any technology that augments or enhances a human or their capabilities. For example, he considered a car, a poem, a pen, and clothing all as forms of media.¹ McLuhan recognized from his studies of media that any technology, once it is put to use, exhibits the same four laws [47]. These state that a new disruptive innovation will enhance or accelerate some aspect of a human’s capabilities, it will retrieve some characteristics or affordances from older innovations, it will make obsolete some existing innovations, and lastly, when overused or used to an extreme, over time, it will reverse the very capabilities it strove to provide and eventually flip into a newer technology. Predicting what will come next is not trivial, and McLuhan also noted that understanding what a technology retrieves requires a deep historical understanding of the technologies that came before it [47].

McLuhan illustrated these four laws using a tetrad (see Figure 1), a visualisation to help researchers understand or even predict the impact a new medium may have. This tetrad is often referred to by media scholars as McLuhan’s tetrad, and it has been commonly used in recent years to evaluate technological innovations [75].

Fig. 1.

The tetrad poses four questions to evaluate a new technology:

(1)

What does the technology enhance or amplify?

(2)

What does the technology make obsolete?

(3)

What does the technology retrieve that had been obsolesced earlier?

(4)

What does the technology reverse or flip into when pushed to extremes or overused?

Table 1 shows representative examples of past disruptive innovations in software engineering and how McLuhan’s tetrad can offer insights into what they enhanced, what they made obsolete, what they retrieved from the past, and what they flipped into over time.

Table 1.

Disruptive Technology Date Emerged	Machine languages 1960s	High-level Prog Langs, OO, Unix 1970s	Internet (Repositories, Libraries, Email, ICQ+Bots, Open Source) 1980s	IDEs, Automated Testing 1990s	Social media, Stack Overflow, GitHub, CI, DevOps 2000s
Enhancement	Programming, Problem-solving	Abstractions, Design composition Pseudocode	Distributed dev (merging, reuse), Communication	Agile, quality, reliability, repeatability	Community building, Community knowledge sharing, Faster deployments
Obsolescence	Manual Computations, Natural language, Paper	Knowledge of machine programming	Tarballs and diffs, Lost source code, Duplicate efforts	Custom tools, Waterfall processes, Business Analysts, Testers	Manuals (technical writers), Systems admins, Emails
Retrieval	Mathematics	Natural language design	Intellectual Property, Communities of Practice	Standardized environments/process	Social developers, Collaborative problem solving, knowledge sharing
Reverse/Flip	High-level programming languages, OS control	Collaborative dev tools, open source	IDEs, Automated testing	Social coding, automated processes	Artificial Development (behaviour/social data-driven)

Table 1. Disruptive Technologies in Software Development: What They Enhanced, What They Made Obsolete, What They Retrieved from Prior Technologies, and What They Flipped into Over Time

McLuhan’s tetrad is used as the first step in our playbook for evaluating new technologies in order to reflect on the impact of prior disruptive technologies (such as Stack Overflow). Figure 2 shows how McLuhan’s tetrad can be used to reflect on the impact that Stack Overflow has had in software engineering. Researchers have studied how Stack Overflow has enhanced (sped up) the task of finding answers to technical questions during coding and debugging tasks [6, 16, 42, 67, 80], how it has enhanced how developers across the world “collaborate” by crowdsourcing this knowledge [5, 8, 12, 32, 44, 77] and how it has been effectively leveraged for retrieving technical information [17] and supporting learning [18, 73]. Researchers have also investigated how Stack Overflow has made other technologies (such as email) obsolete or less important, and how it has retrieved characteristics from prior technologies, such as the support of mentors [30].

Fig. 2.

What Stack Overflow is reversing into is, however, an important and ongoing topic for research. While code snippets from Stack Overflow answers might be seen as a crucial part of developer knowledge, their effective reuse is far from trivial [79, 83]. Some researchers have concerns that developers use code snippets from Stack Overflow without fully understanding how they work, potentially lowering code quality [22, 48, 63, 85] and introducing bugs [2] and security risks [3, 29]. There is also a growing concern that the quality of answers is generally decreasing over time [85]. Because of these problems related to quality and the challenges of customizing snippets from Stack Overflow, developers have a desire for not just finding solutions, but finding solutions that are personalized to their needs/products. Some recent research has investigated how Stack Overflow is “flipping” into generative AI solutions and how generative AI has the potential to reshape software engineering practice [25] and replace community-based question-answering [14].

McLuhan’s tetrad is a provocative tool to guide reflective and predictable research, but more guidance is needed to help decide which phenomena (or actors) and conceptual ideas should be the focus of a study and how to study them. The next part of the playbook provides guidance on these decisions.

3.2 Identifying Research Phenomena Using McGrath’s Research Framework

An essential facet of research design, particularly relevant to software engineering studies, is the coherent structuring of research phenomena, theoretical ideas or constructs, and methodological choices. A pivotal framework that encapsulates these components is Joseph E. McGrath’s triadic domain configuration, comprised of the substantive, conceptual, and methodological domains [45]. Here, we detail each of McGrath’s domains and illustrate their application in the realm of innovative research and software engineering, forming the second part of our playbook for studying disruptive innovations through software engineering research. We continue to reflect on prior research about Stack Overflow to illustrate the playbook.

3.2.1 The Substantive Domain: Embodying Research Phenomena in SE.

The substantive domain serves as the research empirical backbone, forming the focus of the questions posed. It focuses on the specific phenomena or subjects that prompt academic curiosity and investigation. The phenomena are typically the units of analysis in any research study that help frame the focus of the study. McLuhan’s questions about how technology enhances, makes obsolete, retrieves, and reverses should be asked with these phenomena in mind. Some example phenomena that have been the focus of research on Stack Overflow include:

—

Software Artifacts: Looking through libraries of software engineering research, we see that there is extensive research on Stack Overflow that has studied software artifacts as the unit of analysis, notably questions and the quality of answers, for example, see.

—

Individual Developers: Although much of the research on Stack Overflow has focused on the quality or content of questions and answers, some researchers considered the direct impact on developers, for example when considering the emotions developers experience using Stack Overflow [55].

—

Software Teams: Ponzanelli et al. considered teams as the unit of analysis when they studied how integrating Stack Overflow in the IDE improves teamwork [62].

—

Software Projects: Research by Squire et al. studied how companies made the decision to move or not move to Stack Overflow from other channels for project documentation [69].

—

Society: A societal concern of software engineering is to increase diversity and support inclusion. Research by Ford et al. investigated how the design of Stack Overflow may have introduced barriers and led to less participation by some genders [31].

—

Software Processes and Methodologies: Stack Overflow is perceived as an invaluable and authoritative knowledge base by software developers. The discussion threads are also used to help guide code reuse from the answers given on this Q & A site. This phenomenon motivated researchers to investigate Stack Overflow’s contribution to the evolution of software processes. For example, Tang and Nadi [74] investigated how recommendations gleaned from Stack Overflow comment-edit pairs influenced software maintenance practices.

—

Communities: Several studies about Stack Overflow investigated how using it supported crowdsourcing of documentation at a community level [9, 52].

3.2.2 The Conceptual Domain: Developing Ideas and Theoretical Constructs.

Progressing from the empirical realm of the substantive domain, the researcher enters the conceptual domain. This is the domain where new ideas are generated and built upon and where theoretical constructs are leveraged and developed to enable a deeper understanding of the complexities inherent to the substantive domain. Example ideas may include the role or outcomes of an existing or new tool or process.

Often, the “ideas” software engineering researchers formulate about the phenomena they study should be grounded in theoretical concepts to facilitate discussion about construct validity and to provide a solid ground for interpretation of empirical results. Stol et al. highlighted the importance of conceptualisation and suggested that software engineering research should, on the one hand, be informed by existing theories and, on the other hand, aim at the formation of new theories [71]. Sjøberg et al. also described how to build on and extend theories in software engineering [68]. Theories not only help to explain research findings (e.g., why one tool may be better than another for a certain task), but they also provide a basis for other researchers to relate to or build on each other’s work [26].

As our playbook aims to provoke more attention on human and social aspects, the use of theories from both human-computer interaction research and the social sciences can be used to inform and provide context for socio-technical findings in software engineering research. Notably, Lorey et al. cataloged how social science theories, such as the Technology Acceptance Model and the Diffusion of Innovation and Coordination theories, are relevant to software engineering research [41]. These and other theories provide a basis for the ideas that can be explored in software engineering research that considers the disruptive impact of innovative technologies on pertinent socio-technical phenomena.

Returning to our illustrative example, several studies on Stack Overflow have integrated, built on, or developed new theories to explain their research insights in terms of community-based knowledge creation and sharing. For example, Ford et al. provided empirical evidence of the effectiveness of mentoring strategies to support question answering in Stack Exchange [30], while Calefato and colleagues provided empirically-driven guidelines for effective question writing in Stack Overflow.

3.2.3 The Methodological Domain: Guiding Research Strategies.

The methodological domain attends to the practical dimensions of conducting research. It involves the identification and application of appropriate methods and strategies for data collection, analysis, and interpretation. This domain serves to connect the insights gained from the substantive and conceptual domains with the practicalities of empirical investigation.

There are many resources in software engineering to guide the choice of research methods [24, 71, 72]. We note that McGrath’s circumflex, in its original form, considers human and social aspects as a core consideration in the strategy classification it presents—aspects that are often overlooked in SE research. McGrath’s framework, with its focus on human and behavioural aspects, was customized by Storey et al. [72] for software engineering research contexts to account for the prevalence of data studies (data mining and simulation studies) in SE. This extension is referred to as the “Who, What, How Framework,” and it summarizes research strategies into four broad categories: laboratory studies and experiments (strategies that increase the ability to control the behaviour of human subjects), field studies and experiments (strategies that have the potential to increase realism), respondent strategies (that have the potential to increase generalizability), and data studies and experiments (that also have the potential for generalizability and high data precision). The first three strategies in Figure 3 are annotated with a blue person icon to emphasize strategies that directly involve human participants as the unit of observation.

Fig. 3.

3.3 The Research Playbook

McGrath’s triadic domain framework and Storey et al.’s extension for SE underline the interconnected nature of empirical phenomena, theoretical ideas and constructs, and methodological choices in research design. By grounding research investigations in McGrath’s triadic domain framework and McLuhan’s tetrad, researchers can enhance the coherence and robustness of their studies, generating meaningful insights that contribute to the advancement of software engineering.

Figure 4 shows our proposed research playbook. We use the term playbook rather than framework because of the step-by-step guidance it provides on the research process to follow, described below. Although we number and present the steps in order, it is important to note that iteration may and often should occur throughout the process of designing a research program or project. The four steps in the playbook are:

—

Step 1: Disruptive technologies are only disruptive when they are put to use by people for some application. For this step, McLuhan’s tetrad is used to sketch probing questions inspired by the four laws of new technologies. This first step encourages a researcher to step back and reflect on the different possible impacts of disruptive technology in software engineering.

—

Step 2: For this second step, careful attention is paid to which phenomena should be studied, in turn guiding the units of analysis for the study, but also which ideas we have about the impact of the technology on those phenomena. These ideas may come from experience, as software developers ourselves, or they may emerge from other theories in or outside of software engineering. Choosing phenomena that relate to human and social aspects of software engineering is encouraged.

—

Step 3: In this third and critical step, the researcher selects, crafts, clarifies, and justifies their research question(s), carefully considering the different possibilities that the combination of steps 1 and 2 may bring to light. For this step, we suggest creating a table or matrix combining overarching questions posed by McLuhan’s tetrad as rows and the possible phenomena as columns. Cells in the matrix build on ideas (insights) we may have about the phenomena to be studied. This crucial step of brainstorming and selecting research questions involves further contemplation about the potential impact or actionability of the results from studying these questions, as well as identifying the type and novelty of the knowledge contribution that may arise, asking it is more understanding about the problem domain, the design or refinement of a technology, or evaluation of the technology [26].

—

Step 4: For each research question (if more than one), this step involves selecting and justifying research strategies that ideally directly involve human participants.

Fig. 4.

To illustrate the playbook, we apply it to reflect on prior research on the impact of Stack Overflow on software development. Table 2 shows a matrix of research questions where each cell is a question guided by the tetrad that were already asked about the phenomena of documentation and programmers. Other phenomena could have been selected for consideration here, e.g., teams or tools. In Figure 5, we show the different research strategies applied by the noted prior research on Stack Overflow. Each methodological choice is a tradeoff. The choice of a user study means control over human subjects was possible [4]. A respondent strategy, as was used by Ford et al. [31], increased the generalizability of their results on barriers to using Stack Overflow, while the field study used by Merchant et al. [50] on signals in Stack Overflow increased the realism of their study. The choice of a data mining strategy by Calefato et al. [15] had high data precision, making it potentially easier to replicate, but at the expense of control over human subjects and of realism.

Fig. 5.

Using the playbook to reflect on a prior disruptive technology, Stack Overflow, helped us see some patterns in prior research. Notably, most research focuses on the McLuhan enhances dimension, with some research on what Stack Overflow made obsolete. Less research, at least so far, has considered McLuhan’s retrieve and reverse dimensions. Also, we noticed that much of the cited research on Stack Overflow did not consider human and social aspects but rather emphasised technical aspects. This insight is also corroborated by the detailed systematic literature study on Stack Overflow by Meldrum et al. [49].

Table 2.

Triadic Dimensions	Documentation	Individual Developers
	Phenomena to Study (Stack Overflow)
Enhances	How can Stack Overflow be used to augment API documentation? [78]	Does Stack Overflow enhance the skills of student developers? [11]
Obsolesces	Does Stack Overflow replace the need for formal API documentation? [58]	Can a Stack Overflow Bot replace a formal onboarding process in open source? [23]
Retrieves	How does the SO voting feature from sites such as Digg impact documentation quality? [70]	How do badges (from the realm of boy scouts) steer user behaviour in Stack Overflow? [4]
Reverses	Does Stack Overflow lead to incomplete documentation over time? [58]	Does Stack Overflow, if overused, negatively impact student learning? [61]

Table 2. Step 3: This Shows Various Research Questions That Investigated How Stack Overflow Impacted Two Phenomena: Documentation and Developers

In the next two sections, we show how the playbook can guide future research. We delve into two pivotal scenarios that epitomize disruptive innovations in software engineering. These scenarios serve as practical illustrations of how our proposed playbook can be instrumental in guiding and enriching the investigation of emerging technologies in software engineering practices.

4 The Playbook in Action

To demonstrate the utility and versatility of the playbook, our article focuses on two revolutionary technologies that are redefining the landscape of software engineering. The first scenario explores the burgeoning role of large language models (LLMs), particularly generative AI, which are rapidly evolving and increasingly influencing current development practices. This disruption presents a unique opportunity to observe and analyze a disruptive technology as it emerges and integrates into the fabric of software development.

Our second scenario examines the potential of augmented and VR in software engineering. Despite years of development, AR/VR technologies stand on the brink of a paradigm shift in how software is developed and interacted with. The following section will not only explore the implications of AR/VR but will also demonstrate how tools like ChatGPT can be harnessed for brainstorming and conceptualizing research studies in software engineering.

4.1 Applying the Playbook to the Disruption of Generative AI in Code Generation

During the past several years, there has been a growing interest in harnessing the power of AI and machine learning (ML) technologies to enhance and automate various aspects of the software development life cycle [7]. For instance, code completion, a feature that has been embedded in IDEs for years [54], assists developers by anticipating their intentions and providing relevant recommendations while coding [81].

We are also witnessing a pivotal moment in the progression of AI-supported tools as systems such as Copilot [1], ChatGPT-4 [56] and LLaMA [76] herald the arrival of a new epoch characterized by disruption and innovation within the field. With its sophisticated capabilities, Copilot can decipher code context and semantics from minimal input, subsequently producing suggestions for not only the next few lines but also entire functions in certain instances [19]. The extraordinary user experience furnished by these state-of-the-art LLMs is exemplified by their extensive capabilities, superior response quality, and the significant potential to enhance user productivity [66]. Even users initially skeptical have been won over by the transformative influence of these contemporary LLMs on the software development process [66].

In the following, we show how the playbook can help guide and inform research aimed at systematically evaluating the ramifications of generative AI within the complex socio-technical ecosystem of software engineering.

Step 1: McLuhan’s Tetrad Is Used to Brainstorm the Impacts of Applying LLMs on Code Generation. The important point of using McLuhan’s tetrad is to broaden the questions that may be explored in the research. In the following, we show one possible iteration of this step in the playbook. Of course, depending on the context of the researcher and research done to date, what comes to mind in this step may vary greatly.

—

In what ways will LLMs enhance the process of writing code? Faster coding, generated tests, and improved documentation are likely outcomes of using LLMs. In addition to better productivity [37], quality may also be enhanced and could lead to improved time estimates and faster onboarding.

—

What will LLMs make obsolete when used for code generation? LLMs may make tools like Stack Overflow obsolete, reduce the need for search, reduce the need for manually created documentation, and even reduce demands for education about how to code.

—

What might LLMs retrieve that had been obsolesced before? LLMs bring back the ability to effectively explain a solution (of generated code) in natural language (see https://github.com/features/copilot), a more common skill from the earlier days of programming.

—

What will LLMs reverse if used to an extreme for code generation? Although LLMs are thought to improve the abilities of developers, they may reduce their abilities over time should developers trust the output and not spend time understanding the code generated. Developers may eventually have less understanding of why and how the code works. Furthermore, blind trust could lead to lower-quality code, vulnerabilities, and more bugs that are harder to identify and fix, especially if the LLM hallucinates suggestions that are not recognized by the developer.

Figure 6 shows how McLuhan’s tetrad is used to assist in brainstorming the impacts generative AI may have on code generation capabilities and to consider broader side effects, such as loss of developer skill writing code. Using McLuhan’s tetrad as a lens helps expose impacts that did not immediately come to mind when we think about how LLMs will not only enhance code generation but speeding up development time. We note that recently published research focuses mostly on how generative AI enhances or makes obsolete former tooling in SE, with less emphasis on what it will retrieve (such as the use of natural language) or reverse into (such as over reliance and poor quality).

Fig. 6.

Step 2: Deciding Which Phenomena to Study and Identifying Ideas That We Can Build on. The possible research impacts that emerged from the first step helps suggest phenomena and ideas that can be studied, with special attention paid to human and social aspects, as guided by the intention behind the playbook. The phenomena and ideas that become the focus of study are naturally shaped by the prior research, experiences, and interests of the researcher. There is already substantial research on the impacts of LLMs on code generation, and much of this research has focused on understanding the impact of LLM generation on program code quality and accuracy [36]. There is also some research that has investigated the impact on developers, e.g., the good day project at GitHub studied how the use of Copilot impacted the productivity and satisfaction of developers [59]. Our playbook aims to nudge consideration of human and social aspects, thus, we consider the impact of this disruptive technology on both individual developers and teams during this application of the playbook. For this step, we also need to consider what other research has been done, but also look to what signals are coming from the industry about how the technology is having an impact.

Step 3: Creating a Matrix of Possible Research Questions Concerning Phenomena and Ideas That Can Be Studied. Researchers often commit prematurely to research questions—this step of the playbook slows down that process and encourages more careful consideration of the impacts, inspired by the tetrad, crossed with possible phenomena and ideas about those phenomena to brainstorm and inspire new research questions. As mentioned above, here the playbook encourages us to select and justify specific research questions for further study. Each cell contains a potential research question to study. The rows of this matrix correspond to the four McLuhan laws, the tetradic dimensions of enhance, obsolete, retrieve and reverse, and the columns correspond to the phenomena selected for further study: teams and individual developers. The cells are populated with research questions we have brainstormed for further research, in some cases building on the ideas we already have, see Table 3.

Table 3.

Triadic Dimensions	Individual Developers	Team
	Phenomena to Study (LLMs)
Enhances	How do LLMs enhance the productivity for developers in terms of writing code?	Do LLMs help a team focus more on requirements than on writing code?
Obsolesces	Do LLMs make obsolete some coding skills for developers?	Do LLMs make the nerd culture obsolete on the team?
Retrieves	Do LLMs bring back more focus on using natural language when talking about their code?	How can LLMs be integrated as a Bot or agent as a member of the team?
Reverses	When used extensively for code generation, is there a lack of trust by developers in the code they generate using LLMs?	When used extensively for code generation, do LLMs lead to a tangible loss of team knowledge?

Table 3. Step 3: Matrix for Brainstorming Research Questions for the Disruptive Technology of LLMs to Code Generation for the Phenomena of Individual Developer and Team

These questions are what we the authors brainstormed (as prompted by the playbook).

Step 4: Research Method Selection and Justification. As a final step, and based on the research questions shown in the matrix created in step 3, the playbook promotes the selection of research strategies that align with the goals of directly studying human participants (e.g., developers or other stakeholders). In Table 4, we present research strategies aligned with the questions presented in Table 3.

Table 4.

Triadic Dimensions	Individual Developers	Teams
	Research Strategies (LLM)
Enhances	Controlled experiments with human participants, large-scale surveys, mining studies?	Field or Case Study, Ethnography, Behavioral measurements
Obsolesces	Interview and Surveys	Interview and Surveys
Retrieves	Longitudinal studies	Field studies
Reverses	Controlled experiment	Longitudinal case study

Table 4. Step 4: High-Level Research Strategies to Investigate Research Questions for LLMs on Individual Developers and Teams

The impact of generative AI on enhancing developer productivity can be assessed through experiments with human participants, as done by Peng et al. [59] who investigated the impact of using Copilot on a web-development task. Evidence collected in the lab through controlled studies can be complemented by surveys on the perceived impact of generative AI tools on developer productivity. To investigate the impact of LLMs at the team level on requirements engineering, ethnographic field studies might be more suitable.

Interview or survey studies may help identify what coding skills are made obsolete by the adoption of tools-based generative AI and what toxic cultural elements are becoming obsolete because of the emergence of new programming practices supported by these tools at the team level.

As for the questions that consider McLuhan’s retrieve dimension, a longitudinal observational study may assess how Copilot adoption influences how developers formulate solutions using natural language as they prompt. Indeed, recent research on prompt engineering explores how developers craft natural-language inputs to LLMs to produce their desired outputs. A longitudinal case study can be used to study how bots are adopted and used by a team over time.

Controlled experiments may align with research questions about the lack of trust by developers in the code generated by humans vs. LLMs (reverse dimension). This research approach was successfully adopted to investigate human-bot interaction on Stack Overflow by Murgia et al. [53]. A longitudinal case study using interviews may help reveal a potential loss of knowledge at the level of the team over time.

Reflection on Using the Playbook for Research on Generative AI in Code Generation. The playbook was not designed to be prescriptive but rather to aid in an iterative brainstorming process. It was also meant to help us question the choice of research questions and strategies and, along the way, reflect on the impact of any potential research questions and research designs. In this application, the playbook helped identify questions that asked how generative AI could enhance outcomes of code generation but also consider what was made obsolete, what was retrieved, and what it may reverse into. We found it relatively easy to identify what AI may enhance and make obsolete, but what generative AI retrieved took more thought. Ethical issues, which were more of a concern even in the early days of open source and the internet, and chatbots, popular before social media went mainstream, were identified as aspects retrieved from older technologies. In terms of what generative AI may reverse into, we found it relatively easy to speculate what may happen if this technology is overused, e.g., lack of trust may emerge, but what new technology it will flip into is not yet obvious.

Our playbook was designed with the goal to promote more consideration of human and social aspects, often overlooked by much of our research [72]. Selecting human and social phenomena helped steer us in this direction. But even with human or social phenomena at the core of a research question, a researcher may choose a research strategy that does not directly involve human participants as the unit of observation. But this flexibility of how the playbook can be used is a strength, while we hope that using the playbook will nudge our research in the direction of a heavier emphasis on human and social aspects of software engineering.

Step 3 uses McLuhan’s four laws to help brainstorm research questions centered around social and human phenomena. Identifying research questions for all cells in the matrix for the phenomenon we selected was not trivial. This inspired the development of a customized GPT for our playbook, based on ChatGPT, to help open up this brainstorming activity. Next, we apply this GPT to the second use case of the research playbook. Applying the GPT to the above scenario is an exercise we leave for the reader to explore!

4.2 Applying the Playbook to the Disruption of AR/VR on Software Team Communication and Collaboration

In recent years, AR and VR technologies have emerged as potent tools with transformative potential in various fields, including software development. Their application in enhancing team communication and collaboration, especially in remote work contexts, has garnered significant attention [40]. Traditional collaboration tools and methods are evolving rapidly, with AR/VR technologies offering immersive and interactive experiences that could redefine the landscape of team dynamics and productivity [39].

We recognize that applying the playbook is not always straightforward, as mentioned above, especially with very new technologies that show potential for disruption in SE but have not been studied much. To that end, we have developed a specialized GPT model, “MyResearchPlaybook,” designed to provoke our own thinking on a novel research direction by facilitating the brainstorming activities intrinsic to the playbook. See Appendix A for details on this GPT model.

In this section, we demonstrate how MyResearchPlaybook can streamline the investigation of the complex interactions and impacts of AR/VR technologies on software team collaboration. We initiated this process by prompting the specialized GPT model with the question: How will AR/VR tools disrupt software team communication and collaboration?

The subsequent analysis, informed by the use of MyResearchPlaybook, is detailed below. Further insights into the utilization of the GPT model for this research activity can be found in Appendix A.

Step 1: Using McLuhan’s Tetrad to Conceptualise the Influence of AR/VR on Team Dynamics. The application of McLuhan’s tetrad by MyResearchPlaybook aids in expanding the scope and depth of research inquiries. We present an iteration of this process as applied to AR/VR on software team communication and collaboration, noting that we manually refined the output generated by MyResearchPlaybook by adding references to the literature we found that related to the provided suggestions:

—

How will AR/VR enhance communication and collaboration in software teams? These technologies are likely to introduce more immersive and interactive ways of working together, potentially improving understanding, engagement, and efficiency in collaborative tasks [39]. Enhanced spatial awareness and visual communication offered by AR/VR can lead to more effective teamwork and problem-solving [21, 34, 35].

—

What traditional methods might AR/VR make obsolete in software team environments? Traditional video conferencing and text-based communication tools may become less relevant as AR/VR provides more holistic and engaging interaction methods. The need for physical presence in office spaces could also diminish as AR/VR creates virtual spaces that mimic real-world interaction [60].

—

What might AR/VR retrieve that was previously obsolete in team collaboration? AR/VR could bring back a sense of ’physical’ presence and closeness that remote teams have lost, revitalizing the face-to-face interaction dynamics in a digital format [27].

—

What could be the reversal effects of the extreme use of AR/VR in software teams? Over-reliance on these technologies could potentially lead to a decrease in face-to-face social skills and physical team bonding. The high immersion could also lead to difficulties in distinguishing between virtual and real-life interactions, and there might be an increased risk of digital fatigue or disorientation [38].

Step 2: Deciding Which Phenomena to Study and Identifying Ideas We Can Build on (if Any). Our playbook emphasizes the importance of human and social dynamics, particularly in the realm of emerging AR/VR technologies. In applying the playbook to this domain, for this step, we delve into the multifaceted effects of AR/VR on both individual contributors and collaborative teams. This exploration entails a thorough review of existing research while also paying close attention to industry trends and real-world applications of AR/VR in software development environments. It is crucial to understand not only the technological capabilities of AR/VR tools but also their nuanced impact on team dynamics, communication styles, and the overall work culture. By incorporating these insights, we aim to offer a comprehensive perspective on how AR/VR technologies are reshaping the landscape of software team collaboration, highlighting both the opportunities and challenges presented by this disruptive innovation.

Step 3: Creating a Matrix of Possible Research Questions Concerning Phenomena and Ideas That Can Be Studied. As mentioned above, the playbook can encourage us to consider, select, and justify specific research questions for further study. For this scenario and step, MyResearchPlaybook generated these questions for us, where each cell contains a potential research question to study. The rows of this matrix correspond to the McLuhan laws (the tetradic dimensions, as mentioned above), and the columns correspond to the phenomena we selected in Step 2 for further study: AR/VR in individual work and AR/VR in team collaboration, see Table 5.

Table 5.

Triadic Dimensions	Individual Work	Team Collaboration
	Phenomena to Study (AR/VR)
Enhances	How does AR/VR enhance individual productivity and engagement in software development tasks?	How does AR/VR improve collaborative problem-solving and project management in software teams?
Obsolesces	Does AR/VR render traditional desktop-based software development tools obsolete for individual developers?	Does the use of AR/VR in team settings make conventional remote communication tools (like video calls) obsolete?
Retrieves	Does AR/VR bring back a sense of physical presence and interaction in individual remote work settings?	In what ways does AR/VR retrieve traditional face-to-face collaboration dynamics in software teams?
Reverses	When used to an extreme, does AR/VR lead to a disconnection from the physical workspace for individual developers?	Could an over-reliance on AR/VR technologies in software teams lead to a loss of interpersonal skills and team cohesion?

Table 5. Step 3: Matrix for Brainstorming Research Questions for the Disruptive Technology of AR/VR in Software Team Communication and Collaboration for the Phenomena of Individual Work and Team Collaboration

MyResearchPlaybook generated the research questions shown in the Table’s cells.

Step 4: Research Strategy Selection and Justification. As a final step, and based on the research questions outlined in the matrix generated in step 3, we again turn to the MyResearchPlaybook to propose research strategies that align with the goals of directly studying human participants (e.g., software developers and teams) and that address the questions generated in step 3 (see Table 6). The impact of AR/VR on individual developers’ productivity and engagement can be assessed through experiments with human participants. These studies can be complemented by surveys on the perceived impact of AR/VR tools on developer productivity and well-being. To investigate the impact of AR/VR at the team level, field studies might be more appropriate, including case studies and ethnographic approaches to understand the nuanced changes in team dynamics and collaboration.

Table 6.

Triadic Dimensions	Individual Work	Team Collaboration
	Research Strategies (AR/VR)
Enhances	Experiments with human participants, Surveys	Field Studies, Case Studies, Ethnography
Obsolesces	Interviews, Surveys	Interviews, Surveys
Retrieves	Longitudinal studies	Longitudinal studies, Field studies
Reverses	Controlled experiments, Qualitative studies	Qualitative studies, Longitudinal case studies

Table 6. Step 4: High-Level Research Strategies for AR/VR on Individual Work and Team Collaboration

As for the retrieve dimension, a longitudinal study could help assess the long-term effects of AR/VR adoption on individual work habits and team collaboration practices. Interviews or survey studies could be instrumental in identifying the skills and traditional work practices made obsolete by the adoption of AR/VR. To investigate potential negative effects or over-reliance on AR/VR technologies, the reverse dimension, controlled experiments and qualitative studies, such as focus groups or in-depth interviews could be used to gather insights from both individual developers and teams.

Reflection on Using the Playbook for Research on AR/VR on Software Team Communication. Although the MyResearchPlaybook GPT model provided a structured and systematic approach for applying the playbook to the impact of AR/VR on team communication and collaboration, we also found some limitations. Its effectiveness is reliant on the existing body of research and data available. In the rapidly evolving field of AR/VR, this can be a significant limitation, as current research may not yet fully capture the long-term effects or emerging trends. This limitation potentially led to gaps in the playbook’s ability to generate comprehensive and forward-looking research questions for this use case. Furthermore, the generalized framework provided by MyResearchPlaybook may not adequately address the unique and specific contexts within which AR/VR technologies are deployed. The nuances of different software development environments, team cultures, and individual differences in technology adoption are challenging to encapsulate in a standardized model, possibly leading to oversimplified conclusions. Further iterations to refine the questions need to take these different contexts into consideration, but we see how using the playbook helped with an initial version of the research questions.

5 Discussion

In this section, we first suggest how the playbook may provoke a paradigm shift in software engineering research, followed by a discussion of how the playbook prompts a motion towards research that focuses more on human and social aspects. This is followed by a summary of some of the limitations of applying the playbook.

5.1 Provoking a Paradigm Shift in Software Engineering Research

Technological advancements in software engineering are undeniably shaping the future of the discipline. However, as we discuss the ramifications of disruptive technologies, it is paramount to understand that software engineering is fundamentally a socio-technical domain. Beyond the algorithms and architectures, the human and social dimensions provide a comprehensive perspective on the implications of these technologies. We discuss how the McLuhan part of the playbook helps researchers to think of the solutions they develop as extensions of human rather than merely automating what humans do.

Large-scale software projects, collaborative development practices, and the rise of social coding platforms have transformed not only the methodologies but also the dynamics of who contributes and how they collaborate.

Beyond Incrementalism: Generative AI, coupled with the immersive experiences offered by AR and VR, is more than a technological leap; it’s a paradigm shift. These technologies are not just about enhancing existing practices; they are inaugurating a new era of creativity and interaction. We must move beyond the safety of incremental “delta papers” and dare to engage with research that is truly disruptive. This means challenging the traditional benchmarks for evaluating tools and outcomes, urging a broader, more profound, and more meaningful exploration.

Redefining Evaluation and Impact: Rejecting benchmarks as the sole measure of success, we must embrace a more holistic evaluation of technological impacts. Benchmarks, while valuable, often do not capture the multifaceted impact of disruptive technologies on the socio-technical aspects of software development. A more comprehensive evaluation framework that considers user experience, societal impact, and ethical considerations, is critical, especially when disruptive technologies are introduced—what matters and what should be valued has possibly changed.

Framing the Right Questions: The trajectory of our research and innovations in software engineering is heavily influenced by the questions we pose. Inspired by McLuhan’s tetrad, our inquiries should not be limited to the immediate functionalities or obsolescence of a technology. We must probe deeper into the broader socio-technical ripples. For instance, how does the integration of AI in software engineering reshape team hierarchies or inclusivity paradigms? What emerging practices or norms are developer communities adopting in response to these innovations? How do these technological shifts impact perceptions of software reliability, trust, or ethical standards? Furthermore, as technologies like generative AI or AR/VR redefine the software development’s landscape, they challenge established notions of expertise and collaboration. The democratization brought about by these technologies introduces a plethora of voices and perspectives into software engineering. While this diversity is a strength, it also presents challenges in communication, collaboration, and conflict mediation. As we find ourselves at the forefront of these technological breakthroughs, it is crucial to pause and reflect on the wider socio-technical consequences. We need to push our boundaries, daring to ask deeper questions that delve into the complex interplay between humans, technologies, and society.

Designing the Future of Research Creativity: To truly liberate our passion for research in this rapidly evolving landscape, we need to establish structures that foster creativity, interdisciplinary collaboration, and bold exploration. This involves creating academic and industrial environments that support risk-taking, encourage the exploration of unconventional ideas, and facilitate the integration of diverse perspectives.

Disrupting Research with Disruptive Innovations: As we explore disruptive innovations that are impacting software engineering, we are also recognizing how these innovations also impact the very essence of how we do research. The Internet and social media both impacted our ways of doing and sharing research, and likewise, tools like ChatGPT are having a similar if not bigger disruption. We found ourselves using ChatGPT throughout the writing of our article and design of the playbook (somewhat reluctantly at first), but finally embraced the role ChatGPT can play by designing a custom GPT to support the use of the playbook. As mentioned above, our intent is that it should not replace researchers’ creativity and deep reflection, but it can be used to support brainstorming and point to ideas and resources we may not have known or envisioned.

In summary, as we navigate the complexities of disruptive technologies in software engineering, our role as researchers and practitioners demands a provocative, forward-thinking approach. We must challenge existing paradigms, embrace the socio-technical nature of our field, and seek to understand and shape the broader implications of our work. By doing so, we can ensure that our contributions to the field of software engineering are not only technologically advanced but also socially responsible and ethically sound. It is through this holistic understanding and approach that we can truly harness the transformative power of disruptive technologies in software engineering.

5.2 The Playbook and Social/Human Aspects in Software Engineering

In the following, we discuss how the playbook emphasizes the consideration of technological innovations that augment human abilities. McLuhan’s laws of technology provoke questions that specifically ask how a new technology may enhance human capabilities, replace prior augmentation technologies, retrieve older technologies that can be useful again, and over time, how a new technology may reverse the benefit to humans and social systems it was intended to enhance. McGrath’s framework promotes research into the socio-technical phenomena that may be impacted by a new disruptive technology.

McLuhan’s Insight on Technologies as Human Extensions: McLuhan posited that every technology, or “media,” serves as an extension of human capabilities. Introducing a disruptive technology into software engineering is not just about adding a new tool or methodology. When we introduce disruptive technologies like AR, VR, or AI into the software engineering landscape, we are not merely integrating a new tool [33]. We are ushering in entities that interact with, augment, and sometimes challenge human capabilities. For instance, the rise of AR and VR is not just about enhanced visualization; it is about redefining how we interact with software systems. Similarly, AI’s role is not confined to task automation; it has profound implications on developer roles, team dynamics, and the entire software development lifecycle.

Balancing Automation with Human Augmentation: Our focus must extend beyond mere automation. The true potential of disruptive technologies lies in their ability to augment human capabilities, enriching the creative and collaborative aspects of software engineering. The playbook emphasizes this shift from automation to augmentation and considers how these technologies can enhance rather than replace human skills and ingenuity.

McGrath’s Framework and Socio-Technical Phenomenon: Building on McGrath’s research framework, it is evident that the phenomena we study in software engineering are not just technical constructs. They encompass developer communities, collaboration patterns, ethical considerations, user experiences, and societal impacts. The human and social dimensions, especially in the face of disruptive technologies like AR, VR and AI, are not mere footnotes. They form the crux of software engineering research. The human and social facets are not peripheral; they are at the heart of software engineering research, especially in the context of disruptive technologies.

6 Limitations

The playbook is intended as a study design artifact to broaden research questions, open new creative directions, and promote consideration of both human and social aspects when new technological disruptions are adopted or proposed. We developed the playbook by building on McLuhan’s tetrad and McGrath’s research strategy framework, both of which emphasize the impact on human and social aspects, areas frequently overlooked in software engineering research [72].

We note that the use of the playbook is designed for researchers who already possess skills in empirical and design research, along with deep knowledge of related research disciplines. Specifically, McLuhan’s “retrieve” law requires a profound understanding of prior technologies and their impact.

The MyResearchPlaybook GPT, provided as an adjunct to the playbook, necessitates considerable care and responsible usage, requiring significant human oversight and interpretation. The selection of phenomena, dimensions, and research questions, although guided by the playbook or the MyResearchPlaybook GPT, ultimately depends on the researcher’s insights and decisions. The GPT does not identify literature references; therefore, we conducted a brief literature study to identify references that could inspire the research brainstorming activity. These manual steps introduce a subjective element, potentially leading to biases or oversight of critical aspects, but also maintain human creativity as an essential part of our research process.

Finally, it is important to highlight that the MyResearchPlaybook GPT is a provocation and should be used with caution. Since it is based on prior research on technological progressions that were mostly developed in a linear and predictable manner, the suggestions it provides may not keep pace with how technologies such as AI and AR/VR may progress. These new innovations are influenced by a myriad of factors, including economic, social and regulatory aspects, which may lead to non-linear and unforeseen outcomes that the GPT does not consider. Researchers should not assume that the playbook can accurately forecast and analyze the future implications of AI, AR/VR, or other disruptive innovations in software teams.

Additionally, the underlying model of the GPT will evolve, leading to changes in its output. This evolution may affect the consistency and reliability of the suggestions it provides. Therefore, continuous human oversight and critical interpretation are crucial to ensure the responsible use of the MyResearchPlaybook GPT.

7 Conclusion

The software engineering discipline has seen many disruptive technologies that have augmented and extended the capabilities of the engineers who design, write, maintain, and ensure the quality of software programs. As researchers, we have the privilege and opportunity to study the impacts these technologies introduce.

However, often our research is slow to respond to the disruptions new technologies evoke, and we take an overly narrow approach and focus on the short-term technical advantages they introduce. McLuhan recognized this tendency for people to focus on positive influences decades ago, and his observations of the four laws of media led to his tetrad of provoking and non-obvious questions to ask about new technologies. These disruptions, like other forms of media and technologies, all follow the same four laws that McLuhan noted in that they will enhance software engineering capabilities, they will make some existing technologies obsolete, they will retrieve aspects of prior technologies, and they will ultimately reverse the desired outcomes that inspired them when used over time.

The playbook we introduced builds on McLuhan’s four laws and is designed to provoke and prompt new research directions that ask critical questions reflecting on what has come before as well as what will happen, especially to socio-technical aspects of software engineering, when the technologies are used and, possibly overused, over time. We combined McLuhan’s tetrad with McGrath’s research framework that help us craft pertinent research questions about socio-technical aspects in software engineering, while also selecting research strategies that are aligned with studying the phenomena that may matter the most over time—those that relate to human and social aspects.

We applied the playbook in two ways. First, we used it to reflect on a prior disruptive technology on software engineering—Stack Overflow—and to consider some of the landscape of Stack Overflow research that investigated the impact of it on individual developers and development teams. This retrospective application of the playbook not only illustrated the playbook but also revealed that this prior research was predominantly focused on McLuhan’s “enhance” and “obsolete” dimensions, and this research had, especially initially, little emphasis on studying social and human aspects.

We also applied the playbook to two emerging disruptive technologies in software engineering: generative AI and VR/VR. The playbook helps steer towards new research directions that focus on human and social aspects.

Finally, along with the playbook, we provide a customized ChatGPT called MyResearchPlaybook that some researchers may find useful as they brainstorm and design their research questions and studies. We expect we are just at the beginning of these new research assistant tools that will help us expand, accelerate, broaden, and improve our research over time.

In conclusion, software is designed by people for people. How technology will change human lives, capabilities and values changes faster than we can imagine, and demands that we anticipate and pay close attention to the changes that may occur. The playbook we present, we hope, will provide a way for researchers to pause before selecting a particular research question or strategy and to reflect on what else they could study that may be ultimately more important to consider with such an uncertain future in mind.

Acknowledgments

We thank Cassandra Petrachenko for editing our paper, and grateful for the excellent feedback and suggestions from the anonymous reviewers of our paper.

Footnotes

Although McLuhan used the term media more than technology, we use the more familiar term of technology throughout our article.

MyResearchPlaybook is accessible at this link in OpenAI’s library: https://chatgpt.com/g/g-ckghjdz0n-myresearchplaybook

Link to the GitHub repo with the instructions: https://github.com/danielrusso-org/MyResearchPlaybook

A The MyResearchPlaybookGPT: Enhancing Research on Disruptive Technologies

Software engineering is continuously being reshaped by disruptive technologies. These innovations necessitate a multifaceted approach to research, one that comprehends both the technological advancements and their socio-technical implications. In this regard, we introduce “MyResearchPlaybook,” a specialized GPT model, as a significant adjunct to our research playbook and to help, in particular, in the brainstorming activity the playbook invokes. In this section, we describe a playbook GPT we have designed to assist in using the playbook. In Section 4.2 we show how we applied this GPT to assist (not replace!) our activity of applying the playbook to the role of AR/VR in SE.

A.1 GPTs: Tailored AI for Focused Inquiry

GPTs, a subset of the broader Generative Pre-trained Transformer models by OpenAI, are customizable AI tools designed to address specific challenges and tasks. These models extend the capabilities of standard GPTs by focusing on particular domains or applications, offering more nuanced and relevant insights. An example of this is MyResearchPlaybook,² a GPT model configured to assist in exploring the interplay between disruptive technologies and their human-centric impacts. To make the instructions transparent and reusable with other LLMs, we published them in a GitHub repo.³

A.2 MyResearchPlaybook: Bridging McLuhan’s Theory and Software Engineering Research

Drawing upon the theoretical framework of McLuhan’s Triadic Dimensions—enhances, obsoletes, retrieves, and reverses—MyResearchPlaybook aids in constructing a structured matrix for analysis. This matrix serves as a scaffold for developing research questions that are pivotal in understanding the multifaceted effects of disruptive technologies such as AI, AR, VR and their iterations in software engineering.

A.3 Capabilities and Contributions

—

Analytical Matrix Construction: MyResearchPlaybook generates matrices aligning McLuhan’s Dimensions with current research phenomena in software engineering, ensuring a comprehensive exploration of potential impacts.

—

Formulation of Research Questions: It proposes specific research questions at the intersection of each dimension and phenomenon, guiding researchers toward targeted areas of exploration.

—

Strategic Research Planning: The tool suggests initial research strategies in accordance with the ACM SIGSOFT Empirical Standards [65], which are refined based on the researcher’s focus to provide a detailed methodological approach.

—

Adaptability and Contextual Relevance: Acknowledging the diverse nature of questions in disruptive technology research, MyResearchPlaybook adapts its recommendations to various contexts, including empirical studies, literature reviews, and case studies.

In summary, MyResearchPlaybook stands as an innovative addition to our research methodology, introducing a structured, rigorous approach to investigating disruptive technologies in software engineering. Its deployment aims to enhance the analytical depth and breadth of our studies, ensuring that our contributions are not only technologically advanced but also socio-technically informed and ethically sound.

A.4 Limitations of MyResearchPlaybook

While MyResearchPlaybook represents a significant advancement in AI-assisted research in the field of software engineering and disruptive technologies, it is imperative to acknowledge its limitations. This cautionary note serves to highlight areas where reliance on MyResearchPlaybook may require supplementation with human expertise and critical analysis.

—

Dependence on Pre-Existing Knowledge: MyResearchPlaybook operates within the bounds of its training data. This limitation means that the tool may not be aware of the very latest research developments or emerging technologies that have arisen post-training.

—

Lack of Domain-Specific Depth: While MyResearchPlaybook is tailored for research in disruptive technologies, its understanding may not match the depth of knowledge of a seasoned expert in a specific sub-field. It is a generalist tool and may not fully grasp highly specialized or niche areas.

—

Potential for Bias: Like any AI tool, MyResearchPlaybook may inadvertently reflect biases present in its training data. Researchers should be cautious of these potential biases and critically evaluate the suggestions and insights offered by the tool.

—

Absence of Creative Insight: AI models, including MyResearchPlaybook, excel at analyzing and synthesizing existing information but do not possess the creative or innovative capabilities of human researchers. They cannot generate novel theories or hypotheses that have not been previously conceptualized.

—

Need for Human Interpretation: The tool’s outputs require interpretation and contextualization by knowledgeable researchers. MyResearchPlaybook can assist in identifying patterns and generating hypotheses, but the ultimate analysis and decision-making should rest with human researchers.

—

Ethical Considerations: Ethical considerations, particularly in the field of disruptive technologies, often require nuanced understanding and judgement that may be beyond the scope of an AI model like MyResearchPlaybook.

Researchers are advised to use MyResearchPlaybook as a complement to, rather than a replacement for, their expertise and critical thinking skills. The tool should be viewed as one component in a broader methodological toolkit, offering support but not supplanting the essential human elements of curiosity, creativity, and ethical judgement in research.

References

[1]

GitHub Copilot. 2021. https://copilot.github.com

Abstract

1 Introduction

2 Disruptive Software Engineering Technologies Disrupt Research

3 The Design of a Disruptive Research Playbook

3.1 Framing the Research Landscape with McLuhan’s Tetradic Dimensions

3.2 Identifying Research Phenomena Using McGrath’s Research Framework

3.2.1 The Substantive Domain: Embodying Research Phenomena in SE.

3.2.2 The Conceptual Domain: Developing Ideas and Theoretical Constructs.

3.2.3 The Methodological Domain: Guiding Research Strategies.

3.3 The Research Playbook

4 The Playbook in Action

4.1 Applying the Playbook to the Disruption of Generative AI in Code Generation

4.2 Applying the Playbook to the Disruption of AR/VR on Software Team Communication and Collaboration

5 Discussion

5.1 Provoking a Paradigm Shift in Software Engineering Research

5.2 The Playbook and Social/Human Aspects in Software Engineering

6 Limitations

7 Conclusion

Acknowledgments

Footnotes

A The MyResearchPlaybookGPT: Enhancing Research on Disruptive Technologies

A.1 GPTs: Tailored AI for Focused Inquiry

A.2 MyResearchPlaybook: Bridging McLuhan’s Theory and Software Engineering Research

A.3 Capabilities and Contributions

A.4 Limitations of MyResearchPlaybook

References

Cited By

Index Terms

Recommendations

Using ethnographic methods in software engineering research

Disruptive Marketing: What Growth Hackers, Data Punks, and Other Hybrid Thinkers Can Teach Us About Navigating the New Normal

Managing potentially disruptive innovations in software companies

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations