research-article

Public Access

Thought Bubbles: A Proxy into Players’ Mental Model Development

Authors:

Casper HarteveldAuthors Info & Claims

CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

Article No.: 729, Pages 1 - 21

https://doi.org/10.1145/3544548.3581570

Published: 19 April 2023 Publication History

All formats PDF

Abstract

Studying mental models has recently received more attention, aiming to understand the cognitive aspects of human-computer interaction. However, there is not enough research on the elicitation of mental models in complex dynamic systems. We present Thought Bubbles as an approach for eliciting mental models and an avenue for understanding players’ mental model development in interactive virtual environments. We demonstrate the use of Thought Bubbles in two experimental studies involving 250 participants playing a supply chain game. In our analyses, we rely on Situation Awareness (SA) levels, including perception, comprehension, and projection, and show how experimental manipulations such as disruptions and information sharing shape players’ mental models and drive their decisions depending on their behavioral profile. Our results provide evidence for the use of thought bubbles in uncovering cognitive aspects of behavior by indicating how disruption location and availability of information affect people’s mental model development and influence their decisions.

1 Introduction

Complex dynamic systems, such as supply chains, comprise a network of interdependent actors that constantly interact with each other [71]. The behavior of these actors results in emergent dynamics and influences how they interact with the system over time [7, 62]. Some actors show distinct decision patterns [67], and some even exhibit irrational behavior, such as panic-buying and hoarding, in the context of supply chains [84]. Such behaviors and varying decisions have been shown to be the outcome of different mental models about the underlying system [11, 38]. Therefore, it is vital to investigate how decision-makers form mental models of such complex dynamic systems over time and undisturbed in the decision context. Such investigation, however, calls for methods for eliciting mental models and evaluating mental model development over time [91].

In this paper, we present such an elicitation method that we refer to as thought bubbles. This method evokes thought processes in situ and diegetically through multiple open-ended prompts over time as part of an interactive virtual environment, and thus seeks to elicit mental model development in the context of complex dynamic systems. We use the term mental model development to refer to a temporal process that happens during the interaction with complex dynamic systems and over time. The need for a process view stems from mental models not being static constructs [19, 60]. They dynamically change depending on new information or perceptions of the system. Previous studies provide evidence that the interaction forms behavior [67], and that behavior is the outcome of cognitive constructs [30, 58]. Hence, it is not unreasonable to think that the interaction affects mental model development. In addition, we leverage interactive virtual environments because they have been demonstrated to be useful for studying and learning about complex dynamic systems [8, 62, 68] and allow for eliciting mental models diegetically, which mitigates the effect of any out-of-context effects such as pre/post-interaction information or experimenter bias [23].

Eliciting mental models is only the first step in gaining insight into the cognitive aspect of human actions [41]. We also need reliable methods for analyzing the elicited mental model concepts. As our elicited mental models result from textual verbalization of thought processes, we can leverage qualitative analysis to make sense of our elicitation [40, 41, 56]. In grounding our qualitative analysis, we must align our outcomes with existing theories to gain a theoretical understanding of mental model development. We rely on the Situation Awareness (SA) model, introduced by Endsley [28], as a guiding framework for our qualitative analysis. SA framework is a well-established model that describes people’s perception of the elements of the environment, the comprehension of their meaning, and the projection of future states [26]; hence providing a window into their mental model development [29].

We test thought bubbles and our qualitative analysis framework using an interactive virtual environment called gamettes [68] and conduct two experimental studies (Study 1: n=115 and Study 2: n=135) in a supply chain context. These studies are exploratory in nature and serve two purposes. First, we test whether elicitation via thought bubbles can offer us a meaningful outlook on the cognitive aspects of decision-making. In both studies, we examine how experimental manipulations such as disruption location in the supply chain or the level of information sharing affect mental model development. Study 2 also investigates mental model development to help explain cognitive aspects of decisions made by players with different behavioral profiles. Second, we use the results of our qualitative coding in Study 1 as a codebook for the qualitative analysis of Study 2, aiming to test the reliability of SA for analyzing elicited mental model concepts over time. The following are the contributions of this work:

•

We present thought bubbles as a method for collecting qualitative data on human thought processes diegetically and temporally to elicit mental model development;

•

We show a mixed-method approach for analyzing elicited mental models to gain a theoretical understanding of cognitive aspects of human decisions and their mental model formation; and

•

We provide evidence on the effect of disruption location and information sharing on mental model development (Study 1), as well as the effect of information sharing on the mental model development of players with distinct behavioral profiles (Study 2).

2 Related Work

In our review of the related literature, we first study the notion of mental models as a cognitive construct and methods for eliciting mental models. We then look into the study of mental models in the context of dynamic decision-making.

2.1 Mental Models

Mental models are internal and small-scale representations of the real world that people rely on in their interaction with external reality [39, 56]. The notion of mental models, which was first put forth in 1943 by Craik [18], has been extensively researched by cognitive psychologists and tested in various contexts. As a cognitive construct, mental models are compared with the notion of “schemata”, which are sets of expectations or long-term knowledge structures based on prior experiences [10, 52, 75, 93]. Psychologists typically refer to both concepts to better contrast their definitions [9, 47, 76]. The difference is that schemata are inflexible and generic knowledge structures that provide predictive knowledge for routine situations [47, 76]. Mental models, on the other hand, are dynamically formed each time a new data or situation is perceived, and people form richer and more consistent mental models as they gain more experience [19]. When faced with unfamiliar domains, people may rely on analogies to extend knowledge from prior experiences that are well-understood and perceived as similar [16, 39].

Prior research also describes how mental models can be used to run mental simulations (i.e., mentally project into the future) to envision and explore alternative future states before acting [19, 60]. This is where cognition meets behavior; the mental model construct directs actions when people rely on mental simulations in interacting with the environment [58]. From this perspective, understanding behavior requires gaining insight into mental models. However, we cannot directly observe cognitive constructs such as mental models. We need instruments and methodologies to allow people to externalize their thought process [49]. With thought bubbles, we aim to gain insight into the cognitive aspect of human decision-making by eliciting mental models.

Prior research points to the lack of consistent methodologies for mental model elicitation [41]. The literature refers to elicitation approaches in two dimensions: (1) techniques used for extracting thought processes (i.e., verbal, visual, or hybrid); and (2) elicitation context (i.e., situated vs. non-situated) [55]. Verbal elicitation refers to using dialogue or discussion in an interview procedure. In contrast, visual elicitation is related to using graphics to represent an individual’s mental model. The hybrid approach concerns a combination of verbal and visual elicitation [17, 41, 59]. Situated practices involve eliciting mental models in the context (i.e., diegetic). The non-situated approach, on the other hand, involves elicitation in a location removed from the topic of study [55]. Thought bubbles are a form of verbal elicitation that we utilize in situ. By allowing players to provide open-response comments, we leverage the verbalization of players’ thought processes to investigate their mental model development in the context of dynamic decision-making and without introducing framing bias [66, 90].

2.2 Dynamic Decision Making in Interactive Systems

Dynamic decision-making tasks involve taking a series of actions in a dynamic system over time to achieve some goal (e.g., maximizing total profit) [25]. Actions are interdependent, which means later actions depend on the earlier ones, and the environment will change spontaneously or as the result of earlier actions [48, 58]. The study of mental models is fundamental in dynamic decision-making environments. Most studies in this area are motivated by trying to develop a better understanding of mental models to improve decision-making [23]. For example, researchers have studied mental model elicitation from experts to improve system dynamics models for mental model improvement [36]. Others have studied individuals’ understanding of the decision context and showed how mental models affect human rationality in dynamic decisions [74], how different mental models lead to different decisions and performances [38], and how higher similarity between mental models and the decision context results in higher performance [72]. Most of these studies rely on a simulation for the decision context and a form of elicitation technique, as described in the previous section, to capture mental models.

Researchers have proposed various approaches for analyzing elicitation results and studying mental models. From our research perspective, these approaches fall into one of three categories: (1) not applicable to dynamic decision-making, (2) applicable but not practical (e.g., demonstrated by lack of empirical use), and (3) applicable and practical. As an example of the first, Johnson-Laird [51] studied mental models in connection with human reasoning though propositional reasoning, deduction, and syllogistic inference [52, 53, 54]. Despite its prominence, Johnson-Laird’s view is not immediately applicable to dynamic decision-making environments because of its lack of support for studying mental models formed through interaction with the task or the environment [11, 83]. As for the second, Richardson et al. [70] developed a theory of perception, planning, action, and learning specifically for studying mental models in dynamic decision-making. They followed the idea that mental models are multifaceted (i.e., comprising three main sub-models: ends model, means model, and ends-means model [2]) and linked system dynamics feedback theory with models of judgment. Unfortunately, there is not enough evidence in the literature on the empirical use of the theory proposed by Richardson et al. [70].

Finally, as for the third, Endsley [28] introduced the theory of Situation Awareness (SA) in dynamic decision-making and proposed its use for studying mental models [29]. SA model has been extensively and empirically researched through numerous studies by Endsley [30, 31, 32, 33] and others [3, 13, 87]. Therefore, we rely on the SA model for approximating players’ mental model development over time. This idea of leveraging the use of SA to characterize mental models has also been examined by other researchers in inductive reasoning tasks [95]. Endsley describes situation awareness in dynamic decision-making as:

“[...] the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future.” [26]

According to this model, situation awareness is achieved through: (1) perceiving the status, attributes, and dynamics of elements of the environment (perception); (2) understanding the situation based on a synthesis of perceived elements and determining their relevance to decision goals (comprehension); and (3) the ability to project the future events and dynamics of the elements of the system (projection). According to Endsley [29], a situation model ¹ provides a useful window into people’s mental models. While people may have a mental model of how a dynamic system (here a supply chain) works, their interaction with the environment directs how they update their situation model, which eventually directs the selection and revision of their mental model [30]. More specifically, mental models direct attention to key features of the environment and the comprehension, projections, and expectations. Therefore, a situation model is a key to understanding the mental model [29].

Endsley [26, 28] also introduced the Situation Awareness Global Assessment Technique (SAGAT) to measure individuals’ situation awareness. SAGAT involves stopping the simulation in random intervals and querying the subjects with specific questions to determine their SA at that particular time [26]. While we do not use SAGAT, with thought bubbles, we follow the same idea by prompting players in intervals. Our goal is not to measure SA using thought bubbles. Instead, we use the SA framework for making sense of players’ comments during our qualitative coding process resulting from responding to the thought bubbles deployed in gamettes.

3 Gamettes

We introduced gamettes in our previous study [68], as a serious game approach for collecting data on behavioral aspects of human decision-making in supply chain experiments. A gamette is a short game-based scenario that immerses human decision-makers into a specific situation, requiring them to make decisions by responding to a dialog or taking actions. The term gamette is a contraction of “game” with “vignette”, and similar to a vignette, a gamette aims to provide a brief description of a situation, as well as to portray someone. Here, we utilize gamettes to understand an individual’s mental model supporting their decision-making in a drug delivery supply chain. For this, we use the integrated simulation framework proposed by Doroudi et al. [21]. This framework comprises a Flow Simulator for simulating the supply chain dynamics and a gamette environment for engaging human decision-makers with the simulation by immersing them in a specific role and particular state of the supply chain (see Figure 1).

Figure 1:

Within this framework, the Flow Simulator is a multi-agent simulation and the central hub that controls the dynamics of a drug delivery supply chain, including the flow of information and physical products over time. These information and physical flows are driven by the decisions and actions taken by the agents of the system (i.e., manufacturers, wholesalers, and health centers). The Flow Simulator can run in a standalone mode and without any human agents, which means that the Flow Simulator simulates the evolution of the supply chain system and also controls the decisions and actions of the agents through predefined policies. Alternatively, the Flow Simulator can simulate the evolution of the supply chain by fetching information from a gamette client that captures the decisions of human players. Following the approach in our prior work [68], we created a gamette with StudyCrafter², where players take the role of a wholesaler in a drug delivery supply chain. Details of the gamette design are described in Section 5. The same gamette is used for Study 1 and Study 2; the difference is in their experimental design (i.e., Study 1 considers the disruption location and various forms of information sharing; Study 2 considers different behavioral profiles for decision-makers and how they respond to information sharing).

4 Thought Bubbles

We designed thought bubbles with three main characteristics: (1) diegetic, (2) verbal and open-ended input, and (3) over time. Figure 2 demonstrates the design of thought bubbles and its characteristics. To ensure diegetic data collection, we included thought bubbles (see

in Figure 2) as part of a recurrent meeting scene in the gamette where players would first review their performance (see

in Figure 2). This feedback is provided to players through interaction with other Non-Player Characters (NPCs) during the gameplay phase and includes factual information (i.e., historical graphs and the current state of supply chain parameters), without introducing any form of bias into the players’ performance. After performance review, the Boss NPC would deliver the thought bubble prompt (see

in Figure 2). In terms of aesthetics, we did not design a thought bubble in literal terms. Instead, we chose to request players’ thoughts via NPC dialogue as it felt natural to request this prompt as part of the meeting scenario, where the results are discussed, helping keep players in the context of the game.

To evoke players to reflect on their experience, we prompted them by asking, “How do you think we are doing Kate?” (see

in Figure 2) and allowed them to provide open-ended input (see

in Figure 2). We chose an open-ended question and response format to mitigate framing bias [90] and help players to articulate their thought process by engaging associative memory [66]. The analyses we present in this paper center around players’ responses to this question. Finally, players experienced the meeting scene once every four weeks (8 times total) throughout the entire game (see

in Figure 2), allowing us to collect data on their thought process over time. We chose a four-week interval because, in doing so, we could frame the experience as a monthly meeting scene which felt more natural and resulted in the minimum distraction of players from their decision-making task. Moreover, our supply chain experiment includes a lead time of two weeks for ordering decisions (one week for orders to be processed by the manufacturer and one week for players to receive shipments). By considering a four-week interval, we allow players to have sufficient experience and observe the short-term outcome of their decisions before each thought bubble. A video preview of the gamette and the thought bubbles is available in the OSF repository (https://osf.io/btfzx/?view_only=8211d2334d5440a0b75ae947811cb845).

Figure 2:

5 Study 1

The motivation behind Study 1 is to understand the role of human behavior in pharmaceutical supply chains experiencing drug shortages by studying the mental model development of decision makers. Specifically, we aim to elicit—through the thought bubbles—an individual decision maker’s mental model and how this changes over time as they interact with the environment. To this end, we simulated a pharmaceutical supply chain network using the Flow Simulator and used a gamette to immerse human participants into a specific role within this network. We considered a supply chain network that includes two manufacturers, two distributors, and two health centers. Figure 3 illustrates the network structure and the direction of shipments between each entity in this supply chain. We use this network because it permits the analysis of the supply chain network in conjunction with the behavioral dynamics that ensue from having multiple agents within each echelon [22].

5.1 Methods

5.1.1 Hypotheses.

Drug shortages can occur for different reasons. However, most shortages can be traced back to supply chain disruptions [89]. Many researchers have attempted to mitigate the impact of drug shortages through increasing supply chain resilience [88], implementing decision support systems [14], and developing optimal inventory management and ordering policies [5]. Another key, and often understudied, element is the behavior of human decision-makers, which can prolong or aggravate the effect of shortages [22]. Such behaviors are exemplified in our previous research where we showed people tend to deviate from optimal order suggestions [68], and that these behaviors are attributed to hoarding and panic buying when facing shortages [67]. This is due to people’s tendency to ignore the temporal dynamics of the system when forming mental models of the supply chains [37]. Additionally, different disruption locations can create different local dynamics for each role within the supply chain [73, 80]. These local dynamics present unique experiences for players, ultimately affecting their mental models and influencing their inventory management decisions, which contribute to the propagation of shortage effects throughout the rest of the supply chain. So our aim is to explore how players’ mental models are affected when they are in different positions relative to the supply chain disruption, and hypothesize that:

H1. People show differences in their mental model development of a disrupted supply chain, depending on the disruption location.

Previous research on drug shortage management also points to insufficient information sharing between stakeholders in pharmaceutical supply chains as a source for disruption propagation [94], and that many of the costs associated with managing shortages in such supply chains can be mitigated by increasing collaboration and sharing information [69]. Hence, information sharing is considered as one of the essential strategies to improve the resilience of supply chains [50]. Information that is shared between supply chain stakeholders adds to the agents’ management experiences in shaping their mental models of the surrounding supply chain dynamics. Thus, it is important to investigate the influence of information sharing on mental model development and the behavior of decision-makers. We expect that sharing various levels of information results in the development of different mental models. Hence:

H2. People exhibit differences in mental model development of a disrupted supply chain, depending on the level of information being shared with them.

Figure 3:

5.1.2 Experimental Design.

The experimental setting for our study is summarized in Table 1. Players in all conditions play the role of Wholesaler 1 (WS1; see Figure 3), and all other supply chain roles are assigned to agents that the Flow Simulator controls. The simulation agents make decisions based on order-up-to-level policy. According to this policy, each agent orders or produces enough product to bring their inventory position to a predefined level based on a periodic review policy with zero fixed costs [81]. We considered different scenarios for testing the effect of disruption location on players’ mental models (H1), where disruption can happen in either Manufacturer 1 (MN1) or Manufacturer 2 (MN2). We created these disruptions in the form of a manufacturing shutdown which reduces the production capacity of the disrupted manufacturer by 95%.

Table 1:

	Condition 1 (MN1/No-Info)	Condition 2 (MN2/No-Info)	Condition 3 (MN1/Partial-Info)	Condition 4 (MN2/Partial-Info)	Condition 5 (MN1/Complete-Info)	Condition 6 (MN2/Complete-Info)
Player Role	WS1	WS1	WS1	WS1	WS1	WS1
Disrupted Manufacturer	MN1	MN2	MN1	MN2	MN1	MN2
Information Sharing	No	No	MN1 inventory	MN1 inventory	MN1 inventory + HCs behavior and delivery rates	MN1 inventory + HCs behavior and delivery rates
No. of Participants	17	19	20	21	18	20

Table 1: Summary of the experiment settings for Study 1.

We also designed the two health-center agents (HC1 and HC2) in the Flow Simulator with different ordering behaviors to generate different local dynamics for the WS1 role based on disruption location. The difference in health-center behavior is related to how each health-center agent splits orders between wholesalers. While both health-centers receive a constant demand, the HC1 agent splits its orders to its upstream agents by means of a trustworthiness measure, meaning that HC1 orders less from the upstream wholesaler that fails to deliver drugs consistently. HC2, on the other hand, splits its orders always equally regardless of its wholesalers’ trustworthiness. This trust-based behavior of HC1 particularly affects players during the shortage period (weeks 32-36) but differently depending on disruption location. When MN1 is disrupted, in addition to incurring stockout cost for not satisfying demand, players will also experience a decrease in HC1’s demand. When MN2 is disrupted, players experience an increase in HC1’s demand. According to these dynamics, we considered different options for testing the effect of information sharing on mental model development (H2): (1) without; (2) with information sharing on MN1 inventory (Partial Info); and (3) with information sharing on MN1 inventory, health-centers’ ordering behavior, and delivery rates to each health-center (Complete Info). Figure 4 and Figure 5 illustrate our designs for different levels of information sharing.

5.1.3 Participants.

We recruited 115 participants (87 males, 26 females, and 2 not stated). Participants were full-time students enrolled in undergraduate and graduate level courses of Logistics and Supply Chain Management at Northeastern University. The age range is 18 to 32 years (M=22.72, SD=2.01).

5.1.4 Incentive Design.

Previous research points to the importance of incentives in conducting experimental research [57]. To motivate participants to engage with the task and perform well, we offered them a monetary incentive ($50) which was gifted through a raffle. Players who performed better had a higher chance in the raffle. Each participant received one ticket for completing the game plus one ticket for every $1000 in-game profit that they made, more than the average profit of all other players.

Figure 4:

Figure 5:

5.1.5 Procedure.

All participants first visited the study website, where they were formally briefed about the experiment and its purpose. By starting the game, each participant was randomly assigned to one of the six conditions. Table 1 shows the number of participants assigned to each condition. The gamette in all conditions looked the same in all aspects except for the ordering scene, the allocating scene, and part of the tutorial that provided information on health-centers’ ordering behavior. Participants played the role of a character named Kate who was hired as a supply chain director in a wholesaler company. At the beginning of the game, an NPC (Kate’s boss) expresses that the game’s goal is maximizing the company’s profit by minimizing the inventory and stockout costs and maximizing sales revenue. The NPC also informs players, through dialogue, about the sales revenue and cost breakdown, and the lead time of two weeks for orders (one week for orders to be processed by the manufacturer and one week for players to receive shipments).

Each participant first played four weeks of tutorial to familiarize themselves with the game (i.e., how to gather information) and was given instructions about ordering from the manufacturer and allocating to health-centers. Participants were asked to make an ordering decision at each period, but only make allocating decisions when their inventory level was lower than their total demand. In cases where they had enough inventory, the game automatically allocated drugs to each health-center. After finishing the tutorial, all sales and cost data was reset, and they played the game for 35 game weeks, starting at Week 21 of the simulation. After placing the order on Week 24, rather than moving to Week 25, the game transited players to the monthly meeting scene (Figure 2) where they were queried via the thought bubble after receiving information on their performance via NPC dialogue. Players transited to this meeting scene every four weeks, with Week 52 being their last meeting. The disruption started on Week 28 and ended on Week 33. We framed the disruption as a manufacturing shutdown due to COVID-19.

At each period, participants received a shipment from their upstream manufacturer and could review the inventory, demand, and backlog information. Next, if they had limited inventory compared to demand, they were asked to select one of the presented allocation policies: (1) allocate to HC1 first, (2) allocate to HC2 first, or (3) allocate proportionally to each health center’s demand. Finally, they received an order suggestion according to the order-up-to-level policy for making an ordering decision. They had the option to order the suggested amount or modify it. The gamette sends player decisions to the Flow Simulator, which moves the simulation to the next period and sends back the updated parameters for the next round to the gamette. After playing for 35 weeks, through dialogue, an NPC debriefed participants about the study, experimental setting, and their performance.

5.1.6 Data Analysis.

To analyze players’ responses to the thought bubbles prompt (“How do you think we are doing Kate?”), we first applied Initial Coding [78], where we qualitatively coded the open-response comments from players. We followed an inductive approach in our coding practice, where we used a combination of in vivo and constructed codes. Three researchers coded all players’ comments across the eight prompts, leading to 94 initial codes. In further making sense of this data, we considered how these initial codes can lend insight into mental models by looking into existing theories and frameworks. We leveraged the Situation Awareness (SA) model introduced by Endsley [28] (see Section 2.2) to categorize our initial codes. We used three levels of SA to determine whether our generated codes reflect players’ perception of the supply chain (in the game), their comprehension of the state relating to the game’s goals, or their ability to project the future state of the supply chain.

After assigning each code to one of the three SA levels, we determined what aspect of the supply chain (i.e., topic) is described in players’ responses (e.g., profit, cost, inventory, demand, order, general, etc.) and how it is described through either a verb or adjective (i.e., description/action). The descriptions represented the rate of change (e.g., increase, decrease, constant), connotation (e.g., positive, negative, neutral) or specific actions (e.g., over-order, improve, anticipate, etc.) related to the supply chain topic. For example, we determined the comment “inventory cost goes high” to reflect a player’s perception (SA level) of an increase (description) in inventory cost (topic). The process took place by examining the codes through several group meetings. We did not seek to calculate an inter-rater reliability (IRR) as our generated codes were the process and not the product [64]. We use the codes as a passage to get to the aspects of situation awareness and, from there, to get insight into mental model development. Table 2 summarizes the result of our qualitative analysis.

To understand players’ mental model development, we performed quantitative analyses in three steps: (1) counting the number of codes representing each aspect of SA in player comments and performing hypothesis testing to investigate associations; (2) studying patterns by examining the counts of codes temporally and across the eight measurements; and (3) explaining the patterns and associations by examining the topics/descriptions of SA aspects as represented in players’ comments. All comments, codes, and generated categories (i.e., topics and descriptions/actions) through our qualitative analysis are available in the OSF repository (https://osf.io/btfzx/?view_only=8211d2334d5440a0b75ae947811cb845). In Section 5.2 and in Table 2, we refer to participant quotes as “(Player ID, gender, age).” For example, a female player aged 23 with player ID 85 who participated in Study 1 would be displayed as “(PL1-85, female, 23).”

Table 2:

SA Level	Topic	Description/Action	Example
Perception	inventory cost, backlog cost, costs, profit, inventory, demand, backlog, order	increase, decrease, consistent, zero, over-order, under-order	“there is uncertainity [sic] in supply, thats [sic] why our inventory cost goes high, we need to talk with supplier.”–(PL1-18, male, 23) “I think we did pretty well. The profit was constant and the cost was minimal. All orders were satisfied.”–(PL1-112, female, 23)"

Comprehension	general, inventory, demand, backlog, supply line, order	positive, negative, neutral, uncertain	“Awful. Need to introspect and make the necessary changes.”–(PL1-3, male, 24) “We are doing fantasticand meeting demandsregularly with enough saftey [sic] stock in handto satisfy sudden hike in the demand.”–(PL1-2, male, 24)"

Projection	general, profit, inventory, demand, backlog, order, allocation	improve, anticipate problem/uncertainty, increase, decrease, constant, uncertain, proportionally, HC with higher delivery rate, HC2	“We have more inventory since the manufacturer 2 is closed due to COVID-19 so we have to prepare for the incresing [sic] of Healthcare 2 demand.”–(PL1-47, female, 25) “Profit is going down and performing poorly due to high backorder costs. I need to order more.”–(PL1-98, male, 23)"

Table 2: Aspects of mental models identified by applying SA to qualitative responses of players to the thought bubbles.

5.2 Results

Participants wrote on average six words per prompt (Mdn=3, IQR=7). The thought bubble prompt was left unanswered in 5.5% of the instances across the eight measures, spreading across 19 participants. The mean time players spent in the meeting scene was 55.1 seconds (SD=69.7), which included the time they reviewed their performance and responded to the prompt. We report these statistics to illustrate how thought bubbles are utilized. The following sections describe what we learned from players’ responses in Study 1.

We used the outcome of our qualitative analysis and mental model development to compute the frequency of players’ comments that reflected each aspect of SA depending on the disruption location (see Table 3). Chi-square test of independence revealed significant association between disruption location and players’ mental models, as captured by SA via thought bubbles (${\chi }^2_{2, 1011}=12.047$, p = .002), and indicated a small effect (Cramer’s V = .109) [15]. More specifically, players who experienced a disruption in their own supplier (i.e., MN1) wrote significantly more comments reflecting their comprehension of the environment (p < .001). These comments mostly indicate general negative expressions, such as:

“Awful. Need to introspect and make the necessary changes.”–(PL1-3, male, 24)
“very bad.”–(PL1-66, female, NA)
“We’re doing terribly since our supplier is unable to provide more than 20 shipments per weeky [sic] order.”–(PL1-74, male, 21)
“We are not recieving [sic] orders, that’s big problem.”–(PL1-13, male, 24)

Higher comprehension frequency in the comments of this group (i.e., MN1 disruption) does not necessarily reflect a better understanding of the environment and instead shows the level of uncertainty or inability to make sense of the environment towards goals. We also investigated the progression of mental models over time (see Figure 6-A2). We realized that most negative expressions of players with MN1 disruption are mainly observed at the end of the shortage period (i.e., Week 36). This can indicate that the generated dynamics by disruption in MN1 create more uncertainty for players to the extent that they primarily comprehend their environment as negative. Therefore, the source of disruption affects each player’s mental model development relative to their comprehension.

Table 3:

		SA Level
Manipulation		Perception	Comprehension	Projection	Total
Disrupted Location
MN1 (n = 55)	Count	69.0	358.0\rlap^**	38.0\rlap^*	465.0
	Expected	79.1	334.3	51.5	465.0
MN2 (n = 60)	Count	103.0	369.0\rlap^**	74.0\rlap^*	546.0
	Expected	92.9	392.6	60.4	546.0
Information Sharing
Complete (n = 38)	Count	60.0	244.0	34.0	338.0
	Expected	57.5	243.0	37.4	338.0
Partial (n = 41)	Count	80.0\rlap^*	246.0\rlap^**	51.0	377.0
	Expected	64.13	271.1	41.7	377.0
No-Info (n = 36)	Count	32.0\rlap^**	237.0\rlap^**	27.0	296.0
	Expected	50.3	212.8	32.8	296.0
Total	Count	172.0	727.0	112.0	1011.0
	Expected	172.0	727.0	112.0	1011.0

Counts show the presence of each SA level in players’ responses. Expected represents the expected counts for the Chi-square test under the null hypothesis (i.e., no association between manipulations and SA levels.)
^*Chi-square Post-hoc test shows significant association between experimental manipulations and SA level at α = 0.05 with Bonferroni correction.
^**Chi-square Post-hoc test shows significant association between experimental manipulations and SA level at α = 0.01 with Bonferroni correction.

Table 3: Contingency table of frequency of players’ comments representing each aspect of Situation Awareness (SA), separately for implemented manipulations and across the eight thought bubbles prompts in Study 1.

Our analysis also showed that a disruption in MN1 is associated with significantly lower projection, as evidenced by players’ comments (p = .003). Such discrepancy can be mainly attributed to the comments at the end of the shortage period (see Figure 6-A3), where comments from players with a MN1 disruption in Week 36 reflect significantly fewer projections (p = .002). As players with a MN2 disruption experience an increase in their demand, their projections represent topics such as “anticipating uncertain demand”, “anticipating demand to increase,” or “planning to over-order”:

“Doing well so far. I expect it will be difficult once the demand becomes uncertain due to covid19.”–(PL1-12, male, 25)
“demand from HC1 is increasing and we have to supply accordingly.”–(PL1-56, male, 24)

Finally, the chi-square post-hoc test did not show a significant difference in the frequency of players’ perceptions depending on disruption location when considering all thought bubble responses. However, we found that players with MN1 disruption represent significantly lower perception in their responses during Week 36 (p = .005) (see Figure 6-A1). Players experiencing a MN1 disruption seem to only perceive the “increase in their backlog”, “increase in backlog cost”, and “decrease in profit”:

“So bad, first I wanted to keep the demand of HC1, but then I found more demand increase more backlogs which cost more money.”–(PL1-7, male, 22)
“Net profit is negative because of backlog cost.”–(PL1-113, male, NA)

The disruption location affects the state of the supply chain for a given point in time such that players in different groups experience different dynamics. Our results show that these emergent dynamics caused by different disruption locations drive the development of different mental models in how players comprehend the environment and project into the future in general, and how they perceive the system when facing a shortage—thus providing support for H1.

5.2.1 Mental Model Development vs. Level of Information Sharing (H2).

Chi-square test of independence also showed a significant association between the level of information and mental model development (${\chi }^2_{4, 1011}=19.172$, p < .001). The effect size for this finding was small (Cramer’s V = .097). The results of the chi-square post-hoc test (see Table 3) did not reveal a significant association between complete information sharing and the development of mental models. However, our results indicated that the responses of players who only received information on their supplier inventory (i.e., partial information) represented significantly higher perception (p = .003) and significantly lower comprehension (p < .001). As for the perception, players in the Partial-Info group seem to speak to almost all topics more frequently than players in No-Info group, as exemplified in:

“We are in very high loss but may fulfil their requierments [sic] as manufacturing unit have more units readily available.”–(PL1-44, male, 22)
“we have done really well over the past 2-3 weeks. we’ve been able to keep up with demand while maintaining low costs. I will look to keep the inventory low for the time being.”–(PL1-73, male, 21)

The higher perception of players in this group is also reflected in the average count of comments over time. Players in the Partial-Info group show an increase in the average number of comments related to perception when the shortage starts in Week 32 (see Figure 6-B1). On the other hand, the lower frequency of comments related to comprehension in the Partial-Info group is mainly reflective of general positive comments:

“the past several weeks have been good, although manufacturer 2’s accident in Puerto Rico may change things up...”–(PL1-73, male, 21)
“Its [sic] look market is working good but still we keep high inventory due to uncertainity”–(PL1-44, male, 22)

These players seem to be more cautious in interpreting the environment as positive because of having access to partial information. Finally, chi-square post-hoc test indicated that not having access to information is associated with significantly lower perception (p < .001) and significantly higher comprehension (p < .001). Higher comprehension of players in the No-Info group reflects codes “general-positive” and “general-negative”. As the results indicate, not having access to information makes it challenging for players to make sense of the environment. On the other hand, partial information directs players’ attention to perceive more, and complete information has no effect, perhaps due to information overload [28]. These results show partial evidence for H2, where information sharing on supplier inventory affects mental model development of disrupted supply chains.

Figure 6:

5.3 Summary

Study 1 served as the foundation for exploring the use of thought bubbles for eliciting mental model development. We obtained a codebook (Table 2) by examining players’ comments through qualitative analysis and grounding it in SA theory. This codebook represented the aspect of players’ situation models in our decision-making task, which has shown to be directly related to mental models [30]. It is crucial to test the reliability of this approach for eliciting mental model concepts. Hence, we will use our codebook to qualitatively code responses to the thought bubbles prompt for new players in Study 2. Study 1 also demonstrated how experimental manipulations affect mental model development over time. More specifically,

•

a disruption in players’ suppliers affects their mental model development by directing them to comprehend the environment as unfavorable and projecting less about how the future unfolds.

•

receiving information on supplier inventory directs players’ attention to key features of the environment (i.e., more perception) and causes them to be more cautious in how they interpret the situation.

•

sharing no information causes players to perceive the elements of the system less and face difficulty in making sense of the environment.

In examining mental model development in Study 1, however, we did not consider the observed behavior of players. Therefore, in the subsequent study, we examine if the observed decisions of players—as reflected in their behavioral profiles—are the outcome of different mental model development processes.

6 Study 2

To get insight into how mental models drive human decisions, we need to investigate another key aspect influencing mental model development: individual differences. Prior research provides evidence of different decision patterns among supply chain decision makers [67, 85]. Therefore, we conducted Study 2 to explain the cognitive aspect of decisions made by players with different behavioral profiles, while testing the reliability of our generated codebook in Study 1. We also made some changes to the experimental setting of Study 1. First, since we did not find evidence for the effect of complete information sharing, we decided to drop that as a manipulation for Study 2. Second, we decided to limit the disruption location to MN1 to isolate the generated dynamics for easier comparison of mental model developments across players with distinct behavioral profiles.

6.1 Methods

6.1.1 Hypotheses.

Individuals show different behaviors which might be affected by various factors [4]. In the context of dynamic decision-making, one can attribute these behavioral differences to the changes in mental model development [79]. The literature around SA also points to individual characteristics affecting SA [27, 30], which in turn can affect mental model development. Therefore, we expect to see associations between observable behavioral profiles and mental model development as captured via thought bubbles. Hence, we hypothesize:

H3. People show differences in mental model development of a disrupted supply chain dynamics, depending on their behavioral profile.

We retrieve behavioral profiles following the methods we proposed in our previous work [67] (see 6.1.4) to test this hypothesis. Furthermore, in continuation of Study 1, we investigate how information sharing affects mental model formation for different behavioral profiles. While we did not find significant association between complete information sharing and SA levels, we found that access to partial information (i.e., supplier’s inventory level) is associated with significantly higher perception and lower comprehension. In our previous study, we provided empirical evidence on how information sharing affects the behavior of players with different decision patterns [67]. Consequently, we expect the effect of information sharing on observable actions to be reflected in players’ mental model development as captured via thought bubbles. Hence:

H4. People with different behavioral profiles show differences in mental model development of a disrupted supply chain, depending on the level of information being shared with them.

We test these hypotheses using the same approach as in Study 1.

6.1.2 Experimental Design.

Table 4 summarizes the experiment settings for Study 2, where we considered a disruption in MN1 and two options for information sharing: (1) without, and (2) with information sharing on MN1 inventory. This dataset contains players’ responses to the thought bubbles prompt collected following the same procedures described in Section 5.1.5. All other aspects, including supply chain roles and disruption settings, are identical to Study 1 (see Section 5.1.2).

6.1.3 Participants.

Through Prolific ³, we recruited 135 (57 males, 74 females, 2 non-binary, 2 not stated) online participants. We only allowed participation for working professionals who had an undergraduate degree or higher and reported English as their first language or were fluent in it. Each participant spent, on average, 58 minutes (SD=19.35) playing and received a $7.5 reward for their participation. The age range is 21 to 71 years (M=34.23, SD=9.86). We provided participants with the same incentive as in Study 1 (see Section 5.1.4) and followed the same procedures (see Section 5.1.5).

Table 4:

	Player Role	Disrupted Manufacturer	Information Sharing	No. of Participants
				Hoarders	Reactors	Followers	Total
Condition 1 (No-Info)	WS1	MN1	No	31	21	9	61
Condition 2 (Info)	WS1	MN1	Yes	25	27	8	60
No. of Participants				56	48	17	121

Table 4: Summary of the experiment settings in Study 2.

6.1.4 Data Analysis.

The three researchers involved in data analysis of Study 1 qualitatively coded players’ open-response comments (a total of 1093 comments) by using the previously generated categories as a codebook (see Table 2). Each researcher used a tuple in the form <SA Level, Topic, Discription/Action> to code each comment within the dataset. In case a player’s comment reflected multiple topics or SA aspects simultaneously, we used multiple tuples to represent them. Our goal was to evaluate whether data are being interpreted in the same way relative to the previous analysis in Study 1. Therefore, per the recommendations of McDonald et al. [64], we attempted to measure inter-rater reliability (IRR) by calculating Fleiss’ kappa (κ) [35]. After the first round of coding, we obtained κ=0.75. While this showed a relatively high agreement rate, we examined the codes once more to address sources of disagreement. After our second attempt, we obtained κ=0.84, which we considered sufficient for our hypothesis testing.

The steps for our quantitative analysis were similar to those mentioned in Study 1 (see 5.1.6). However, to decide which set of codes to use for the rest of our analysis, we considered two heuristics: (1) using the majority vote (i.e., codes in which two out of three coders agreed upon), and (2) using codes from the master coder (i.e., first author) in addition to the majority vote. During our analysis, we realized that both heuristics show similar results. Therefore, for simplicity, we chose to move forward with the first heuristic. Players’ comments and qualitative codes for Study 2 can be accessed in the OSF repository (https://osf.io/btfzx/?view_only=8211d2334d5440a0b75ae947811cb845). To obtain the behavioral profile of players, we followed the methods from our prior study [67]. We first filtered outlier players (n=14) and then used a Hidden Markov Model (HMM) to infer players’ response modes in their deviation from order suggestions (see Figure 4 for an example of order suggestion). Finally, through cluster analysis of sequences of response modes, we found three behavioral profiles: (1) Hoarders who deviate from order suggestions and over-order more frequently, (2) Reactors who follow order suggestions but react to the disruption news and deviate from recommendation only after disruption, and (3) Followers who almost always follow order suggestions. For more detailed description of the methods for retrieving behavioral profiles we refer to [67]. Table 4 shows the count of players characterized by each behavioral profile across the two conditions. Moving forward, we investigate the mental model development (as captured with thought bubbles) of these three profiles in testing our hypotheses.

6.2 Results

Similar to Study 1, we investigated how thought bubbles were utilized among Study 2 participants. On average, players wrote five words in each of their comments (Mdn=2.00, IQR=5.25). The thought bubble prompt was left unanswered in 3.8% of the instances across the eight measures, which were spread across 12 players. Players spent, on average, 56.4 seconds (SD=47.3), where they reviewed their performance and responded to the prompt.

6.2.1 Mental Model Development vs. Behavioral Profiles (H3).

We first investigated the mental model development of players with different behavioral profiles. Table 5 shows the frequency of players’ comments that reflect each aspect of SA and separated for Hoarders, Reactors, and Followers. Chi-square test of independence showed a significant association between behavioral profiles and players’ situation model (${\chi }^2_{4, 1069}=18.132$, p = .001), and indicated a weak dependence (Cramer’s V = .092). Specifically, Hoarders’ comments demonstrate significantly lower comprehension (p = .005). According to our prior study [67], Hoarders over-order more frequently compared to other players. Such behavior is associated with a poor understanding of the system’s dynamics, especially when facing a complex decision-making environment [84]:

“Terrible, the ordering system has made it too confusing to track where the saline is in the system, so I don’t know how I can be expected to turn a profit like this”–(PL2-30, male, 27)
“Bad. The slow supply of saline has messed things up. I’ll have to keep ordering more.”–(PL2-56, female, 32)

Even if they comprehend their situation as positive, they seem to give more weight to future uncertainties and, as a result, behave proactively. Followers, on the other hand, show significantly higher comprehension (p < .001) and lower perception (p < .001) in their comments (See Table 5). The comprehension aspect of the Followers’ situation model is mainly reflected in the code “general-positive” as exemplified in comments such as “good”, “excellent”, “great”, “very good”. This could explain why Followers trust the order suggestions and do not deviate: in their mind everything is going well and there is no reason to change. In addition, because they comprehend the state as positive more frequently, they may have less incentive to pay attention to the elements of the system, which can explain their lower perception. Next, we examined the progression of mental model development as evidenced by SA levels (Figure 7). The disruption seems to have triggered an increase in perception, especially for Hoarders and Reactors, which is later reduced during the shortage period (see Figure 7-A). The majority of the comments reflecting perceptions during the disruption period center around “backlog-increase” or “profit-decrease” codes (see Figure 9-A1):

“The reduced supply from our manufacturers is affecting our supply chain. We could not meet the demands in the past month.”–(PL2-26, male, 32)
“We need to get more order to deliver to out customer as we have a backlog we need to supply.”–(PL2-90, male, not stated)

Hoarders also show a peak in their projection level right after being notified about the disruption on Week 28 (see Figure 7-C), which is mainly related to “anticipating problem/uncertainty” (see Figure 9-C1):

“So far everything has been smooth, but I anticipate a shortage of inventory due to Covid at our supplier, so I have upped the most recent order to be prepared.”–(PL2-105, female, 54)
“Good, ordered more inventory this past week incase [sic] supply chain runs into issues over the next few weeks due to covid.”–(PL2-39, female, 22)

As it appears again, Hoarders are more proactive when facing uncertainty. Reactors fall behind in their projections, although showing more projections toward the end (see Figure 7-C). Their projections predominantly concern improvement as captured by codes “general-improve” and “profit-improve” (see Figure 9-C2):

“Badly but hopefully the situation is stabilising”–(PL2-32, female, 62)
“The problem will be solved very soon”–(PL2-68, male, 48)

Such comments, however, illustrate a lack of clear strategy in Reactors’ projections. While they show similar trends in perception levels as Hoarders (Figure 7-A), Reactors’ mental model development over time does not seem to enable them to be strategic about the future. In sum, these results support H3: we observe that behavioral profiles result in a different mental model development.

Table 5:

			SA Level
	Profile/Manipulation		Perception	Comprehension	Projection	Total
Behavior Profile	Hoarder	Count	88.0	387.0\rlap^*	56.0	531.0
		Expected	75.5	404.8	50.6	531.0
	Reactor	Count	60.0	333.0	42.0	435.0
		Expected	61.8	331.6	41.5	435.0
	Follower	Count	4.0\rlap^**	95.0\rlap^**	4.0	103.0
		Expected	14.6	78.5	9.8	103.0
Information Sharing per Behavior Profile	Hoarder
	Info (n = 25)	Count	57.0\rlap^**	163.0\rlap^**	36.0\rlap^*	256.0
		Expected	42.4	186.5	26.9	256.0
	No-Info (n = 31)	Count	31.0\rlap^**	224.0\rlap^**	20.0\rlap^*	275.0
		Expected	45.5	200.4	29.0	275.0
	Reactor
	Info (n = 27)	Count	21.0\rlap^*	185.0\rlap^**	14.0	220.0
		Expected	30.3	168.4	21.2	220.0
	No-Info (n = 21)	Count	39.0\rlap^*	148.0\rlap^**	28.0	215.0
		Expected	29.6	164.5	20.7	215.0
	Follower
	Info (n = 8)	Count	3.0	46.0	2.0	51.0
		Expected	1.9	47.03	1.9	51.0
	No-Info (n = 9)	Count	1.0	49.0	2.0	52.0
		Expected	2.0	47.9	2.01	52.0
	Total	Count	152.0	815.0	102.0	1069.0
		Expected	152.0	815.0	102.0	1069.0

Counts show the presence of each SA level in players’ responses. Expected represents the expected counts for the Chi-square test under the null hypothesis (i.e., no association between manipulations and SA levels.)
^*Chi-square Post-hoc test shows significant association between behavioral profiles or experimental manipulations and SA level at α = 0.05 with Bonferroni correction.
^**Chi-square Post-hoc test shows significant association between behavioral profiles or experimental manipulations and SA level at α = 0.01 with Bonferroni correction.

Table 5: Contingency table of frequency of players’ comments representing each aspect of Situation Awareness (SA), separately for implemented manipulations and across the eight thought bubbles prompts in Study 2.

Figure 7:

6.2.2 Mental Model Development vs. Information Sharing per Behavioral Profile (H4).

To study how information sharing affects the mental model development of players with different behavioral profiles, we followed the same steps as in previous analyses. First, we counted the presence of each SA aspect in Hoarders’ comments, where we found a significant association between information sharing and SA levels (${\chi }^2_{2, 531}=21.216$, p < .001), indicated by a small effect (Cramer’s V = .200). According to chi-square post-hoc test (Table 5), being a Hoarder and having access to information is associated with significantly higher perception (p < .001), significantly less comprehension (p < .001), and significantly higher projection (p = .005). In addition, comments from Hoarders in the Info group show fluctuations of SA aspects over time (see Figure 8-A1-3). We can see an increase in the perception aspect of these Hoarders after the disruption notification (i.e., Week 28 - see Figure 8-A1). We noticed that comments from Hoarders in the Info group do not directly reflect the perception of the supplier inventory. However, their comments do show signs of becoming more proactive and perhaps using the information on supplier inventory to hoard before shortage affects their supply:

“Good, ordered more inventory this past week incase [sic] supply chain runs into issues over the next few weeks due to covid.”–(PL2-39, female, 22)
“we have as much available as possible heading into uncertain times.”–(PL2-50, female, 35)

This pattern is also represented in the projection aspect of Hoarders in the Info group (Figure 8-A3). It seems access to information affects their mental model development by adding to their uncertainties:

“The shutdown of the factory was a suprise [sic]. I’ll have to make sure we keep a good supply.”–(PL2-56, female, 32)
“Okay, but concerned about future supply issues.”–(PL2-60, female, 35)

As mentioned before, poor understanding of the system’s dynamics makes hoarding behavior more likely. Hoarders in Study 2 also represent this poor understanding through generic descriptions/adjectives such as “general-positive”, “general-negative”, “general-neutral” (see Figure 9-B2). Providing information to Hoarders does not necessarily increase their understanding of the system. It does, however, seem to reduce their lack of understanding by directing their attention to other aspects (through increased perception) and leveraging information to make larger safety stocks in preparation for future uncertainties (i.e., projection).

We also found a significant association between information sharing and mental model development for Reactors (${\chi }^2_{2, 435}=14.122$, p < .001), which was indicated by a small effect (Cramer’s V = .180). In addition, the chi-square post-hoc test revealed that being a Reactor and having access to information is associated with significantly lower perception (p = .004) and higher comprehension (p < .001)(see Table 5), which is the exact opposite of Hoarders. While the results for projection is not significant, average counts of SA aspects over time (see Figure 8-B1-3), display an increasing trend in the number of comments reflecting a projection for Reactors in both groups (Figure 8-B3). Reactors in the Info group show an increase in their perception followed by a decline after the shortage (Figure 8-B1). The content of their perceptions mainly reflects “profit-decrease” and “backlog-increase” (see Figure 10-A2):

“We incured [sic] a lot of losses this month. we are not able to meet up with the demands from the health centers.”–(PL2-26, male, 32)
“not great our backlog is large”–(PL2-4, female, 36)

While Reactors in the Info group show significantly higher comprehension, the content of their comprehension is mainly summarized in “general-positive”, “general-negative”, and “general-neutral” codes similar to Hoarders. These codes are represented in comments such as “good”, “perfect”, “fine”, “bad”, “Not good”, and “terrible”. Such comments can be again a sign of poor understanding of the system’s dynamics. In contrast with Hoarders, it seems information sharing does not direct Reactors’ attention to other elements as much (through perception), which affects how they comprehend their state. Reactors in the Info-group also show a declining trend in the counts of their comprehension-related codes over time (Figure 8-B2). Reactors start to hoard when facing the disruption, but it seems that those who have access to their supplier inventory rather wait and see how things unfold instead of being proactive. Both Hoarders and Reactors engage in hoarding behavior as represented in their actions. However, they show different hoarding patterns that seem to originate from different mental model development, which is affected by information sharing.

Finally, the chi-square test of independence did not show a significant association between information sharing and mental model development of Followers (${\chi }^2_{2, 103}=1.085$, p = .58, Cramer’s V = .103). We report the chi-square test result to be consistent with previous analyses. However, we acknowledge that the contingency table for Followers (Table 5) violates the assumption of the chi-square test by having expected frequencies less than 5. We also performed Fisher’s exact test [34], which similarly revealed no significant association (p = .65). It seems information sharing does not affect Followers mental model development. They continue to believe that ordering the same as the system’s suggestion is the best strategy. The average counts of SA aspects over time also did not reveal a specific trend (Figure 8-C1-3). These results supports our hypothesis (H4) by showing how information sharing is associated with mental model development depending on the behavioral profile.

Figure 8:

6.3 Summary

Study 2 demonstrated the reliability of our qualitative coding approach for measuring mental model concepts as evidenced by a sufficiently high agreement rate (κ=0.84). We also connected mental model development to player actions by characterizing Hoarders, Reactors, and Followers from the lens of their observed behavior. Our results demonstrated that differences in decision patterns result from differences in mental model development. In particular, Hoarders’ proactive behavior is the outcome of more uncertainty about the future as they fail to make sense of the situation. In contrast, the Followers’ mental model development directs them to interpret the situation positively and trust the order suggestions. Disruption also affects players’ mental model development over time, directing Hoarders’ and Reactors’ attention to elements of the environment, evidenced by increased perception (see Figure 7-A).

Study 2 also complemented Study 1 by providing evidence on how the effect of information sharing on observed behavior is, in fact, the outcome of the mental model development of players over time. More specifically,

•

Hoarders exhibit higher perception, lower comprehension, and higher projection when having access to their supplier inventory, leading them to be more proactive and stockpile.

•

Reactors display lower levels of perception, and higher comprehension, which directs them towards less strategic behavior and merely react to uncertainties.

•

Followers’ mental model development does not seem to be affected by information sharing.

These results are aligned with evidence we found in our previous research for the effect of information sharing on the observed behavior of different player types [67]. Using thought bubbles, however, we opened a window on mental model development (as a cognitive construct) over time and demonstrated how information sharing plays a role in explaining such behaviors from a cognitive aspect.

7 Discussion and Conclusion

The results from our studies provided empirical support for the use of thought bubbles. A striking finding was the complete opposite patterns in mental model development of Hoarders and Reactors. Our results not only complemented the behavioral patterns from our prior research [67], but provided a deeper theoretical understanding of the cognitive aspects of human decisions in their interaction with dynamic systems. Understanding cognitive constructs of mental models is instrumental in making sense of human behavior and has been of interest to the HCI field for decades [46]. This is an elusive task and requires elicitation of mental models and analyzing elicited concepts. We used thought bubbles for mental model elicitation by evoking players’ thinking and capturing verbalization of their thought processes. Of course, many other researchers have also studied elicitation of mental models [41, 55]. Our work is distinguished from these studies in the sense that we attempted to elicit mental models in (1) a diegetic setting—in the context and during the interaction, (2) using an open-ended prompt to collect verbalization of thought processes, and (3) over time.

The concept of elicitation is fundamental in studying mental models. In the HCI context, numerous studies on eliciting mental models exist [6, 40]. However, most of them approach the elicitation process in a non-diegetic form, for example, using interviews before or after the interaction. While a comparison with non-diegetic elicitation techniques was out of the scope of our work, we showed how diegetic elicitation allows for dynamic assessment of mental model development over time, and by having minimal interruption to the user. The idea of diegetic elicitation pertains to prior research studying how elicitation procedures affect mental models. Jones et al. [55] provided evidence on how the interview process affects mental model representations and found out that out-of-context elicitation pertains to mental models stored in long-term memory. Doyle et al. [24] argued that interviewers or additional information after the task can impact subjects’ mental models, and thus, subjects should be isolated from any out-of-context influences. Therefore, non-diegetic elicitation may measure mental models that are different from what players relied on during the interaction. In addition, considering the context of our study (i.e., dynamic decision-making), it made sense to elicit mental models in situ and diegetically, as in this context people continuously interact with the decision environment which affects their mental model development. We designed thought bubbles as part of a meeting scene (Figure 2) to ensure keeping players in the context.

We chose to elicit mental models through a verbal process, asking players to articulate their thinking. We found inconsistencies in prior research regarding the value of verbal elicitation. Some researchers advocated for the use of textual data and argued that language is key to understanding mental models [12]. Others have pointed to the complex nature of mental models as a cognitive construct that makes it difficult for individuals to articulate them [66]. While we acknowledge that not all aspects of mental models can be verbalized, our results demonstrated that the parts that are verbalized provide much insight, especially about the mental model development of people with different behavioral profiles. We accomplished this by designing thought bubbles with an open-ended prompt to mitigate contextual and framing biases [66] and to avoid taking players out of context. Although one might be skeptical of players’ engagement with thought bubbles in this format, despite a somewhat negative trend, we found evidence for continuous engagement across the eight prompts (see Appendix C). An interesting finding was differences in the average number of words per comment depending on behavioral profile and manipulations. For example, hoarders in the Info group and Reactors in the No-info group wrote more words per comment (see Figure 12).

Using thought bubbles, we collected data on players’ thought processes over time and in intervals. Mental model elicitation over time is particularly important, considering that mental models are dynamic constructs [19]. They dynamically form based on new information and as the result of interaction with the environment. Therefore, elicitation over time allowed us to consider this dynamic nature and opened up an opportunity to study mental model development as a process. Of course, the way mental model measurement should be conceptualized (i.e., as a process or an outcome) depends on the research question and the underlying task. However, prior studies also seem to advocate for the process view on mental models, pointing to their dynamic nature [83], or because of measurement errors arguing that measuring the change in mental models through repeated elicitation is preferred [24]. Our results demonstrated the benefit of elicitation over time by showing how experimental manipulations and behavioral profiles influence the mental model development of players over time, as represented by the aspect of their situation model (see Figures 6-8).

From a methodological perspective, our approach for testing thought bubbles involved three main elements: (1) game environment, (2) dynamic decision-making task, and (3) interaction with a complex dynamic system. Future research can leverage thought bubbles by adapting any of these elements with some considerations. First, while games are particularly useful setting for knowledge elicitation [91], we can envision implementing thought bubbles in a non-game environment; for example, as part of a simulation interface where subjects provide textual input (in intervals) on their thought process. However, we argue that the characteristics of simulation games (here gamettes)—including their realism in representing the system, ability to foster communication, and active involvement [62]—can lead players to engage with thought bubbles differently in a game setting compared to a non-game one. Future research can study how well a non-game interface can capture thought processes in relation to the task with thought bubbles.

Second, we studied the use of thought bubbles in a dynamic decision-making task. When studying a different type of task, one must ask if the focus is on measuring mental models as a process or an outcome [83]. If the focus is on the mental model representation and their content or structure in relation to the task—while we advocate leveraging thought bubbles for diegetic elicitation—querying subjects in intervals might be less relevant. Although considering the studies that cast doubt on the externalization of mental models as a static construct [24], and prefer the process view on mental models [83], future research must scrutinize to what extent eliciting mental model development as a process can be justified for other types of tasks. Another key consideration is whether the goal is to understand mental models or improve the mental models of the task. While improving mental models was not the purpose of our study, we showed how experimental manipulations such as information sharing affect mental model development. Therefore, future research should look into how manipulations can improve mental models and enhance performance, perhaps by incorporating smart nudging interventions [65]. Another way is to leverage AI for different framing of the prompt, especially as prior research argued how wording and framing of questions could affect mental models [24].

The type of task also affects the choice of framework for analyzing elicitation results. We used the Situation Awareness model introduced by Endsley [28] as it maps well to mental models theory and dynamic decision-making context, and hence, suited our study in which participants interacted with a dynamic decision task. However, when the task is not dynamic decision-making (e.g., gestural interaction with a display [82]), SA might be less relevant. While we advocate for grounding the qualitative analysis process in theories, the choice of underlying theoretical framework in non-dynamic context requires more scrutiny. We demonstrated the reliability of our approach by using our generated codebook from Study 1 to a new dataset (Study 2) and providing insight on mental model development of players with distinct behavioral profiles.

Finally, we studied players’ mental model development in their interaction with an interactive environment simulating a complex dynamic system (i.e., supply chain simulation that evolved both spontaneously and as the result of players’ actions). However, HCI encompasses many forms of interaction [44]. Future research can study other types of interaction in which thought bubbles can be leveraged, perhaps with some adaptions. For example, researchers can investigate the use of thought bubbles for diegetic elicitation through non-verbal queries such as card sorting [63], or diagrammatic representations [77]. In addition, recent advances in AI have inspired many researchers to study human-AI interaction [1, 86] and investigate mental models of AI [40, 92]. Thought bubbles can be utilized to advance our understanding of users’ mental model development of AI over time. Thought bubbles can also be viewed as a form of reflection, and reflection can be used to improve learning [45]. Therefore, future research can look into opportunities in which thought bubbles can be utilized for reflection in game-based learning technologies [43]. For all this to work, we need to think about how to scale up the use of thought bubbles with respect to qualitative analysis, which is a time-consuming process. Future studies can look into leveraging natural language processing (NLP) for qualitative analysis [20, 42, 61]. Last but not least, thought bubbles can potentially elicit more than mental models of the tasks at hand by uncovering cognitive aspects such as attitudes and motivation, which is another interesting future avenue, as reflected in the following quote:

“This is bad. I guess I was scared about having enough supply. I over ordered and now we are paying for it in surplus inventory. I should have gone to veterinarian school like my family wanted! But no! I was all like ”I’m going to be a great supply chain manager one day””

Acknowledgments

This research was supported with funding from the National Science Foundation (NSF: 1638302 & 2028449). We further thank the StudyCrafter and Drug Shortage teams at Northeastern University.

A Distribution of Qualitative Codes Over Time for Behavioral Profiles

Figure 9:

B Distribution of Qualitative Codes Over Time for Behavioral Profiles and Depending On Information Sharing

Figure 10:

C Average Count of Word Per Comments

Figure 11:

Figure 12:

Footnotes

Here we use Situation Awareness and Situation Model interchangeably.

StudyCrafter is a platform where users can easily create, play, and share gamified projects. It can be accessed at https://studycrafter.com

www.prolific.co

Supplementary Material

MP4 File (3544548.3581570-video-preview.mp4)

Video Preview

Download
20.21 MB

MP4 File (3544548.3581570-talk-video.mp4)

Pre-recorded Video Presentation

Download
192.54 MB

MP4 File (3544548.3581570-video-figure.mp4)

Video Figure

Download
75.10 MB

References

[1]

Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300233

Abstract

1 Introduction

2 Related Work

2.1 Mental Models

2.2 Dynamic Decision Making in Interactive Systems

3 Gamettes

4 Thought Bubbles

5 Study 1

5.1 Methods

5.1.1 Hypotheses.

5.1.2 Experimental Design.

5.1.3 Participants.

5.1.4 Incentive Design.

5.1.5 Procedure.

5.1.6 Data Analysis.

5.2 Results

5.2.1 Mental Model Development vs. Level of Information Sharing (H2).

5.3 Summary

6 Study 2

6.1 Methods

6.1.1 Hypotheses.

6.1.2 Experimental Design.

6.1.3 Participants.

6.1.4 Data Analysis.

6.2 Results

6.2.1 Mental Model Development vs. Behavioral Profiles (H3).

6.2.2 Mental Model Development vs. Information Sharing per Behavioral Profile (H4).

6.3 Summary

7 Discussion and Conclusion

Acknowledgments

A Distribution of Qualitative Codes Over Time for Behavioral Profiles

B Distribution of Qualitative Codes Over Time for Behavioral Profiles and Depending On Information Sharing

C Average Count of Word Per Comments

Footnotes

Supplementary Material

References

Cited By

Index Terms

Recommendations

Acoustic Methods for Assessment of Bubbles Produced by Marine Plants

Building Maintenance Management: Development of an Integrated Model of Procedures

Gestão de Recursos Energéticos Para Apoio à Operação Dos Virtual Power Players

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

HTML Format

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations