Introduction

Through interactions with the surrounding world, humans learn about things that they can influence though their own actions as well as things they cannot change through their behavior. Perceived controllability is a belief that one can control the outcome of a specific situation through their behavior [1] and plays an important role in how we behave and feel [2]^, which can be seen, for example, in learned helplessness theory [3]. Classically, controllability has been thought to be estimated or inferred from the learned relationship between one’s own action toward the environment and following contingent outcomes [4]. We perceive a situation as controllable if our action and outcome are correlated in which the correlation between an action and an outcome is computed retrospectively by using previous action-outcome history [5]. We will call this kind of controllability inference ‘single-agent controllability inference’, meaning that there is only one agent that interacts with the environment during decision making. Then, if people perceive the world as controllable, they are motivated to exhibit more proactive and cognitively demanding strategies because they expect that the outcome depends on their behavior, while low controllability promotes reactive or helplessness behavior because in the uncontrollable situation [1,3,4]. Animal studies and neuroimaging studies have shown that single-agent controllability inference and its influence on decision making involve a corticostriatal dopaminergic system such that the striatum detects the controllable situation and the ventromedial prefrontal cortex (vmPFC) modulates behavior according to controllability [3,4].

However, unlike the situation in the single-agent controllability inference, the usual decision-making situation in our real life involves multiple agents who can influence the decision outcome, including myself and other people. In this case, to identify whether I have controllability over the outcome or not, we have to exclude the possibility that other people except myself might have controllability by observing other people. We will call this kind of controllability inference ‘multi-agent controllability inference’ in contrast to the single-agent controllability inference. Considering classical studies showing the influence of controllability in various aspects of our behavior and emotion, establishing a mechanism of multi-agent controllability inference is important because it is much more common than single-agent controllability inference. However, no previous studies have investigated computational models of multi-agent controllability inference and how the brain implements this inference process. Therefore, in this study, we investigated these questions using the novel multi-agent Bayesian controllability inference (MABC) model and model-based fMRI analyses.

Based on previous studies, we formulated the following hypotheses. First, we expected that to infer multi-agent controllability, people might have to leverage other people’s action-outcome correlation information in addition to one’s own action-outcome correlation information. If so, our next question would be the computational mechanism of integrating these multiagents’ (both self and others’) behavioral information. Similar to the Bayesian integration model of multiple sensory information [6], we expected that the multi-agent controllability inference would be explained as a Bayesian inference of posterior controllability given prior and the multiple agents’ action-outcome observations. As a possible candidate neural mechanism, we expected that multi-agent controllability inference would recruit corticostriatal circuits, meaning the generalization of these regions’ roles from single-agent controllability to multi-agent controllability. Furthermore, considering that integrating others’ action-outcome information might be crucial in multi-agent controllability, we hypothesized that recruitment of regions previously known for processing others’ behavioral information might be necessary, such as the temporoparietal junction (TPJ). This region is known for theory of mind [7] and integrating others’ dissent information during decision making [8]. Finally, as in previous single-agent studies, we expected that this inferred multi-agent controllability would also facilitate motivated behavior and influence emotion. In particular, considering that utilizing value is cognitively costly (especially when values of actions are similar) [9], we expected that a motivation to utilize value would be increased by multi-agent controllability and that the emotion that arises from social situations, such as guilt, would be affected by controllability.

To test these hypotheses, we designed a novel task involving two participants (N = 103, Fig 1A, Materials and Methods) in which only one participant had control over the outcome (reward or loss), meaning that only one person’s behavior can influence the outcome. The controller or person in control changed every 20–30 trials throughout the task, and the person who had true controllability was not revealed to the participants; thus, the participants had to infer the true controllability structure using the action-outcome history of both the participants themselves and the other person. Note that from the perspective of one participant (e.g., player 1; among two participants), blocks of trials in which a participant had true controllability were called “self-controllable” blocks, and the blocks of trials in which another person had controllability were called “other-controllable” blocks. Participants were occasionally asked about who would be in control in the current context (causality choice, Fig 1) or asked about their mood. Behavioral analyses, including computational modeling and functional MRI analyses (fMRI, N = 30), based on our computational model were performed.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Experimental paradigm and graphical summary.

(A) Two-person decision-making task with changing controllability. Two participants simultaneously performed the task in a separate room. Although they expected to be playing with each other, they actually played with an artificial reinforcement learning agent. Participants were instructed to perform the two-alternative choice task such that each person would make a choice separately and the resulting outcome (either reward or loss) would be the only common element for them. Participants were told that in every trial, only one person among them had control over the outcome (“controller” or “person in control”), while another person’s choice would not influence the outcome (“person without controllability”), and the controller could change after some trials. There was always an optimal option, but only for the controller, that resulted in reward with high probability (80% reward/20% loss, actual probability not known to participant) and a suboptimal option that resulted in loss with high probability (80% loss/20% reward); there was no optimal option for the person not in control. Therefore, the choice made by the controller was important for both of them, while the choice made by the person not in control did not matter. The controller was switched every 20–30 trials. To assess perceived controllability, participants were asked who caused the outcome every two or three trials (causality choice, Task 2), and they were also asked to rate their current mood every two or three trials (Task 1). (B) True controllability structure and the location of optimal choices in self-controllable blocks. The gray shaded area represents the self-controllable block. The optimal option only existed in self-controllable blocks. Note that in this figure, task 1 (100 trials) and task 2 (100 trials) were concatenated for visualization (1–100 trials in this figure are task 1 and 101–200 trials are task 2). (C) Conceptual summary of the experiment. An agent observed two kinds of interactions–an interaction between self and environment and the interaction between other person and environment. The TPJ and striatum integrate both types of information to infer multi-agent controllability. Then, inferred controllability information was utilized to influence motivated behavior in the vmPFC.

https://doi.org/10.1371/journal.pcbi.1009549.g001

Results

Behavioral effect of others’ action-outcome contingency on controllability perception

Participants correctly identified who controlled the situation (true controllability) significantly better than chance (mean controllability accuracy (sd): 59.5% (10); t[9.70] = 102.00, p<0.0001, CIs: 0.076 to 0.115 in one-sample t-test testing whether difference between accuracy and chance level is above 0).

Then, we expected that the correlated relationship between participants’ own action and outcome would increase perceived controllability, while the correlated action and outcome of other persons would decrease perceived controllability. Here, a correlated relationship means that one’s specific action (e.g., left) is consistently associated with the same outcome (reward), while the other action (right) is consistently associated with another outcome (loss). The correlation index (CIx) of each trial was defined such that CIx = 1 at trial k means that a participant showed correlated action-outcome for two consecutive trials (trial k-1 and k), whereas CIx = 0 means the opposite (Materials and Methods). Both CIx of self (CIx_self) and other person (CIx_other) was calculated in every trial. By using a mixed-effect logistic regression, we tested the fixed effect of CIx_self, CIx_other and their interaction on participants’ perception as a controller (selecting “Self” in the causality choice). Consistent with previous studies, we found that observing correlated own action and outcome increased the probability of perceiving oneself as controllable (beta = 1.48, t[4080] = 11.91, p<0.0001, Fig 2A). Importantly, however, observing correlated others’ action and outcome decreased the probability of perceiving oneself as controllable (beta = -1.78, t[4080] = -15.04, p<0.0001, Fig 2A), showing that others’ behavioral information is utilized during controllability inference in addition to their own behavioral information. The interaction effect was not significant (beta = 0.29, t[4080] = 1.86, p = 0.0626). Note that the correlation between CIx_self and CIx_other did not reach statistical significance (Pearson r_partial = -0.00, p = 0.866).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Behavioral analysis results.

(A) Opposite effect of correlated self- and others’ action outcomes on perceived controllability. Mixed-effect logistic regression showed that correlated action-outcome of oneself (CIx_self = 1) significantly increased the probability of perceiving oneself as controller, while correlated action-outcome of other person (CIx_{other = 1}) significantly decreased this probability, meaning that both action-outcome information is utilized in multiagent controllability inference. A-O in this figure represents action outcome. (B) Difference in perceived controllability between reward and loss. The proportion of trials perceived as self-controllable was higher after reward than after loss. (C) Effect of perceived controllability on optimal choice. The proportion of optimal choices was higher in the trials after participants selected that they had caused the outcome than after selecting other persons who had caused the outcome.

https://doi.org/10.1371/journal.pcbi.1009549.g002

Furthermore, we found that reward biases people to perceive them as a more controller (t[102] = 9.14, p<0.0001, CIs: 0.185 to 0.287, d = 0.901, paired t-test comparing the proportion of selecting oneself as a controller between reward and loss trials preceding causality rating trials, Fig 2B). Therefore, we added reward and its interaction with both CIx as a fixed effect in previous mixed effect logistic regression, which not only showed a similar effect of CIx_self and CIx_other (positive and negative, respectively; all p<0.0001) but also showed a positive interaction between reward and CIx_self (beta = 0.96, t[4077] = 5.27, p<0.0001), meaning that reward amplifies CIx_self’s effect on controllability perception (or inference).

Finally, considering the effect of controllability on motivation, we expected that when participants perceived themselves as a controller, their motivation to make better choices would increase optimal choices. Indeed, participants were more likely to make optimal choices when they believed they had control (compared optimal choice proportion between trials right after selecting “Self” in the causality choice and trials right after selecting “Other” in the causality choice using paired t-test, t[102] = 4.88, p<0.0001, CIs: 0.078 to 0.184, d = 0.481, Fig 2C). Here, the optimal choice was defined as choosing an option that was designed to have a higher probability of reward than the other option when one had true controllability. Therefore, this optimal choice can be defined only within self-controllable blocks. Additionally, by considering the possibility that participants might have not learned which option is optimal in early trials of self-controllable blocks, we performed the same analysis only using the late self-controllable block (final 10 trials of self-controllable blocks), which showed a consistent result as the previous one (t[85] = 2.24, p = 0.0280, CIs: 0.011 to 0.186, d = 0.260, paired t-test).

Multiagent Bayesian controllability inference model

Behavior analyses suggest that observations regarding the action-outcome relationship for both oneself and others are utilized for controllability inference. We derived a controllability inference under a multi-agent setting as a Bayesian inference of posterior controllability given both the observed self- and other’s action-outcome history and the prior, which is similar to the multisensory integration model (see S1 Text for posterior derivation). We also showed that although our task situation involves only two agents, this model can be applied to a situation involving three or more people (see S1 Text for this generalization). A key aspect of this multi-agent Bayesian controllability inference model (MABC model) is that an agent’s posterior controllability is determined by the ratio between the likelihood of the current outcome given one’s own action (self-likelihood; e.g., “how likely is it that my choice has resulted in the current reward if I assume that I was in control?) and the likelihood of the current outcome given other people’s action (other-likelihood; e.g., “how likely is it that other people’s choice resulted in the current reward if I assume that the other person was in control?”) such that a relative increase in self-likelihood increases the inferred controllability (Fig 3A and 3B). These likelihoods can be represented by (1) and (2) respectively. These likelihoods were computed using the learned action-outcome relationship (p(outcome|action)) in which learning was based on the classical static-learning rate reinforcement learning rule [10].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Computational model of multi-agent controllability inference.

(A) Schematic description. Action of self or action of another person caused an outcome according to the true controllability. Then, an agent obtains observations including outcome, own action and other’s action. In the multi-agent controllability inference module, the agent computes self-likelihood and other-likelihood using this information. Finally, both likelihoods are integrated in a Bayesian fashion with bias to estimate multi-agent controllability. In our task, this estimated multi-agent controllability was utilized not only to make causality choices but also in the next trial as a predicted controllability, which increased value utilization during the value-action conversion to make value-dependent choices. (B) Example trajectory of the inferred controllability. This plot shows the example trajectory of the controllability (blue line) extracted from the winning MABC model. Red “x” is the participant’s causality choice, and the black line is the true controllability. (C) Effect of correlated action outcome on inferred controllability. Similar to the behavior analysis, correlated own action-outcome increased inferred multi-agent controllability, while correlated others’ action-outcome decreased it. (D) Bias of the controllability inference. Both biases were significant such that bias_pos>0 while bias_neg<0, meaning that people tend to prioritize self-likelihood during multi-agent controllability inference after reward, which increases perceived controllability, while the opposite tendency appears after loss. (E) Controllability-dependent value utilization. τ was larger than zero, meaning that the predicted controllability increases the value utilization during decision making.

https://doi.org/10.1371/journal.pcbi.1009549.g003

Then, they were combined to make posterior controllability inferences as follows: (3)

The above equation shows that the posterior inferred controllability increases as the relative likelihood () increases. A detailed description of the MABC model can be found in the Materials and Methods. Each likelihood computation was accomplished by using a learned action-outcome relationship (Fig 3A and 3B).

Using this model as our core controllability inference hypothesis, we constructed thirty-one computational models that include models that do not consider others’ behavior (S2A Fig). The winning model was the MABC model with reward biases and controllability-guided value utilization (protected exceedance probability (PEP) = 1, Materials and Methods). This model successfully replicated the effect of CIx_self and CIx_other on perceived controllability in our previous behavioral analyses (Fig 3C and S1 Text). Bias parameters represent the degree of relative weight on self-likelihood compared to other-likelihood during multi-agent controllability inference. A bias parameter greater than 0 means that they would overestimate their own controllability by overweighing the self-likelihood compared to other-likelihood and vice versa. bias_pos is the bias after reward, and bias_neg is the bias after loss. Consistent with the behavior analysis results, participants’ inferred controllability was overestimated after reward by overweighing self-likelihood (t[102] = 4.47, p<0.0001, CIs: 0.126 to 0.326, d = 0.441, one-sample t-test of bias_pos, Fig 3D) while underweighting self-likelihood when the outcome was loss (t[102] = -5.03, p<0.0001, CIs: -0.357 to -0.155, d = -0.495, one-sample t-test of bias_neg, Fig 3D).

Furthermore, in a winning MABC model, high inferred controllability of the previous trial (predicted controllability) increased value utilization, meaning an increased deterministic tendency of choosing an option with high subjective value. More specifically, predicted controllability increased the inverse temperature of the decision to increase value utilization, and the degree of inverse temperature modulation by predicted controllability was parametrized by τ (τ>0 means that predicted controllability increases the inverse temperature and vice versa; note that τ is not an inverse temperature itself but its modulator, Materials and Methods). τ was significantly above 0 in the one-sample t-test, meaning that predicted controllability increased value utilization (t[102.00] = 8.66, p<0.0001, CIs: 0.568 to 0.906, d = 0.854, Fig 3E).

Reward-driven controllability overestimation bias improves adaptability at the cost of illusory control

Essentially, overestimation bias might cause the illusion of control in an uncontrollable environment, ultimately wasting one’s effort. Indeed, bias_pos was positively correlated with the illusion of control (proportion of selecting “Self” in the causality choice when the true controller was the other, Pearson r_partial = 0.39; p<0.0001, Fig 4, left), while negative bias was not (p>0.5; see S1 Text for more explanation). Note that all correlation analyses we performed were partial correlations using sex as a covariate. Then, what is the advantage of this bias? We expected that in an environment where the controller often changes, positive bias might facilitate a fast recognition of change from an uncontrollable environment to a controllable environment, leading to an increased optimal choice by value utilization.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Effect of reward-driven positive inferential bias.

Participants with high positive bias exhibited a higher level of predicted controllability both in other-controllable blocks and self-controllable blocks compared to the level of predicted controllability in participants with low positive bias (middle). Although such bias induced an illusion of control in other-controllable blocks (left), it helped participants to rapidly regain controllability in self-controllable blocks (middle) and increased decision optimality in those blocks (right).

https://doi.org/10.1371/journal.pcbi.1009549.g004

Consistent with our expectation, positive bias was significantly correlated with choice optimality in the second half of the self-controllable block (Pearson r_partial = 0.26; p = 0.0080, Fig 4, right), while the first half was not (Pearson r_partial = -0.18; p = 0.0648), showing that a controllability-induced value-dependent decision only influences the optimal choice when one has sufficiently learned about the value of options. Additionally, a negative bias was not related to choice optimality (p>0.05 for both the first and second halves). These results show that although positive bias increases the illusion of control when another person has true controllability, it improves adaptability by promoting fast recognition of the return of true controllability.

Neural substrate of multi-agent controllability estimation

In the model-based fMRI analyses using log-odds of inferred multi-agent controllability (log-controllability ratio, LCR in short) estimated from the winning MABC model as a parametric modulator (GLM1, see Materials and Methods), we found activity of the left TPJ, [–50, –22, 26], cluster-level FWE-corrected p = 0.032, Z-value of the peak voxel = 4.37, Fig 5A and S2 Table) and the right striatum ([18, 8, –6], cluster-level FWE-corrected p<0.001, Z-value of the peak voxel = 4.82, Fig 5A). These controllability signals were greater at 1 s after the outcome presentation compared to the time right after the outcome (0 s) (t[29.00] = 3.84, p = 0.0006, CIs: 0.194 to 0.635, d = 0.702, for the right striatum and t[29.00] = 2.66, p = 0.0125, CIs: 0.052 to 0.395, d = 0.486, for the left TPJ, in paired t-tests between 0 s and 1 s after outcome, Fig 5A).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Neural mechanism of multi-agent controllability inference.

(A) Regions involved in multi-agent controllability inferences. The BOLD signals of the right striatum (lower left) and left TPJ (right left) covaried with the inferred controllability estimated from the winning MABC model. Both regions’ controllability inference-related signals increased from 0 s to 1 s after the outcome (bar graphs with gray color). There was a unidirectional Granger causality between two regions (yellow arrow) (B) Bayesian model selection results. Bayesian model selection showed that the MABC-based neural model explains the activity of both regions much better than a single-agent controllability model (PEP = 1 for the MABC-based model in both regions), meaning that these regions are related to utilizing multiagent information rather than single-agent information only.

https://doi.org/10.1371/journal.pcbi.1009549.g005

In the following region of interest (ROI) analyses, we further found unidirectional Granger causality between these regions (from TPJ to striatum, S7 Fig and S1 Text).

Controllability estimation improves value utilization in the vmPFC

The winning MABC model showed that the predicted controllability of the next trial improves value utilization. Basically, people choose more deterministically the option with higher subjective value when the value difference (VD) between the high-valued option and low-valued option is high. Improving value utilization means that the effect of VD on deterministic decisions would be increased by the predicted controllability, meaning a positive interaction between VD and predicted controllability. At the neural level, we expected that this effect would be represented as a similar positive interaction effect, meaning an increase in the VD signal by the predicted controllability.

In GLM1, parametric modulators, including VD, predicted controllability and their interaction, were locked at the cue presentation time (Materials and Methods). We identified that the value utilization region that encodes VD involves the vmPFC ([0, 52, –12], cluster-level FWE-corrected p<0.001, Z-value of the peak voxel = 5.18, red-yellow area in Fig 6A and S2 Table), which signal predicted participants’ value-dependent choice (choosing option with higher value, beta = 0.15, t[5973] = 4.51, p<0.0001, in the mixed-effect logistic regression, Fig 6C). Importantly, predictive controllability increased this VD signal in the vmPFC ([–2, 34, –14], small-volume corrected for the vmPFC cluster involved in VD, p = 0.012, Z-value of the peak voxel = 4.01, green area in Fig 6A and S2 Table), and this interaction effect was correlated with the value-utilization parameter (τ, Pearson r_partial = 0.38, p = 0.0441, Fig 6B), suggesting that controllability increases value utilization by enhancing the value difference signal in the vmPFC.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Neural mechanism of controllability-dependent policy change and reward-driven adaptability.

(A) Region increasing value utilization in response to increased multi-agent controllability. The BOLD signal in the vmPFC covaried with VD (red and yellow). Furthermore, small regional activity within the vmPFC (green) exhibited a positive interaction effect between predicted controllability and VD, meaning that this region’s VD-related activity was amplified by the predicted controllability. (B) Relationship between VD-related activity amplification and controllability-dependent policy modulation. The mean beta of the positive interaction effect within the vmPFC, which is the degree of VD signal amplification by the predicted controllability, was positively correlated with the parameter τ of the winning MABC model. (C) Neural mechanism of reward-driven policy change mediated by controllability bias. Mediation analysis results showed that the controllability inference signal in both the TPJ and striatum mediates the effect of reward (0 s after the outcome) on the vmPFC signal of the next trial, which was involved in value utilization. This was further supplemented by the fact that vmPFC signals increased actual value-dependent choices (red arrow). Note that black solid arrows represent significant relationships in the mediation analysis, and the red solid arrow represents significant relationships in logistic regression predicting value-dependent choice by vmPFC signal.

https://doi.org/10.1371/journal.pcbi.1009549.g006

Discussion

Identifying controllability becomes more complex when there are other agents who can influence the outcome compared to the single-agent setting. In this study, using the novel multi-agent Bayesian controllability model, we showed that people perform Bayesian inference to integrate multiple agents’ action-outcome profiles. MABC model-based fMRI analyses and the Bayesian model selection of neural models showed that the left TPJ and the right striatum perform multi-agent controllability inference by integrating both self- and other-likelihood information. Then, multi-agent controllability information increased motivated behavior, especially by facilitating value utilization in the vmPFC. We also found likelihood-encoding regions such that a decrease in other-likelihood was detected in the right MTG, while the self-likelihood computation involved frontostrial regions. Finally, our multi-agent controllability inference was positively biased toward the self, which increased the illusion of control. However, bias helped adaptability in the controllability-changing environment, whose effect was mediated by the multi-agent controllability inferencing areas. Furthermore, feelings of guilt in daily life were related to controllability inference bias (high negative bias and low positive bias). Therefore, in people with excessive guilt, increasing positive bias and decreasing negative bias might help reduce their guilty feelings.

An important contribution of our study is the identification of novel behavioral and neural mechanisms of inferencing controllability under a multi-agent setting. Previous single-agent controllability studies revealed that during an interaction with the environment, a correlated or deterministic relationship between an agent’s own action and outcome increases perceived controllability [4,5]. However, many decision-making situations involve multiple agents; thus, we usually consider other agents that could influence the outcome even when their existence is uncertain [13]. From the artificial intelligence perspective, recent studies of multi-agent reinforcement learning also show the importance of considering other agents because it is useful to make globally optimal decisions and resource allocation [14]. Interestingly, one recent study reported that a sense of controllability was reduced in the presence of others [15]. In that study, participants had to decide when to stop inflating a balloon to avoid a burst that resulted in a large loss. There were social trials with the presence of other persons involved in the decision-making situation (regardless of whether other persons can influence decisions or not) and nonsocial trials without other persons [15]. The authors showed that the sense of agency is reduced in social trials by mentalizing others and that the activity of mentalizing regions (e.g., the TPJ and the precuneus) was increased in social trials [15]. Although the authors suggest that the inference of controllability is influenced by the presence of others, they reported that the presence of others only “decreased” the sense of controllability because they did not consider how contingency between others’ action and outcome influences multi-agent controllability [15]. However, we showed that perceived controllability could be either increased or decreased by integrating the observed action-outcome relationship of other people. In particular, our MABC model showed that the Bayesian posterior of multi-agent controllability depends on the relative strength between self-likelihood information and other-likelihood information, meaning that one can feel low controllability even when their own action and environmental outcome is contingent if other-likelihood is high. This model is similar to the model of multisensory integration [6]. In multisensory integration, multiple sensory cues are integrated according to their precision or reliability to infer Bayesian posterior [6]. In this context, self-likelihood and other likelihood can be seen as the reliability of the controllability cue from each agent (self and other people). Another notable point is that the MABC model is not only applicable to situations with two-agent situations (self and other) but is also applicable to situations with a larger number of agents (more than two). In this case, other likelihood represents the likelihood that any other people except me caused the outcome (S1 Text). However, this generalized version of the MABC model might be unrealistic if there are too many agents (e.g., more than 10) because this model assumes that one agent simultaneously updates all other agents’ action-outcome contingency. Therefore, we should modify this model to enable realistic computation of other-likelihood in the large-sized group. For example, an attention-based learning mechanism can be applied to other-likelihood learning with an excessive number of agents [16].

From the neural mechanism perspective, we found that regions involved in single-agent controllability inference and controllability-dependent behavioral modification also perform a similar role in multi-agent controllability inferences and related behavioral change. Single-agent controllability studies have suggested that the corticostriatal circuit plays an important role [1,4] such that the striatum infers one’s controllability and the vmPFC mediates controllability-dependent behavior [3,17]. Our results generalized the role of these two regions from single-agent controllability to multi-agent controllability. In particular, MABC model-based fMRI analyses and Bayesian model selection showed that the striatum not only utilized its own action-outcome information but also integrated other people’s action-outcome information. Although this is the first study to show striatal involvement in multi-agent controllability inference, studies have shown that the striatum is involved in social comparison, such as comparing one’s reward gain with others [18], which is similar to the comparison between self-likelihood and other-likelihood in our MABC model. Furthermore, we also showed that using this multi-agent controllability information, the vmPFC increases value utilization in decision making. Interestingly, a recent study by Wang and colleagues also showed that the vmPFC is involved in the controllability’s influence on value [19], which is consistent with our result. However, in that study, controllability inflated value itself rather than increasing value utilization. Another study showed that vmPFC activity after decision outcome was modulated by internal control [2]. Those studies and our study show various ways that a vmPFC mediates the influence of controllability on decision making. In summary, our results suggest that the striatum and vmPFC play a generalized role in inferencing controllability and mediating controllability-dependent behavior in both single-agent and multi-agent settings.

The involvement of the left TPJ was the region that was not reported in previous studies of single-controllability inference. The left TPJ is necessary for reasoning about someone else’s belief [20] and has been known to be involved in understanding social intention [21]. A recent study also reported that both the left and right TPJ are involved in dissenting other people’s consensus opinions [8], and another study showed that left and right TPJ activity is involved in the mentalizing process during decision making in the presence of others [15]. These results are consistent with our result in that the left TPJ integrates other-likelihood information, which is a kind of reasoning regarding the likelihood of the current outcome from another person’s perspective.

Granger causality analysis showed predictive causality from the TPJ to the striatum, while there was no predictive causality in the reverse direction, suggesting the possibility that a multi-agent controllability inference signal in the TPJ might be predictive of a controllability inference signal in the striatum. However, because Granger causality does not tell us about the effective connectivity between regions [22], our results should be treated with great caution, and future studies might investigate their relationship using analyses such as dynamic causal modeling [23]. We speculate that controllability inference completed in these two regions might be conveyed to the vmPFC to increase value utilization.

In summary, we showed that the striatum and vmPFC are likely to act as controllability sensors and effectors, respectively, in both single-agent and multi-agent settings. However, in the multi-agent setting, an inference of controllability also involves the TPJ, and this inference within the TPJ and striatum utilizes both others’ action-outcome information and one’s own action-outcome information. The final notable point is that while the TPJ was involved in utilizing integrated self- and other-likelihood information, pure other-likelihood information was related to the right MTG, a region involved in simulating others’ minds [24].

One interesting extension that we can infer from our result regarding how multi-agent controllability inference influences behavior (value utilization) is the possible influence of the multi-agent environment on learned helplessness behavior. As the expected value of control theory suggests [25], there is no need to make deliberative and motivated behavior when one perceives no controllability over an environment. Therefore, many single-agent studies, including animal studies, have shown that if an agent’s own action does not seem to reliably change the environment, they exhibit learned helplessness behavior [3]. However, learned helplessness in humans usually occurs in multi-agent situations, such as social life, especially in highly competitive groups, and our results indicate that this social learned helplessness can be induced even in situations where environmental change highly correlates with one’s own action if another’s action-outcome correlation is higher. We expect that such social learned helplessness might involve the TPJ-striatum-vmPFC circuit, which can be investigated in future studies.

We also identified a neurocomputational mechanism regarding how reward-driven controllability overestimation bias leads to adaptive behavior under volatile true controllability. Positive bias facilitates the recognition of self-controllability when one regains true controllability and is able to make better decisions by increasing value utilization, which also reinforces self-likelihood, meaning a reward-controllability positive loop. However, this bias costed an illusion of control. When true controllability is variable and the controller is uncertain, failing to recognize a situation in which one has true controllability harms the outcome for both oneself and for the other person, while wasting inefficient effort when one does not have control only costs one’s own effort. We suspect that this asymmetry makes the fast recognition of controllability more important than the illusion of control, which might justify the presence of the positive bias. We also found that reward increases the controllability signal in the TPJ and striatum to increase value utilization in the vmPFC, which is consistent with our MABC model.

Furthermore, inference bias also played an important role in emotion. A recent study showed that one’s internal control belief increases the amount of happiness and pride after receiving reward [2], and another study showed a similar influence of controllability on guilt-related feelings, such as regret [26]. We reported similar results indicating that multi-agent controllability amplifies the outcome-related mood and that this effect was mediated by striatal activity. These biases overestimate self-controllability after a positive outcome and reduce it after a negative outcome bias one’s mood toward a positive direction. Again, like social learned helplessness we mentioned previously, this suggests how feeling in the social situation is influenced by others’ action-outcome relationship. Studies have suggested that such biases in perceptions of control are characteristics of the normal human brain that are beneficial for mental health [27,28]. Consistently, we have found that people with lower bias are prone to feel more guilt in their daily lives. Although speculative, we suspect that the low positive bias in social life could be related to depression, which is known to exhibit excessive guilt in social life [29].

The relationship between bias and guilt could be related to our study design in which both agents share the outcome. In this scenario, an agent might feel guilty because they inferred that their choice induced loss of another person. However, it is also possible that their guild came from the perception that they induced the loss of themselves. Our study design does not allow us to discriminate these two possibilities. Therefore, this interesting question could be investigated in future studies such that the outcome only goes to one of two agents.

One point that should be noted when interpreting our study is that our design assumed a situation such that only one person can control the situation. However, in the real world, there are other situations in which no one controls the outcome or the cooperation of multiple people, including oneself, influences the outcome, such as in cooperative games in game theory [30]. We expect that extending our model to such a situation could be interesting, and we speculate that in a cooperative situation, there could be multiple levels of controllability, including the overall controllability of the group and relative controllability of the self within the group. Furthermore, one of the limitations of our task is that although we had two sets of tasks having different controllability changing schedules that were randomly assigned across participants, both sets started with a self-controllable block, which might have allowed people to learn a self-controllable state first and an uncontrollable state next. However, this might have resulted in an undesirable blocking effect.

In conclusion, our study advanced previous studies by showing a novel neurocomputational mechanism of controllability inference in the multi-agent setting and its influence on behavior and emotion. The MABC model showed that people utilize both self-and other people’s action-outcome information to compute self-likelihood and other-likelihood by recruiting the frontostriatal regions and right MTG, respectively. Then, multi-agent controllability was inferred by integrating these two likelihood information in the TPJ and striatum. This inferred controllability information was leveraged to increase value utilization in the vmPFC, which resulted in optimal behavior. These results generalize the role of the striatum and vmPFC from single-agent controllability to multi-agent controllability such that involvement of the TPJ in addition to the striatum might help integrate self- and other-related information. We finally showed that innate positive bias facilitated behavioral adaptation in the controllability-changing environment at the cost of illusory control. Furthermore, such bias was related to negative social emotions such as guilt.

In the complex social world with multiple people involved in decision-making situations, identifying our controllability is difficult, and various kinds of psychiatric disorders, such as depression and anxiety, could arise under such complexity. Therefore, by showing how we perceive controllability in the multi-agent setting and how it affects our subsequent behavior and emotion, we believe that our results could benefit research on psychiatric disorders.

Materials and methods

Supporting information

Acknowledgments

The language of this manuscript was edited by American Journal Experts (AJE). The authors acknowledge R.R. for helpful discussions on developing the paradigm.