Data Storytelling in Data Visualisation: Does it Enhance the Efficiency and Effectiveness of Information Retrieval and Insights Comprehension?

For this study, we opted for a topic of broad interest: climate change and global warming, which are increasingly significant global issues. Some associated data can be challenging for the general public to comprehend without advanced visualisation skills [87]. We chose to use open-source data from the "Our World in Data" (2020) website¹ to help us determine if data storytelling could be beneficial in situations where participants’ understanding of the visualisation is crucial [87].

All visualisations without data storytelling elements (referred to as conventional data visualisations) were generated using Tableau software² [5, 41]. In contrast, visualisations featuring data storytelling elements (referred to as data stories) were modified using LucidChart³, a free graphic design tool, to incorporate these elements. The design of the visualisations was jointly agreed upon by all the co-authors of the paper in multiple iterations, making sure the data storytelling elements above were applied. Details about the specific visualisations used in the study are provided below.

First, participants were asked demographic questions, namely: i) self-reported gender, ii) age, iii) highest level of education, iv) region, v) career, and vi) whether their career is related to data analysis. We then inquired about participants’ self-assessed experience in data analysis and data visualisation (e.g., "choose the circumstance that best describes your experience with data analysis"). To enhance the response rate for demographic and background questions, we situated them in the initial section of the survey (Figure 2-Section 1).

The objective of this section was to estimate each participant’s visualisation literacy (VL) to address RQ5 (see Figure 2-Section 2). To estimate a participant’s VL pertinent to this study, we employed a modified version of the Visualisation Literacy Assessment Test (VLAT), developed by Lee et al. [56]. The original test covers 12 types of visualisations, each with 6 questions. However, some of these were not applicable to the visualisations featured in our study (e.g., a map of the United States). To avoid participant fatigue and focus on the specific visualisation types used in our study – line charts, bar charts, stacked bar charts, pie charts, stacked area charts, and bubble charts – we selected only the VLAT questions relevant to these visualisation types. This resulted in six sets of questions, each related to one of the six visualisation types. This approach ensured that the assessment measured participants’ abilities in the context of the visualisations they encountered later. We employed the corrected score of the VLAT to neutralise the impact of guessing, as suggested by Frary [34] and Lee et al. [56].

Conventional visualisations versus data stories. This consisted of the core questions of this survey (see Figure 2-Section 3). We designed 6 pairs of visualisations, each showcasing different data but conveying the same insight within a pair. Specifically, we created six conventional visualisations (cv1-6) alongside six alternate versions that applied data storytelling principles (ds1-6). These data stories were created following the transformation process to convert conventional visualisations to those with data storytelling elements, as proposed by [99] and detailed in Section 3.1. Two researchers participated in several discussion sessions to collaboratively identify the insights from each visualisation based on the data and metadata provided in the Our World in Data repository (phase i) to then apply the transformation process (phases ii and iii), ensuring that aspects such as the type of chart, decluttering actions, emphasis techniques, annotations, and explanatory titles all effectively contributed to conveying the insights. Later, a third researcher validated the insights and the data stories, and further participated in discussion sessions with the other researchers to agree on the final set of visualisations. All visualisations included the complete range of data storytelling elements discussed above. Two example conventional visualisation–data story pairs (cv1–ds1 and cv2–ds2) are shown in Figure 3. To facilitate a balanced exposure to both conventional and data storytelling visualisations, each participant was randomly presented with three items from each condition, and in a random sequence. For instance, one participant might be shown visualisations cv1, cv2, cv3 paired with ds4, ds5, ds6, while another could encounter cv4, cv5, cv6 and ds1, ds2, ds3. This method not only ensured an even distribution of both conventional visualisations and data stories but also helped mitigate any learning or practice effects. No explicit indication was provided to participants regarding which visualisation was conventional and which was a data story.

The data visualisations were designed to be diverse in terms of both visualisation complexity and frequency of use across varied audiences. To achieve this, and as described in the previous section, we included a range of chart types: line charts, bar charts, pie charts, stacked bar charts, stacked area charts, and bubble charts. According to Segel and Heer [85], line charts, bar charts, and pie charts are the most commonly used visualisation techniques and are relatively easy to understand [56]. Stacked bar charts and stacked area charts have been found to be slightly more challenging for participants, as indicated by the difficulty scores calculated by Lee et al. [56], yet they are also commonly used. Bubble charts have been considered more difficult to interpret, according to their difficulty scores, and are less commonly used [56]. This assortment of chart types provided a comprehensive range of visualisations (see details in Table 1). In our study, the distribution mirrored the commonality of these charts: we included three chart types that are most common, two that are relatively common, and one that is less commonly used. The choice of chart for the conventional visualisations was also influenced by the specific data being displayed. We intentionally designed the conventional mirroring of the original data source made public through Our World in Data.

We included text annotations, data point annotations, colour emphasis, and explanatory titles in all visualisations with data storytelling elements. Moreover, since the change of chart type can significantly affect visualisation and different chart types require distinct visualisation skills from participants [56], we limited the change of chart types to half of the visualisation sets (3 sets) (see details in Table 1, columns 1 and 2).

Types of questions. Each time a participant was presented with a visualisation, they were asked to respond to four multiple-choice questions. Each question was categorical, and the correct response was only one out of four predefined options. For each pair of visualisations, the same set of questions was asked consistently for both versions –the conventional visualisation and the data story – to different participants.

We defined these questions based on Bloom’s taxonomy [6]. Bloom’s taxonomy has been extensively used to categorise questions into varying levels of complexity (i.e., knowledge, comprehension, application, analysis, synthesis, and evaluation). It has also been proposed for assessing the kinds of tasks that data visualisation can enable [2, 15], as well as for supporting the development of visualisation skills [16, 63]. Following the guidelines proposed by Burns et al. [15], we designed two types of questions based on the first two levels (knowledge and comprehension) of Bloom’s taxonomy. We selected these two levels because they align with the goal of this research (RQ3). Further levels require other types of interventions and study designs that go beyond the purpose of the current study. In the context of information visualisation, knowledge (Level 1) questions in Bloom’s taxonomy serve to assess whether users can identify and select relevant data points, categories, or trends from a data set. This aligns with our goal to assess whether data storytelling supports information retrieval tasks. Comprehension questions (Level 2) relate to questions that assess the ability to understand, interpret, and derive meaningful insights from the data. Data comprehension, in this sense, goes beyond mere retrieval and involves interpretation, assessment, and decision-making. It involves identifying various data points, making comparisons, and understanding the underlying meaning, which is aligned with the comprehension task. Questions of this type were subdivided into two sub-types. The first sub-type asked participants about a single insight that the data visualisation aimed to provide and allowed the participants to choose the correct answer. The second sub-type requires participants to gain a comprehensive understanding of multiple insights. The question presents four statements, and the participants were required to identify the invalid statement.

For each pair of visualisations, we added two information retrieval (Level 1) questions and two comprehension (Level 2) questions (see Table 1, column 3). Table 2 (row 1) shows an example of an information retrieval question that requires the participants to retrieve a single data point from the visualisation and choose the correct value among the response options [1]. Table 2 (row 2) shows an example of a comprehension question that asks the participants to identify a single insight by comparing various data points. The comprehension question sub-types were equally distributed across all pairs of visualisations and depended on the number of insights that could be extracted from the data shown in each visualisation pair (see Table 1, column 3). Table 2 (row 3) presents an example in which a participant needs to identify the incorrect insight among the other three correct insights communicated in the conventional visualisation or the data story. Two researchers collaboratively designed these questions independently of the type of visualisation (conventional or data story). A third researcher validated these questions. Discussion sessions were conducted among the three researchers to reach a consensus on the final set of questions. They ensured there was an alignment between the questions asked and the information highlighted in the visualisations with data storytelling elements.

The time taken to complete each question was recorded to assess the efficiency of data storytelling (RQ1) [36, 105], while the correctness rate of responses was recorded to shed light on its effectiveness (RQ2) [36, 105].

In this section, we revealed to participants that three earlier visualisations contained data storytelling elements. We clarified this by displaying a side-by-side comparison of a conventional visualisation and a data story, both illustrated in Figure 1. We annotated each storytelling element within the visualisations to help participants associate the terms with specific visual cues. The highlighted elements were: a) annotated data points, b) colour/thickness emphasis, c) text annotations, and d) an explanatory title. Participants then shared their preferences on the most useful – or not useful at all – elements for extracting information, with the option to further explain their choices through an open-ended question.

We defined and calculated the following metrics to answer our research questions. All those metrics were calculated both for data storytelling and conventional data visualisation:

We employed the Shapiro-Wilk W-test to assess the normality of the correct rate, average completion time, and visualisation literacy scores [80]. Our analysis provided strong evidence that these variables were not normally distributed.

To address RQ1, we utilised R to compute the median and interquartile range (IQR) of the average success time metric, visualising the results with a box plot. Subsequently, we applied the Wilcoxon signed-rank test, suitable for paired data and capable of accounting for individual differences by using each participant as their own control, to evaluate whether the success time metric under data storytelling significantly differed from those observed with conventional data visualisations for the same participants. We adopted a p-value threshold of less than 0.05 to determine significance as guided by [67, 91]. We use the \(r (Z/\sqrt {N})\) as our effect size measure due to its suitability for non-parametric paired data comparison. For Q2, we replicated the methodological approach used for RQ1, focusing on the correct rate per participant as our metric of interest.

To address RQ3, we segmented the data according to question type (i.e., information retrieval and comprehension), as well as sub-types of comprehension questions (i.e., single insight versus multiple insights). We calculated the median and IQR for these segments and depicted the statistics using box plots. We then applied the same statistical tests as those used for RQ1 and RQ2 to assess differences in efficiency and effectiveness between data storytelling and conventional visualisations across varying tasks and levels of comprehension complexity. To account for multiple comparisons, we adjusted the p-values using the Bonferroni correction, multiplying the original p-values by 4 [44].

To address RQ4, we calculated the frequency of participant preferences for data storytelling elements that may have enabled quick identification of relevant information or facilitated accurate identification of insights. Additionally, we conducted a thematic analysis of participant explanations for the perceived utility of data storytelling elements, coding each of the 102 responses [13]. One researcher initially led this qualitative analysis, identifying emerging themes, while a second researcher cross-verified these themes and collaborated with the rest of the authors in finalising them, thereby ensuring both the reliability and validity of the thematic interpretation [38].

To address RQ5, the relationship between VL and task performance metrics was examined across two visualisation conditions: data storytelling and conventional visualisation. Two multiple linear regression analyses were conducted. The first regression analysis investigated the association between the participants’ VL and task efficiency, measured as the average success time (AvgSuccessTime). Additionally, an interaction term (VLAT × Condition[DS]) was included to assess whether the visualisation condition influenced this relationship. Similarly, the second regression analysis explored the association between participants’ VL and task effectiveness, measured as the correct rate (CorrectRate). We also included the same interaction term to evaluate the potential influence of the visualisation condition on this relationship. In both regression analyses, β₀ represented the intercept with the conventional visualisation condition served as the baseline, while Condition[DS] denoted the data storytelling condition. The assumptions of homoscedasticity, normality, independence of residuals, linearity, and multicollinearity were validated for both regressions.

We found that the average success time for information retrieval questions was actually higher when participants were presented with data storytelling visualisations (median: 31.859 seconds, IQR: 21.166) compared to conventional visualisations (median: 26.259 seconds, IQR: 14.208) for information retrieval questions, but this difference was not significant after p-value correction (U = 3517, Z = 2.229, p = 0.1027, r = 0.155).

Contrary to this, although the average success time for comprehension questions (considering both sub-types together) was slightly lower with data storytelling (median: 33.461 seconds, IQR: 18.327) compared to conventional visualisations (median: 40.297 seconds, IQR: 22.341), this difference was not significant (U = 1408, Z = -2.296, p = 0.0861, r = 0.16).

Separating the comprehension questions further into single insight and multiple insight questions revealed that the average success time of single insight questions was lower with data storytelling visualisation (median: 25.15 seconds, IQR: 15.161) compared to conventional visualisations (median: 29.394 seconds, IQR: 20.138) and this difference was significant following p-value correction with a small effect size (U = 1192, Z = -2.686, p = 0.0283, r = 0.187). Notwithstanding, data storytelling decreased the average success time of single insight comprehension questions by an average of 4 seconds. We found that the average success time for multiple insight comprehension questions was only slightly lower with data storytelling (median: 43.812 seconds, IQR: 31.031) compared to conventional visualisations (median: 49.232 seconds, IQR: 29.053); however, this difference was also not statistically significant (U = 1285, Z = -1.378, p = 0.6757, r = 0.096).

In sum, conventional visualisations actually facilitated quicker responses in information retrieval tasks compared to data storytelling. Although data storytelling showed a trend towards reduced success times in comprehension tasks, particularly for questions targeting single insights, the differences were not significant.

In evaluating data storytelling as a more effective medium for communicating both critical data points and insights and facilitating comprehension, we found strong and consistent evidence in its favour for all the types of questions posed to participants. First, our findings indicated a higher correct rate for information retrieval questions when participants utilised data storytelling (median: 1, IQR: 0.222) as opposed to conventional visualisations (median: 0.778, IQR: 0.444). This difference was significant and had a moderate effect size(U = 1716, Z = 4.157, p = 1e-04, r = 0.29), with data storytelling enhancing the correct rate by an average of 0.15. Furthermore, we noted a higher correct rate in comprehension questions using data storytelling (median: 0.778, IQR: 0.444) compared to conventional visualisations (median: 0.556, IQR: 0.556), a difference that was also significant and exhibited a moderate effect size(U = 2973.5, Z = 4.104, p = 1e-04, r = 0.286). Data storytelling increased the correct rate by a mean of 0.19, implying a more substantial improvement in comprehension questions relative to information retrieval questions.

Upon dividing the comprehension questions into single and multiple insights sub-types, we observed a significantly higher correct rate for single insight questions for data storytelling (median: 1, IQR: 0.444) compared to conventional visualisations (median: 0.556, IQR: 0.889), with a significant difference and a moderate effect size (U = 1783, Z = 4.231, p < 0.0001, r = 0.295) and a mean increase in the correct rate of 0.22. Similarly, data storytelling proved more effective for multiple insights comprehension questions (median: 0.556, IQR: 0.889) compared to conventional visualisations (median: 0.556, IQR: 0.889), albeit to a lesser yet significant degree with a small effect size (U = 1613, Z = 2.667, p = 0.0304, r = 0.186), improving the correct rate by an average of 0.15. These findings illustrate a pronounced enhancement in the correct rate for single insight questions through data storytelling compared to both single and multiple insights comprehension questions.

In sum, our findings emphasise the benefits of data storytelling in enhancing both information retrieval and comprehension, especially manifest in single insight comprehension tasks, thereby underscoring its potential critical role in enabling accurate and insightful data comprehension.

The first theme referred to the usefulness of data storytelling visual elements to identify key information to support the understanding of the data and reduce complexity (N=24). Visual elements like "added brief explanations" [textual annotations] (P15), "colour schemes" [colour emphasis](P26, P29), and "clear labelling" [annotated data points] (P27) helped participants to "identify the important elements" on the graph more readily (P15, P18, P22). The participants emphasised the role of textual elements, whether annotations or explanatory titles, in making the data "easier to understand" (P66, P67). These storytelling elements were not merely facilitative but often essential for understanding the graph: ("some graphics are harder to interpret, and the text [annotations] helped me to understand them [graphics] better" – P49), as they "helped to give facts rather than estimations" (P19, P36).

Another theme that emerged referred to the perceived role of data storytelling elements in enhancing their efficiency when identifying key information (N=26). For example, some participants explained that these elements acted as guides that helped them "identify more efficiently the relevant information" (P60, P67, P68) and "[convey] the main idea and the key points faster" (P67), sometimes even obviating the need for detailed chart analysis: "Without analysing the chart I would get the information needed just by reading the textual annotations" – P61. This efficiency could be attributed to the use of data storytelling elements to focus their visual attention. Textual elements combined with colour emphasis helped the participants to "focus [their] attention" (P70) and find answers "way quicker" (P65). One participant elaborated on this as follows: "I tend to get distracted if data is only represented by numbers or plain text. Colours and Explanatory titles were really helpful" – P56.

Colour emphasis also played a role in helping participants "find answers way quicker" (P65) and making the data "stand out" (P69), effectively directing their attention towards accurate information: "without them, I would not know what I am looking at." – P43; "it points to the correct data analysis; without it, you cannot tell what it is about or stands for" – P8. Participants also noted that helping them to focus their attention on what was important in a visualisation was particularly beneficial when operating under time constraints: "breaking down the data with colour seems to help; it drew my attention, especially when you’re being timed" – P44. These findings highlight the perceived role of data storytelling elements in focusing their attention and improving their efficiency when identifying key information.

Participants (N=10) also emphasised the importance of data storytelling elements in enhancing the readability and depth of understanding associated with data visualisations. One participant (P2) noted that "short text indicating values or indications enhances the readability", highlighting the role of textual elements in making the graphs more accessible. The perceptions of the contributions of the added context provided by data storytelling elements in enabling quicker data comprehension were expressed by another participant (P6): "It helped me to understand the data more quickly, and it gave me more context of it". Participants confirmed that the textual elements contributed to a more holistic understanding of the data; they "give additional information to the graph" (P7); "give a detailed look on the data" (P3) and help "give facts rather than estimations" (P12). Moreover, the storytelling elements added a layer of depth, making the information retrieved "much more complete" (P14). These comments suggest that storytelling elements can improve the ease of data interpretation and serve as complementary information to make the data more accessible and contextual by highlighting relevant information.

Participants (N=21) also perceived that the storytelling elements enhanced their ability to quickly and accurately interpret the data. Colour emphasis and annotated data points were acknowledged for their role in making data "easier to interpret" (P42, P43, P47) and supporting "more accurate" interpretation (P51). Participants noted that explanatory titles, textual annotations and colour emphasis made it "easier to locate information" (P26), enabling them to answer questions with greater confidence and accuracy ("with the storytelling elements was easier find the information to respond the questions correctly" –P24; "it made my interpretations more accurate" –P51). This was echoed by P16, who felt that such elements present the information at a glance to "speed up the process" of reaching the correct answer.

Similarly, textual annotations and annotated data points were perceived as helpful to "reinforce the graphic information" (P41) and "confirming assumptions" (P40), thereby allowing for more accurate responses to questions. Interestingly, while some participants (P26, P37) felt that they could have eventually figured out their responses without these storytelling elements, they acknowledged that data storytelling elements made the identification of key information faster and more focused. This perspective was echoed by P41, who felt that data storytelling "reinforces the graphic information and helps to understand better and integrate any data that may not appear immediate at first analysis". These perspectives suggest that data storytelling elements make the data interpretation process more precise, allowing participants to validate their initial interpretations and derive insights with greater confidence.

In contrast to the perceived benefits of data storytelling elements highlighted above, some participants (N=16) found that these elements were not always effective for interpreting data visualisations. Such elements, particularly textual elements (i.e., annotated data points, textual annotations and explanatory titles), could introduce confusion ("actually made the data more confusing to look at" –P85, "it’s just a lot of information and it can be a bit overwhelming" – P97) or serve as distractions if not used effectively ("the highlights [textual annotations] are kind of distracting from other data if done for only one thing" –P87, "the [textual annotations] distracted me from the correct visualisation of the data." –P100). Some participants (N=6) also questioned the need for these visual elements, arguing that the data was self-explanatory and that storytelling elements did not add significant value ("The data pretty much explains itself... the ones with data storytelling did not make that much difference." –P86) or that the non-storytelling graph was "easier to interpret", "clear" and "more visually appealing" (P88, P92, P98). Some participants (N=3) expressed concerns about the potential for these elements to be misleading or distracting. P90 elaborated on this as follows: "I preferred to base my responses on graphs because I didn’t know if the information about the visualisations [textual annotations] were true to the presented data", while another found the highlights "distracting from the correct visualisation of the data" (P100). These comments suggest that while data storytelling elements could be beneficial, these should be carefully crafted to avoid information overload, particularly when the data is easier to interpret.