Evaluating Alarm Classifiers with High-confidence Data Programming

Alarm fatigue is a well-known problem of physiologic monitoring in the hospital setting [11]. Bedside monitors continuously measure heart rhythm, heart rate, respiratory rate, blood oxygen, and other signals—and produce an alarm when the signals appear abnormal. Many of these alarms are not actionable or informative and often overwhelm the caregivers. The end result is that the clinicians react slowly, if at all, to alarms that have a small but nonzero probability of representing a critical patient need [5]. Ideally, the clinicians should only be alerted by the most important and actionable alarms, be able to interact with other informative alarms as they prefer, whereas the rest of the alarms are suppressed.

Alarm fatigue can be mitigated by reducing or prioritizing alarms via threshold tuning, customization, delaying, integration, and other methods [18]. Researchers have proposed novel algorithms for monitoring and suppressing unnecessary alarms based on advanced data processing [1, 8, 10]. At the core of algorithmic methods of reducing alarm fatigue is the classification of alarms into multiple classes such as priority, actionability, informativeness, suppressibility, delayability, and so on. This classification needs to be carefully balanced with the possibility of missing critically important and urgent alarms.

The clinical investigation and deployment of alarm classification systems is predicated on their expected performance. For example, when deploying an alarm suppression system, hospital policy makers need to ensure that its specificity to critical alarms is above a certain level, so that most of the critical alarms will continue to be reported. Measuring the performance of an alarm classifier typically requires a representative dataset of alarms correctly labeled with the classes of interest (e.g., suppressible vs. non-suppressible).

It is time-consuming and expensive to create highly accurate labeled datasets for the evaluation and tuning of alarm classifiers. A common way to do so is to perform an observational study [3] of many patients and manually label each period of time when a certain type of alarm would be, for instance, actionable. Such a study is a major commitment when it comes to an initial deployment of a novel alarm classifier, in part due to the significant effort of manually labeling thousands of alarms. Furthermore, it is impractical to perform an observational study for every adjustment of the settings of an alarm classifier throughout its lifecycle. This cost can be reduced with patient simulations [7], but precise and realistic simulations of human physiology are notoriously difficult and expensive to construct.

This article proposes a cheap and rapid method of estimating the performance of a clinical alarm classification system in the absence of a dataset with highly accurate labels. Our method can support early-stage low-cost evaluations of alarm classifiers in a variety of ways. For example, it can prioritize observational studies of classifiers with a higher potential to alleviate alarm fatigue so that the manual labeling effort is spent optimally. It can also guide the tuning of the classifier’s settings towards effective alarm management, reducing the risk of missing critically important alarms.

A key step of our method is to probabilistically label patient data according to the emerging paradigm of data programming [16]. We start with a dataset of unlabeled patient data, typically abundant in most clinical settings, several alarm classes of interest, and an alarm classifier with tunable settings. We then collect clinical intuitions about this alarm type and encode them as labeling functions—weak classifiers of these alarms that can abstain and need not be comprehensive or non-contradictory. Often, a labeling function represents an intuitive heuristic such as “if the heart rate is high, then the alarm is actionable”. The outputs of the labeling functions can be combined in a variety of ways, via majority vote or a generative model, resulting in labels of varied confidence for each data point. Finally, using the high-confidence subset of the labeled data, we estimate the uncertain true class-wise rates (e.g., sensitivity and specificity if we consider only two classes of alarms) of the alarm classifier.

To communicate the uncertainty of our estimates, we mathematically develop confidence bounds on the true rates of the alarm classifier. These bounds indicate, for a given level of confidence, the interval of possible values for the true rate that one could obtain from a hypothetical observational study of a given size. These bounds account for a finite number of samples for each class, potentially incorrect weak labels, and the uncertainty in estimating the confidence in our weak labels. Accounting for the confidence estimation uncertainty is a novel contribution in this journal extension of our earlier work [14].

We validated our method with five datasets of clinical alarms, derived from a 551-hour alarm labeling effort from Children’s Hospital of Philadelphia. One of these datasets was used in our prior work [14], while the remaining four allow for novel experiments. We evaluated the performance of alarm actionability classifiers that threshold the key physiological signals such as SpO\(_2\) and heart rate. Our method’s estimated confidence bounds almost always contain the true-label-based specificity and sensitivity—and vary appropriately based on the amount of available data. This application demonstrated our method of estimating the sensitivity-specificity tradeoffs in an alarm classifier without investing hundreds of hours into labeling the alarm data.

In summary, this article makes three research contributions:

The remainder of this article proceeds as follows. The next section presents a motivating scenario for low-cost evaluation of alarm classifiers. Section 3 discusses the literature related to evaluating alarm classifiers. Section 4 introduces the minimal background and precisely formulates the mathematical problem at the heart of our method, which then is described in the following section. The five alarm datasets of are described in Section 6, and its results can be found in Section 7. The article concludes with a summary in Section 8.

This article focuses on clinical alarms produced by physiological monitors in a hospital setting. By an “alarm” we mean a combination of waveforms and patient data that have triggered a monitor to notify the clinicians. An alarm classifier is some algorithm or device that categorizes the alarms in a clinically useful manner. For example, one algorithm may determine which alarms are actionable and prioritize their delivery to the nursing staff. Another algorithm may find suppressible alarms and remove them from the feed. Yet another algorithm may discover equipment malfunction alarms (as opposed to the patient-related ones) and notify the technical personnel or the manufacturer. Our work considers all such algorithms to be alarms classifiers.

Most alarm classifiers have a direct impact on patient treatment and, therefore, need to be evaluated to justify their deployment and use. For example, to justify using an alarm suppression system, the decision-makers need to ensure that it suppresses some fraction of false alarms and avoids suppressing too many true alarms. This evaluation would quantify the true class-wise rates: the proportion of samples labeled correctly for each class. In a two-class setting, the true rates are sensitivity (for the positive class) and specificity (for the negative class).

Consider a scenario to motivate our work. A hospital is considering the deployment of an alarm suppression system, which has tunable parameters. The effect of these parameters on the hospital’s population of patients is not known: will the reduction in superfluous alarms be worth the risk of missing important and actionable alarms? To answer this question, the hospital would need to perform a controlled observation study in which a sample of alarms is collected, labeled, and used to test the suppression system with different parameter settings. Such a study is particularly time-consuming because the collected alarms (likely thousands of them) would need to be manually labeled. This study may indicate that the expected suppression performance is far below acceptable, and the labeling effort would be wasted. On the other hand, the classifier may perform so well that the hospital may decide to skip the observational study and, instead, advance to an interventional pilot study.

Thus, the hospital could benefit from cheaper and faster ways of predicting the range of hypothetical outcomes of an observation study for an alarm suppression system, ideally avoiding the manual labeling of alarms. As the hospital continues to collect alarm data, it wants to gain increasing confidence in its prediction, until the point when some decision can made. This scenario is possible not only for alarm suppression systems, but also for other algorithmic classifiers of alarms, for example, whether an alarm is actionable, what priority it should have, or whether it was caused by clinical events or technical issues. Since the classes of alarms and patient populations differ and change over time, clinicians need to repeatedly predict classifier performance, and therefore a general estimation approach is needed. This general scenario of evaluating an alarm classifier without perfect labels is shown in Figure 1.

There exists a vast literature on clinical alarms and unsupervised/weakly-supervised learning. We focus on the areas particularly relevant to our goal of evaluating alarm classifiers.

Alarm fatigue is a serious and well-known problem of physiological monitors [5, 11]. To address it, a variety of approaches to suppress or prioritize alarms have been proposed based on techniques from signal processing, statistics, and machine learning [1, 8, 10]. Many of such approaches need patient data labels to be designed or trained, and all of them need the labels to be evaluated. Our article introduces a lightweight way of performing these evaluations without investing in high-quality labels. Note that our proposed method is not specific to any physiological input, unlike many alarm classification techniques.

The gold standard for evaluating the clinical effectiveness of an alarm classifier is an interventional study, in which the researchers deploy the system and measure its effects compared to a control group. To estimate the true rates of a classifier (e.g., its sensitivity and specificity), a controlled observational study would be sufficient: alarms are collected from physiological monitors, manually labeled by the clinical experts, and fed into the classifier. Then the true rates are computed by comparing its outputs with the manual labels. Both types of studies require substantial time and effort, in part due to the need to label each alarm by hand. For example, nurses may need to review video feeds of patients to determine the actionability of alarms [9]. Our work is not meant to replace either type of studies; instead, we aim at prioritizing, guide, and reduce the risk of observational studies by providing an early and cheap estimation of the expected performance of an alarm classifier.

The Area under the Receiver Operating Characteristic Curve (AuROC) is a metric often used to evaluate the quality of medical classifiers. There exists literature where the AuROC can be estimated in the form of confidence intervals from parameters such as error rates and the number of positive/negative samples [4]. Another AuROC estimation method from test scores via bootstrapping [2]. However, these methods require a labeled dataset which is prohibitively expensive to collect. Our approach can be used to generate confidence intervals for sensitivity/specificity using a weakly labeled dataset, from which AuROC can be calculated.

Alarms can be labeled with high-precision methods using patient simulations and computer-aided clinical trials [6]. To provide realistic data, these methods require detailed physiological models, a building which is an expensive and lengthy task. For clinical alarms, an appropriate model is rarely available. Our method is related to observational studies in the same way as computer-aided clinical trials are related to traditional clinical trials. That is, we perform a virtual algorithmic evaluation of alarm classifiers. After that, our results can provide the basis for an observational study or a clinical trial of the algorithm or a device that classifies alarms.

Recently, a quick and inexpensive way of labeling data has emerged, known as data programming [16]. A key element of data programming is a set of quantitative intuitions about how the data corresponds to labels. For example, a clinician might say, “when a patient over 60 years old has had a heart rate over 120 beats for over a minute, such an alarm is a high priority.” These intuitions, algorithmically represented as labeling functions, are allowed to be incomplete, sometimes incorrect, and contradictory. A labeling function returns a class label or an “abstain” verdict for any input. Given a diverse combination of many labeling functions and an unlabeled dataset, data programming algorithms produce probabilistic labels—a label and a confidence between 0 and 1—for each sample in the dataset. A prominent data programming tool Snorkel [15] estimates an optimal weight for each labeling function by using a generative graphical model and a prior on the class balance. Our approach encodes clinical intuitions about classes of alarms (e.g., suppressible vs. non-suppressible) as labeling functions, feeds them along with alarm data into Snorkel, and relies on the resulting probabilistic labels to quantify the uncertainty in the classifier performance.

Extending the data programming work, adversarial data programming generates data in addition to labeling it [12, 13]. A Generative Adversarial Network (GAN) is used to estimate the weight of each labeling function as well as the dependencies between them given a set of labeling functions and an unlabeled dataset. The weights and dependencies are used by the GAN’s generator to produce labeled samples that come from the data distribution. This technique is related to patient simulations and computer-aided clinical trials in that realistic labeled data can be generated for the purpose of evaluating an alarm classifier. Data generation can be used in an extension of our work to address applications with small datasets or imbalanced datasets.

This article builds on our previous approach to estimating true rates in alarm suppression systems [14]. Compared to that, we have extended the scope from evaluating suppression systems to evaluating any alarm classifier—and correspondingly extended our evaluation with four new alarm datasets of varying sizes. We have also extended the technical approach to account for the uncertainty in estimating the label confidence in a finite sample. This extension allows us to vary the bound width appropriately datasets with few samples, which did not satisfy the assumptions of the earlier approach.

We start with an unlabeled dataset of alarms (\(\mathcal {I}\)) and consider a hypothetical, unavailable dataset of alarms (\(\mathcal {I}^*\)) of a known size collected during a hypothetical observation study from the same population as \(\mathcal {I}\). We are evaluating an alarm classifier (\(S\)) that takes a sample and assigns one of \(J\) classes to it. This alarm classifier has tunable parameters affecting its output (e.g., an alarm threshold for high heart rate), and in the rest of the article, we treat each alarm classifier with different parameter values as a separate classifier \(S\).

Our goal is to quantify how well the alarm classifier \(S\) would perform on the hypothetical dataset \(\mathcal {I}^*\). We measure the classifier performance with its true rate for each class. For any class \(j\), a true rate \(R_j\) is the fraction of samples labeled \(j\) that are indeed of class \(j\). In the case of only two classes, the positive true rate is called sensitivity, and the negative true rate is called specificity. We aim at predicting the true rates for each class on the hypothetical dataset \(\mathcal {I}^*\). We refer to these hypothetical, unknown rates as \(R^*_j\), and each of them is, conceptually, determined on a subset of \(\mathcal {I}^*\) containing all samples with label \(j\), which we refer to as \(\mathcal {I}^*_j\). Since we do not have access to \(\mathcal {I}^*\), we will predict \(R^*_j\) indirectly using \(\mathcal {I}\).

To use dataset \(\mathcal {I}\), we first need to overcome the absence of true labels for it. So one sub-task will be to create a probabilistic labeler—an algorithm that takes a sample and assigns it an estimated label and a confidence in that label (a number between 0 and 1). We refer to the assigned label as \(f(x)\) and its confidence as \(g(x)\) for any sample \(x\). We will build that labeler from labeling functions, which encode rules of thumb and heuristics acquired from clinicians. Each labeling function takes a sample and either abstains or assigns one of the classes to it. Labeling functions can contradict each other or abstain in different combinations. Once a probabilistic labeler is created, each sample \(x\) in the dataset \(\mathcal {I}\) will be characterized by three values: a true (unknown) label \(f^*(x)\), an estimated label \(f(x)\), and a confidence \(g(x)\) in the estimated label. In practice, we only have access to the latter two.

Our second sub-task will be to structure \(\mathcal {I}\) based on the estimated labels. For each label \(j\), we form a high-confidence set \(\mathcal {I}_j\) of samples \(x\) from \(\mathcal {I}\) that the probabilistic labeler labeled with \(j\) (i.e., \(f(x) = j\)) and high confidence, (i.e., \(g(x) \ge 1-\epsilon\) for some constant \(\epsilon\) that we pick up front).

On each high-confidence set \(\mathcal {I}_j\), we can calculate the estimated true rate of \(S\) on class \(j\), denoted as \(R_j\). To predict the hypothetical rate \(R^*_j\), we will come up with an interval \(C_j\) of possible true rates around \(R_j\). This interval should contain the value of \(R^*_j\) with at least the containment probability \(p_j\) (a.k.a. the significance level), given to us by the user.

To summarize, the technical problem addressed in this article is to create the smallest interval containing the hypothetical rate \(R_j^*\) with a probability at least \(p_j\). This problem can be stated mathematically as follows:

This section describes our approach for producing confidence bounds for the class-wise true rates of an alarm classifier. Figure 2 summarizes the five steps of our approach:

To evaluate the performance of our approach, we conducted experiments on five alarm datasets: low SpO\(_2\), low/ high respiratory rate (RR), and low/high heart rate (HR)¹ alarms. We consider alarm classifiers that label an alarm as either suppressible or non-suppressible. For low vital sign alarms, the classifier labels an alarm as suppressible if the vital sign measurement is above a specified threshold at the time of the alarm; otherwise it is labeled non-suppressible. For high vital sign alarms, the classifier labels an alarm as suppressible if the vital sign measurement is below a specified threshold, otherwise, it labels non-suppressible. We consider the context where the threshold of these alarm classifiers were improperly configured, resulting in a large volume of suppressible alarms being missed. Our goal is to establish and visualize the connection between the alarm classifiers’ threshold and its sensitivity/specificity, given an unlabeled dataset of patient vitals data. This visualization will be compared with the sensitivity/specificity computed from the manually-annotated alarm data, which is our case stands in for a hypothetical observational study. In this section, we overview the dataset and data preprocessing approach, introduce the labeling functions collected for labeling alarms, and describe the implementation details of our approach.

This section presents the results of our experiments described in the previous section.³ Specifically, we evaluate the performance of the confidence bounds for the sensitivity and specificity of alarm suppression classifiers produced by our approach. We also evaluate the bounds for the tradeoff curve between sensitivity and specificity, because this curve may inform clinical choices of alarm classification parameters. A successful application of our approach would result in bounds that contain the hypothetical sensitivity/specificity and tighten when more data is made available. We compare our confidence bounds to those produced by the confidence uncertainty-unaware approach from earlier work [14].

The parameters of our approach are set as follows. We allow for a 10% chance of the confidence bounds not containing the true sensitivity/specificity, i.e., significance level \(p_0 = p_1 = 0.1\). Label uncertainty \(\epsilon _j\) determines the high-confidence subsets \(\mathcal {I}_j\) that we optimize over in our approach when computing the confidence bounds. We search the space \(\epsilon _j \in [0.001, 0.5]\) to find the tightest bounds.⁴

The confidence bounds for sensitivity, specificity, and the tradeoff between the two using different probabilistic labeling methods are depicted in Figures 3–5, respectively. To draw the tradeoff bounds, for each threshold we plot the \(({\it specificity} + c_0, {\it sensitivity} + c_1)\) for the upper-bound and \(({\it specificity} - c_0, {\it sensitivity} - c_1)\) for the lower-bound. Since we have access to true labels (i.e., the labels extracted from the alarm annotations), we use them to plot the true curve for the sensitivity/specificity/tradeoff, which represents the results of an observational study. Furthermore, we use the true labels to implement a fourth probabilistic labeling method, an oracle labeler, that, for each alarm, yields its true label and a 99% confidence. The oracle bounds represent the result of the approach if we were able to perfectly label the alarms dataset, eliminating the labeling and confidence uncertainty. Oracle bounds always contain the true curve and are sufficiently wide enough to account for sampling uncertainty; therefore, we seek bounds that are at least as wide as the oracle bounds. Table 2 summarizes the average width of the sensitivity and specificity bounds as well as the percentage of the true curve contained in the sensitivity, specificity, and tradeoff confidence bounds. We defer to Appendix C for the hyperparameters that produce the confidence bounds in the aforementioned table and figures. We note that only a subset of the thresholds for each alarm classifier can plausibly be adopted into a clinical setting, and hence only portions of these curves are clinically relevant. In discussion with pediatric physicians, we defined the following clinically relevant regions: 80 to 95 for low SpO\(_2\), 0 to 30 for low RR, 60 to 120 for high RR, 50 to 120 for low HR, and 160 to 220 for high HR.

Our main observation is that our approach produces confidence bounds with high containment. Furthermore, the results demonstrate that bound containment improves as progressively more accurate and data efficient probabilistic labeling methods are used. Our confidence uncertainty-aware approach yields 100% containment for 11, 11, and 15 out of 15 bounds using majority vote, uninformed, and informed generative models, respectively. While most majority vote bounds have high containment, we can understand the instances where it is low by looking at the low SpO\(_2\) case study tradeoff bounds (Figure 5, top row). The bounds bow outward unlike traditional tradeoff curves, and if we flipped the labels it would bow inward and improve containment. This implies that the majority vote labeled incorrectly with high-confidence as we anticipated in Section 5.3. The uninformed generative model is disadvantaged when the actual class balance is heavily skewed. This can explain the lower containment in the high HR and the low SpO\(_2\) case. Finally, since all bounds produced by an informed generative model achieve 100% containment of the true curve, our approach has effectively captured this outcome of a hypothetical observational study.

By comparing the left and the right columns in Figures 3–5, we conclude that accounting for confidence uncertainty trades bound width for higher containment. The confidence uncertainty-unaware approach yields 100% containment for 7, 9, and 7 out of 15 bounds using majority vote, uninformed, and informed generative models, respectively. In the cases where 100% containment is not achieved, the confidence bounds produced by our approach have better containment. On average, the improvement is \(0.187\, \pm \, 0.12\), \(0.224 \, \pm \, 0.17\), and \(0.437 \, \pm \, 0.16\) for each labeling method, and \(0.284 \pm 0.19\) overall. However, our approach’s bounds are wider than the confidence uncertainty-unaware bounds in all cases by \(0.178 \, \pm \, 0.14\) on average. Confidence uncertainty widens the bounds in order to capture more of the hypothetical observational study result.

We compare each approach to their respective oracle bounds and find that the confidence uncertainty-aware approach yields bounds that are good approximations of the oracle bounds. Our approach’s bounds are wider than the oracle bounds in 24 out of 30 cases by \(0.129\pm 0.12\) on average. The confidence uncertainty-unaware approach’s bounds are wider than the oracle bounds in 13 out of 30 cases by \(0.127\pm 0.14\) on average. Our uncertainty-aware bounds better account for the sampling uncertainty than the uncertainty-unaware approach. Furthermore, our approach using an informed generative model is able to achieve the same containment as the oracle bounds (100% containment) in all cases, whereas the confidence uncertainty-unaware approach achieves oracle containment in less than half of the cases. Hence, the bounds produced by our approach, though slightly wider, converge to the quality of the best possible confidence bounds producible by our approach.

The width of our bounds varies appropriately with the amount of provided data. For example, the bounds for larger dataset cases, low SpO\(_2\), high RR, and high HR, are of moderate width. However, small dataset cases, low RR and low HR, have wide bounds. Bound width is directly affected by the number of samples used (determined by the dataset \(\mathcal {I}\) and label uncertainty \(\epsilon _j\)) and the choice of significance level \(p_j\) (which follows from Corollary 1). We explore the effect of the sample count per class in the dataset on the bound width using our approach with an informed generative model in Figure 6. This figure shows that having more data yields tighter bounds.

Recall the context of the case study is to better pick alarm thresholds of a previously poorly calibrated alarm-generating device. In practice, to help determine the best threshold for alarm suppression classifiers, hospital policy makers could use the sensitivity/specificity tradeoff. A good threshold would produce specificity close to 1 (i.e., not suppress any non-suppressible alarms) while maximizing sensitivity (i.e., silence as many suppressible alarms as possible). Suppose policy makers decide to allow a minimum of 90% specificity. For the low vital sign alarms, the minimum threshold using our approach using an informed generative model and the true curve, respectively, are 92 and 91 for SpO\(_2\), 27 and 28 for RR, and 114 and 103 for HR. For the high vital sign alarms, the respective maximum thresholds are 70 and 69 for RR, and 178 and 178 for HR. The thresholds selected using our approach are sufficiently accurate approximations, if not exact, to those that would have been selected in an observational study.

Limitations: Our confidence bounds are accurate when suppression accuracy is relatively consistent on different high-confidence labels, as stated in Assumption 1. This assumption may be violated in contexts with few available samples or when high-confidence labeling is particularly biased/inaccurate — and then our theoretical guarantees might not hold. For example, there may exist a subset of the patient population for which the clinician-designed labeling functions always label incorrectly. Alternatively, the high-confidence datasets may be biased by only containing alarms of a patient subset not representative of the entire patient population. Our conclusions are expected to generalize to the setting of our data collection, i.e., actionable alarms for pediatric medical-floor patients. To apply our method in a different setting, one may need to elicit different/more labeling functions, and so the tightness and accuracy of the confidence bounds may vary.

In this article, we proposed an approach for estimating the performance of a clinical alarm classification system without access to labeled alarm data. Probabilistic labels obtained via generative modeling were used as a proxy for the unknown true labels to estimate the average true rates. We then quantified confidence bounds for these estimates. Finally, we evaluated our approach in case studies of low SpO\(_2\), low/high respiratory rate, and low/high heart rate alarms. The results show that we outperform prior work and produce confidence bounds that contain most, if not all, of the true curves generated in an observational study. For future work, we plan to analyze the impact of the number of weak labeling functions used in our approach, automate the extraction of weak labeling functions that satisfy the assumptions of generative models, as well as explore unsupervised confidence calibration of generative models for data programming.

We start with an unlabeled dataset of alarms \(\mathcal {I}\) and a hypothetical, unavailable dataset of alarms \(\mathcal {I}^*\). We pick some subset of high-confidence samples:

Recall our assumption of consistent true rates:

Then,

Let \(\delta _j = 1 - \frac{1}{|\mathcal {I}_j|} \sum _{x \in \mathcal {I}_j} g(x)\) and \(\gamma _j = \frac{(c_j - \delta _j) \sqrt {|\mathcal {I}^*_j|}}{\sqrt {|\mathcal {I}_j|} + 2 \sqrt {|\mathcal {I}^*_j|}}\). Then for any alarm classifier \(S\) it holds that:

For the corollary, we are given desired significance level \(p_j\):

Solving the equation above for \(c_j\) yields the following equation for the bound size,

The twelve guidelines for deciding whether an alarm is suppressible or non-suppressible from Section 5.2 are encoded as labeling functions (LFs) in the following ways.

Table C.1 summarizes the hyperparameters that produce the confidence bounds in Figures 3–5. Recall that \(\epsilon _j\) is the allowed level of label uncertainty in both approaches, and \(\gamma _j\) is an admissable free parameter that applies only to the confidence uncertainty unaware approach [14].