research-article

Open access

Information Leakage Games: Exploring Information as a Utility Function

Authors:

Mário S. Alvim,

Konstantinos Chatzikokolakis,

Yusuke Kawamoto,

Catuscia PalamidessiAuthors Info & Claims

ACM Transactions on Privacy and Security, Volume 25, Issue 3

Article No.: 20, Pages 1 - 36

https://doi.org/10.1145/3517330

Published: 09 April 2022 Publication History

All formats PDF

Abstract

A common goal in the areas of secure information flow and privacy is to build effective defenses against unwanted leakage of information. To this end, one must be able to reason about potential attacks and their interplay with possible defenses. In this article, we propose a game-theoretic framework to formalize strategies of attacker and defender in the context of information leakage, and provide a basis for developing optimal defense methods. A novelty of our games is that their utility is given by information leakage, which in some cases may behave in a non-linear way. This causes a significant deviation from classic game theory, in which utility functions are linear with respect to players’ strategies. Hence, a key contribution of this work is the establishment of the foundations of information leakage games. We consider two kinds of games, depending on the notion of leakage considered. The first kind, the QIF-games, is tailored for the theory of quantitative information flow. The second one, the DP-games, corresponds to differential privacy.

1 Introduction

A fundamental problem in computer security is the leakage of sensitive information due to the correlation of secret information with observable information publicly available, or in some way accessible, to the attacker. Typical examples are side-channel attacks, in which (observable) physical aspects of the system, such as the execution time of a decryption algorithm, may be exploited by the attacker to restrict the range of the possible (secret) decryption keys. The branch of security that studies the amount of information leaked by a system is called quantitative information flow (QIF), and it has seen growing interest over the past decade (i.e., see [9, 30, 49, 63]).

Another prominent example is the privacy breach in public databases, where even if personal identifiers such as an individual’s name and date of birth are removed, the information stored in the records may be enough to uniquely re-identify a person. The problem of privacy has become more prominent with the advent of Big Data technologies, and the study of attacks and protection methods is a very active field of research. We mention in particular the differential privacy (DP) approach [32, 35], which has become extremely popular in the last decade.

It has been recognized that randomization can be very useful to obfuscate the link between secrets and observables. Examples include various anonymity protocols (e.g., the dining cryptographers [25] and Crowds [59]), and the typical mechanisms for DP such as Laplace and geometric noise [35]. The defender (the system designer or the user) is therefore typically probabilistic. As for the attacker (the party interested in inferring sensitive information), most works in the literature on QIF consider only passive attacks, limited to observing the system’s behavior. Notable exceptions are the works of Boreale and Pampaloni [17] and of Mardziel et al. [53], which consider adaptive attackers who interact with and influence the system. We note, however, that Boreale and Pampaloni [17] do not consider probabilistic strategies for the attacker. As for Mardziel et al. [53], although their model allows them, none of their extensive case studies uses probabilistic attack strategies to maximize leakage. This may seem surprising, since, as mentioned before, randomization is known to be helpful (and, in general, crucial) for the defender to undermine the attack and protect the secret. Thus, there seems to be an asymmetry between attacker and defender with respect to probabilistic strategies. Our thesis is that there is indeed an asymmetry, but that does not mean that the attacker has nothing to gain from randomization: when the defender can change his own strategy according to the attacker’s actions, it becomes advantageous for the attacker to try to be unpredictable and, consequently, adopt a probabilistic strategy. For the defender, although randomization is useful for the same reason, it is also useful because it potentially reduces information leakage, which constitutes the utility of the attacker. This latter aspect introduces the asymmetry mentioned previously.

In the present work, we consider scenarios in which both attacker and defender can make choices that influence the system during an attack. We aim, in particular, at analyzing the attacker’s strategies that can maximize information leakage, and the defender’s most appropriate strategies to counter the attack and keep the system as secure as possible. As argued before, randomization can help both the attacker and defender by making their moves unpredictable. The most suitable framework for analyzing this kind of interplay is, naturally, game theory. In particular, we consider zero-sum games, and employ measures of information leakage as the utility function (with the positive sign for the attacker and the negative one for the defender). In this context, randomization is naturally captured by the notion of mixed strategies, and the optimal strategies for the defender and attacker are expressed by the notion of Nash equilibria.

However, it is important to note that the notion of information is fundamentally different from the typical utility functions modeled in game theory, such as money, time, or similar resources. These traditional quantities are linear with respect to the combination of strategies, hence the utility of a mixed strategy can be soundly defined as expectation. In classical game theory, this concept of utility has been formalized by the axioms of von Neumann and Morgenstern [65]. For instance, consider a lottery where the prize is stashed in two identical treasure chests, one containing 100 gold coins and the other containing 200 gold coins, and assume that when a participant wins, a hidden fair coin is flipped to decide what chest he should get his prize from. The winner expects to get 150 golden coins, and the corresponding benefit from that does not depend on his exactly knowing from which chest the prize is coming. Note that that is still true even if he never gets to explicitly know the result of the coin flip, nor of what was the chosen chest: no matter from where the golden coins came, the corresponding (monetary) utility to him is the same.

In contrast, the concept of information has a rather different nature, which relates to the asymmetry mentioned before. To understand this point, consider a scenario in which we can pose “yes”/“no” questions to one of two oracles and our utility is the amount of information we can extract from that oracle. Assume we know that one of the oracles always tells the truth, whereas the other always lies. For instance, if we ask “Is Alice single?” to the truthful oracle and get “yes” as an answer, we know for sure that Alice is single, and if we get the same answer from the lying oracle, we know for sure that Alice has a partner. Note that the information utility of the answers is maximum, as long as we know the oracle they came from, because they entirely determine the truth or falsehood of the fact we asked about. Now suppose that we cannot directly query any of these perfectly informative oracles, but instead we have to ask our question to some intermediate person who will first flip a hidden, fair coin to decide which oracle to pose the question to. The person then poses the question to the oracle, hidden from us, obtains an answer, and lets us know that the answer was, say, “yes.” How useful is this answer? Not useful at all, indeed, as we do not know what oracle it came from! The critical insight is that, contrarily to what happens with quantities like money, time, or other similar resources, the “average” of two highly informative answers is not necessarily a highly informative answer. More precisely, information is non-linear with respect to hidden choice (in this example, the hidden coin flip that decides among strategies). For this reason, traditional game theory as formalized by von Neumann–Morgenstern axioms fails to adequately capture games in which utility is information.

Consequently, in this work, we consider a new kind of games, which we call information leakage games. In these games, the defender can mix two (pure) strategies, by choosing one strategy or the other, without necessarily revealing his choice to the attacker. As seen in the preceding example, the lack of knowledge of the defender’s choice may reduce the amount of information that the attacker can infer. Namely, the information leaked with a mixed strategy is never larger than the leakage of each original pure strategy, which means that utility is a quasi-convex function of the defender’s strategy. As for the attacker’s strategies, the properties of the utility depend on the particular definition of leakage, and we will discuss the various instances in the next section. In any case, our setting departs from the standard game theory, in which utility is a linear function on the strategies of both players. Nevertheless, we show that our games still have Nash equilibria, namely pairs of defender-attacker strategies where neither player has anything to gain by unilaterally changing his own strategy. This means that there exists an “optimal” balance of strategies between defender and attacker, and we will propose algorithms to compute these equilibria using a more efficient variant of the well-known subgradient method for two variables. The efficiency improvement is achieved thanks to the specificity of our problem, and it can be considered our original contribution. Specifically, we are able to rewrite the minimax problem in a more convenient form (c.f. Proposition 3.3), involving a function that can be locally maximized easily. We then prove that the two standard sufficient conditions for the convergence of the subgradient method (bounded subgradients and bounded distance between the initial and optimal points) hold in our case. Finally, we show that not only our method converges to a saddle point but also that an approximate solution for the strategy provides an “approximate saddle point” (c.f. Theorem 3.4).

1.1 Specific Information Leakage Frameworks

The literature on information leakage in the probabilistic setting can be divided into two main lines of research: quantitative information flow (QIF) [9, 10, 49, 63] and differential privacy (DP) [33, 34, 35]. Our game-theoretic approach encompasses both of them, and we will refer to them specifically as QIF-games and DP-games, respectively. For reasoning about information leakage, we adopt the information-theoretic view, which is very popular in the QIF community. In this setting, a system or a mechanism is modeled as an information-theoretic channel, where the secrets are the inputs and the observables are the outputs. We will use this model both for QIF-games and for DP-games. We assume that both attacker and defender can influence with their actions the choice of the channel used in the game. In other words, we assume that channels are determined by two parameters that represent the actions, or strategies, of the two players.

In the case of QIF, the typical measures of leakage are based on the concept of vulnerability, which quantifies how easily the secret can be discovered (and exploited) by the attacker. There are various models of attackers and corresponding notions of vulnerability proposed in the literature, but for the sake of generality, here we abstract from the specific model. Following earlier work [8] and Boreale and Pampaloni [17], we adopt the view that vulnerability is any convex and continuous function. Such a class of functions has been shown in earlier work [8] to be the unique family presenting a set of fundamental information-theoretic properties, and subsuming most previous measures of the QIF literature, including Bayes vulnerability (a.k.a. min-vulnerability [24, 63]), Shannon entropy [61], guessing entropy [54], and g-vulnerability [10]. It is important to note that quasi-convexity is implied by convexity, and therefore this definition of vulnerability respects the assumption of our games. As for the attacker, he knows his own choice, thus the information that he can infer by mixing two strategies is the average of that of the individual strategies, weighted on the percentages by which they appear in the mix. This means that utility is a linear (more precisely, affine) function of his strategies. In conclusion, in the QIF-games, the utility is convex on the defender and linear on the attacker.

In contrast, the property of DP is usually defined for a (probabilistic) mechanism associated with a specific query on datasets. We say that a mechanism \(\mathcal {K}\) is ε-differentially private if for any two neighbor datasets x and x′ that differ only for one record, the ratio of the probabilities that \(\mathcal {K}\), applied respectively on x and x′, reports the same answer y, is bounded by e^ε. Here we adopt a more abstract view: following the approach in other works [23, 48], we assume that x and x′ range over a generic domain \(\mathcal {X}\) endowed with a binary relation ∼ that generalizes the neighborhood relation. For the sake of uniformity, we will also model the mechanisms as information-theoretic channels, whose typical element is the probability of reporting a certain y when applied to a given x. In the DP literature, the parameter ε is used to control the privacy provided by a mechanism: the higher its value, the more secret information the mechanism can reveal. Here we follow that approach and adopt as the measure of payoffs the minimum parameter ε for which a mechanism is ε-differentially private.

For what concerns the composition of strategies, we remark an important difference between the QIF and DP settings. Whereas the measure of leakage in QIF can be—and often is—defined as an average on all secrets and observables, in DP it is a worst-case measure, in the sense that only the worst possible case counts (namely, the minimum ε). In line with this philosophy, the composition of two mechanisms in DP is not linear with respect to the leakage, even in the case of visible choice: the leakage is determined by the worst of the two, at least for non-trivial combinations. We will call this property quasi-max, and we will see that it is a particular case of quasi-convexity. Because of this general non-linearity, it makes sense to explore for DP also the case in which the defender uses visible choice (i.e., he makes known to the attacker which mechanism he is using). In the case of DP, we therefore consider two cases for the defender’s choice: visible and invisible. For the attacker, however, we consider only visible choice, as it is natural to assume that he knows what choices he has made. We show that also for DP, and in all of these possible scenarios, Nash equilibria exist, and we provide algorithms to compute them.

Note that we do not consider the case of visible choice for the defender in QIF because in that setting, for visible choice, the leakage is linear. Since the leakage would be linear in both the attacker’s and defender’s strategies, the games would be just ordinary games, and the existence of Nash equilibria and their computation would derive from standard game theory. The interested reader can find an analysis of this kind of game in earlier work [6]. Table 1 summarizes the various scenarios.

Table 1.

1.2 Contributions

The main contributions of this work are the following:

•

We define a general framework of information leakage games to reason about the interplay between attacker and defender in QIF and DP scenarios. The novelty w.r.t. standard game theory is that the utility of mixed strategies is not the expected utility of pure strategies.

•

We extend the existence of Nash equilibria to DP-games (their existence for QIF-games is proved in a preliminary version [5] of this article).

•

We provide methods for finding Nash equilibria of QIF-games by solving a convex optimization problem, and for computing optimal strategies for DP-games by solving a sequence of linear programs. Our implementation of those algorithms for solving QIF-games and DP-games is freely available online as part of the LIBQIF tool [1].

•

We examine the difference between our information leakage games and standard game theory, from a principled point of view, by analyzing the axioms of von Neumann and Morgenstern that constitute the foundations of the latter, and show why they cannot encompass the notion of utility as information.

•

As a case study on QIF-games, we consider the Crowds protocol in a MANET (Mobile Ad-hoc NETwork). We study the case in which the attacker can add a corrupted node as an attack, the defender can add an honest node as a countermeasure, and we compute the defender component of the Nash equilibrium.

•

As a case study on DP-games, we illustrate how to design a local DP mechanism using a real dataset on criminal records. Specifically, we consider a DP-game between a defender who discloses his non-sensitive attributes to a data curator, and an attacker (a data analyst) who queries the data curator about one of the non-sensitive information to extract the defender’s secret that is correlated with the disclosed non-sensitive information. By computing the Nash equilibrium of this DP-game, we construct a privacy mechanism for the defender.

1.2.1 Relation with Preliminary Version.

A preliminary version of this work, considering only the QIF-games, appeared in earlier work [5]. The main novelties of this article to that version are the following:

•

We revise and present in more detail the algorithms for computing the Nash equilibria (Section 3.3).

•

We introduce the notion of DP-games (Section 4.1) and study its theory, showing the existence of Nash equilibria (Sections 4.2 through 4.4).

•

We provide an algorithm for computing optimal strategies for DP-games (Section 4.5).

•

We show a case study on DP-games to illustrate how to design a local DP mechanism for an adaptive disclosure of non-sensitive information correlated with a secret (Section 6.2).

•

We present proofs for our technical results (Appendix B).

1.2.2 Relation with Our Subsequent Work.

There are two subsequent papers based on the preliminary conference version [5] of this article. These were published in the preliminary conference version [7] and in the full journal version [6]. In these two papers, the focus was on the relation between the QIF-games that we consider here and other kinds of games (sequential games, games with visible choice for the defender). In particular, we built a taxonomy of the various kinds of games.

In the present article, on the contrary, the focus is on the foundational aspects of a class of games that subsumes QIF and DP and the contrast with the foundations of traditional game theory (as established by von Neumann–Morgenstern axioms). Furthermore, the extension to the case of DP is new to this article and was not present in the previous works [5–7].

1.3 Related Work

There is extensive literature on game theory models for security and privacy in computer systems, including network security, physical security, cryptography, anonymity, location privacy, intrusion detection, and economics of security and privacy. See the work of Manshaei et al. [52] for a survey.

In many studies, security games have been used to model and analyze utilities between interacting agents, especially attackers and defenders. In particular, Korzhyk et al. [51] present a theoretical analysis of security games and investigate the relation between Stackelberg and simultaneous games under various forms of uncertainty. In application to network security, Venkitasubramaniam and Tong [64] investigate anonymous wireless networking, formalized as a zero-sum game between the network designer and the attacker. The task of the attacker is to choose a subset of nodes to monitor so that the anonymity of routes is minimum whereas the task of the designer is to maximize anonymity by choosing nodes to evade flow detection by generating independent transmission schedules.

Khouzani et al. [47] present a framework for analyzing a trade-off between usability and security. They analyze guessing attacks and derive the optimal policies for secret picking as Nash/Stackelberg equilibria. Khouzani and Malacaria [43, 44, 45] study the problem of optimal channel design in the presence of hard and soft constraints. They employ entropies (the dual of vulnerability) and, to capture the widest class of leakages, they consider the property of core-concavity, a generalization of concavity that includes entropies that are not concave (e.g., like the Rényi entropies when α > 1). They show the existence of universally optimal strategies in the context of a two-player zero-sum game with incomplete information and that the defender’s Bayes-Nash equilibrium strategies are solutions of convex programming problems. Furthermore, they show that for any choice of the entropy measure, this problem can be solved via convex programming with zero duality gap, for which the Karush-Kuhn-Tucker (KKT) conditions can be used. In summary, their work uses game theory to support the design of optimal channels within operational constraints, whereas ours focuses on modeling the interplay between attacker and defender regarding information leakage of given channels, and to reason about their optimal strategies.

Concerning costs of security, Yang et al. [68] propose a framework to analyze user behavior in anonymity networks. The utility is modeled as a combination of weighted cost and anonymity utility. They also consider incentives and their impact on users’ cooperation.

Some security games have considered leakage of information about the defender’s choices. For example, Alon et al. [2] present two-player zero-sum games where a defender chooses probabilities of secrets while an attacker chooses and learns some of the defender’s secrets. Then they show how the leakage on the defender’s secrets influences the defender’s optimal strategy. Xu et al. [67] present zero-sum security games where the attacker acquires partial knowledge on the security resources the defender is protecting, and show the defender’s optimal strategy under such attacker’s knowledge. More recently, Farhang and Grossklags [37] presented two-player games where utilities are defined taking account of information leakage, although the defender’s goal is different from our setting. They consider a model where the attacker incrementally and stealthily obtains partial information on a secret, whereas the defender periodically changes the secret after some time to prevent a complete compromise of the system. In particular, the defender is not attempting to minimize the leak of a certain secret, but only to make it useless (for the attacker). Hence, their model of defender and utility is totally different from ours.

The research area of QIF has been dedicated to the development of theories to quantify and mitigate the amount of information leakage, including foundational works [12, 18, 29, 30, 49, 63], verification of QIF properties [16, 27, 28, 31, 50], and mitigation of information leakage [11, 44, 45, 46]. In this research area, several studies [4, 17, 40, 53, 56] have quantitatively modeled and analyzed the information leakage in interactive systems by using different approaches than ours. Boreale and Pampaloni [17] model adaptive attackers interacting with systems and investigate the relationship between adaptive and non-adaptive adversaries, although they do not deal with probabilistic strategies for the attacker. Mardziel et al. [53] formalize adaptive attackers using probabilistic finite automata and analyze the information leakage of several case studies using a probabilistic programming language, whereas they do not show any extensive case studies that use probabilistic attack strategies to maximize leakage. Alvim et al. [4] shows that interactive systems can be modeled as information-theoretic channels with memory and feedback without using game theory to model the adaptiveness of agents. Kawamoto and Given-Wilson [40] show a quantitative model of information leakage in scheduler-dependent systems, although they assume only a passive observer who receives traces interleaved by some scheduler.

To the best of our knowledge, however, no other work has explored games with utilities defined as information-leakage measures, except the preliminary version of this article [5], which introduced the QIF-games (called information leakage games there), and our subsequent work [6, 7], which extended the QIF-games to sequential games, and also considered the case in which the defender’s choice is visible to the attacker.

Finally, in game theory, Matsui [55] uses the term information leakage game with a meaning different than ours, namely as a game in which (part of) the strategy of one player may be leaked in advance to the other player, and the latter may revise his strategy based on this knowledge.

1.4 Plan of the Article

In Section 2, we review fundamental concepts from game theory, QIF, and DP. In Section 3, we formalize the concept of QIF-games, in which utility is the vulnerability of a channel, and provide a series of examples of such games. We then derive the existence of Nash equilibria for these games and provide a method for computing them. In Section 4, we formalize the concept of DP-games, in which utility is the level ε of DP provided by a system. We demonstrate that such utility functions are quasi-convex or quasi-max, depending, respectively, on whether the defender’s action is hidden from or visible to the attacker. We derive the existence of Nash equilibria for these games and provide a method to compute them. In Section 5, we compare our information leakage games with the axiomatic formulation of standard game theory and show why the latter does not capture our games. In Section 6, we apply our framework to two case studies: a version of the Crowds protocol and a design of local DP mechanisms for an adaptive disclosure of information correlated with secrets. Finally, in Section 7, we present our final remarks. The proofs of the formal results are presented in Appendix B.

2 Preliminaries

In this section, we review some basic notions from game theory, QIF, and DP.

We use the following notation. Given a set \(\mathcal {I}\), we denote by \(\mathbb {D}\mathcal {I}\) the set of all probability distributions over \(\mathcal {I}\). Given \(\mu \in \mathbb {D}\mathcal {I}\), its support\({\sf supp}(\mu)\) is the set of its elements with positive probabilities (i.e., \({\sf supp}(\mu) = \lbrace i \in \mathcal {I}: \mu (i)\gt 0 \rbrace\)). We write i ← μ to indicate that a value i is sampled from a distribution μ.

2.1 Convexity and Quasi-Convexity

We recall that a set \(\mathcal {S}\subseteq \mathbb {R}^n\) is convex if \(ps_0 + (1-p)s_1 \in \mathcal {S}\) for all \(s_0, s_1\in \mathcal {S}\) and p ∈ [0, 1]. Let S be a convex set. A function \(f:\mathcal {S}\rightarrow \mathbb {R}\) is convex if f(ps₀ + (1 − p)s₁) ≤ pf(s₀) + (1 − p)f(s₁) for all \(s_0, s_1\in \mathcal {S}\) and p ∈ [0, 1], and it is concave if − f is convex. We say that f is quasi-convex if f(ps₀ + (1 − p)s₁) ≤ max {f(s₀), f(s₁)} for all \(s_0, s_1\in \mathcal {S}\) and p ∈ [0, 1], and it is quasi-concave if − f is quasi-convex. It is easy to see that if f is convex then it is also quasi-convex, and if it is concave then it is also quasi-concave.

2.2 Two-Player, Simultaneous Games

We recall basic definitions from two-player games, a model for reasoning about the behavior of two strategic players. We refer to the work of Osborne and Rubinstein [58] for more details.

In a game, each player disposes of a set of actions that he can perform and obtains some payoff (gain or loss) depending on the outcome of the actions chosen by both players. The payoff’s value to each player is evaluated using a utility function. Each player is assumed to be rational—that is, his choice is driven by the attempt to maximize his own utility. We also assume that the set of possible actions and the utility functions of both players are common knowledge.

In this article, we only consider finite games, namely the cases in which the set of actions available to each player is finite. Furthermore, we only consider simultaneous games, in which each player chooses actions without knowing the actions chosen by the other.

Formally, a finite simultaneous game between a defender and an attacker is defined as a tuple \((\mathcal {D}, \mathcal {A}, {\it u}_{\sf d}, {\it u}_{\sf a})\), where \(\mathcal {D}\) is a nonempty set of the defender’s actions, \(\mathcal {A}\) is a nonempty set of the attacker’s actions, \({\it u}_{\sf d}: \mathcal {D}\times \mathcal {A}\rightarrow \mathbb {R}\) is the defender’s utility function, and \({\it u}_{\sf a}: \mathcal {D}\times \mathcal {A}\rightarrow \mathbb {R}\) is the attacker’s utility function.

Each player may choose an action deterministically or probabilistically. A pure strategy of the defender (respectively, attacker) is a deterministic choice of an action—that is, an element \(d\in \mathcal {D}\) (respectively, \(a\in \mathcal {A}\)). A pair (d, a) of the defender’s and attacker’s pure strategies is called a pure strategy profile. The values \({\it u}_{\sf d}(d, a)\) and \({\it u}_{\sf a}(d, a)\) respectively represent the defender’s and the attacker’s utilities.

A mixed strategy of the defender (respectively, attacker) is a probabilistic choice of an action, defined as a probability distribution \(\delta \in \mathbb {D}\mathcal {D}\) (respectively, \(\alpha \in \mathbb {D}\mathcal {A}\)). A pair (δ, α) of the defender’s and attacker’s mixed strategies is called a mixed strategy profile. The defender’s and the attacker’s expected utility functions for mixed strategies are respectively defined as follows:

A defender’s mixed strategy \(\delta \in \mathbb {D}\mathcal {D}\) is a best response to an attacker’s mixed strategy \(\alpha \in \mathbb {D}\mathcal {A}\) if \({\it U}_{\sf d}(\delta , \alpha) = \max _{\delta ^{\prime }\in \mathbb {D}\mathcal {D}}{\it U}_{\sf d}(\delta ^{\prime }, \alpha)\). Symmetrically, \(\alpha \in \mathbb {D}\mathcal {A}\) is a best response to \(\delta \in \mathbb {D}\mathcal {D}\) if \({\it U}_{\sf a}(\delta , \alpha) = \max _{\alpha ^{\prime }\in \mathbb {D}\mathcal {A}}{\it U}_{\sf d}(\delta , \alpha ^{\prime })\). A mixed strategy Nash equilibrium is a profile (δ^*, α^*) such that δ^* is a best response to α^* and vice versa. This means that in a Nash equilibrium, no unilateral deviation by any single player provides better utility to that player. If δ^* and α^* are point distributions concentrated on some pure strategies \(d^*\in \mathcal {D}\) and \(a^*\in \mathcal {A}\), respectively, then (δ^*, α^*) is a pure strategy Nash equilibrium, and is denoted by (d^*, a^*). Although every finite game has a mixed strategy Nash equilibrium, not all games have a pure strategy Nash equilibrium.

2.3 Zero-Sum Games and Minimax Theorem

A game \((\mathcal {D},\, \mathcal {A},\, {\it u}_{\sf d}, {\it u}_{\sf a})\) is zero-sum if for any \(d\in \mathcal {D}\) and any \(a\in \mathcal {A}\), \({\it u}_{\sf d}(d, a) = -{\it u}_{\sf a}(d, a)\)—that is, the defender’s loss is equivalent to the attacker’s gain. For brevity, in zero-sum games we denote by u the attacker’s utility function \({\it u}_{\sf a}\), and by U the attacker’s expected utility \({\it U}_{\sf a}\).¹ Consequently, the goal of the defender is to minimize U, and the goal of the attacker is to maximize it.

In simultaneous zero-sum games, a Nash equilibrium corresponds to a solution of the minimax problem—that is, a mixed strategy profile (δ^*, α^*) such that U(δ^*, α^*) = min _δmax _αU(δ, α). (This is equivalent to the maximin problem, which looks for (δ^*, α^*) such that U(δ^*, α^*) = max _αmin _δU(δ, α).) Von Neumann’s minimax theorem ensures that such a solution always exists and is stable.

Theorem 2.1 (Von Neumann’s Minimax Theorem).

Let \(\Delta \subset \mathbb {R}^m\) and \(A \subset \mathbb {R}^n\) be compact convex sets, and \({\it U}: \Delta \times A\rightarrow \mathbb {R}\) be a continuous function such that U(δ, α) is convex in δ ∈ Δ and concave in α ∈ A. Then it is the case that

\begin{equation*} \min _{\delta \in \Delta } \max _{\alpha \in A} {\it U}(\delta , \alpha) = \max _{\alpha \in A} \min _{\delta \in \Delta } {\it U}(\delta , \alpha). \end{equation*}

Moreover, under the conditions of Theorem 2.1, there exists a saddle point (δ^*, α^*) s.t., for all δ ∈ Δ and α ∈ A, U(δ^*, α) ≤ U(δ^*, α^*) ≤ U(δ, α^*).

2.4 Quantitative Information Flow

Next we recall the standard framework of quantitative information flow (QIF), which is used to measure the amount of information leakage in a system [9]. Notably, we define notions of “secret,” an adversary’s “prior knowledge” about the secret (or simply, “prior”), and a “vulnerability measure” to gauge how well the adversary can exploit her knowledge about the secret. We also define “channels,” probabilistic mappings used to model systems with observable behavior that changes the adversary’s probabilistic knowledge, making the secret more vulnerable and hence causing information “leakage.” Throughout this article, we remain faithful to terminology and a presentation style that is well established in the field of QIF [9].

A secret is some piece of sensitive information the defender wants to protect, such as a user’s password, social security number, or current geo-location. The attacker often has some partial knowledge about the secret values, which is represented as a probability distribution on secrets called a prior distribution (or simply a prior). We denote by \(\mathcal {X}\) the set of all possible secrets, and we typically use π to denote a prior that belongs to the set \(\mathbb {D}{\mathcal {X}}\) of distributions over \(\mathcal {X}\).

The vulnerability of a secret is a measure of the usefulness of the attacker’s knowledge about the secret. In this article, we consider a very general notion of vulnerability, following Alvim et al. [8], and define a vulnerability measure \(\mathbb {V}\) to be any continuous and convex function of type \(\mathbb {D}{\mathcal {X}} \rightarrow \mathbb {R}\). It has been shown in Alvim et al. [8] that these functions coincide with the set of g-vulnerabilities, and are, in a precise sense, the most general information measures w.r.t. a set of basic axioms.²

Systems can be modeled as information-theoretic channels. A channel\(C : \mathcal {X}\times \mathcal {Y}\rightarrow \mathbb {R}\) is a function in which \(\mathcal {X}\) is a finite set of input values, \(\mathcal {Y}\) is a finite set of output values, and C(x, y) represents the conditional probability of the channel producing output \(y \in \mathcal {Y}\) when input \(x \in \mathcal {X}\) is provided. Every channel C satisfies 0 ≤ C(x, y) ≤ 1 for all \(x\in \mathcal {X}\) and \(y\in \mathcal {Y}\), and \(\sum _{y\in \mathcal {Y}} C(x,y) = 1\) for all \(x\in \mathcal {X}\).

A prior distribution \(\pi \in \mathbb {D}{\mathcal {X}}\) and a channel C with inputs \(\mathcal {X}\) and outputs \(\mathcal {Y}\) induce a joint distribution p_{X, Y}(x, y) = π(x)C(x, y) on \(\mathcal {X}\times \mathcal {Y}\), with marginal probabilities p_X(x) = ∑_yp_{X, Y}(x, y) and p_Y(y) = ∑_xp_{X, Y}(x, y), and conditional probabilities \(p_{X \mid y}(x{\mid }y) = {p_{X,Y}(x,y)}{p_{Y}(y)}\) if p_Y(y) ≠ 0. For a given y (s.t. p_Y(y) ≠ 0), the conditional probability distribution p_X∣y is called the posterior distribution on \(\mathcal {X}\) given y. When clear from the context, we may omit subscripts on probability distributions, writing, for example, p(y), p(x, y), and p(x | y) for p_Y(y), p_{X, Y}(x, y), and p_X|y(x | y), respectively.

A channel C in which \(\mathcal {X}\) is a set of secret values and \(\mathcal {Y}\) is a set of observable values produced by a system can be used to model computations on secrets. Assume that the attacker has prior knowledge π about the secret value, knows how a channel C works, and can observe the channel’s outputs. Then the effect of the channel is to update the attacker’s knowledge from the prior π to a collection of posteriors p_X∣y, each occurring with probability p(y).

Given a vulnerability measure \(\mathbb {V}\), a prior π, and a channel C, the posterior vulnerability\(\mathbb {V}\!\left[\pi ,C\right]\) is the vulnerability of the secret after the attacker has observed the output of C. Formally,

\begin{align} \mathbb {V}\!\left[\pi ,C\right] \stackrel{\mathrm{def}}{=}&\, \mathop{\mathbb{E}}\limits_{y \leftarrow p_{Y}} \mathbb {V}[{p_{X \mid y}}]\!. \end{align}

(1)

The information leakage of a channel C under a prior π is a comparison between the vulnerability of the secret before the system was run, called the prior vulnerability, and the posterior vulnerability of the secret. The leakage reflects how much the observation of the system’s outputs increases the utility of the attacker’s knowledge about the secret. It can be defined either additively (by \(\mathbb {V}\!\left[\pi ,C\right]-\mathbb {V}\!\left[\pi \right]\)), or multiplicatively (by \({\mathbb {V}\!\left[\pi ,C\right]}{\mathbb {V}\!\left[\pi \right]}\)). Additive leakage is a measure of an adversary’s absolute gain of information, and the resulting value retains the same meaning as the original vulnerability functions, such as the adversary’s “monetary gain” or “damage caused.” Multiplicative leakage, however, is a dimensionless scalar representing the adversary’s relative gain of information, usually interpreted as a percentage.

2.5 Differential Privacy

Next we briefly recall the notion of of differential privacy (DP) [32, 35]. Consider a system that takes inputs from a set \(\mathcal {X}\), endowed with an adjacency relation\(\sim \subseteq \mathcal {X}\times \mathcal {X}\) representing which pairs of inputs should be considered (almost) “indistinguishable.” We say that x and x′ are adjacent if x ∼ x′. Intuitively, in the discrete case, a system (or mechanism) that takes inputs from \(\mathcal {X}\) is differentially private if the probability of its producing any output given input \(x \in \mathcal {X}\) is roughly the same as the probability of its producing the same output given any other input \(x^{\prime } \in \mathcal {X}\) adjacent to x. Using the channel notation to represent a system/mechanism, this intuition is formalized as follows.

Definition 1 (ε-differential Privacy).

Let ε ≥ 0 and \(\sim \subseteq \mathcal {X}\times \mathcal {X}\) be an adjacency relation of secrets. A channel \(C: \mathcal {X}\times \mathcal {Y}\rightarrow \mathbb {R}\) provides ε-differential privacy if for any pair of values x, x′ s.t. x ∼ x′, and for any output value \(y\in \mathcal {Y}\),

\begin{equation*} C(x, y) \le e^\varepsilon C(x^{\prime }, y), \end{equation*}

where C(x, y) (respectively, C(x′, y)) is the probability of producing y on input x (respectively, x′).

Note that our definition of DP is in line with the influential Pufferfish framework by Kifer and Machanavajjhala [48], in which a mechanism’s (secret) input domain can be arbitrary, and the adjacency relation can be customized to explicitly define what is considered to be secret. Traditional DP is a particular instantiation of this definition in which the (secret) input set consists in datasets, and the adjacency relation is defined to reflect the usual distinction between the presence or absence of any single individual in a dataset. It is noteworthy that our abstract definition of DP also encompasses local DP, first formalized by Kasiviswanathan et al. [38]. More precisely, one only needs to define the set of secrets to be, for example, values of an attribute for a single individual, and the adjacency relation to be that in which any two distinct values are adjacent.

3 QIF-games

In this section, we investigate QIF-games, in which the utility function is the information leakage of a channel. We demonstrate that the utility function is convex on mixed strategies, and that there exist Nash equilibria for these games. We then show how to compute these equilibria.

3.1 Definition of QIF-games

We introduce our games through a couple of examples to motivate the use of game theory. In the first example (the two millionaires problem) the utility happens to behave like in standard game theory, but it is just a particular case. In general, there is a fundamental difference, and we will illustrate this with our second example (the binary-sum game).

3.1.1 The Two Millionaires Problem.

The two millionaires problem was introduced by Yao [69]. In the original formulation, there are two “millionaires,” Alex and Don, who want to discover who is the richest among them, but neither wants to reveal to the other the amount of money they have.

We consider a (conceptually) asymmetric variant of this problem, where Alex is the attacker and Don is the defender. Don wants to learn whether or not he is richer than Alex, but does not want Alex to learn anything about the amount x of money he has. For this purpose, Don sends x to a trusted server, Jeeves, who in turn asks Alex, privately, what is her amount a of money. Jeeves then checks which among x and a is greater, and sends the result y back to Don.³ However, Don is worried that Alex may intercept Jeeves’ message containing the result of the comparison, and exploit it to learn more accurate information about x by tuning her answer a appropriately (since, given y, Alex can deduce whether a is an upper or lower bound on x). Note that this means, effectively, that a is the choice of action given to Alex in the game. We assume that Alex may get to know Jeeves’ reply, but not the messages from Don to Jeeves.⁴ See Figure 1.

Fig. 1.

We will use the following information flow terminology: the information that should remain secret (to the attacker) is called high, and what is visible to (and possibly controllable by) the attacker is called low. Hence, in the program run by Jeeves, a is a low input and x is a high input. The result y of the comparison (since it may be intercepted by the attacker) is a low output. The problem is to avoid the flow of information from x to y (given a).

One way to mitigate this problem is to use randomization. Assume that Jeeves provides two different programs to ensure the service. Then, when Don sends his request to Jeeves, he can make a random choice d among the two programs 0 and 1, sending d to Jeeves along with the value x. Now, if Alex intercepts the result y, it will be less useful to her since she does not know which of the two programs has been run. As Don, of course, knows which program was run, the result y will still be just as useful to him.⁵

To determine the best probabilistic strategy that Don should apply to select the program, we analyze the problem from a game-theoretic perspective. For simplicity, we assume that x and a both range in {0, 1}. The two alternative programs that Jeeves can run are shown in Figure 2.

Fig. 2.

The combined choices of Alex and Don determine how the system behaves. Let \(\mathcal {D}= \lbrace 0,1\rbrace\) represent Don’s possible choices (i.e., the program to run), and \(\mathcal {A}= \lbrace 0,1\rbrace\) represent Alex’s possible choices (i.e., the value of the low input a). We shall refer to the elements of \(\mathcal {D}\) and \(\mathcal {A}\) as actions. For each possible combination of actions d and a, we can construct a channel C_da with inputs \(\mathcal {X}= \lbrace 0,1\rbrace\) (the set of possible high-input values) and outputs \(\mathcal {Y}= \lbrace T, F\rbrace\) (the set of possible low-output values), modeling the behavior of the system from the point of view of the attacker. Intuitively, each channel entry C_da(x, y) is the probability that the program run by Jeeves (which is determined by d) produces output \(y\in \mathcal {Y}\) given that the high input is \(x \in \mathcal {X}\) and that the low input is a. Figure 3 presents the four channel matrices representing all possible behaviors of the program. Note that channels C₀₁ and C₁₀ do not leak any information about the input x (output y is constant), whereas channels C₀₀ and C₁₁ completely reveal x (output y is in a bijection with x).

Fig. 3.

We now investigate how the defender’s and the attacker’s strategies influence the leakage of the system. For that, we can consider the (simpler) notion of posterior vulnerability, since, for any given prior, the value of leakage is in a one-to-one, monotonic correspondence with the value of posterior vulnerability.⁶ For this example, we consider posterior Bayes vulnerability [24, 63], defined as

\begin{align*} \mathbb {V}^{\mathrm{Bayes}}\!\left[\pi ,C\right]\overset{def}{=}\mathop{\mathbb{E}}\limits_{y \leftarrow p_{Y}}\mathop{\text{max}}\limits_x\;{p{(x\,|\,y)}}\mathop{\sum}\limits_y\;\;\mathop{\text{max}}\limits_x\;\;\pi{(x)}C{(x,y)} \end{align*}

which measures the expected probability of the attacker guessing the secret correctly in one try after having observed the output of the system. It can be shown that \(\mathbb {V}^{\mathrm{Bayes}}\,[\pi ,C]\) coincides with the converse of the Bayes error.

For simplicity, we assume a uniform prior distribution π_u. It has been shown that, in this case, the posterior Bayes vulnerability of a channel C can be computed as the sum of the greatest elements of each column of C, divided by the size of the high-input domain [21]. Namely,

\begin{align*} \mathbb {V}^{\mathrm{Bayes}}\!\left[\pi _u,C\right] = \frac{\sum _{y}\max _xC(x,y)}{|\mathcal {X}|}. \end{align*}

Simple calculations yield the results \(\mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{00}]= \mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{11}] = 1\) and \(\mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{01}]= \mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{10}] = {1}{2}\), which are summarized in the utility table of Figure 4, which is similar to that of the well-known “matching pennies” game.

Fig. 4.

As in standard game theory, there may not exist an optimal pure strategy profile. The defender, as well as the attacker, can then try to minimize/maximize the system’s vulnerability by adopting mixed strategies δ and α, respectively. A crucial task is evaluating the vulnerability of the system under such mixed strategies. This evaluation is naturally performed from the attacker’s point of view, who knows his own choice a, but not the defender’s choice d (hidden choice).

To define precisely the vulnerability of a mixed strategy, let us introduce some notation: given a channel matrix C and a scalar k, k C is the matrix obtained by multiplying every element of C by k. Given two compatible channel matrices C₁ and C₂, namely matrices with the same indices of rows and columns,⁷ C₁ + C₂ is obtained by adding the cells of C₁ and C₂ with same indices. Note that if μ is a probability distribution on \(\mathcal {I}\), then \({\mathop{\mathbb {E}}\limits_{i\leftarrow \mu }} C_{i}\) is a channel matrix.

From the attacker’s point of view, the resulting system is the convex combination

\begin{align} C_{\delta a} \overset{def}{=}{\mathop{\mathbb {E}}\limits_{d\leftarrow \delta }} C_{da}\, \end{align}

(2)

(i.e., a probabilistic choice between the channels representing the defender’s actions).

The overall vulnerability of the system is then given by the vulnerability of C_δa, averaged over all attacker’s actions.

We now define formally the ideas illustrated previously using a simultaneous game (Section 2.2).

Definition 2 (

QIF-game) A QIF-game is a simultaneous game \((\mathcal {D}, \mathcal {A}, C, \mathbb {V}),\) where \(\mathcal {D}, \mathcal {A}\) are the finite sets of actions of the attacker and the defender, respectively; C = {C_da}_da is a family of channel matrices of same type indexed on pairs of actions \(d\in \mathcal {D},a\in \mathcal {A}\); and \(\mathbb {V}\) is a vulnerability measure. The game is zero-sum, and for a prior π, the utility for the attacker of a pure strategy profile (d, a) is given by \(\mathbb {V}\!\left[\pi ,C_{d a}\right]\). The utility \(\mathbb {V}(\delta ,\alpha)\) for the attacker of a mixed strategy profile (δ, α) is defined as

\begin{equation*} \mathbb {V}(\delta ,\alpha)\overset{def}{=}\mathop {\mathbb {E}}\limits_{\substack {a\leftarrow \alpha }}\hspace{-2.125pt} \mathbb {V}\!\left[\pi ,C_{\delta a}\right] =\textstyle {\sum _{a}\:} \alpha (a) \,\mathbb {V}\!\left[\pi ,C_{\delta a}\right], \end{equation*}

where C_δa is the hidden-choice channel composition defined as in (2).

In our example, δ can be represented by a single number p: the probability that the defender chooses d = 0 (i.e., Program 0). From the point of view of the attacker, once he has chosen a, the system will look like a channel C_pa = p C_0a + (1 − p) C_1a. For instance, in the case a = 0, if x is 0 Jeeves will send T with probability 1, but, if x is 1, Jeeves will send F with probability p and T with probability 1 − p. Similarly for a = 1. Figure 5 summarizes the various channels modeling the attacker’s point of view. It is easy to see that \(\mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{p0}] = {(1+p)}{2}\) and \(\mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{p1}] = {(2-p)}{2}\). In this case \(\mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{pa}]\) coincides with the expected utility with respect to p—that is, \(\mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{pa}] = p\,\mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{0a}]+(1- p)\,\mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{1a}]\).

Fig. 5.

Assume now that the attacker chooses a = 0 with probability q and a = 1 with probability 1 − q. The utility is obtained as the expectation with respect to the strategy of the attacker, hence the total utility is \(\mathbb {V}^{\mathrm{Bayes}}(p,q) = {q \,(1+p)}{2} + {(1-q)\,(2-p)}{2}\), which is affine in both p and q. By applying standard game-theoretic techniques, we derive that the optimal strategy is \((p^*,q^*) = ( {1}{2}, {1}{2})\).

In the preceding example, things work just like in standard game theory. However, the next example fully exposes the difference of our games with respect to those of standard game theory.

3.1.2 The Binary-Sum Game.

The previous example is an instance of a general scenario in which a user, Don, delegates to a server, Jeeves, a certain computation that also requires some input from other users. Here we will consider another instance, in which the function to be computed is binary-sum ⊕. We assume that Jeeves provides the programs in Figure 6. The resulting channel matrices are represented in Figure 7.

Fig. 6.

Fig. 7.

We consider again Bayes posterior vulnerability as utility. Simple calculations yield values \(\mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{00}]= \mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{11}] = \mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{01}]= \mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{10}] = 1\). Thus, for the pure strategies, we obtain the utility table shown in Figure 8. This means that all pure strategies have the same utility 1 and therefore they are all equivalent. In standard game theory, this would mean that also the mixed strategies have the same utility 1, since they are defined as expectation. In our case, however, the utility of a mixed strategy of the defender is convex on the distribution,⁸ so it may be convenient for the defender to adopt a mixed strategy. Let p and 1 − p be the probabilities of the defender choosing Program 0 and Program 1, respectively. From the point of view of the attacker, for each of his choices of a, the system will appear as the probabilistic channel C_pa represented in Figure 9. Simple calculations yield \(\mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{p0}] = \mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{p1}] = 1-p\) if \(p \le {1}{2}\), and \(\mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{p0}] = \mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{p1}] = p\) if \(p \ge {1}{2}\). However, with respect to a mixed strategy of the attacker, the utility is still defined as expectation. Since in this case the utility is the same for a = 0 and a = 1, it remains the same for any strategy of the attacker. Formally, \(\mathbb {V}^{\mathrm{Bayes}}(p,q) = q\, \mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{p0}] + (1-q) \, \mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{p1}] = \mathbb {V}^{\mathrm{Bayes}}\,[\pi _{u},C_{p0}]\), which does not depend on q and it is minimum for \(p= {1}{2}\). We conclude that the point of equilibrium is \((p^*,q^*)=( {1}{2},q^*)\) for any value of q^*.

Fig. 8.

Fig. 9.

3.2 Existence of Nash Equilibrium for QIF-games

In our earlier work [6], it was proved that the posterior vulnerability of a convex combination of channels is smaller than or equal to the convex combination of their vulnerabilities. As a consequence, posterior vulnerability is a convex function of the strategy of the defender, and using von Neumann’s minimax theorem we are able to derive the existence of the Nash equilibrium for QIF-games.

Theorem 3.1 (Convexity of Vulnerability w.r.t. Channel Composition [6]).

Let \(\lbrace C_{i}\rbrace _{i \in \mathcal {I}}\) be a family of channels of the same type, all with domain \(\mathcal {X}\). Then, for every prior distribution \(\pi \in \mathbb {D}{\mathcal {X}}\), every vulnerability \(\mathbb {V}\), and any probability distribution μ on \(\mathcal {I}\), we have that

\begin{align*} \mathbb {V}\bigl [\pi , \mathop{\mathbb{E}}\limits_{i\leftarrow \mu } C_{i} \bigr ] \le \mathop{\mathbb{E}}\limits_{i\leftarrow \mu } \mathbb {V}\!\left[\pi ,C_{i}\right]{.} \end{align*}

As formalized in the next result, the existence of Nash equilibria for QIF-games immediately follows from the preceding theorem.

Corollary 3.2 (Existence of Nash Equilibrium for.

QIF-games) For any (zero-sum) QIF-game, there exists a Nash equilibrium, which in general is given by a mixed strategy.

3.3 Computing Equilibria for QIF-games

Recall that the utility of a mixed strategy profile (δ, α), given by Definition 2, is

\begin{equation*} \mathbb {V}(\delta ,\alpha) = \mathop{\mathbb{E}}\limits_{a\leftarrow \alpha } \mathbb {V}\bigl [\pi , \mathop{\mathbb{E}}\limits_{d\leftarrow \delta } C_{da} \bigr ]\,, \end{equation*}

and that \(\mathbb {V}(\delta ,\alpha)\) is convex on δ and affine on α. Theorem 2.1 guarantees the existence of an equilibrium (i.e., a saddle point) (δ^*, α^*), which is a solution of both the minimax and the maximin problems. The goal in this section is to compute (i) a δ^* that is part of an equilibrium, which is important to optimize the defense, and (ii) the utility \(\mathbb {V}(\delta ^*,\alpha ^*)\), which is important to provide an upper bound on the effectiveness of an attack when δ^* is applied.

This is a convex-concave optimization problem for which various methods have been proposed in the literature. If \(\mathbb {V}\) is twice differentiable (and satisfies a few extra conditions), then the Newton method can be applied [20]; however, many \(\mathbb {V}\)’s of interest, and notably the Bayes vulnerability, one of the most popular vulnerability measures, are not differentiable. For non-differentiable functions, Nedić and Ozdaglar [57] propose an approximate iterative method that computes a subgradient and updates both α and δ at each step in the direction of the subgradient, moving toward the point of equilibrium. The process stops when the change in the value of \(\mathbb {V}(\delta ,\alpha)\) induced by the update of α and δ is “small enough,” meaning that we are close to \(\mathbb {V}(\delta ^*,\alpha ^*)\), and the current values of α and δ are returned as approximations of δ^* and α^*. Some precautions must be taken to ensure stability—that is, to avoid swinging around the point of equilibrium: the size of the updates needs to become smaller as we get closer to it. Clearly, there is a trade-off between the precision of the approximation and the convergence speed.

In case we are interested in computing only the optimal strategy of the defender, however, we propose a method more efficient than the one of Nedić and Ozdaglar [57]. Our method is iterative like the one in the work of Nedić and Ozdaglar [57], but in contrast to their work [57], at every step it performs the update only on the δ component, whereas it computes the exact \(\text{argmax}\) on the α component, thus improving efficiency. For this purpose, we exploit the fact that \(\mathbb {V}(\delta ,\alpha)\) is affine on α (not just concave). This means that, for a fixed δ, maximizing \(\mathop{\mathbb{E}}_{a\leftarrow \alpha } \mathbb {V}\!\left[\pi ,\mathop{\mathbb{E}}_{d\leftarrow \delta } C_{da}\right]\) simply involves selecting an action a with the highest \(\mathbb {V}\!\left[\pi ,\mathop{\mathbb{E}}_{d\leftarrow \delta } C_{da}\right]\) and assigning probability 1 to it.

Proposition 3.3.

The minimax problem \(\delta ^* = \text{argmin}_\delta \max _\alpha \mathbb {V}(\delta ,\alpha)\) is equivalent to

\begin{equation} \delta ^* = \text{argmin}_\delta f(\delta) \quad \mbox{where} \quad f(\delta) = \max _a\mathbb {V}\!\left[\pi ,\textstyle \mathop{\mathbb{E}}_{d\leftarrow \delta } C_{da}\right]. \end{equation}

(3)

It is important to point out that solving (3) produces a δ^* that is the defender’s mixed strategy of an equilibrium, but in general it does not produce the attacker’s mixed strategy α^* of the equilibrium. For the latter, we would need to solve the maximin problem.

Note that f is a function of a single variable δ, and our problem reduces to computing its minimum value. We now proceed to apply the subgradient descent method for this task. We start by recalling the notions of subgradient and projection on the simplex.

Definition 3 (Subgradient).

Let U be a convex open set in the Euclidean space \(\mathbb {R}^n\). A subgradient of a convex function \(f:U\rightarrow \mathbb {R}\) at a point x₀ in U is any vector \(v\in \mathbb {R}^n\) such that, for all x ∈ U,

\begin{equation*} f(x)-f(x_{0})\ge v\cdot (x-x_{0}), \end{equation*}

where “ · ” denotes the dot product between vectors.

Definition 4 (Projection).

The n-simplex \(\mathbb {S}^n\) is the set of all vectors in \(\mathbb {R}^n\) that represent probability distributions. Given a vector \(v\in \mathbb {R}^n\), the projection of v on \(\mathbb {S}^n\), which we will denote by P(v), is the probability distribution \(\delta \in \mathbb {S}^n\) that is closest, in the Euclidean sense, to v.

Note that if \(v\in \mathbb {S}^n,\) then P(v) = v. To compute the projection, we can use the efficient algorithms proposed in the work of Wang and Carreira-Perpinán [66].

We are now ready to define our iterative method for the approximation of δ^* and \(\mathbb {V}(\delta ^*,\alpha ^*)\). The idea is to start from an arbitrary fully supported distribution over \(\mathcal {D}\), for instance, the uniform distribution \(u_\mathcal {D}\), and then compute the next δ from previous one following (3). More precisely, at step k + 1, we compute δ^{(k + 1)} from δ^(k) according to the following inductive definition:

\begin{equation} \begin{array}{c} \delta ^{(1)} = u_\mathcal {D}, \\ \delta ^{(k+1)} = P\left(\delta ^{(k)} - s_k h^{(k)}\right), \\ {\mbox{where}}\; s_k = {0.1}{\sqrt {k}}~ \mbox{ and} h^{(k)} \mbox{is any subgradient of} f(\delta) = \max _a\mathbb {V}\!\left[\pi ,\mathop{\mathbb{E}}_{d\leftarrow \delta } C_{da}\right] \mbox{at point} \delta ^{(k)}. \end{array} \end{equation}

(4)

The step size s_k has to be decreasing in k to avoid the swinging effect, as explained earlier. There are various choices for s_k that guarantee convergence [19]. We have chosen \(s_k= {0.1}{\sqrt {k}}\) because we have determined heuristically that it gives good performance.

We use the same stopping criterion of Boyd and Mutapcic [19, Section 3.4]. Namely, at each step k, we compute the following value l^(k), which is guaranteed to be a lower bound on f(δ^*):

\begin{equation} l^{(k)} = \frac{ 2\sum _{i=i}^{k} s_i f(\delta ^{(i)}) - \frac{|\mathcal {X}|-1}{|\mathcal {X}|} - \sum _{i=i}^{k} s_i^2\Vert h^{(i)}\Vert _2^2 }{2\sum _{i=i}^{k} s_i } . \end{equation}

(5)

Hence, we can stop the process when f(δ^(k)) − l^(k) ≤ ϵ, for some given ϵ > 0, and then the value returned is \(\hat{\delta }= \delta ^{(k)}\).⁹ The parameter ϵ determines the desired level of approximation, as formally stated in Theorem 3.4. Before stating the result, we need to recall the Lipschitz property.

Definition 5 (Lipschitz).

Let (A, d_A) and (B, d_B) be two metric spaces, and let G be a non-negative real constant. A function F: A → B is G-Lipschitz if for all a, a′ ∈ A,

\begin{equation*} d_B (F(a),F(a^{\prime })) \le G\cdot d_A(a,a^{\prime }). \end{equation*}

We also say that F is Lipschitz if it is G-Lipschitz for some G.

Theorem 3.4 (Convergence).

If \(\mathbb {V}\) is Lipschitz, then the sequence {δ^(k)}_k in (4) converges to a δ^* that is part of an equilibrium. Moreover, when the stopping condition is met (i.e., f(δ^(k)) − l^(k) < ϵ), then the approximate solution \(\hat{\delta }=\delta ^{(k)}\) returned by the algorithm, and for an equilibrium (δ^*, α^*),

\begin{equation*} \mathbb {V}(\hat{\delta }, \alpha) - \epsilon \le \mathbb {V}(\hat{\delta },\alpha ^*) \le \mathbb {V}(\delta ,\alpha ^*) + \epsilon \qquad \forall \delta ,\alpha . \end{equation*}

This implies that the computed strategy is ϵ-close to the optimal one, namely

\begin{equation*} |\mathbb {V}(\hat{\delta },\alpha ^*) - \mathbb {V}(\delta ^*,\alpha ^*) | \le \epsilon . \end{equation*}

The convergence crucially depends on the G-Lipschitzness assumption, which ensures the boundness of the subgradients and therefore that there is no “swinging effect” around the point of equilibrium. One may be tempted to also find a relation between the constant G and the convergence rate, but, unfortunately, there is no such relation in general. Essentially, this is because G is a global bound, whereas the convergence rate depends on the steepness around the point of equilibrium and how well it combines with the rate by which the step size diminishes.

Notably, the Lipschitz condition is satisfied by the large class of the g-vulnerability measures [10] when the “set of guesses” (or “actions”) is finite. The set of guesses \(\mathcal {W}\) is a parameter of a g-vulnerability measure, along with the gain function \(g:\mathcal {W}\times \mathcal {X}\rightarrow \mathbb {R}\). One example of g-vulnerability with finite \(\mathcal {W}\) is that of Bayes vulnerability: in this case, \(\mathcal {W}=\mathcal {X}\) and g(w, x) = 1 if w = x, and g(w, x) = 0 otherwise.

In general, if \(\mathcal {W}\) is finite, the posterior g-vulnerability \(\mathbb {V}\) is given by

\begin{equation*} \textstyle \mathbb {V}\!\left[\pi ,C\right] = \sum _{y}\, \max _{w}\, \sum _{x}\, \pi (x) \,C(x,y) \,g(w,x). \end{equation*}

The preceding \(\mathbb {V}\) is piecewise linear, and therefore G-Lipschitz, where G is the maximum of the norms of the subgradients at the point distributions. In general, a subgradient vector h^(k) is given by

\begin{equation*} h^{(k)}_d = \delta ^{(k)}(d) \,{\textstyle \sum _y}\, \pi (x^*_y)\, C_{da^*}(x^*_y,y), \end{equation*}

where \(a^*,x^*_y\) are (any of) the ones giving the max in the branches of f(δ^(k)).

Note that if \(\mathbb {V}\) is piecewise linear, then also f is piecewise linear. Hence, in the case of g-vulnerability, the convex optimization problem could be transformed into a linear one using a standard technique, then solved by linear programming. However, typically \(\mathbb {V}\) has a large number of max branches, and consequently this conversion can generate a huge number of constraints. In our experiments, we found that the subgradient method described earlier is significantly more efficient than linear programming (although the latter has the advantage of giving an exact solution).

However, not all vulnerabilities are Lipschitz; Shannon entropy, for instance, is not. Note that Shannon entropy can be expressed as g-vulnerability, but the corresponding set of guesses \(\mathcal {W}\) is infinite [8].

Finally, although the subgradient method is general, it might be impractical in applications where the number of attacker or defender actions is very large. Application-specific methods could offer better scalability in such cases; we leave the development of such methods as future work.

4 DP-games

In this section, we investigate DP-games, in which the utility function is the level of privacy of a channel, namely the minimum ε for which the channel is ε-differentially private.

We demonstrate that such utility functions are quasi-convex or quasi-max (rather than linear) on mixed strategies, depending, respectively, on whether the defender’s action is hidden from or visible to the attacker, and derive the existence of Nash equilibria for these games. We then show how to compute these equilibria.

4.1 Definition of DP-games

We begin by formalizing the level of DP of a channel, following an alternative characterization of DP based on max-divergence [36]. Let \(\mathcal {X},\mathcal {Y}\) be two sets (the domains of “secrets” and “observables,” respectively), and let ∼ be a symmetric binary relation on \(\mathcal {X}\)—that is, \(\sim \subseteq \mathcal {X}\times \mathcal {X}\) and such that x ∼ x′ iff x′ ∼ x. We say that a channel \(C:\mathcal {X}\times \mathcal {Y}\rightarrow \mathbb {R}\) is conforming to ∼ if for all x, \(x^{\prime } \in \mathcal {X}\) and \(y \in \mathcal {Y}\), x ∼ x′ implies that C(x, y) = 0 iff C(x′, y) = 0.

Note that the notions of domain and adjacency relation are more abstract than in the typical definition of DP. This is because we want to capture both the cases of the standard (central) DP and the local DP. In the central DP, the domain of secrets consists of datasets, and the adjacency relation x ∼ x′ holds iff x and x′ differ for one record. In the local DP, the domain of secrets consists of all possible values of a record, and the adjacency relation x ∼ x′ holds for all values x and x′ different from each other.

*Definition 6* (Differential Privacy Level of a Channel).

As shown in the work of Dwork and Roth [36], the relation with the standard notion of DP (cf. Definition 1) is that a channel C provides ε-differential privacy iff \(\mathbb {V}^{\rm DP}[C] \le \varepsilon\). In other words, \(\mathbb {V}^{\rm DP}[C]\) is the least value ε for which C is ε-differentially private. This presentation eases mathematical treatment, and effectively means that (i) \(0 \le \mathbb {V}^{\rm DP}[C] \lt \infty\), and (ii) \(\mathbb {V}^{\rm DP}[C] = \infty\) iff C is not conforming to ∼. Moreover, as usual, it is the case that the higher \(\mathbb {V}^{\rm DP}[C]\), the less private channel C is.

We will use the level of privacy (or more appropriately, non-privacy, or leakage) \(\mathbb {V}^{\rm DP}\) as the definition of utility in DP-games. In the next sections, we will see that the properties of \(\mathbb {V}^{\rm DP}\) are rather different from those of vulnerability in QIF-games.

4.2 DP-games vs. QIF-games

Intuitively, DP and QIF have a similar “philosophy”: in both cases, the attacker is trying to infer information about the secrets from the observables.¹⁰ The notions of attackers in QIF-games and DP-games differ significantly. In both cases, the attacker is trying to gain some knowledge about the secrets from the observables, but the modalities are very different. Specifically:

(A)

In DP-games, the prior does not appear in the definition of the payoff like it does in QIF-games. In DP-games, the payoff is simply the minimal ε such that ε-DP holds.

(B)

In DP-games, the attacker is not trying to maximize her expected posterior probability of success on all observables like in QIF-games, but rather the maximum distinguishability (i.e., the ratio of the likelihoods) of two secrets for an observable whatsoever. This is what we call “worst-case principle.”

(C)

From the game-theoretic point of view, this worst-case principle makes a big difference. In fact, we show that it implies that the leakage is not a convex function as in QIF-games, but only quasi-convex. Furthermore, even in the case of visible choice, the leakage is not linear as in QIF-games, but only quasi-max, which is a particular case of quasi-convexity.

4.3 Formal Definition of DP-games

Recall that for QIF-games, we defined (cf. (2)) the hidden-choice composition C_{δ, a} resulting from a mixed strategy δ by the defender and a pure strategy a by the attacker as

\begin{align*} C_{\delta a} \stackrel{\mathrm{def}}{=}\mathop{\mathbb{E}}_{d\leftarrow \delta } C_{da}. \end{align*}

In our DP-games, we will also consider the visible-choice composition modeling the case in which the attacker is able to identify the action taken by the defender. This is because even in the case of visible choice, the level of privacy is not linear w.r.t. the channel composition. Hence, visible-choice DP-games are not captured by standard game theory.

To define the latter formally, let us introduce some notation. Given a family \(\lbrace M_{i}\rbrace _{i \in \mathcal {I}}\) of compatible matrices s.t. each M_i has type \(\mathcal {X}\times \mathcal {Y}_{i} \rightarrow \mathbb {R}\), their concatenation \(\mathop { {\Diamond }}_{i \in \mathcal {I}}\) is the matrix having all columns of every matrix in the family, in such a way that every column is tagged with the matrix it came from. Formally, \(\left(\mathop { {\Diamond }}_{i \in \mathcal {I}} M_{i} \right)(x,(y,j)) = \,M_{j}(x,y)\), if \(y \in \mathcal {Y}_{j}\), and the resulting matrix has type \(\mathcal{X}\times{(\bigsqcup_{i\epsilon\mathcal{I}}\mathcal{Y}_i)} \rightarrow \mathbb{R}\).¹¹ Figure 10 provides an example of the concatenation of two matrices.

Fig. 10.

We now will define the “visible choice” among channel matrices, representing the situation in which a channel is probabilistically picked from a family of channels, and the choice of channel is revealed to the attacker. Formally, given \(\lbrace C_{i}\rbrace _{i \in \mathcal {I}}\) of compatible channels s.t. each C_i has type \(\mathcal {X}\times \mathcal {Y}_{i} \rightarrow \mathbb {R}\), and a distribution μ on \(\mathcal {I}\), their visible choice composition \({\bigsqcup\!\!\!\!\cdot}_{\;i \leftarrow \mu }\) is defined as

\begin{align} \mathop{\bigsqcup\!\!\!\!\!\!\cdot}_{i \leftarrow \mu } C_{i} = \mathop { {\Diamond }}_{i \in \mathcal {I}} \;\mu (i)\, C_{i}. \end{align}

(6)

It can be shown that the resulting matrix \({\bigsqcup\!\!\!\!\cdot}_{\;i \leftarrow \mu } C_{i}\) has type \(\mathcal {X}\times \left(\bigsqcup _{i \in \mathcal {I}} \mathcal {Y}_{i} \right) \rightarrow \mathbb {R}\), and it is also a channel [7]. Figure 11 depicts the visible choice among two channel matrices C₁ and C₂, with probability \({1}{3}\) and \({2}{3}\), respectively.

Fig. 11.

It is important to notice that both the result of a hidden choice composition as in (2) and of a visible choice composition as in (6) cannot increase the maximum level of DP provided by the operand channels, as formalized in Theorem 4.1 ahead. Intuitively, that is a consequence of the fact that both operators produce a new channel matrix only by concatenating, scaling, and summing up columns of the original channel matrices, and these three transformations cannot increase the maximum ratio between any two elements in a same column of the resulting matrix.

We are now ready to formalize the concept of DP-games.

Definition 7 (.

DP-game) A DP-game is a simultaneous game \((\mathcal {D}, \mathcal {A}, C),\) where \(\mathcal {D}, \mathcal {A}\) are the finite sets of actions of the attacker and the defender, respectively, and C = {C_da}_da is a family of compatible channel matrices, all conforming to a symmetric binary relation ∼ on their inputs, indexed on pairs of actions \(d\in \mathcal {D},a\in \mathcal {A}\). The game is zero-sum, and the utility for the attacker of a pure strategy profile (d, a) is given by \(\mathbb {V}^{\rm DP}[C_{d a}]\).

When the defender’s choice is hidden from the attacker, the attacker’s utility of a mixed strategy profile (δ, α) is defined as

\begin{equation*} \mathbb {V}^{\rm DP}(\delta ,\alpha)\stackrel{\mathrm{def}}{=}\mathbb {V}^{\rm DP}\left[{\bigsqcup\!\!\!\!\!\!\cdot _{\;a \leftarrow \alpha }}\,C_{\delta a}\right], \end{equation*}

where \(C_{\delta a} \stackrel{\mathrm{def}}{=}\mathop{\mathbb{E}}_{d\leftarrow \delta } \,C_{da}\) is the hidden choice as defined in (2).

When the defender’s choice is visible to the attacker, the attacker’s utility of a mixed strategy profile (δ, α) is defined as

\begin{equation*} \mathbb {V}^{\rm DP}(\delta ,\alpha)\stackrel{\mathrm{def}}{=}\mathbb {V}^{\rm DP}\left[{\bigsqcup\!\!\!\!\!\!\cdot _{\;a \leftarrow \alpha }}\,{\bigsqcup\!\!\!\!\!\!\cdot _{\;d \leftarrow \delta }}\,C_{d a}\right]. \end{equation*}

Note that in our definition of DP-games, the defender is interested in minimizing maximum leakage, whereas the adversary is interested in maximizing this same quantity. Note that it is reasonable for the defender always to try to minimize maximum leakage, as the guarantees of DP are intended to hold independently of the adversary. However, it would be reasonable to consider scenarios in which the adversary would not be necessarily interested in maximizing the channel’s maximum leakage, for example, because he adopts a different notion of relation ∼ than that used by the defender. In such a case, the resulting game would not be, in general, zero-sum. A study of such games, however, is beyond the scope of this work, and we therefore assume here that defender and attacker agree on the utility measure to be minimized/maximized.

Now we present an example of a DP-game and show that \(\mathbb {V}^{\rm DP}\) is not convex w.r.t. hidden choice.

Example 8 (\(\mathbb {V}^{\rm DP}\) is not convex w.r.t. hidden choice)

Let us consider a DP-game comprised of the action sets \(\mathcal {D}= \lbrace 0, 1\rbrace\), \(\mathcal {A}= \lbrace 0, 1\rbrace\), and four channels C_da shown in Figure 12 over an input domain \(\mathcal {X}=\lbrace x_0, x_1\rbrace\) and an output domain \(\mathcal {Y}=\lbrace y_0, y_1\rbrace\). Assume also that x₀ ∼ x₁. It is easy to see that the DP levels of these channels are \(\mathbb {V}^{\rm DP}[C_{00}]=\mathbb {V}^{\rm DP}[C_{11}]= \ln {\max \lbrace {0.90}{0.10}, {0.90}{0.10}\rbrace }=2.197\) and \(\mathbb {V}^{\rm DP}[C_{01}]=\mathbb {V}^{\rm DP}[C_{10}]= \ln {\max \lbrace {0.03}{0.01}, {0.99}{0.97}\rbrace }=1.099\). Now suppose that the attacker chooses a = 0, and that the defender chooses either d = 0 or d = 1 with probability \({1}{2}\). By Definition 6,

\begin{align*} \mathbb {V}^{\rm DP}[{\textstyle {1}{2}\,C_{0 0} + {1}{2}\,C_{0 1}}] &= \textstyle \max \bigl \lbrace \, \ln {0.455}{0.065},\, \ln {0.935}{0.545} \,\bigr \rbrace = 1.946 \\ \textstyle {1}{2}\,\mathbb {V}^{\rm DP}[C_{0 0}] + {1}{2}\,\mathbb {V}^{\rm DP}[C_{0 1}] &= \textstyle {1}{2} \cdot 2.197 + {1}{2} \cdot 1.099 = 1.648 {.} \end{align*}

Hence, we obtain \(\mathbb {V}^{\rm DP}[{\textstyle \sum _{d} {1}{2}\,C_{0 d}}] = 1.946 \gt 1.648 = \sum _{d} {\textstyle {1}{2}}\,\mathbb {V}^{\rm DP}[C_{0 d}]\), which implies that \(\mathbb {V}^{\rm DP}\) is not a convex function w.r.t. the defender’s hidden choice.

Fig. 12.

4.4 Existence of Nash Equilibrium for DP-games with Hidden Choice

We first show that the utility functions of DP-games are quasi-convex w.r.t. hidden choice composition, and quasi-max w.r.t. visible choice (the term quasi-max is just our way of indicating the property expressed in (2) of the following theorem).

Theorem 4.1 (\(\mathbb {V}^{\rm DP}\) is Quasi-convex/quasi-max on Channel Composition)

Let \(\lbrace C_{i}\rbrace _{i \in \mathcal {I}}\) be a family of compatible channels such that each C_i is conforming to a symmetric binary relation ∼ over their input set, and μ be a probability distribution on \(\mathcal {I}\). Then:

(1)

\(\mathbb {V}^{\rm DP}\) is quasi-convex w.r.t. hidden choice: \(\mathbb {V}^{\rm DP}[\mathop{\mathbb{E}}_{i\leftarrow \mu } C_{i} ] \le \max _{i\in {\sf supp}(\mu)} \mathbb {V}^{\rm DP}[C_{i}].\)

(2)

\(\mathbb {V}^{\rm DP}\) is quasi-max w.r.t. visible choice: \(\mathbb {V}^{\rm DP}[ {\bigsqcup\!\!\!\!\!\!\cdot _{\;i \leftarrow \mu }}C_i ] = \max _{i\in {\sf supp}(\mu)} \mathbb {V}^{\rm DP}[C_i].\)

We are now ready to provide an upper bound on the utility under any strategy in a DP-game with hidden choice.

Proposition 4.2 (Upper Bound on the Utility for Hidden Choice).

Now we prove the existence of a Nash equilibrium in DP-games as follows. Interestingly, there is an optimal strategy for the attacker that is independent of the defender’s strategy due to the fact that the attacker’s strategy is visible to himself.

Theorem 4.3 (Nash Equilibria for.

DP-games with Hidden Choice) For any DP-game with hidden choice, there exists a Nash equilibrium. Moreover, an arbitrary mixed strategy α^* such that \({\sf supp}(\alpha ^*) = \mathcal {A}\) is an optimal strategy for the attacker.

4.5 Computing Equilibria for DP-games with Hidden Choice

In this section, we show how to compute optimal strategies in DP-games with hidden choice of the defender’s action. Recall that by Theorem 4.3, an optimal strategy for the attacker is an arbitrary mixed strategy α^* such that \({\sf supp}(\alpha ^*) = \mathcal {A}\). To compute an optimal strategy for the defender, we use the following result.

Proposition 4.4 (Optimal Strategy for.

DP-games with Hidden Choice) For any DP-game with hidden choice, an optimal strategy for the defender can be obtained by linear programming.

Example 9 (Optimal strategy for a \(\texttt {DP}\)-game with hidden choice)

We use the DP-game with hidden choice in Example 8 to illustrate the DP level \(\mathbb {V}^{\rm DP}\) when changing the defender’s mixed strategy δ in Figure 13. Let α^* be an arbitrary mixed strategy such that \({\sf supp}(\alpha ^*) = \mathcal {A}\). Then the level of \(\mathbb {V}^{\rm DP}\) for α^* is the maximum of those for the pure strategies a = 0, 1; hence, α^* is an optimal strategy for the attacker. The defender’s optimal strategy δ^* is given by δ^*(0) ≈ 0.14 and δ^*(1) ≈ 0.86.

Fig. 13.

4.6 Nash Equilibria for DP-games with Visible Choice

Analogously, we obtain the following theorem under a visible choice of the defender’s action.

Theorem 4.5 (Nash Equilibrium for.

DP-games with Visible Choice) For any DP-game with visible choice, there exists a Nash equilibrium. Moreover, an arbitrary mixed strategy α^* such that \({\sf supp}(\alpha ^*) = \mathcal {A}\) is an optimal strategy for the attacker, and a pure strategy \(d^* \in \text{argmin}_d \max _a(\mathbb {V}^{\rm DP}[C_{da}])\) is an optimal strategy for the defender.

Example 10 (Optimal Strategy for a \(\texttt {DP}\)-game with Visible Choice)

To illustrate the Nash equilibria for DP-games with visible choice, we again use the DP-game in Example 8. By Definition 6, \(\mathbb {V}^{\rm DP}[C_{00}] \approx 2.20\), \(\mathbb {V}^{\rm DP}[C_{01}] \approx 1.10\), \(\mathbb {V}^{\rm DP}[C_{10}] \approx 1.10\), and \(\mathbb {V}^{\rm DP}[C_{11}] \approx 1.95\). Hence, the defender’s optimal strategy is the pure strategy \(d^* = \text{argmin}_d \max _{a} \mathbb {V}^{\rm DP}[C_{da}] = 1\).

Finally, we compare the DP-games with visible choice with those with hidden choice in terms of \(\mathbb {V}^{\rm DP}\) as follows.

Proposition 4.6 (Visible Choice ≥ Hidden Choice).

5 Information-leakage Games Vs. Standard Game Theory Models

In this section, we elaborate on the differences between our information leakage games and standard approaches to game theory. In particular, we discuss (1) why the use of vulnerability as a utility function makes QIF-games and DP-games non-standard w.r.t. von Neumann–Morgenstern’s treatment of utility, (2) why the use of concave utility functions to model risk-averse players does not capture the behavior of the attacker in QIF-games, and (3) how QIF-games differ from traditional convex-concave games.

5.1 Von Neumann–Morgenstern’s Treatment of Utility

In their treatment of utility, von Neumann and Morgenstern [65] demonstrated that the utility of a mixed strategy equals the expected utility of the corresponding pure strategies when a set of axioms is satisfied for player’s preferences over probability distributions (a.k.a. lotteries) on payoffs. Since in our information leakage games the utility of a mixed strategy is not the expected utility of the corresponding pure strategies, it is relevant to identify how exactly our framework fails to meet von Neumann–Morgenstern axioms.

Let us first introduce some notation. Given two mixed strategies σ, σ′ for a player, we write σ⪯σ′ (or σ′⪰σ) when the player prefers σ′ over σ, and σ ∼ σ′ when the player is indifferent between σ and σ′. Then, von Neumann–Morgenstern axioms can be formulated as follows [60]. For every mixed strategies σ, σ′, and σ′′:

Completeness: It is either the case that σ⪯σ′, σ⪰σ′, or σ ∼ σ′.

Transitivity: If σ⪯σ′ and σ′⪯σ′′, then σ⪯σ′′.

Continuity: If σ⪯σ′⪯σ′′, then there exist p ∈ [0, 1] s.t. p σ + (1 − p) σ′′ ∼ σ′.

Independence: If σ⪯σ′, then for any σ′′ and p ∈ [0, 1], we have p σ + (1 − p) σ′′⪯p σ′ + (1 − p) σ′′.

The utility function \(\mathbb {V}[\pi ,C]\) for QIF-games (for any fixed prior π on secrets), and the utility function \(\mathbb {V}^{\rm DP}[C]\) for DP-games are both total functions on \(\mathcal {C}\) ranging over the reals, and therefore satisfy axioms A1, A2, and A3. However, neither satisfy A4, as the next example illustrates.

Example 11.

Consider the following three channel matrices from input set \(\mathcal {X}= \lbrace 0,1\rbrace\) to output set \(\mathcal {Y}=\lbrace 0,1\rbrace\), where δ is a small positive constant.

\begin{align*} {\begin{array}{|c|c|c|} \hline C_{1} & y = 0 & y = 1 \\ \hline x = 0 & 1-2\delta & 2\delta \\ x = 1 & 2\delta & 1-2\delta \\ \hline \end{array} \qquad \begin{array}{|c|c|c|} \hline C_{2} & y = 0 & y = 1 \\ \hline x = 0 & 1-\delta & \delta \\ x = 1 & \delta & 1-\delta \\ \hline \end{array} \qquad \begin{array}{|c|c|c|} \hline C_{3} & y = 0 & y = 1 \\ \hline x = 0 & \delta & 1-\delta \\ x = 1 & 1-\delta & \delta \\ \hline \end{array} } \end{align*}

In a QIF-game, if we focus on posterior Bayes vulnerability on a uniform prior π_u, it is clear that an attacker would prefer C₂ over C₁, because

\begin{equation*} \mathbb {V}^{\mathrm{Bayes}}\!\left[\pi _{u},C_{1}\right] = 1 - 2\delta \lt 1 - \delta = \mathbb {V}^{\mathrm{Bayes}}\!\left[\pi _{u},C_{2}\right]. \end{equation*}

Similarly, in a DP-game, the attacker would also prefer C₂ over C₁, since

\begin{equation*} \mathbb {V}^{\rm DP}[C_{1}] = {(1-2\delta)}{2\delta } \lt {(1-\delta)}{\delta } = \mathbb {V}^{\rm DP}[C_{2}]. \end{equation*}

However, for the probability \(p = {1}{2}\), we would have the following hidden composition.

\begin{align*} {\begin{array}{|c|c|c|} \hline p\,C_{1} + (1-p)\,C_{3} & y = 0 & y = 1 \\ \hline x = 0 & {(1-\delta)}{2} & {(1+\delta)}{2} \\ x = 1 & {(1+\delta)}{2} & {(1-\delta)}{2} \\ \hline \end{array} \quad \text{and} \quad \begin{array}{|c|c|c|} \hline p\,C_{2} + (1-p)\,C_{3} & y = 0 & y = 1 \\ \hline x = 0 & {1}{2} & {1}{2} \\ x = 1 & {1}{2} & {1}{2} \\ \hline \end{array} } \end{align*}

But notice that the channel p C₁ + (1 − p) C₃ clearly reveals no less information about the secret than channel p C₂ + (1 − p) C₃, and the utility of the corresponding QIF-game would be

\begin{equation*} \mathbb {V}^{\mathrm{Bayes}}\!\left[\pi _{u},p\,C_{1} + (1-p)\,C_{3}\right] = {(1+\delta)}{2} \gt {1}{2} = \mathbb {V}^{\mathrm{Bayes}}\!\left[\pi _{u},p\,C_{2} + (1-p)\,C_{3}\right]. \end{equation*}

Similarly, for a DP-game, we would have

\begin{equation*} \mathbb {V}^{\rm DP}[p\,C_{1} + (1-p)\,C_{3}] = {(1+\delta)}{(1-\delta)} \gt 1 = \mathbb {V}^{\rm DP}[p\,C_{2} + (1-p)\,C_{3}]. \end{equation*}

Hence, in both kinds of games, we have C₁⪯C₂, but p C₁ + (1 − p) C₃⪰p C₂ + (1 − p) C₃, and the axiom of independence is not satisfied.

It is actually quite natural that neither vulnerability nor DP satisfies independence: a convex combination of two “informative” channels (i.e., high-utility outcomes) can produce a “non-informative” channel (i.e., a low-utility outcome), whereas we can never obtain a “more informative” channel from a convex combination of “non-informative” channels. As a consequence, the traditional game-theoretic approach to the utility of mixed strategies does not apply to our information leakage games.

5.2 Risk Functions

At first glance, it may seem that our QIF-games could be expressed with some clever use of the concept of a risk-averse player (which, in our case, would be the attacker), which is also based on convex utility functions (cf. [58]). There is, however, a crucial difference: in the models of risk-averse players, the utility function is convex on the payoff of an outcome of the game, but the utility of a mixed strategy is still the expectation of the utilities of the pure strategies (i.e., it is linear on the distributions). In contrast, the utility of mixed strategies in our QIF-games is convex on the distribution of the defender. This difference arises precisely because in QIF-games utility is defined as the vulnerability of the channel perceived by the attacker, and, as we discussed, this creates an extra layer of uncertainty for the attacker.

5.3 Convex-Concave Games

Another well-known model from standard game theory is that of convex-concave games, in which each of two players can choose among a continuous set of actions yielding convex utility for one player and concave for the other. In this kind of game, the Nash equilibria are given by pure strategies for each player.

A natural question would be why not represent our systems as convex-concave in which the pure actions of players are the mixed strategies of our QIF-games. Namely, the real values p and q that uniquely determine the defender’s and the attacker’s mixed strategies, respectively, in the two millionaires game of Section 3.1, could be taken to be the choices of pure strategies in a convex-concave game in which the set of actions for each player is the real interval [0, 1].

This mapping from QIF-games to convex-concave, however, would not be natural. One reason is that utility is still defined as expectation in the standard convex-concave, in contrast to QIF-games. Consider two strategies p₁ and p₂ with utilities u₁ and u₂, respectively. If we mix them using the coefficient q ∈ [0, 1], the resulting strategy q p₁ + (1 − q) p₂ will have utility u = q u₁ + (1 − q) u₂ in the standard convex-concave game, whereas in our case the utility would in general be strictly smaller than u. The second reason is that a pure action corresponding to a mixed strategy may not always be realizable. To illustrate this point, consider again the two millionaires game, and the defender’s mixed strategy consisting of choosing Program 0 with probability p and Program 1 with probability 1 − p. The requirement that the defender has a pure action corresponding to p implies the existence of a program (on Jeeves’ side) that internally makes a probabilistic choice with bias p and, depending on the outcome, executes Program 0 or Program 1. However, it is not granted that Jeeves disposes of such a program. Furthermore, Don would not know what choice has actually been made, and thus the program would not achieve the same functionality (i.e., let Don know who is the richest). (Note that Jeeves should not communicate to Don the result of the choice, because of the risk that Alice intercepts it.) This latter consideration underlines a key practical aspect of QIF-games, namely the defender’s advantage over the attacker due to his knowledge of the result of his own random choice (in a mixed strategy). This advantage would be lost in a convex-concave representation of the game since the random choice would be “frozen” in its representation as a pure action.

6 Case Studies

6.1 The Crowds Protocol as a QIF-game

In this section, we apply our game-theoretic analysis to the case of anonymous communication on a mobile ad hoc network (MANET). In such a network, nodes can move in space and communicate with other nearby nodes. We assume that nodes can also access some global (wide-area) network, but such connections are neither anonymous nor trusted. Consider, for instance, smartphone users who can access the cellular network but do not trust the network provider. The goal is to send a message on the global network without revealing the sender’s identity to the provider. For that, users can form a MANET using some short-range communication method (e.g., Bluetooth) and take advantage of the local network to achieve anonymity on the global one.

Crowds [59] is a protocol for anonymous communication that can be employed on a MANET for this purpose. Note that although more advanced systems for anonymous communication exist (e.g., Onion Routing), the simplicity of Crowds makes it particularly appealing for MANETs. The protocol works as follows: the initiator (i.e., the node who wants to send the message) selects some other node connected to him (with uniform probability) and forwards the request to him. A forwarder, upon receiving the message, performs a probabilistic choice: with probability p_f he keeps forwarding the message (again, by selecting uniformly a user among the ones connected to him), whereas with probability 1 − p_f he delivers the message on the global network. Replies, if any, can be routed back to the initiator following the same path in reverse order.

Anonymity comes from the fact that the detected node (the last in the path) is most likely not the initiator. Even if the attacker knows the network topology, he can infer that the initiator is most likely a node close to the detected one, but if there are enough nodes we can achieve some reasonable anonymity guarantees. However, the attacker can gain an important advantage by deploying a node himself and participating to the MANET. When a node forwards a message to this corrupted node, this action is observed by the attacker and increases the probability of the forwarding node being the initiator. Nevertheless, the node can still claim that he was only forwarding the request for someone else; hence, we still provide some level of anonymity. By modeling the system as a channel, and computing its posterior Bayes vulnerability [63], we get the probability that the attacker correctly guesses the identity of the initiator, after performing his observation.

In this section, we study a scenario of 30 nodes deployed in an area of 1 × 1 km, in the locations illustrated in Figure 14. Each node can communicate with others up to a distance of 250 m, forming the network topology shown in the graph. To compromise the anonymity of the system, the attacker plans to deploy a corrupted node in the network; the question is which is the optimal location for such a node. The answer is far from trivial: on the one side being connected to many nodes is beneficial, but at the same time these nodes need to be “vulnerable,” being close to a highly connected clique might not be optimal. At the same time, the administrator of the network is suspecting that the attacker is about to deploy a corrupted node. Since this action cannot be avoided (the network is ad hoc), a countermeasure is to deploy a deliverer node at a location that is most vulnerable. Such a node directly delivers all messages forwarded to it on the global network; since it never generates messages, its own anonymity is not an issue; it only improves the anonymity of the other nodes. Moreover, since it never communicates in the local network, its operation is invisible to the attacker. But again, the optimal location for the new deliverer node is not obvious, and most importantly, the choice depends on the choice of the attacker.

Fig. 14.

To answer these questions, we model the system as a QIF-game where the actions of attacker and defender are the locations of newly deployed corrupted and honest nodes, respectively. We assume that the possible locations for new nodes are the nine ones shown in Figure 14. For each pure strategy profile (d, a), we construct the corresponding network and use the PRISM model checker to construct the corresponding channel C_da, using a model similar to the one of Shmatikov [62]. Note that the computation considers the specific network topology of Figure 14, which reflects the positions of each node at the time when the attack takes place; the corresponding channels need to be recomputed if the network changes in the future. As a leakage measure, we use the posterior Bayes vulnerability (with uniform prior π), which is the attacker’s probability of correctly guessing the initiator given his observation in the protocol. According to Definition 2, for a mixed strategy profile (δ, α) the utility is \(\mathbb {V}(\delta ,\alpha)= {\mathbb {E}_{a\leftarrow\alpha }}\mathbb {V}\;[\pi ,C_{\delta a}]\).

The utilities (posterior Bayes vulnerability %) for each pure profile are shown in Table 2. Note that the attacker and defender actions substantially affect the effectiveness of the attack, with the probability of a correct guess ranging between \(5.46\%\) and \(9.5\%\). Based on Section 3.3, we can compute the best strategy for the defender, which turns out to be (probabilities expressed as %)

\begin{equation*} \delta ^* = (34.59, 3.48, 3.00, 10.52, 3.32, 2.99, 35.93, 3.19, 2.99). \end{equation*}

This strategy is part of an equilibrium and guarantees that for any choice of the attacker the vulnerability is at most \(8.76\%\), and is substantially better than the best pure strategy (location 1) which leads to the worst vulnerability of \(9.32\%\). As expected, δ^* selects the most vulnerable locations (1 and 7) with the highest probability. Still, the other locations are selected with non-negligible probability, which is important for maximizing the attacker’s uncertainty about the defense. Finally, note that for this specific δ^*, the adversary’s best response is the pure strategy 1.

Table 2.

6.2 Design of an LDP Mechanism Using a DP-game

In this section, we illustrate how to use our DP-game framework in the design of a privacy mechanism. We use a real dataset related to the COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) risk assessment instrument developed by Equivant (former Northpointe). The tool provides several scores for individuals, including recidivism risk and violent recidivism risk, calculated using criminal history, jail and prison time, and demographics. The data we use in our case study is taken from the ProPublica’s study evaluating COMPAS’ accuracy and bias on data from individuals from Broward County, in the United States, between 2013 and 2014 [13].

In our case study, we consider that a defender is an individual who may participate in an independent study about criminal recidivism. The defender has access to his own COMPAS data and is willing to disclose to a data curator the following non-sensitive attributes for the purpose of research: his ethnicity (z₁), gender (z₂), language (z₃), and marital status (z₄). However, he wishes to hide the status of his own agency_text attribute (x), taken from the COMPAS dataset, which is considered sensitive information. This attribute can take one of four values: BC (Broward County), DRRD (indicating that the individual is under the Day Report Rentry Division, which helps reintegrate offenders back into the community following release from jail), pretrial, or probation. However, the defender is aware that there are correlations between the non-sensitive values z₁, z₂, z₃, z₄ and the sensitive value x, as shown in Figure 15. Therefore, the defender is interested in obfuscating his non-sensitive values before revealing them, to mitigate any possible leakage of information about x.

Fig. 15.

To do that, the defender may employ a local DP mechanism and add noise to the values of z₁, z₂, z₃, and z₄ before reporting them to a data curator. We assume that a data analyst—here taking the role of the attacker—is allowed to query the data curator about only one of the four attributes z₁, z₂, z₃, or z₄. This assumption is justified by the fact that the combination of all four attributes could reveal too much information about x. We can capture this scenario with the following protocol:

(1)

The defender obfuscates his own record (z₁, z₂, z₃, z₄) using one of four available privacy mechanisms M₁, M₂, M₃, M₄, and sends it to the data curator. Each mechanism obfuscates the whole record (z₁, z₂, z₃, z₄) at once, producing a new record \((z_1^{\prime }, z_2^{\prime }, z_3^{\prime }, z_4^{\prime })\) of randomized values to be reported to the data curator. Hence, the defender’s action d is taken from the index set \(\mathcal {D}=\lbrace 1,2,3,4\rbrace\) of the available mechanisms M₁, M₂, M₃, M₄. We denote by M_d the mechanism that the defender uses.

(2)

The data curator receives and stores the randomized record \((z_1^{\prime }, z_2^{\prime }, z_3^{\prime }, z_4^{\prime })\) from the defender.

(3)

The attacker (i.e., the data analyst) can ask for the data curator to reveal only one of the four sanitized values \(z_1^{\prime }\), \(z_2^{\prime }\), \(z_3^{\prime }\) or \(z_4^{\prime }\), and we denote the disclosed value by y. The attacker’s action a is taken from the set \(\mathcal {A}=\lbrace 1,2,3,4\rbrace\) of the attribute indices of \(z_1^{\prime }\), \(z_2^{\prime }\), \(z_3^{\prime }\), \(z_4^{\prime }\).

We assume that when the defender selects a privacy mechanism, he does not know which attribute z_a the attacker will request. Symmetrically, the attacker does not know what privacy mechanism M_d the defender uses. By using our framework of DP-games, we design an optimal privacy mechanism by computing an optimal strategy for the defender.

Let \(\mathcal {Z}_i\) be the range of the attribute z_i’s possible values, and \(\mathcal {Z}= \prod _{i=1}^{4} \mathcal {Z}_i\). To construct the defender’s privacy mechanism, we define \((\epsilon , \mathcal {Z}_i)\)-randomized response \(\mathit {RR}_{\epsilon }^{\mathcal {Z}_i}: \mathcal {Z}_i\rightarrow \mathbb {D}\mathcal {Z}_i\) by

\begin{align*} \mathit {RR}_{\epsilon }^{\mathcal {Z}_i}(y \,|\, z) &= {\left\lbrace \begin{array}{ll} {e^\varepsilon }{(|\mathcal {Z}_i| + e^\varepsilon - 1)} & \text{if}\;\;{y = z} \\ {1}{(|\mathcal {Z}_i| + e^\varepsilon - 1)} & \text{if}\;\;{y \in \mathcal {Z}_i\setminus \lbrace z\rbrace .}\end{array}\right.} \end{align*}

Then, the defender’s actions are realized by mechanisms \(M_d: \mathcal {Z}\rightarrow \mathbb {D}\mathcal {Z}\) that apply \((0.1, \mathcal {Z}_d)\)-randomized response \(\mathit {RR}_{0.1}^{\mathcal {Z}_d}\) to z_d and \((2.0, \mathcal {Z}_j)\)-randomized response \(\mathit {RR}_{2.0}^{\mathcal {Z}_j}\) to z_j for all j ≠ d. Note that each mechanism M_d adds significantly more noise to attribute z_d than to all other attributes. For instance, \(M_3(z_1, z_2, z_3, z_4) = (\mathit {RR}_{2.0}^{\mathcal {Z}_1}(z_1), \mathit {RR}_{2.0}^{\mathcal {Z}_2}(z_2), \mathit {RR}_{0.1}^{\mathcal {Z}_3}(z_3), \mathit {RR}_{2.0}^{\mathcal {Z}_4}(z_4))\).

To design an optimal privacy mechanism for the defender, we consider a \(\texttt {DP}\)-game. To set the game properly, we need to consider the full mechanisms, namely those that operate on the secrets. Remember that the domain of secrets consists of the four possible values of the agency_text attribute (and the adjacency relation is the usual one for local DP: x ∼ x′ for all pairs of different values x and x′). For each choice a of one of the z_i’s attributes, and each choice d of the mechanism M_d that obfuscates the z_i’s, the corresponding full mechanism C_da is obtained by composing the probabilistic relation from x to z_a with M_d. Note that this is similar to the addition of noise to the result of a query in standard DP. The main difference is that a query is deterministic, whereas in our case the relation between x and z_a is probabilistic. For this reason, the addition of the noise M_d has to be done by multiplying the channel representing the probabilistic relation and M_d. This kind of channel composition is called cascade.

More formally, for a pure strategy profile \((d, a)\in \mathcal {D}\times \mathcal {A}\), the channel \(C_{da}: \mathcal {X}\times \mathcal {Z}_a\rightarrow \mathbb {R}\) is the cascade composition of the channels representing the correlation p(z_i | x) and the randomized response p(y | z_i):

\begin{align*} C_{da}(x, y) &= {\left\lbrace \begin{array}{ll}{\textstyle \sum _{z \in \mathcal {Z}_a}}\, \mathit {RR}_{0.1}^{\mathcal {Z}_a}(y \,|\, z)\, p(z \,|\, x) & \text{if}\;\;{d=a} \\ {\textstyle \sum _{z \in \mathcal {Z}_a}}\, \mathit {RR}_{2.0}^{\mathcal {Z}_a}(y \,|\, z)\, p(z \,|\, x) & \text{otherwise}.\end{array}\right.} \end{align*}

Note that the output ranges over \(\mathcal {Z}_a\), since the data curator provides the attacker with the data of only one of the four attributes.

For each channel C_da, the DP level \(\mathbb {V}^{\rm DP}[ C_{da} ]\) is given by Definition 6 and shown in Table 3.

Table 3.

When the defender’s choice is hidden, we compute a solution for the DP-game using the algorithm shown in Proposition 4.4. We obtain the following optimal mixed strategy \(\delta ^* \in \mathbb {D}\mathcal {D}\) for the defender:

\begin{equation*} \delta ^* = (0.5714, 0.0183, 0.0000, 0.4103). \end{equation*}

Then the defender’s optimal privacy mechanism \(M^*: \mathcal {Z}\rightarrow \mathbb {D}\mathcal {Z}\) is given by \(M^* = \mathop{\mathbb{E}}_{d\leftarrow \delta ^*} [M_d],\) and its DP level is \(\mathbb {V}^{\rm DP}[ M^* ] = 0.3892\).

By using M^*, each attribute is obfuscated by the following mechanism:

\begin{align*} p(y \,|\, z_1) &= 0.5714 \,\mathit {RR}_{0.1}^{\mathcal {Z}_1}(y \,|\, z_1) + (1 - 0.5714) \,\mathit {RR}_{2.0}^{\mathcal {Z}_1}(y \,|\, z_1) \\ p(y \,|\, z_2) &= 0.0183 \,\mathit {RR}_{0.1}^{\mathcal {Z}_2}(y \,|\, z_2) + (1 - 0.0183) \,\mathit {RR}_{2.0}^{\mathcal {Z}_2}(y \,|\, z_2) \\ p(y \,|\, z_3) &= \mathit {RR}_{2.0}^{\mathcal {Z}_3}(y \,|\, z_3) \\ p(y \,|\, z_4) &= 0.4103 \,\mathit {RR}_{0.1}^{\mathcal {Z}_4}(y \,|\, z_4) + (1 - 0.4103) \,\mathit {RR}_{2.0}^{\mathcal {Z}_4}(y \,|\, z_4). \end{align*}

Then the optimal mechanism M^* adds less perturbation to the attributes z₂ (gender) and z₃ (language). This can be explained by observing p(z₂ | x) and p(z₃ | x) in Figure 15(b) and (c). Clearly, z₂ and z₃ leak little information on the secret x, so the defender need not add much noise to z₂ or z₃.

Note that to construct this optimal mechanism M^*, the defender needs to know the correlation p(z_a | x) for each \(a\in \mathcal {A}\). (This is reasonable in our scenario, since the defender has access to the original COMPAS dataset and can compute the correlations from it). In contrast, the attacker need not know the correlation p(z_a | x) at all, since an arbitrary mixed strategy α^* with \({\sf supp}(\alpha ^*)=\mathcal {A}\) is optimal for the attacker. (Similarly, note that this is reasonable in our scenario, and this lack of knowledge is one of the reasons the data analyst is interested in collecting data from individuals in the COMPAS dataset in the first place.)

When the defender’s choice is visible, we compute a solution for the DP-game using Theorem 4.5. The pure strategy d = 4 is optimal for the defender, whereas an arbitrary mixed strategy α^* with \({\sf supp}(\alpha ^*)=\mathcal {A}\) is optimal for the attacker. The DP level of the equilibrium is given by 0.5994.

7 Conclusion and Future Work

In this article, we explored information leakage games, in which a defender and an attacker have opposing goals in optimizing the amount of information revealed by a system. In particular, we discussed QIF-games, in which utility is information leakage of a channel, and we introduced DP-games, in which utility is the level of DP provided by the channel. In contrast to standard game theory models, in our games the utility of a mixed strategy is either a convex, quasi-convex, or quasi-max function of the distribution of the defender’s actions, rather than the expected value of the utilities of the pure strategies in the support. Nevertheless, the important properties of game theory, notably the existence of a Nash equilibrium, still hold for our zero-sum information leakage games, and we provided algorithms to compute the corresponding optimal strategies for the attacker and the defender.

As future research, we would like to extend information leakage games to scenarios with repeated observations—that is, when the attacker can repeatedly observe the outcomes of the system in successive runs, under the assumption that both the attacker and the defender may change the channel at each run. This would incorporate the sequential and parallel compositions of QIF [39] in information leakage games. Furthermore, we would like to consider the possibility of adapting the defender’s strategy to the secret value, as we believe that in some cases this would provide a significant advantage to the defender. We would also like to deal with the cost of attack and of defense, which would lead to non-zero-sum games. Finally, we are also interested in incorporating our information leakage games in signaling games [26], which model asymmetric information among players and are applied to economics and behavioral biology.

Footnotes

Conventionally in game theory, the utility u is set to be that of the first player, but we prefer to look at the utility from the point of view of the attacker to be in line with the definition of utility as vulnerability, which we will define in Section 2.4.

More precisely, if posterior vulnerability is defined as the expectation of the vulnerability of posterior distributions, the measure respects the data-processing inequality and yields non-negative leakage iff vulnerability is convex.

The reason to involve Jeeves is that Alex may not want to reveal a to Don either.

⁴

That may be the case because, say, although all messages are exchanged using encryption, Jeeves’ secret key has been hacked by Alice, whereas Don’s remains protected.

⁵

Note that d should not be revealed to the attacker: although d is not sensitive information in itself, knowing it would help the attacker figure out the value of x.

⁶

More precisely, for any fixed prior π, the additive leakage of a channel C is obtained by a simple shifting of the posterior vulnerability \(\mathbb {V}\!\left[\pi ,C\right]\) by a term of \(-\mathbb {V}\!\left[\pi \right]\) corresponding to the prior vulnerability. Similarly, the multiplicative leakage of a channel C is obtained by a simple scaling of the posterior vulnerability \(\mathbb {V}\!\left[\pi ,C\right]\) by a factor of \({1}{\mathbb {V}\!\left[\pi \right]}\) corresponding to the prior vulnerability. Both transformations consist of a monotonic bijection between the corresponding leakage notions and posterior vulnerability.

⁷

Note that two channel matrices with different column indices can always be made compatible by adding appropriate columns with 0-valued cells in each of them.

⁸

The convexity of posterior vulnerability w.r.t. the strategy of the defender is formally shown in Theorem 3.1 ahead.

⁹

Note that f(δ^k) is not necessarily decreasing, so an optimization would be to keep track of the best values of f(δ^(k)) and l^(k), stop when f(δ^(best)) − l^(best) < ϵ, and return \(\hat{\delta }= \delta ^{(\text{best})}\).

¹⁰

The interpretation of DP in terms of inference can be found, for instance, in several works [3, 15, 22, 35, 48].

¹¹

For \(\mathcal {I}= \lbrace i_{1}, i_{2}, \ldots , i_{n} \rbrace\), \(\bigsqcup _{i \in \mathcal {I}} \mathcal {Y}_{i} = \mathcal {Y}_{i_{1}} \sqcup \mathcal {Y}_{i_{2}} \sqcup \cdots \sqcup \mathcal {Y}_{i_{n}}\) denotes the disjoint union\(\lbrace (y,i) \mid y \in \mathcal {Y}_{i}, i \in \mathcal {I}\rbrace\) of the sets \(\mathcal {Y}_{i_{1}}\), \(\mathcal {Y}_{i_{2}}, \ldots , \mathcal {Y}_{i_{n}}\).

¹²

In the work of Kawamoto and Murakami [41, 42], a similar inequality is used to prove their main theorems.

B Proofs

This appendix contains the proof of the formal results of this article.

Proof for Corollary 3.2

Given a mixed strategy profile (δ, α), the utility \(\mathbb {V}(\delta , \alpha)\) given in Definition 2 is affine (hence concave) on α. Furthermore, by Theorem 3.1, \(\mathbb {V}(\delta , \alpha)\) is convex on δ. Hence, we can apply von Neumann’s minimax theorem (Section 2.3), which ensures the existence of a saddle point (i.e., a Nash equilibrium).□

Proof for Proposition 3.3

It is sufficient to show that for all mixed strategy δ for the defender:

\begin{equation*} \max _\alpha \mathbb {V}(\delta ,\alpha) = \max _a \,\mathbb {V}\!\left[\pi ,{\textstyle \mathop{\mathbb{E}}_{d\leftarrow \delta }} C_{da}\right]\!. \end{equation*}

Let α be an arbitrary mixed strategy for the attacker. Then we have that

\begin{align*} \mathbb {V}(\delta ,\alpha) &= \textstyle {\sum _{a}\:}\alpha (a) \,\mathbb {V}\!\left[\pi ,\mathop{\mathbb{E}}\limits_{d\leftarrow \delta }\, C_{da}\right] \\ &\le \textstyle {\sum _{a}\:}\alpha (a) (\max _a \,\mathbb {V}\!\left[\pi ,\mathop{\mathbb{E}}\limits_{d\leftarrow \delta }\,C_{da}\right]) & \text{(by}\;\;{ \alpha (a)\ge 0)} \\ &= ({\textstyle \max _a} \,\mathbb {V}\!\left[\pi ,\textstyle \mathop{\mathbb{E}}\limits_{d\leftarrow \delta }\,C_{da}\right])(\textstyle {\sum _{a}\:}\alpha (a))) \\ &= {\textstyle \max _a} \,\mathbb {V}\!\left[\pi ,\textstyle \mathop{\mathbb{E}}\limits_{d\leftarrow \delta }\,C_{da}\right]. \end{align*}

This holds for any α, hence \(\max _\alpha \mathbb {V}(\delta ,\alpha) \le \max _a \,\mathbb {V}\!\left[\pi ,\mathop{\mathbb{E}}\limits_{d\leftarrow \delta }\,C_{da}\right]\); the (≥) case is trivial since we can take α to be the point distribution on any a.□

Proof for Theorem 3.4

Our iterative method for computing the minimum value f^* = min _δf(δ) is an application of the subgradient descent method, which can be applied to any convex function.

We start by showing that f(δ) is convex. Let \(\delta _1,\delta _2\in \mathbb {D}\mathcal {D}\), and c ∈ [0, 1] be a convex coefficient:

\begin{align*} f(c\delta _1 + (1-c) \delta _2) &= \max _a\mathbb {V}\!\left[\pi ,\textstyle {\sum _{d}\:}(c\delta _1(d) + (1-c) \delta _2(d))\,C_{da} \right] & \text{(by (3))} \\ &\le \max _a(c \mathbb {V}\!\left[\pi ,\textstyle {\sum _{d}\:} \delta _1(d)\,C_{da}\right] \, + \, (1-c)\mathbb {V}\!\left[\pi ,\textstyle {\sum _{d}\:}\delta _2(d)\,C_{da}\right]) & \hspace{-12.75pt}\text{(by Theorem 3.1)} \\ &\le c \max _a \mathbb {V}\!\left[\pi ,\textstyle {\sum _{d}\:} \delta _1(d)\,C_{da}\right] \, + \, (1-c) \max _a \mathbb {V}\!\left[\pi ,\textstyle {\sum _{d}\:}\delta _2(d)\,C_{da}\right] \\ &= c f(\delta _1) + (1-c) f(\delta _2)\;. & \text{(by (3))} \end{align*}

The proof of convergence of the subgradient decent method applied to f is provided by Boyd and Mutapcic [19, Sections 3 and 5]. We only need to ensure that the two crucial conditions of this proof, stated in that work [19, Section 3.1], are met.

First, the proof requires that the norm of all subgradients ‖h^(k)‖₂ is bounded by some constant G. For this, it is sufficient to show that f(δ) is Lipschitz; the latter follows from the fact that \(\mathbb {V}(\pi)\) itself is assumed to be Lipschitz.

Second, the proof requires that there is a known number R bounding the distance between the initial point δ⁽¹⁾ and the optimal one, namely ‖δ⁽¹⁾ − δ^*‖₂ ≤ R. Since we start from the uniform distribution \(u_\mathcal {D}\), we take R to be the distance between δ⁽¹⁾ and the point in the simplex maximally distant from it, which is a point distribution δ^point = (1, 0, …). In other words, for \(n = |\mathcal {X}|\), we take

\begin{equation*} R = \Vert \delta ^{(1)} - \delta ^{\text{point}} \Vert _2 = \sqrt { \big (1 - {1}{n} \big)^2 + \big (0 - {1}{n} \big)^2 (n-1) } = \sqrt { {(n-1)}{n} } . \end{equation*}

Note also that the “diminishing step size” s_k that we use is among those known to guarantee convergence [19, Sections 3 and 5].

The fact that l^(k) is a lower bound on f^* comes from the work of Boyd and Mutapcic [19, Section 3.4], using the preceding R. As a consequence, the stopping criterion guarantees that

\begin{equation} f(\hat{\delta }) - f(\delta ^*) \le \epsilon . \end{equation}

(7)

Let α be an arbitrary mixed strategy for the attacker. From Proposition 3.3 and the preceding,

\begin{align*} \mathbb {V}(\hat{\delta }, \alpha) - \epsilon & (\hat{\delta }) - \epsilon & \text{(by def. of} f \text{in (3))} \\ & \le f(\delta ^*) & \text{(by (7))} \\ & = \mathbb {V}(\delta ^*,\alpha ^*) & \text{(by (3))} \\ & \le \mathbb {V}(\hat{\delta },\alpha ^*). & (\text{since} (\delta ^*,\alpha ^*) \text{is a saddle point}) \end{align*}

Similarly for \(\mathbb {V}(\hat{\delta },\alpha ^*) \le \mathbb {V}(\delta ,\alpha ^*) + \epsilon\), which concludes the proof.□

Proof for Proposition 4.2

The proof is based on the quasi-convexity and quasi-max of \(\mathbb {V}^{\rm DP}\):

\begin{align*} \mathbb {V}^{\rm DP}\bigl [{\bigsqcup\!\!\!\!\!\!\cdot _{\;a \leftarrow \alpha }} C_{\delta a}\bigr ] &= {\textstyle \max _{a\in {\sf supp}(\alpha)}} \mathbb {V}^{\rm DP}[C_{\delta a}] & \left(\text{by Theorem~4.1 (2)}\right) \\ &\le {\textstyle \max _{a\in {\sf supp}(\alpha)}} {\textstyle \max _{d\in {\sf supp}(\delta)}} \mathbb {V}^{\rm DP}[C_{da}]{.} & \left(\text{by Theorem~4.1 (1)}\right) \end{align*}

□

Proof for Theorem 4.3

By Theorem 4.1 (2), it holds for any mixed strategy profile (δ, α) that

\begin{equation*} \mathbb {V}^{\rm DP}(\delta ,\alpha)\stackrel{\mathrm{def}}{=}\mathbb {V}^{\rm DP}\bigl [{\bigsqcup\!\!\!\!\!\!\cdot _{\;a \leftarrow \alpha }}\,C_{\delta a}\bigr ] = {\textstyle \max _{a\in {\sf supp}(\alpha)}}\, \mathbb {V}^{\rm DP}[C_{\delta a}] \le {\textstyle \max _{a\in \mathcal {A}}}\, \mathbb {V}^{\rm DP}[C_{\delta a}]. \end{equation*}

Hence, for any δ, an arbitrary mixed strategy α^* s.t. \({\sf supp}(\alpha ^*) = \mathcal {A}\) maximizes \(\mathbb {V}^{\rm DP}(\delta ,\alpha)\). Therefore,

\begin{equation*} \textstyle \min _{\delta } \max _{\alpha } \mathbb {V}^{\rm DP}(\delta , \alpha) = \min _{\delta } \mathbb {V}^{\rm DP}(\delta , \alpha ^*) = \max _{\alpha } \min _{\delta } \mathbb {V}^{\rm DP}(\delta , \alpha) {} \end{equation*}

(i.e., there exists a Nash equilibrium).□

Proof for Proposition 4.4

Let α^* be an arbitrary mixed strategy such that \({\sf supp}(\alpha ^*) = \mathcal {A}\). By Theorem 4.3,

\begin{align*} \textstyle \min _{\delta } \max _{\alpha } \mathbb {V}^{\rm DP}(\delta , \alpha) =&\, \textstyle \max _{\alpha } \min _{\delta } \mathbb {V}^{\rm DP}(\delta , \alpha) \\ =&\, \textstyle \min _{\delta } \mathbb {V}^{\rm DP}(\delta , \alpha ^*) \\ =&\, \textstyle \min _{\delta } \max _{a\in \mathcal {A}} \mathbb {V}^{\rm DP}(\delta , a) \\ =&\, \textstyle \min _{\delta } \max _{a\in \mathcal {A}} \max _{x,x^{\prime },y} {\textstyle \frac{ \sum _{d} \delta (d) C_{da}(x,y) }{ \sum _{d} \delta (d) C_{da}(x,y) }}. \end{align*}

The preceding program is an instance of generalized fractional programming[14], in which we minimize the largest of a family of ratios of continuous functions:

\begin{align*} \min _{\delta \in \mathbb {D}{\mathcal {D}}} \max _{j \in J} {\textstyle \frac{ f_j(\delta) }{ g_j(\delta) }}, \quad \text{where }\quad ~ J &= \mathcal {A}\times \mathcal {X}\times \mathcal {X}\times \mathcal {Y},\\ f_{j}(\delta) &= {\textstyle \sum _{d}}\, \delta (d) C_{da}(x,y), \\ g_{j}(\delta) &= {\textstyle \sum _{d}}\, \delta (d) C_{da}(x^{\prime },y), \qquad j=(a,x,x^{\prime },y)\in J . \end{align*}

We can solve this program using the Dinkelbach-type algorithm [14] as follows. Define

\begin{align*} \lambda _k &= \max _{j \in J} {\textstyle \frac{ f_j(\delta _{k-1}) }{ g_j(\delta _{k-1}) }}, \\ F_k(\delta) &= \max _{j \in J} \left[ f_j(\delta) - \lambda _k g_j(\delta) \right] , \\ \delta _k &= \text{argmin}_{\delta \in \mathbb {D}{\mathcal {D}}} F_k(\delta) . \end{align*}

We start from a uniform δ₀, and iterate using the preceding formulas, until F_k(δ_k) = 0 for some k ≥ 1, in which case δ_k is guaranteed to be the optimal solution with value λ_k. Note that for each iteration k ≥ 1, λ_k is constant (computed from δ_{k − 1}). Moreover, the optimization problem for δ_k, which we need to solve in every iteration, requires to minimize the max of linear functions, and can be transformed into the following linear program:

\begin{align*} &\text{variables }~ \delta (d), z \\ &\text{minimize }~ z \quad \quad \quad \text{s.t. }~ 1 = {\displaystyle \sum _{d}}\, \delta (d) ~\mbox{ and }~ z \ge f_j(\delta) - \lambda _k g_j(\delta) \quad \forall j\in J , \end{align*}

since f_j(δ), g_j(δ) are linear on δ. Therefore, an optimal strategy for the DP-game is obtained by solving a sequence of linear programs.□

Proof for Theorem 4.5

Similarly to Theorem 4.3, an arbitrary mixed strategy α^* such that \({\sf supp}(\alpha ^*) = \mathcal {A}\) maximizes the attacker’s payoff independently of δ. Hence, for any δ and α,

\begin{equation*} \mathbb {V}^{\rm DP}(\delta ,\alpha) = {\textstyle \max _{d\in {\sf supp}(\delta)}\, \max _{a\in \mathcal {A}}}\,\ \mathbb {V}^{\rm DP}[C_{da}]. \end{equation*}

This is minimized for a point distribution δ^* s.t. δ^*(d^*) = 1 for \(d^* \in \text{argmin}_d \max _{a} \mathbb {V}^{\rm DP}[C_{da}]\).□

Proof for Proposition 4.6

For any \(a\in \mathcal {A}\) and any \(\delta \in \mathbb {D}\mathcal {D},\) we have

\begin{align} \mathbb {V}^{\rm DP}\left[{{\bigsqcup\!\!\!\!\!\!\cdot _{\;d \leftarrow \delta }}} {C_{d a}}\right] \,\, =&\,\, {\textstyle \max _{d \in {\sf supp}(\delta)}} \,\mathbb {V}^{\rm DP}[C_{d a}] & \text{(by Theorem~4.1 (2))}\nonumber \nonumber\\ \ge &\,\, \, \mathbb {V}^{\rm DP}\left[\mathop{\mathbb{E}}\limits_{d\leftarrow \delta }\, C_{d a}\right]{.}& \text{(by Theorem~4.1 (1))} \end{align}

(8)

Then for any \(\alpha \in \mathbb {D}\mathcal {A}\), any \(\delta \in \mathbb {D}\mathcal {D}\), and any \(a^\dagger \in \text{argmax}_{a\in {\sf supp}(\alpha)} \mathbb {V}^{\rm DP}[\mathop{\mathbb{E}}\limits_{d\leftarrow \delta }\, C_{d a}]\), we obtain

\begin{align*} \mathbb {V}^{\rm DP}\biggl [{\bigsqcup\!\!\!\!\!\!\cdot _{\;\substack {d \leftarrow \delta -PLXBCR- a \leftarrow \alpha }}} C_{da}\biggr ] \; =&\; {\textstyle \max _{a\in {\sf supp}(\alpha)}} \mathbb {V}^{\rm DP}\left[{{\bigsqcup\!\!\!\!\!\!\cdot _{\;d \leftarrow \delta }}} {C_{d a}}\right]& \text{(by Theorem~4.1 (2))}\\ \;\ge &\;\mathbb {V}^{\rm DP}\left[{{\bigsqcup\!\!\!\!\!\!\cdot _{\;d \leftarrow \delta }}} {C_{d a^\dagger }}\right] \\ \;\ge &\;\mathbb {V}^{\rm DP}\left[\mathop{\mathbb{E}}\limits_{d\leftarrow \delta }\, C_{d a^\dagger }\right]&\text{(by (8))}\\ \;=&\;{\textstyle \max _{a\in {\sf supp}(\alpha)}} \mathbb {V}^{\rm DP}\left[\mathop{\mathbb{E}}\limits_{d\leftarrow \delta }\, C_{d a}\right]&(\text{by def. of} a^{\dagger })\\ \;=&\;\mathbb {V}^{\rm DP}\left[{\bigsqcup\!\!\!\!\!\!\cdot _{\;a \leftarrow \alpha }}\, \mathop{\mathbb{E}}\limits_{d\leftarrow \delta } {C_{d a}}\right]. & \text{(by Theorem~4.1 (2))} \end{align*}

Therefore the theorem follows immediately.□

Acknowledgments

The authors are thankful to Arman Khouzani and Pedro O. S. Vaz de Melo for valuable discussions, and to Shintaro Miura for pointing out various related literature.

References

[1]

GitHub. n.d. LIBQIF: A Quantitative Information Flow Toolkit Library. Retrieved February 23, 2022 from https://github.com/chatziko/libqif.

Google Scholar

[2]

Noga Alon, Yuval Emek, Michal Feldman, and Moshe Tennenholtz. 2013. Adversarial leakage in games. SIAM J. Discrete Math. 27, 1 (2013), 363–385.

Digital Library

Google Scholar

[3]

Mário S. Alvim, Miguel E. Andrés, Konstantinos Chatzikokolakis, Pierpaolo Degano, and Catuscia Palamidessi. 2015. On the information leakage of differentially-private mechanisms. J. Comp. Security 23, 4 (2015), 427–469.

Digital Library

Google Scholar

[4]

Mário S. Alvim, Miguel E. Andres, and Catuscia Palamidessi. 2012. Information flow in interactive systems. J. Comp. Security 1, 20 (2012), 3–50.

Crossref

Google Scholar

[5]

Mário S. Alvim, Konstantinos Chatzikokolakis, Yusuke Kawamoto, and Catuscia Palamidessi. 2017. Information leakage games. In Proc. of GameSec(LNCS, Vol. 10575). Springer, 437–457.

Google Scholar

[6]

Mário S. Alvim, Konstantinos Chatzikokolakis, Yusuke Kawamoto, and Catuscia Palamidessi. 2018. A game-theoretic approach to information-flow control via protocol composition. Entropy 20, 5 (2018), 382.

Crossref

Google Scholar

[7]

Mário S. Alvim, Konstantinos Chatzikokolakis, Yusuke Kawamoto, and Catuscia Palamidessi. 2018. Leakage and protocol composition in a game-theoretic perspective. In Proc. of POST(LNCS, Vol. 10804). Springer, 134–159.

Crossref

Google Scholar

[8]

Mário S. Alvim, Konstantinos Chatzikokolakis, Annabelle McIver, Carroll Morgan, Catuscia Palamidessi, and Geoffrey Smith. 2016. Axioms for information leakage. In Proc. of CSF. 77–92.

Google Scholar

[9]

Mário S. Alvim, Konstantinos Chatzikokolakis, Annabelle McIver, Carroll Morgan, Catuscia Palamidessi, and Geoffrey Smith. 2020. The Science of Quantitative Information Flow. Springer.

Crossref

Google Scholar

[10]

Mário S. Alvim, Konstantinos Chatzikokolakis, Catuscia Palamidessi, and Geoffrey Smith. 2012. Measuring information leakage using generalized gain functions. In Proc. of CSF. 265–279.

Google Scholar

[11]

Arthur Américo, M. H. R. Khouzani, and Pasquale Malacaria. 2019. Deterministic channel design for minimum leakage. In Proc. of CSF. IEEE, Los Alamitos, CA, 428–441.

Google Scholar

[12]

Arthur Américo, M. H. R. Khouzani, and Pasquale Malacaria. 2020. Conditional entropy and data processing: An axiomatic approach based on core-concavity. IEEE Trans. Inf. Theory 66, 9 (2020), 5537–5547.

Crossref

Google Scholar

[13]

J. Angwin, J. Larson, S. Mattu, and L. Kirchner. 2016. How we analyzed the COMPAS recidivism algorithm. ProPublica. Retrieved February 23, 2022 from https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm.

Google Scholar

[14]

Ana Isabel Barros. 2013. Discrete and Fractional Programming Techniques for Location Models. Vol. 3. Springer Science & Business Media.

Google Scholar

[15]

Gilles Barthe and Boris Köpf. 2011. Information-theoretic bounds for differentially private mechanisms. In Proc. of CSF. IEEE, Los Alamitos, CA, 191–204.

Google Scholar

[16]

Fabrizio Biondi, Yusuke Kawamoto, Axel Legay, and Louis-Marie Traonouez. 2019. Hybrid statistical estimation of mutual information and its application to information flow. Formal Aspects Comput. 31, 2 (2019), 165–206.

Digital Library

Google Scholar

[17]

Michele Boreale and Francesca Pampaloni. 2015. Quantitative information flow under generic leakage functions and adaptive adversaries. Log. Methods Comput. Sci. 11, 4 (2015), 166–181.

Crossref

Google Scholar

[18]

Michele Boreale, Francesca Pampaloni, and Michela Paolini. 2015. Asymptotic information leakage under one-try attacks. Math. Struct. Comput. Sci. 25, 02 (2015), 292–319.

Crossref

Google Scholar

[19]

Stephen Boyd and Almir Mutapcic. 2006. Subgradient methods. Lecture Notes of EE364b, Winter Quarter 2007. Stanford University, Stanford, CA.

Google Scholar

[20]

Stephen Boyd and Lieven Vandenberghe. 2004. Convex Optimization. Cambridge University Press, New York, NY.

Crossref

Google Scholar

[21]

Christelle Braun, Konstantinos Chatzikokolakis, and Catuscia Palamidessi. 2009. Quantitative notions of leakage for one-try attacks. In Proc. of MFPS(ENTCS, Vol. 249). Elsevier, 75–91.

Google Scholar

[22]

Konstantinos Chatzikokolakis, Miguel E. Andrés, Nicolás E. Bordenabe, and Catuscia Palamidessi. 2013. Broadening the scope of differential privacy using metrics. In Proc. of PETS(LNCS, Vol. 7981). Springer, 82–102.

Google Scholar

[23]

Konstantinos Chatzikokolakis, Daniel Gebler, Catuscia Palamidessi, and Lili Xu. 2014. Generalized bisimulation metrics. In Proc. of CONCUR(LNCS, Vol. 8704). Springer, 32–46.

Google Scholar

[24]

Konstantinos Chatzikokolakis, Catuscia Palamidessi, and Prakash Panangaden. 2008. On the Bayes risk in information-hiding protocols. J. Comp. Security 16, 5 (2008), 531–571.

Digital Library

Google Scholar

[25]

David Chaum. 1988. The dining cryptographers problem: Unconditional sender and recipient untraceability. J. Cryptol. 1 (1988), 65–75.

Crossref

Google Scholar

[26]

In-Koo Cho and David M. Kreps. 1987. Signaling games and stable equilibria. Q. J. Econ. 102, 2 (1987), 179–221.

Crossref

Google Scholar

[27]

Tom Chothia, Yusuke Kawamoto, and Chris Novakovic. 2013. A tool for estimating information leakage. In Proc. of CAV. 690–695.

Google Scholar

[28]

Tom Chothia, Yusuke Kawamoto, and Chris Novakovic. 2014. LeakWatch: Estimating information leakage from Java programs. In Proc. of ESORICS, Part II. 219–236.

Google Scholar

[29]

Tom Chothia, Yusuke Kawamoto, Chris Novakovic, and David Parker. 2013. Probabilistic point-to-point information leakage. In Proc. of CSF. IEEE, Los Alamitos, CA, 193–205.

Google Scholar

[30]

David Clark, Sebastian Hunt, and Pasquale Malacaria. 2005. Quantitative information flow, relations and polymorphic types. J. Log. Comput. 18, 2 (2005), 181–199.

Digital Library

Google Scholar

[31]

David Clark, Sebastian Hunt, and Pasquale Malacaria. 2007. A static analysis for quantifying information flow in a simple imperative language. J. Comp. Security 2007 (2007), 1–49.

Crossref

Google Scholar

[32]

Cynthia Dwork. 2006. Differential privacy. In Proc. of ICALP(LNCS, Vol. 4052). Springer, 1–12.

Google Scholar

[33]

Cynthia Dwork. 2011. A firm foundation for private data analysis. Commun. ACM 54, 1 (2011), 86–96.

Digital Library

Google Scholar

[34]

Cynthia Dwork and Jing Lei. 2009. Differential privacy and robust statistics. In Proc. of STOC. ACM, New York, NY, 371–380.

Google Scholar

[35]

Cynthia Dwork, Frank Mcsherry, Kobbi Nissim, and Adam Smith. 2006. Calibrating noise to sensitivity in private data analysis. In Proc. of TCC(LNCS, Vol. 3876). Springer, 265–284.

Google Scholar

[36]

Cynthia Dwork and Aaron Roth. 2014. The algorithmic foundations of differential privacy. Found. Trends Theor. Comp. Sci. 9, 3–4 (2014), 211–407.

Digital Library

Google Scholar

[37]

Sadegh Farhang and Jens Grossklags. 2016. FlipLeakage: A game-theoretic approach to protect against stealthy attackers in the presence of information leakage. In Proc. of GameSec. 195–214.

Google Scholar

[38]

Shiva Prasad Kasiviswanathan, Homin K. Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam D. Smith. 2008. What can we learn privately? In Proc. of FOCS. IEEE, Los Alamitos, CA, 531–540.

Google Scholar

[39]

Yusuke Kawamoto, Konstantinos Chatzikokolakis, and Catuscia Palamidessi. 2017. On the compositionality of quantitative information flow. Log. Methods Comput. Sci. 13, 3:11 (2017), 1–31.

Google Scholar

[40]

Yusuke Kawamoto and Thomas Given-Wilson. 2015. Quantitative information flow for scheduler-dependent systems. In Proc. of QAPL. 48–62.

Google Scholar

[41]

Yusuke Kawamoto and Takao Murakami. 2019. Local distribution obfuscation via probability coupling. In Proc. of Allerton 2019. IEEE, Los Alamitos, CA, 718–725.

Google Scholar

[42]

Yusuke Kawamoto and Takao Murakami. 2019. Local obfuscation mechanisms for hiding probability distributions. In Proc. of ESORICS, Part I(LNCS, Vol. 11735). Springer, 128–148.

Google Scholar

[43]

M. H. R. Khouzani and P. Malacaria. 2016. Relative perfect secrecy: Universally optimal strategies and channel design. In Proc. of CSF. 61–76.

Google Scholar

[44]

M. H. R. Khouzani and Pasquale Malacaria. 2017. Leakage-minimal design: Universality, limitations, and applications. In Proc. of CSF. IEEE, Los Alamitos, CA, 305–317.

Google Scholar

[45]

M. H. R. Khouzani and Pasquale Malacaria. 2018. Optimal channel design: A game theoretical analysis. Entropy 20, 9 (2018), 675.

Crossref

Google Scholar

[46]

M. H. R. Khouzani and Pasquale Malacaria. 2019. Generalized entropies and metric-invariant optimal countermeasures for information leakage under symmetric constraints. IEEE Trans. Inf. Theory 65, 2 (2019), 888–901.

Digital Library

Google Scholar

[47]

M. H. R. Khouzani, Piotr Mardziel, Carlos Cid, and Mudhakar Srivatsa. 2015. Picking vs. guessing secrets: A game-theoretic analysis. In Proc. of CSF. IEEE, Los Alamitos, CA, 243–257.

Google Scholar

[48]

Daniel Kifer and Ashwin Machanavajjhala. 2014. Pufferfish: A framework for mathematical privacy definitions. ACM Trans. Database Syst. 39, 1 (2014), Article 3, 36 pages.

Digital Library

Google Scholar

[49]

Boris Köpf and David A. Basin. 2007. An information-theoretic model for adaptive side-channel attacks. In Proc. of CCS. ACM, New York, NY, 286–296.

Google Scholar

[50]

Boris Köpf and Andrey Rybalchenko. 2010. Approximation and randomization for quantitative information-flow analysis. In Proc. of CSF. IEEE, Los Alamitos, CA, 3–14.

Google Scholar

[51]

Dmytro Korzhyk, Zhengyu Yin, Christopher Kiekintveld, Vincent Conitzer, and Milind Tambe. 2011. Stackelberg vs. nash in security games: An extended investigation of interchangeability, equivalence, and uniqueness. J. Artif. Intell. Res. 41 (2011), 297–327.

Crossref

Google Scholar

[52]

Mohammad Hossein Manshaei, Quanyan Zhu, Tansu Alpcan, Tamer Bacşar, and Jean-Pierre Hubaux. 2013. Game theory meets network security and privacy. ACM Comput. Surv. 45, 3 (2013), Article 25, 39 pages.

Digital Library

Google Scholar

[53]

Piotr Mardziel, Mário S. Alvim, Michael W. Hicks, and Michael R. Clarkson. 2014. Quantifying information flow for dynamic secrets. In Proc. of S&P. 540–555.

Google Scholar

[54]

James Massey. 1994. Guessing and entropy. In Proc. of ISIT. IEEE, Los Alamitos, CA, 204.

Google Scholar

[55]

Akihiko Matsui. 1989. Information leakage forces cooperation. Games Econ. Behav. 1, 1 (1989), 94–115.

Crossref

Google Scholar

[56]

David Mestel. 2019. Quantifying information flow in interactive systems. In Proc. of CSF. IEEE, Los Alamitos, CA, 414–427.

Google Scholar

[57]

Angelia Nedić and Asuman Ozdaglar. 2009. Subgradient methods for saddle-point problems. J. Optim. Theory Appl. 142, 1 (2009), 205–228.

Digital Library

Google Scholar

[58]

Martin J. Osborne and Ariel Rubinstein. 1994. A Course in Game Theory. MIT Press, Cambridge, MA.

Google Scholar

[59]

Michael K. Reiter and Aviel D. Rubin. 1998. Crowds: Anonymity for Web transactions. ACM Trans. Inf. Syst. Secur. 1, 1 (1998), 66–92.

Digital Library

Google Scholar

[60]

Ariel Rubinstein. 2012. Lecture Notes in Microeconomic Theory (2nd ed.). Princeton University Press, Princeton, NJ.

Google Scholar

[61]

Claude E. Shannon. 1948. A mathematical theory of communication. Bell Syst. Tech. J. 27 (1948), 379–423.

Crossref

Google Scholar

[62]

Vitaly Shmatikov. 2002. Probabilistic analysis of anonymity. In Proc. of CSFW. 119–128.

Google Scholar

[63]

Geoffrey Smith. 2009. On the foundations of quantitative information flow. In Proc. of FOSSACS(LNCS, Vol. 5504). Springer, 288–302.

Google Scholar

[64]

Parvathinathan Venkitasubramaniam and Lang Tong. 2012. A game-theoretic approach to anonymous networking. IEEE/ACM Trans. Netw. 20, 3 (2012), 892–905.

Digital Library

Google Scholar

[65]

John von Neumann and Oskar Morgenstern. 2007. Theory of Games and Economic Behavior. Princeton University Press, Princeton, NJ.

Google Scholar

[66]

Weiran Wang and Miguel A. Carreira-Perpinán. 2013. Projection onto the probability simplex: An efficient algorithm with a simple proof, and an application. arXiv preprint arXiv:1309.1541 (2013).

Google Scholar

[67]

Haifeng Xu, Albert Xin Jiang, Arunesh Sinha, Zinovi Rabinovich, Shaddin Dughmi, and Milind Tambe. 2015. Security games with information leakage: Modeling and computation. In Proc. of IJCAI. 674–680.

Google Scholar

[68]

Mu Yang, Vladimiro Sassone, and Sardaouna Hamadou. 2012. A game-theoretic analysis of cooperation in anonymity networks. In Proc. of POST. 269–289.

Google Scholar

[69]

Andrew C. Yao. 1982. Protocols for secure computations. In Proc. of FOCS. 160–164.

Google Scholar

Cited By

View all

Xue LXiong S(2024)Flexible metal foil/polymer sandwich composites for electromagnetic interference shielding with anti-wind–sand environment toleranceJournal of Materials Science: Materials in Electronics10.1007/s10854-024-13427-135:25Online publication date: 6-Sep-2024
https://doi.org/10.1007/s10854-024-13427-1
Sakib SAmariucai GGuan Y(2023)Variations and Extensions of Information Leakage Metrics with Applications to Privacy Problems with Imperfect Statistical Information2023 IEEE 36th Computer Security Foundations Symposium (CSF)10.1109/CSF57540.2023.00007(407-422)Online publication date: Jul-2023
https://doi.org/10.1109/CSF57540.2023.00007

Recommendations

Fair and private rewarding in a coalitional game of cybersecurity information sharing

Cybersecurity information sharing is a key factor of cyber threat intelligence, allowing organisations to detect and prevent malicious behaviours proactively. However, stimulating organisations to participate and deterring free‐riding in such sharing is ...
Attacker Investigation System Triggered by Information Leakage
IIAI-AAI '15: Proceedings of the 2015 IIAI 4th International Congress on Advanced Applied Informatics

While a considerable amount of research has been devoted to preventing leakage of classified information, little attention has been paid to identifying attackers who steal information. If attackers can be identified, more precise countermeasures can be ...
Bypassing information leakage protection with trusted applications

Insider threats are an increasing concern for most modern organizations. Information leakage is one of the most important insider threats, particularly according to its potential financial impact. Data Leakage Protection (DLP) systems have been ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Privacy and Security

ACM Transactions on Privacy and Security Volume 25, Issue 3

August 2022

288 pages

ISSN:2471-2566

EISSN:2471-2574

DOI:10.1145/3530305

Editor:
Ninghui Li
Purdue University, USA

Issue’s Table of Contents

This work is licensed under a Creative Commons Attribution-NoDerivs International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 April 2022

Accepted: 01 January 2022

Revised: 01 January 2022

Received: 01 December 2020

Published in TOPS Volume 25, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

JSPS and Inria
PEPS 2018 project MAGIC
ANR project REPAS
Epistemic Interactive Concurrency (EPIC) from the STIC AmSud Program
CNPq
CAPES
FAPEMIG
ERATO HASUO Metamathematics for Systems Design Project
JSPS KAKENHI
ERC grant HYPATIA under the European Union Horizon 2020 research and innovation programme

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
2,199
Total Downloads

Downloads (Last 12 months)604
Downloads (Last 6 weeks)90

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Xue LXiong S(2024)Flexible metal foil/polymer sandwich composites for electromagnetic interference shielding with anti-wind–sand environment toleranceJournal of Materials Science: Materials in Electronics10.1007/s10854-024-13427-135:25Online publication date: 6-Sep-2024
https://doi.org/10.1007/s10854-024-13427-1
Sakib SAmariucai GGuan Y(2023)Variations and Extensions of Information Leakage Metrics with Applications to Privacy Problems with Imperfect Statistical Information2023 IEEE 36th Computer Security Foundations Symposium (CSF)10.1109/CSF57540.2023.00007(407-422)Online publication date: Jul-2023
https://doi.org/10.1109/CSF57540.2023.00007

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

1 Introduction

1.1 Specific Information Leakage Frameworks

1.2 Contributions

1.2.1 Relation with Preliminary Version.

1.2.2 Relation with Our Subsequent Work.

1.3 Related Work

1.4 Plan of the Article

2 Preliminaries

2.1 Convexity and Quasi-Convexity

2.2 Two-Player, Simultaneous Games

2.3 Zero-Sum Games and Minimax Theorem

2.4 Quantitative Information Flow

2.5 Differential Privacy

3 QIF-games

3.1 Definition of QIF-games

3.1.1 The Two Millionaires Problem.

3.1.2 The Binary-Sum Game.

3.2 Existence of Nash Equilibrium for QIF-games

3.3 Computing Equilibria for QIF-games

4 DP-games

4.1 Definition of DP-games

4.2 DP-games vs. QIF-games

4.3 Formal Definition of DP-games

4.4 Existence of Nash Equilibrium for DP-games with Hidden Choice

4.5 Computing Equilibria for DP-games with Hidden Choice

4.6 Nash Equilibria for DP-games with Visible Choice

5 Information-leakage Games Vs. Standard Game Theory Models

5.1 Von Neumann–Morgenstern’s Treatment of Utility

5.2 Risk Functions

5.3 Convex-Concave Games

6 Case Studies

6.1 The Crowds Protocol as a QIF-game

6.2 Design of an LDP Mechanism Using a DP-game

7 Conclusion and Future Work

Footnotes

B Proofs

Acknowledgments

References

Cited By

Index Terms

Recommendations

Fair and private rewarding in a coalitional game of cybersecurity information sharing

Attacker Investigation System Triggered by Information Leakage

Bypassing information leakage protection with trusted applications

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

HTML Format

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations