A Realization Approach to Lossy Network Compression of a Tuple of Correlated Multivariate Gaussian RVs ^†

Abstract

Examined in this paper is the Gray and Wyner source coding for a simple network of correlated multivariate Gaussian random variables, $Y_1:\Omega \to {\mathbb{R}}^{p_1}$ and $Y_2:\Omega \to {\mathbb{R}}^{p_2}$. The network consists of an encoder that produces two private rates $R_1$ and $R_2$, and a common rate $R_0$, and two decoders, where decoder 1 receives rates $(R_1,R_0)$ and reproduces $Y_1$ by ${\widehat{Y}}_1$, and decoder 2 receives rates $(R_2,R_0)$ and reproduces $Y_2$ by ${\widehat{Y}}_2$, with mean-square error distortions $\mathbb{E}\left|\right|Y_i-{\widehat{Y}}_i{\left|\right|}_{{\mathbb{R}}^{p_i}}^2\le {\Delta }_i\in [0,\infty ],i=1,2$. Use is made of the weak stochastic realization and the geometric approach of such random variables to derive test channel distributions, which characterize the rates that lie on the Gray and Wyner rate region. Specific new results include: (1) A proof that, among all continuous or finite-valued random variables, $W:\Omega \to \mathbb{W}$, Wyner’s common information, $C(Y_1,Y_2)={inf}_{{\mathbf{P}}_{Y_1,Y_2,W}:{\mathbf{P}}_{Y_1,Y_2|W}={\mathbf{P}}_{Y_1|W}{\mathbf{P}}_{Y_2|W}}I(Y_1,Y_2;W)$, is achieved by a Gaussian random variable, $W:\Omega \to {\mathbb{R}}^n$ of minimum dimension n, which makes the two components of the tuple $(Y_1,Y_2)$ conditionally independent according to the weak stochastic realization of $(Y_1,Y_2)$, and a the formula $C(Y_1,Y_2)=\frac{1}{2}{\sum }_{j=1}^{n}ln\left(\frac{1+d_j}{1-d_j}\right),$ where $d_i\in (0,1),i=1,\dots ,n$ are the canonical correlation coefficients of the correlated parts of $Y_1$ and $Y_2$, and a realization of $(Y_1,Y_2,W)$ which achieves this. (2) The parameterization of rates that lie on the Gray and Wyner rate region, and several of its subsets. The discussion is largely self-contained and proceeds from first principles, while connections to prior literature is discussed.

1. Introduction

In their seminal paper, Source Coding for a Simple Network [1], Gray and Wyner characterized the lossless rate region for a tuple of finite-valued random variables, and the lossy rate region for a tuple of arbitrary distributed random variables. Many extensions and generalizations followed Gray and Wyner’s fundamental work. Wyner [2] introduced an operational definition of the common information between a tuple of sources that generate symbols with values in finite spaces. Wyner’s operational definition of common information is defined as the minimum achievable common message rate on the Gray and Wyner lossless rate region. Witsenhausen [3] investigated bounds for Wyner’s common information, and sequences of pairs of random variables in this regard [4]. Gács and Körner [5] introduced another definition of common randomness between a tuple of jointly independent and identically distributed random variables. Benammar and Zaidi [6,7] characterized the Gray–Wyner rate region, when there is side information at the decoders, under various scenarios that include both receivers and reproduce the source symbols without distortion. Insightful application examples for binary sources are considered in [7] (Section 4.2). In their previous work, Benammar and Zaidi [8,9] characterized the rate distortion function of the Heegard and Berger [10] problem, with two sources and side information at the two decoders (under a degraded set-up). Connections between the Gray and Wyner lossy source coding network and the notions of empirical and strong coordination capacity for arbitrary networks were developed by Cuff, Permuter and Cover [11] and the references therein, where the authors elaborated on the usefulness of the common information between the different network nodes.

Viswanatha, Akyol and Rose [12], and Xu, Liu and Chen [13], explored the connection of Wyner’s common information and the Gray and Wyner lossy rate region, to generalize Wyner’s common information to its lossy counterpart, for random variables taking values in arbitrary spaces. They characterized Wyner’s lossy common information as the minimum common message rate on the Gray and Wyner lossy rate region, when the sum rate is arbitrarily close to the rate distortion function with joint decoding for the Gray and Wyner lossy network. Applications to encryption and secret key generation are discussed by Viswanatha, Akyol and Rose in [12] (and references therein).

The current paper is focused on the calculations of rates that lie in the Gray and Wyner rate region [1], for two sources that generate symbols, according to the model of jointly independent and identically distributed multivariate correlated Gaussian random variables $Y_1:\Omega \to {\mathbb{R}}^{p_1},Y_2:\Omega \to {\mathbb{R}}^{p_2}$, and square-error fidelity at the two decoders. The current literature on methods and algorithms to compute such rates are subject to a number of limitations which often prevent their practical usefulness:

(1) Rates that lie in the Gray and Wyner rate region are only known for the special case of a tuple of scalar-valued Gaussian random variables with square error distortion, i.e., $p_1=p_2=1$ [1,12,13].

(2) Wyner’s lossy common information is only computed in closed form, for the special cases of a tuple of scalar-valued Gaussian random variables, [12,13].

(3) Important generalizations to a tuple of sources that generate multivariate Gaussian symbols, require new derivations often of considerable difficulty.

(4) Realizations of the optimal test channel distributions and their structural properties of the various rate distortion functions (RDFs), which are involved in the Gray and Wyner characterization of the rate region, are not developed.

(5) A proof that the Gray and Wyner for jointly Gaussian sources, is characterized by a Gaussian auxiliary random variable W is still missing from past literature.

It is known from [1] that the Gray and Wyner rate region can be parameterized by an auxiliary random variable $W:\Omega \to \mathbb{W}$, via several rate distortion functions. Moreover, subsets of the Gray and Wyner rate region are parameterized by W which satisfies conditional independence (1).

$$\begin{array}{c}\hfill {\mathbf{P}}_{Y_1,Y_2|W}={\mathbf{P}}_{Y_1|W}{\mathbf{P}}_{Y_2|W}.\end{array}$$

(1)

The current paper makes use of the canonical variable form and the weak stochastic realization of the tuple of random variables $(Y_1,Y_2)$, introduced in Section 2 to characterize subsets of the Gray and Wyner rate region, which are parameterized by jointly Gaussian random variables $(Y_1,Y_2,W)$ with $W:\Omega \to {\mathbb{R}}^n$, where n is a finite number, while in some cases, the minimum dimension of W is clarified. The weak stochastic realization is developed to deal with the fundamental issue that, for the Gray and Wyner network, one is given the joint distribution ${\mathbf{P}}_{Y_1,Y_2}$, while the characterization of the RDFs involves the specification of the test channel distributions, that achieve these RDFs, and the actual construction of realizations of all random variables involved, that induce the test channel distributions. Furthermore, Wyner’s common information between $Y_1$ and $Y_2$ involves the construction of a joint distribution ${\mathbf{P}}_{Y_1,Y_2,W}$ where W is the auxiliary random variable that makes $Y_1$ and $Y_2$ conditionally independent, i.e., (1) holds.

The rest of the section serves mainly to review the Gray and Wyner characterization of the lossy rate region and the characterization of Wyner’s lossy common information.

1.1. Literature Review

(a) The Gray and Wyner source coding for a simple network [1].

Consider the Gray and Wyner source coding for a simple network, as shown in Figure 1, for a tuple of jointly independent and identically distributed multivariate Gaussian random variables $(Y_1^N,Y_2^N)=\{(Y_{1,i},Y_{2,i}):i=1,2,\dots ,N\}$,

$$\begin{array}{c}\hfill Y_{1,i}:\Omega \to {\mathbb{R}}^{p_1}={\mathbb{Y}}_1,\hspace{1em}{Y}_{2,i}:\Omega \to {\mathbb{R}}^{p_2}={\mathbb{Y}}_2,\hspace{1em}i=1,\dots ,N\end{array}$$

(2)

with square error distortion functions at the two decoders,

$$\begin{array}{c}\hfill D_{Y_1}(y_1^N,{\widehat{y}}_1^N)=\frac{1}{N}\sum _{i=1}^N\left|\right|y_{1,i}-{\widehat{y}}_{1,i}{\left|\right|}_{{\mathbb{R}}^{p_1}}^2,\hspace{1em}{D}_{Y_2}(y_2^N,{\widehat{y}}_2^N)=\frac{1}{N}\sum _{i=1}^N\left|\right|y_{2,i}-{\widehat{y}}_{2,i}{\left|\right|}_{{\mathbb{R}}^{p_2}}^2\end{array}$$

(3)

where $\left|\right|·{\left|\right|}_{{\mathbb{R}}^{p_i}}^2$ are Euclidean distances on ${\mathbb{R}}^{p_i},i=1,2$.

The encoder takes as its input the data sequences $(Y_1^N,Y_2^N)$ and produces at its output three messages, $(S_0,S_1,S_2)$, with binary bit representations $(N{R}_0,N{R}_1,N{R}_2)$, respectively. There are three channels, channel 0, channel 1, channel 2, with capacities $(C_0,C_1,C_2)$ (in bits per second), respectively, to transmit the messages to two decoders. Channel 0 is a common channel and channel 1 and channel 2 are the private channels which connect the encoder to each of the two decoders. Message $S_0$ is a common or public message that is transmitted through the common channel 0 with capacity $C_0$ to decoder 1 and decoder 2; $S_1$ is a private message which is transmitted through the private channel 1 with capacity $C_1$ to decoder 1; and $S_2$ is a private message, which is transmitted through the private channel 2 with capacity $C_2$ to decoder 2.

Decoder 1 aims to reproduce $Y_1^N$ by ${\widehat{Y}}_1^N$ subject to an average distortion and decoder 2 aims to reproduce $Y_2^N$ by ${\widehat{Y}}_2^N$, subject to an average distortion, where $({\widehat{Y}}_{1,i},{\widehat{Y}}_{2,i})=({\widehat{y}}_{1,i},{\widehat{y}}_{2,i})\in {\widehat{\mathbb{Y}}}_1\times {\widehat{\mathbb{Y}}}_2\subseteq {\mathbb{Y}}_1\times {\mathbb{Y}}_2,i=1,\dots ,N$, that is,

$$\begin{array}{c}\hfill \mathbf{E}\left\{D_{Y_1}(Y_1^N,{\widehat{Y}}_1^N)\right\}\le {\Delta }_1,\hspace{1em}\mathbf{E}\left\{D_{Y_2}(Y_2^N,{\widehat{Y}}_2^N)\right\}\le {\Delta }_2,\hspace{1em}({\Delta }_1,{\Delta }_2)\in [0,\infty ]\times [0,\infty ].\end{array}$$

(4)

Gray and Wyner characterized the rate region, denoted by ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$, via a coding scheme that uses the auxiliary random variable W, and the family of probability distributions

$$\begin{array}{c}\hfill \mathcal{P}\triangleq \left\{{\mathbf{P}}_{Y_1,Y_2,W}(y_1,y_2,w),\hspace{1em}\hspace{1em}{y}_1\in {\mathbb{Y}}_1,y_2\in {\mathbb{Y}}_2,w\in \mathbb{W}|\hspace{1em}\hspace{1em}{\mathbf{P}}_{Y_1,Y_2,W}(y_1,y_2,\infty )={\mathbf{P}}_{Y_1,Y_2}(y_1,y_2)\right\}\end{array}$$

(5)

such that the joint probability distribution ${\mathbf{P}}_{Y_1,Y_2,W}(y_1,y_2,w)$ on ${\mathbb{Y}}_1\times {\mathbb{Y}}_2\times \mathbb{W}$, has a $(Y_1,Y_2)$−marginal probability distribution ${\mathbf{P}}_{Y_1,Y_2}(y_1,y_2)$ on ${\mathbb{Y}}_1\times {\mathbb{Y}}_2$ that coincides with the probability distribution of $(Y_1,Y_2)$.

For the source and distortion function specified in (2) and (3), we apply the weak stochastic realization to construct the family of distributions $\mathcal{P}$, which is parameterized by the auxiliary random variable W. Use is made of the characterization of ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$ is described in terms of an auxiliary random variable, as follows.

Theorem 1

(Theorem 8 in [1]). Let ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$ denote the Gray and Wyner rate region of the simple network shown in Figure 1.

Suppose there exists ${\widehat{y}}_i\in {\widehat{\mathbb{Y}}}_i$ such that $\mathbf{E}\left\{d_{Y_i}(Y_i,{\widehat{y}}_i)\right\}<\infty$, for $i=1,2$.

For each ${\mathbf{P}}_{Y_1,Y_2,W}\in \mathcal{P}$ and ${\Delta }_1\ge 0,{\Delta }_2\ge 0$, define the subset of Euclidean $3-$ dimensional space

$$\begin{array}{c}\hfill {\mathcal{R}}_{GW}^{{\mathbf{P}}_{Y_1,Y_2,W}}({\Delta }_1,{\Delta }_2)=\left\{\left(R_0,R_1,R_2\right)|\hspace{1em}\hspace{1em}{R}_0\ge I(Y_1,Y_2;W),\hspace{1em}\hspace{1em}{R}_1\ge R_{Y_1|W}\left({\Delta }_1\right),\hspace{1em}\hspace{1em}{R}_2\ge R_{Y_2|W}\left({\Delta }_2\right)\right\}\end{array}$$

(6)

where $R_{Y_i|W}\left({\Delta }_i\right)$ is the conditional rate distortion function of $Y_i^N$, conditioned on $W^N$, at decoder i, for $i=1,2$, and $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$ is the joint rate distortion function of the joint decoding of $(Y_1^N,Y_2^N)$ (all single letters). Let

$$\begin{array}{c}\hfill {\mathcal{R}}_{GW}^{\ast }({\Delta }_1,{\Delta }_2)={\left(\bigcup _{{\mathbf{P}}_{Y_1,Y_2,W}\in \mathcal{P}}{\mathcal{R}}_{GW}^{{\mathbf{P}}_{Y_1,Y_2,W}}({\Delta }_1,{\Delta }_2)\right)}^c\end{array}$$

(7)

where ${\left(·\right)}^c$ denotes the closure of the indicated set. The achievable Gray–Wyner lossy rate region is given by

$$\begin{array}{c}\hfill {\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)={\mathcal{R}}_{GW}^{\ast }({\Delta }_1,{\Delta }_2).\end{array}$$

(8)

Gray and Wyner [1] (Theorem 6) also showed that, if $(R_0,R_1,R_2)\in {\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$, then

$$\begin{array}{cc}\hfill R_0+R_1+R_2& \ge R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2),\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill R_0+R_1& \ge R_{Y_1}\left({\Delta }_1\right),\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill R_0+R_2& \ge R_{Y_2}\left({\Delta }_2\right)\hfill \end{array}$$

(11)

where $R_{Y_i}\left({\Delta }_i\right)$ is the rate distortion function of $Y_i^N$ at decoder i, for $i=1,2$, and $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$ is the joint rate distortion function of $(Y_1^N,Y_2^N)$ at the two decoders. The inequality in (9) is called the Pangloss Bound of the Gray–Wyner lossy rate region ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$. The set of triples $(R_0,R_1,R_2)\in {\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$ that satisfy the equality $R_0+R_1+R_2=R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$ is called the Pangloss Plane of the Gray–Wyner lossy rate region ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$.

Gray and Wyner proved that ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$, is also determined from [1] ((4) of page 1703, Equation (42)),

$$\begin{array}{c}\hfill T({\alpha }_1,{\alpha }_2)=\underset{{\mathbf{P}}_{Y_1,Y_2,W}\in \mathcal{P}}{inf}\left\{I(Y_1,Y_2;W)+{\alpha }_1{R}_{Y_1|W}\left({\Delta }_1\right)+{\alpha }_2{R}_{Y_2|W}\left({\Delta }_2\right)\right\}\end{array}$$

(12)

where $0\le {\alpha }_i\le 1,i=1,2,{\alpha }_1+{\alpha }_2\ge 1$, and where for each ${\mathbf{P}}_{Y_1,Y_2,W}\in \mathcal{P}$, the conditional distribution ${\mathbf{P}}_{Y_1,Y_2|W}$ is defined, from which follows the ${\mathbb{Y}}_i-$ marginals ${\mathbf{P}}_{Y_i|W},i=1,2$.

(b) Wyner’s common Information of finite-valued random variables.

Wyner [2] introduced an operational definition of the common information between a tuple of random variables $(Y_1^N,Y_2^N)$ that takes values in finite spaces.

The first approach of Wyner’s operational definition of common information between sequences $Y_1^N$ and $Y_2^N$ is defined as the minimum achievable common message rate $R_0$ on the Gray–Wyner network of Figure 1.

Wyner’s single letter information theoretic characterization of the infimum of all achievable message rates $R_0$, called Wyner’s common information, is defined by

$$\begin{array}{c}\hfill C(Y_1,Y_2)=\underset{{\mathbf{P}}_{Y_1,Y_2,W}:\hspace{1em}{\mathbf{P}}_{Y_1,Y_2|W}={\mathbf{P}}_{Y_1|W}{\mathbf{P}}_{Y_2|W}}{inf}I(Y_1,Y_2;W).\end{array}$$

(13)

Here, ${\mathbf{P}}_{Y_1,Y_2,W}$ is any joint probability distribution on ${\mathbb{Y}}_1\times {\mathbb{Y}}_2\times \mathbb{W}$ with $(Y_1,Y_2)$−marginal ${\mathbf{P}}_{Y_1,Y_2}$, such that W makes $Y_1$ and $Y_2$ conditionally independent, that is ${\mathbf{P}}_{Y_1,Y_2,W}\in \mathcal{P}$.

(c) Minimum common message rate and Wyner’s lossy common information for arbitrary random variables.

Viswanatha, Akyol and Rose [12], and Xu, Liu and Chen [13] explored the connection of Wyner’s common information and the Gray–Wyner lossy rate region, to provide a new interpretation of Wyner’s common information to its lossy counterpart.

The following characterization was derived by Xu, Liu and Chen [13] (an equivalent characterization was also derived by Viswanatha, Akyol and Rose [12]).

Theorem 2

(Theorem 4 in [13]). Suppose there exists ${\widehat{y}}_i\in {\widehat{\mathbb{Y}}}_i$ such that $\mathbf{E}\left\{d_{Y_i}(Y_i,{\widehat{y}}_i)\right\}<\infty$, for $i=1,2$.

Let $C_{GW}(Y_1,Y_2;{\Delta }_1,{\Delta }_2)$ denote the minimum common message rate $R_0$ on the Gray–Wyner lossy rate region ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$, with a sum rate not exceeding the joint rate distortion function, ${\sum }_{i=0}^2{R}_i\ge R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$, while satisfying the average distortions.

Then, $C_{GW}(Y_1,Y_2;{\Delta }_1,{\Delta }_2)$ is characterized by the optimization problem

$$\begin{array}{c}\hfill C_{GW}(Y_1,Y_2;{\Delta }_1,{\Delta }_2)=inf\hspace{1em}I(Y_1,Y_2;W)\end{array}$$

(14)

such that the following identity holds

$$\begin{array}{c}\hfill R_{Y_1|W}\left({\Delta }_1\right)+R_{Y_2|W}\left({\Delta }_2\right)+I(Y_1,Y_2;W)=R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)\end{array}$$

(15)

where the infimum is over all random variables W taking values in $\mathbb{W}$, which parameterize the source distribution via ${\mathbf{P}}_{Y_1,Y_2,W}$, having a ${\mathbb{Y}}_1\times {\mathbb{Y}}_2-$ marginal source distribution ${\mathbf{P}}_{Y_1,Y_2}$, and induce joint distributions ${\mathbf{P}}_{W,Y_1,Y_2,{\widehat{Y}}_1,{\widehat{Y}}_2}$ which satisfy the constraints.

It is shown in [12,13] that there exists a distortion region such that $C_{GW}(Y_1,Y_2;{\Delta }_1,{\Delta }_2)=C_W(Y_1,Y_2)$, i.e., it is independent of the distortions $({\Delta }_1,{\Delta }_2)$, and $C_W(Y_1,Y_2)=C(Y_1,Y_2)$, i.e., it is equal to Wyner’s information theoretic characterization of common information between $Y_1$ and $Y_2$, defined by (13). However, their proofs that W is finite-dimensional Gaussian relies on the assumption that W is continuous-valued.

The next theorem is derived by Xu, Liu and Chen [13].

Theorem 3

(Theorem 5 in [13]). Let $(Y_1,Y_2)$ be a pair of random variables with distribution ${\mathbf{P}}_{Y_1,Y_2}$ on the alphabet space ${\mathbb{Y}}_1\times {\mathbb{Y}}_2$, where ${\mathbb{Y}}_1$ and ${\mathbb{Y}}_2$ are arbitrary measurable spaces that can be discrete or continuous.

Let W be any random variable achieving $C(Y_1,Y_2)$ defined by (13).

Let the reproduction alphabet ${\widehat{\mathbb{Y}}}_1={\mathbb{Y}}_1$, ${\widehat{\mathbb{Y}}}_2={\mathbb{Y}}_2$ and two per-letter distortion measures $d_{Y_1}(y_1,{\widehat{y}}_1),d_{Y_2}(y_2,{\widehat{y}}_2)$ satisfy

$$\begin{array}{c}\hfill d_{Y_i}(y_i,{\widehat{y}}_i)>d_{Y_i}(y_i,y_i)=0,\hspace{1em}\hspace{1em}{y}_i\ne {\widehat{y}}_i,\hspace{1em}\hspace{1em}i=1,2\end{array}$$

(16)

If the following conditions are satisfied:

(1) For any $y_1\in {\mathbb{Y}}_1,y_2\in {\mathbb{Y}}_2$ and $w\in \mathbb{W}$, ${\mathbf{P}}_{W|Y_1,Y_2}>0$;

(2) There exists an ${\widehat{y}}_i\in {\widehat{\mathbb{Y}}}_i$, such that

$$\begin{array}{c}\hfill \mathbf{E}\left\{d_{Y_i}(Y_i,{\widehat{y}}_i)\right\}<\infty ,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}i=1,2\end{array}$$

(17)

then there exists a strictly positive vector $\gamma =({\gamma }_1,{\gamma }_2)\in (0,\infty )\times (0,\infty )$, such that, for $0\le ({\Delta }_1,{\Delta }_2)\le \gamma$,

$$\begin{array}{c}\hfill C_{GW}(Y_1,Y_2;{\Delta }_1,{\Delta }_2)=C_W(Y_1,Y_2)=C(Y_1,Y_2).\end{array}$$

(18)

Moreover, $C_{GW}(Y_1,Y_2;{\Delta }_1,{\Delta }_2)$ is constant on ${\mathcal{D}}_W=\left\{({\Delta }_1,{\Delta }_2)\in [0,\infty ]\times [0,\infty ]:0\le ({\Delta }_1,{\Delta }_2)\le \gamma \right\}$.

The analog of the above theorem is also derived by Viswanatha, Akyol and Rose in [12] (Lemma 1). A subset of the Pangloss plane is derived by Gray and Wyner [1] (Theorem 9).

For a bivariate Gaussian random variables, i.e., $p_1=p_2=1$, with square-error distortions, Viswanatha, Akyol and Rose in [12], and Xu, Liu and Chen [13] computed $C_{GW}(Y_1,Y_2;{\Delta }_1,{\Delta }_2)$, by using Xiao’s and Luo’s [14] (Theorem 6) the closed-form expression of joint rate distortion function $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$. In addition, for the bivariate Gaussian random variables, with symmetric square-error distortions, i.e., ${\Delta }_1={\Delta }_2=\Delta$, Gray and Wyner [1] (Section 2.5, (B)), computed a rate-triple $(R_0,R_1,R_2)\in {\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$ that lies on the Pangloss plane.

1.2. Main Theorems and Discussion

What follows is a brief summary of the main theorems derived in this paper, and relations to the literature.

Theorem 9 shows that, among all joint distributions ${\mathbf{P}}_{Y_1,Y_2,W}$ induced by a tuple of multivariate correlated Gaussian random variables $(Y_1,Y_2)$, and an arbitrary random variable $W:\Omega \to \mathbb{W}$, continuous or discrete-valued, Wyner’s common information $C(Y_1,Y_2)$, defined by (13), is minimized by a triple $(Y_1,Y_2,W)$ which induces a jointly Gaussian distribution ${\mathbf{P}}_{Y_1,Y_2,W}$, and $W:\Omega \to \mathbb{W}={\mathbb{R}}^n$ is a finite-dimensional Gaussian random variable. In particular, Theorem 9 gives the weak stochastic realization of $(Y_1,Y_2)$, and the construction of the random variable W, which induce a joint distribution ${\mathbf{P}}_{Y_1,Y_2,W}$ that achieves the minimum of $I(Y_1,Y_2;W)$ such that W makes $Y_1$ and $Y_2$ conditionally independent.

Then, use is made of Theorem 9, Section 2.2, such as Definition 1 of the canonical variable form and the weak stochastic realization to derive Wyner’s common information $C(Y_1,Y_2)$ defined by (13), and the optimal realization of the triple $(Y_1,Y_2,W^{\ast })=(Y_1,Y_2,W^{\ast })$ that achieves $C(Y_1,Y_2)$, as stated in the next theorem.

Theorem 4.

Consider a tuple of Gaussian random variables $Y_i:\Omega \to {\mathbb{R}}^{p_i}$, with $Q_{Y_i}>0$, for $i=1,2$, $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)})$, $Q_{(Y_1,Y_2)}\ge 0$, and apply Algorithm A1 (and the notation therein) to decompose and transform the random variables into a canonical variable form (with abuse of notation, the transform random variables are denoted by $(Y_1,Y_2)\in G(0,Q_{\mathrm{cvf}})$.), $(Y_1,Y_2)\in G(0,Q_{\mathrm{cvf}})$, using the material and notation of Section 2.2, i.e., Definition 1.

(a) Then,

$$\begin{array}{cc}\hfill C(Y_1,Y_2)=& C(Y_{11},Y_{21})+C(Y_{12},Y_{22})+C(Y_{13},Y_{23})\hfill \\ \hfill =& \left\{\begin{array}{ccc}0,\hfill & if\hfill & p_{13}>0,\hspace{1em}{p}_{23}>0,\hspace{1em}{p}_{11}=p_{12}=p_{21}=p_{22}=0,\hfill \\ \frac{1}{2}\sum _{i=1}^{n}ln\left(\frac{1+d_i}{1-d_i}\right),\hfill & if\hfill & p_{12}=p_{22}>0,\hspace{1em}{p}_{11}=p_{21}=0,\hspace{1em}{p}_{13}\ge 0,\hspace{1em}{p}_{23}\ge 0,\hfill \\ +\infty ,\hfill & if\hfill & p_{11}=p_{21}>0\hfill \end{array}\right.\hfill \end{array}$$

(19)

where $(p_{11},p_{12},p_{13})$ and $(p_{21},p_{22},p_{23})$ are the dimensions of the canonical variable decomposition of the tuple $(Y_1,Y_2)$, and

$$\begin{array}{cc}\hfill C(Y_{11},Y_{21})=& +\infty ,\hspace{1em}\hspace{1em}if\hspace{1em}\hspace{1em}{p}_{11}=p_{21}>0;\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill C(Y_{13},Y_{23})=& 0,\hspace{1em}\hspace{1em}if\hspace{1em}\hspace{1em}{p}_{13}>0\hspace{1em}\hspace{1em}and\hspace{1em}\hspace{1em}{p}_{23}>0;\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill C(Y_{12},Y_{22})=& \frac{1}{2}\sum _{i=1}^{n}ln\left(\frac{1+d_i}{1-d_i}\right),\hspace{1em}\hspace{1em}if\hspace{1em}\hspace{1em}n=p_{12}=p_{22}>0.\hfill \end{array}$$

(22)

Thus, $C(Y_{12},Y_{22})$ is the most interesting value if defined.

(b) The random variable $W^{\ast }$ defined below is such that $C(Y_1,Y_2)$ of part (a) is attained.

$$\begin{array}{c}{W}^{\ast }:\Omega \to {\mathbb{R}}^n,\hspace{1em}\hspace{1em}n\in {\mathbb{Z}}_{+},\hfill \\ n_1=p_{11}=p_{21},\hspace{1em}\hspace{1em}{n}_2=p_{12}=p_{22},\hspace{1em}\hspace{1em}{n}_1+n_2=n,\hfill \\ W^{\ast }=\left(\begin{array}{c}{W}_1^{\ast }\hfill \\ W_2^{\ast }\hfill \end{array}\right),\hspace{1em}\hspace{1em}\hspace{1em}{W}_1^{\ast }:\Omega \to {\mathbb{R}}^{n_1},\hspace{1em}\hspace{1em}{W}_2^{\ast }:\Omega \to {\mathbb{R}}^{n_2},\hspace{1em}\hspace{1em}\hfill \\ W_1^{\ast }=Y_{11}=Y_{21},\hfill \end{array}$$

(23)

$$W_2^{\ast }=L_1{Y}_{12}+L_2{Y}_{22}+L_{3}V,\hspace{1em}\hspace{1em}see Theorem \mathit{11}.\left(b\right)for the formulas of L_1, L_2, L_3$$

(24)

where the following properties hold:

$$then\hspace{1em}(Y_1,Y_2,W^{\ast })\in G(0,Q_s\left(I\right)),\hspace{1em}see \left(81\right)for{Q}_s\left(I\right),$$

(25)

$$(F^{Y_{11},Y_{12},Y_{13}},F^{Y_{21},Y_{22},Y_{23}}|F^{W_1^{\ast },W_2^{\ast }})\in \mathrm{CI},$$

(26)

$$F^{W_1^{\ast }}\subseteq (F^{Y_{11}}\vee F^{Y_{21}}),F^{W_2^{\ast }}\subseteq (F^{Y_{12}}\vee F^{Y_{22}}),$$

(27)

$$C(Y_1,Y_2)=I(Y_1,Y_2;W^{\ast }).$$

(28)

(c) The following operations are defined, using (a),

$$W^{\ast }=\left(\begin{array}{c}{W}_1^{\ast }\hfill \\ W_2^{\ast }\hfill \end{array}\right),$$

(29)

$$W_1^{\ast }=Y_{11}=Y_{21},$$

(30)

$$W_2^{\ast }=L_1{Y}_{12}+L_2{Y}_{22}+L_{3}V,\hspace{1em}\hspace{1em}see \left(103\right), \left(104\right) for the formulas of\hspace{1em}{L}_1,\hspace{1em}{L}_2,\hspace{1em}{L}_3;$$

(31)

$$Z_{12}=Y_{12}-\mathbf{E}\left[Y_{12}\right|F^{W_2^{\ast }}]=Y_{12}-Q_{Y_{12},W_2^{\ast }}{Q}_{W_2^{\ast }}^{-1}{W}_2^{\ast },\hspace{1em}$$

(32)

$$Z_{22}=Y_{22}-\mathbf{E}\left[Y_{22}\right|F^{W_2^{\ast }}]=Y_{22}-Q_{Y_{22},W_2^{\ast }}{Q}_{W_2^{\ast }}^{-1}{W}_2^{\ast },$$

(33)

$$Z_{13}=Y_{13},\hspace{1em}\hspace{1em}{Z}_{23}=Y_{23},\hspace{1em}\hspace{1em}(the components Z_{11} and Z_{21} do not exist),$$

(34)

$$Z_1=\left(\begin{array}{c}{Z}_{12}\hfill \\ Z_{13}\hfill \end{array}\right),\hspace{1em}{Z}_2=\left(\begin{array}{c}{Z}_{22}\hfill \\ Z_{23}\hfill \end{array}\right)$$

(35)

and these imply,

$$Y_{11}=W_1^{\ast }=Y_{21},$$

(36)

$$Y_{12}=Q_{Y_{12},W_2^{\ast }}{Q}_{W_2^{\ast }}^{-1}{W}_2^{\ast }+Z_{12},\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{Y}_{22}=Q_{Y_{22},W_2^{\ast }}{Q}_{W_2^{\ast }}^{-1}{W}_2^{\ast }+Z_{22},$$

(37)

$$Y_{13}=Z_{13},\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{Y}_{23}=Z_{23};$$

(38)

equivalently

$$\begin{array}{c}\left(\begin{array}{c}{Y}_{11}\hfill \\ Y_{12}\hfill \\ Y_{13}\hfill \end{array}\right)=\left(\begin{array}{cc}{I}_{n_1}\hfill & 0\hfill \\ 0\hfill & Q_{Y_{12},W_2^{\ast }}{Q}_{W_2^{\ast }}^{-1}\hfill \\ 0\hfill & 0\hfill \end{array}\right)\left(\begin{array}{c}{W}_1^{\ast }\hfill \\ W_2^{\ast }\hfill \end{array}\right)+\left(\begin{array}{cc}0\hfill & 0\hfill \\ I_{n_2}\hfill & 0\hfill \\ 0\hfill & I_{n-n_1-n_2}\hfill \end{array}\right)\left(\begin{array}{c}{Z}_{12}\hfill \\ Z_{13}\hfill \end{array}\right),\hfill \\ \left(\begin{array}{c}{Y}_{21}\hfill \\ Y_{22}\hfill \\ Y_{23}\hfill \end{array}\right)=\left(\begin{array}{cc}{I}_{n_1}\hfill & 0\hfill \\ 0\hfill & Q_{Y_{22},W_2^{\ast }}{Q}_{W_2^{\ast }}^{-1}\hfill \\ 0\hfill & 0\hfill \end{array}\right)\left(\begin{array}{c}{W}_1^{\ast }\hfill \\ W_2^{\ast }\hfill \end{array}\right)+\left(\begin{array}{cc}0\hfill & 0\hfill \\ I_{n_2}\hfill & 0\hfill \\ 0\hfill & I_{n-n_1-n_2}\hfill \end{array}\right)\left(\begin{array}{c}{Z}_{22}\hfill \\ Z_{23}\hfill \end{array}\right).\hfill \end{array}$$

The derivation of Theorem 4 is presented in Section 3.2, after several of the tools are presented, such as, weak stochastic realizations and minimal realizations.

Remark 1.

Relation of Theorem 4 to the literature.

(a) Corollary 1 in [15] gives an expression analogous to the case (20), which is expressed in terms of the correlation coefficients, ${\rho }_i\in (-1,1)$ and not the canonical correlation coefficients $d_i\in (0,1)$. Similarly, [16], under Lemma 1, reproduces Corollary 1 in [15], with the correlation coefficients ${\rho }_i$ replaced by their absolute values $|{\rho }_i|$.

(b) The derivation in [15,16] is based on the use of rate distortion functions of Gaussian random variables with square-error distortion functions, which presupposes the that auxiliary RV $W\to \mathbb{W}$ takes continuous values.

(c) Refs. [15,16] do not provide a realization of the triple $(Y_1,Y_2,W^{\ast })$, as given in Theorem 4 (which is based on applying the parametrization of Theorem 8).

On the other hand, the derivation of Theorem 4 is based on Theorem 9, which shows that, among all joint distributions ${\mathbf{P}}_{Y_1,Y_2,W}$ induced by a tuple of multivariate correlated Gaussian random variables $(Y_1,Y_2)$, and an arbitrary random variable $W:\Omega \to \mathbb{W}$, continuous or discrete-valued, Wyner’s common information $C(Y_1,Y_2)$, defined by (13), is minimized by a triple $(Y_1,Y_2,W)$ which induces a jointly Gaussian distribution ${\mathbf{P}}_{Y_1,Y_2,W}$, and $W:\Omega \to \mathbb{W}={\mathbb{R}}^n$ is finite-dimensional Gaussian random variable.

(d) The derivation of Theorem 4 contains many intermediate results which are applicable to the problems considered in [15,16], such as Relaxed Wyner’s Common Information in [17]. These are discussed in Section 4.3.

Theorem 5 gives a parametric characterization of the Gray and Wyner rate region ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$, with respect to the variance matrix of the triple of jointly Gaussian random variables $(Y_1,Y_2,W)$.

Theorem 5.

Consider a tuple of Gaussian random variables $Y_i:\Omega \to {\mathbb{R}}^{p_i}$, with $Q_{Y_i}>0$, for $i=1,2$, $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)})$ (not necessarily in canonical variable form), with induced Gaussian measure ${\mathbf{P}}_0=G(0,Q_{(Y_1,Y_2)})$ on the space $({\mathbb{R}}^{p_1}\times {\mathbb{R}}^{p_2},B\left({\mathbb{R}}^{p_1}\right)\otimes B\left({\mathbb{R}}^{p_2}\right))$, and square error distrortion functions $D_{Y_1}(y_1,{\widehat{y}}_1)=\left|\right|y_1-{\widehat{y}}_1{\left|\right|}_{{\mathbb{R}}^{p_1}}^2,\hspace{1em}{D}_{Y_2}(y_2,{\widehat{y}}_2)=\left|\right|y_2-{\widehat{y}}_2{\left|\right|}_{{\mathbb{R}}^{p_2}}^2$.

The following hold.

(a) There exists a Gaussian measure ${\mathbf{P}}_1=G(0,Q_{(Y_1,Y_2,W)})$ defined on the space $({\mathbb{R}}^{p_1}\times {\mathbb{R}}^{p_2}\times {\mathbb{R}}^n,B\left({\mathbb{R}}^{p_1}\right)\otimes B\left({\mathbb{R}}^{p_2}\right)\otimes B\left({\mathbb{R}}^n\right)),n\in {\mathbb{Z}}_{+}$ associated with the Gaussian random variables $(Y_1,Y_2))$, $W:\Omega \to {\mathbb{R}}^n,W\in G(0,Q_W)$ such that ${\mathbb{P}}_1{|}_{{\mathbb{R}}^{p_1}\times {\mathbb{R}}^{p_2}}=G(0,Q_{(Y_1,Y_2)})$. Moreover, a realization of the random variables $(Y_1,Y_2,W)$ with induced measure ${\mathbf{P}}_1=G(0,Q_{(Y_1,Y_2,W)})$ is

$$\left(\begin{array}{c}{Y}_1\hfill \\ Y_2\hfill \end{array}\right)=Q_{(Y_1,Y_2),W}{Q}_W^{†}W+\left(\begin{array}{c}{Z}_1\hfill \\ Z_2\hfill \end{array}\right)$$

(39)

$$(Z_1,Z_2)\in G(0,Q_{(Z_1,Z_2)}),\hspace{1em}\hspace{1em}(Z_1,Z_2)\hspace{1em}\hspace{1em}independent of\hspace{1em}\hspace{1em}W,$$

(40)

$$Q_{(Y_1,Y_2),W}=\mathbf{E}\left[\left(\begin{array}{c}{Y}_1\hfill \\ Y_2\hfill \end{array}\right)W^T\right]=\left(\begin{array}{c}{Q}_{Y_1,W}\hfill \\ Q_{Y_2,W}\hfill \end{array}\right)$$

(41)

where $Q_W^{†}$ is the pseudoinverse of $Q_W$.

(b) For Gaussian auxiliary random variables given in part (a), the Gray–Wyner rate region ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$ is determined from

$$\begin{array}{cc}\hfill T^G({\alpha }_1,{\alpha }_2)=& \underset{(Y_1,Y_2,W)\in G(0,Q_{(Y_1,Y_2,W)})of\left(68\right)}{inf}\{I(Y_1,Y_2;W)\hfill \end{array}$$

$$\begin{array}{cc}\hfill & +{\alpha }_1{R}_{Y_1|W}\left({\Delta }_1\right)+{\alpha }_2{R}_{Y_2|W}\left({\Delta }_2\right)\}\hfill \\ \hfill =& \underset{(Y_1,Y_2,W)\in G(0,Q_{(Y_1,Y_2,W)}),of\left(\right),\left(\right)}{inf}\{I(Y_1,Y_2;W)\hfill \end{array}$$

(42)

$$\begin{array}{cc}\hfill & +{\alpha }_1{R}_{Y_1|W}\left({\Delta }_1\right)+{\alpha }_2{R}_{Y_2|W}\left({\Delta }_2\right)\}\hfill \\ \hfill =& \underset{Q_{Y_1,Y_2|W},Q_{Y_i|W},i=1,2}{inf}\{\frac{1}{2}ln{\left(\frac{det\left(Q_{(Y_1,Y_2)}\right)}{det\left(Q_{Y_1|Y_2,W}\right)det\left(Q_{Y_2|W}\right)}\right)}^{+}\hfill \end{array}$$

(43)

$$+{\alpha }_1{R}_{Y_1|W}\left({\Delta }_1\right)+{\alpha }_2{R}_{Y_2|W}\left({\Delta }_2\right)\}$$

(44)

$$subject to,\hspace{1em}\hspace{1em}{Q}_{(Y_1,Y_2)}\ge Q_{(Y_1,Y_2)|W},\hspace{1em}\hspace{1em}{Q}_{Y_1|Y_2,W}=Q_{Y_1|W}-Q_{Y_1,Y_2|W}{Q}_{Y_2|W}^{†}{Q}_{Y_1,Y_2|W}^T$$

(45)

where $0\le {\alpha }_i\le 1,i=1,2,{\alpha }_1+{\alpha }_2\ge 1$, $I(Y_1,Y_2;W)=H(Y_1,Y_2)-H\left(Y_1\right|Y_2,W)-H\left(Y_2\right|W)$, and $R_{Y_i|W}\left({\Delta }_i\right),i=1,2$ are given in Theorem 13.(b), and ${\left(·\right)}^{+}=max\{1,·\}$.

The derivation of Theorem 5 is presented in Section 4.4, after the structural properties of RDFs, $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2),R_{Y_i|W}\left({\Delta }_i\right),R_{Y_i}\left({\Delta }_i\right),i=1,2$, of Theorem 12, Theorem 13, Theorem 14 are presented. From Theorem 5, follow simplified characterizations of subsets of the rate region ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$, such as rates that lie on Pangloss Plane, and rates that correspond to W that make $Y_1$ and $Y_2$ conditional independent, i.e., W is such that ${\mathbf{P}}_{Y_1,Y_2|W}={\mathbf{P}}_{Y_1|W}{\mathbf{P}}_{Y_2|W}$.

Utilizing the structural properties of RDFs, $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2),R_{Y_i|W}\left({\Delta }_i\right),R_{Y_i}\left({\Delta }_i\right),i=1,2$, of Theorem 12, Theorem 13, Theorem 14, and Theorem 4, the next theorem is obtained, which gives the formula of Wyner’s lossy common information $C_{GW}(Y_1,Y_2;{\Delta }_1,{\Delta }_2)=C_W(Y_1,Y_2)$.

Theorem 6.

Consider a tuple $(Y_1,Y_2)$ of Gaussian random variables in the canonical variable form of Definition 1. Restrict attention to the correlated parts of these random variables, as defined in Theorem 8, by (77)–(79), and subset of the distortion region is defined by

$${\mathcal{D}}_W=\left\{({\Delta }_1,{\Delta }_2)\in [0,\infty ]\times [0,\infty ]|\hspace{1em}\hspace{1em}0\le {\Delta }_1\le n(1-d_1),\hspace{1em}\hspace{1em}0\le {\Delta }_2\le n(1-d_1)\right\},$$

(46)

$$\forall j\in {\mathbb{Z}}_n,\hspace{1em}\hspace{1em}{d}_j\in (0,1).$$

(47)

Then, Wyner’s lossy common information (calculation of expression in Theorem 3) is given by

$$\begin{array}{cc}\hfill C_{GW}(Y_1,Y_2;{\Delta }_1,{\Delta }_2)=& C_W(Y_1,Y_2)\hfill \end{array}$$

(48)

$$\begin{array}{cc}\hfill =& C(Y_1,Y_2)=\frac{1}{2}\sum _{j=1}^{n}ln\left(\frac{1+d_j}{1-d_j}\right),\hspace{1em}\hspace{1em}({\Delta }_1,{\Delta }_2)\in {\mathcal{D}}_W\hfill \end{array}$$

(49)

The derivation of Theorem 6 is presented in Section 4.2 and makes use of a degenerate version of the realization of the triple $(Y_1,Y_2,W^{\ast })$ given in Theorem 4, and the RDFs $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2),R_{Y_i|W}\left({\Delta }_i\right),i=1,2$.

Remark 2.

By Theorem 5, a subset of the Gray–Wyner rate region is obtained by replacing $(Y_1,Y_2,W)\in G(0,Q_{(Y_1,Y_2,W)})$ of (39) and (40) by W that makes $Y_1$ and $Y_2$ conditionally independent, i.e., $(Z_1,Z_2)\in G(0,Q_{(Z_1,Z_2)})$ and $(Z_1,Z_2,W)$ mutually independent (e.g., $Q_{(Z_1,Z_2)}$ is block-diagonal).

1.3. Structure of the Paper

Section 2 introduces the mathematical tools of the geometric approach to Gaussian random variables, the weak stochastic realization of conditional independence (Section 2.4).

Section 3 contains the problem statement, the solution procedure and the weak realization of a tuple of multivariable random variables $(Y_1,Y_2)$ such that another multivariate Gaussian random variable W makes $Y_1$ and $Y_2$ conditionally independent (Section 2.5). $C_W(Y_1,Y_2)=C(Y_1,Y_2)$.

Section 4 is concerned with the characterization of the Gray–Wyner rate region ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$, the characterization of rates that lie on the Pangloss Plane, and Wyner’s lossy common information. This section includes calculations of the rate distortion functions $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$, $R_{Y_i|W}\left({\Delta }_i\right),R_{Y_i}\left({\Delta }_i\right),i=1,2$, the weak stochastic realizations of the random variables $(Y_1,Y_2,{\widehat{Y}}_1,{\widehat{Y}}_2,W)$ which achieve these rate distortion functions, for jointly multivariate Gaussian random variables with square-error distortion functions.

Section 5 includes remarks on possible extensions.

Appendix A.3 makes use of a matrix equality and a determinant inequality first obtained by Hua LooKeng in 1952, which are used to carry out the optimization problem of Wyner’s lossy common information $C_W(Y_1,Y_2)=C(Y_1,Y_2)$.

2. Probabilistic Properties of Tuples of Random Variables

The reader finds in this section the basic properties associated with:

(1) the transformation of a tuple of Gaussian multivariate random variables $(Y_1,Y_2)$ in their canonical variable form, and

(2) The parameterization of all jointly Gaussian distributions ${\mathbf{P}}_{Y_1,Y_2,W}(y_1,y_2,w)$ by a zero mean Gaussian random variables $W:\Omega \to {\mathbb{R}}^k\equiv \mathbb{W}$ such that (a) W makes the multivariate random variables $(Y_1,Y_2)$ conditional independent, and (b) The marginal distribution ${\mathbf{P}}_{Y_1,Y_2,W}(y_1,y_1,\infty )={\mathbf{P}}_{Y_1,Y_2}(y_1,y_2)$ coincides with the joint distribution of the multivariate random variables $(Y_1,Y_2)$.

2.1. Notation of Elements of Probability Theory

The notation used in this paper is briefly specified. Denote by ${\mathbb{Z}}_{+}=\{1,2,\dots ,\}$ the set of the integers and by $\mathbb{N}=\{0,1,2,\dots ,\}$ the set of the natural integers. For $n\in {\mathbb{Z}}_{+}$ denote the following finite subsets of the above defined sets by ${\mathbb{Z}}_n=\{1,2,\dots ,n\}$ and ${\mathbb{N}}_n=\{0,1,2,\dots ,n\}$.

Denote the real numbers by $\mathbb{R}$ and the set of positive and of strictly positive real numbers, respectively, by ${\mathbb{R}}_{+}=[0,\infty )$ and ${\mathbb{R}}_{++}=(0,\infty )\subset \mathbb{R}$. The vector space of n-tuples of real numbers is denoted by ${\mathbb{R}}^n$. Denote the Borel $\sigma$-algebra on this vector space by $B\left({\mathbb{R}}^n\right)$, hence, $({\mathbb{R}}^n,B\left({\mathbb{R}}^n\right))$ is a measurable space.

The expression ${\mathbb{R}}^{n\times m}$ denotes the set of n by m matrices with elements in the real numbers, for $n,\hspace{1em}m\in {\mathbb{Z}}_{+}$. For the symmetric matrix $Q\in {\mathbb{R}}^{n\times n}$, the inequality $Q\ge 0$ denotes that for all vectors $u\in {\mathbb{R}}^n$ the inequality $u^{T}Qu\ge 0$ holds. Similar, $Q>0$ denotes that for all $u\in {\mathbb{R}}^n\setminus \left\{0\right\}$, $u^{T}Qu>0$. The notation $Q_1\le Q_2$ denotes that $Q_2-Q_1\ge 0$.

Consider a probability space denoted by $(\Omega ,F,\mathbb{P})$ consisting of a set $\Omega$, a $\sigma$-algebra F of subsets of $\Omega$, and a probability measure $\mathbb{P}:F\to [0,1]$.

A real-valued random variable is a function $X:\Omega \to \mathbb{R}$ such that the following set belongs to the indicated $\sigma$-algebra, $\{\omega \in \Omega |X\left(\omega \right)\in (-\infty ,u]\}\in F$ for all $u\in \mathbb{R}$. A random variable taking values in an arbitrary measurable space $(\mathbb{X},B(\mathbb{X}\left)\right)$ is defined correspondingly by $X:\Omega \to \mathbb{X}$ and $X^{-1}\left(A\right)=\{\omega \in \Omega |X\left(\omega \right)\in A\}\in B\left(\mathbb{X}\right)$, for all $A\in B\left(\mathbb{X}\right)$. The measure (or distribution if $\mathbb{X}$ is an Euclidean space) induced by the random variable on $(\mathbb{X},B(\mathbb{X}\left)\right)$ is denoted by ${\mathbf{P}}_X$ or $\mathbf{P}\left(dx\right)$. The $\sigma$-algebra generated by a random variable $X:\Omega \to \mathbb{X}$ is defined as the smallest $\sigma$-algebra containing the subsets $X^{-1}\left(A\right)\in F$ for all $A\in B\left(\mathbb{X}\right)$. It is denoted by $F^X$. The real-valued random variable X is called G-measurable for a $\sigma$-algebra $G\subseteq F$ if the subset $\{\omega \in \Omega |X\left(\omega \right)\in (-\infty ,u]\}\in G$ for all $u\in \mathbb{R}$. Denote the set of positive random variables which are measurable on a sub-$\sigma$-algebra $G\subseteq F$ by,

$$L_{+}\left(G\right)=\{X:\Omega \to {\mathbb{R}}_{+}=[0,\infty )|X\hspace{1em}isG-measurable\}.$$

The tuple of sub-$\sigma$-algebras $F_1,\hspace{1em}{F}_2\subseteq F$ is called independent if $\mathbb{E}\left[X_1{X}_2\right]=\mathbb{E}\left[X_1\right]\mathbb{E}\left[X_2\right]$ for all $X_1\in L_{+}\left(F_1\right)$ and all $X_2\in L_{+}\left(F_2\right)$. The definition can be extended to any finite set of independent sub-$\sigma$-algebras.

2.2. Geometric Approach of Gaussian Random Variables and Canonical Variable Form

The purpose of this section is to introduce the geometric approach of a tuple of finite-dimensional Gaussian random variables using the canonical variable form of the tuple introduced by H. Hotelling, [18]. The use of the geometric approach of two Gaussian random variables with respect to the computation of mutual information is elaborated by Gelfand and Yaglom in [19], making reference to an insight due to Kolmogorov. However, the canonical variable form is not given in [19].

An ${\mathbb{R}}^n$-valued Gaussian random variable with as parameters the mean value $m_X\in {\mathbb{R}}^n$ and the variance $Q_X\in {\mathbb{R}}^{n\times n}$, $Q_X=Q_X^T\ge 0$, is a function $X:\Omega \to {\mathbb{R}}^n$ which is a random variable and such that the measure of this random variable equals a Gaussian measure described by its characteristic function,

$$\mathbf{E}[exp\left(i{u}^{T}X\right)]=exp(i{u}^T{m}_X-\frac{1}{2}{u}^T{Q}_{X}u),\hspace{1em}\forall \hspace{1em}u\in {\mathbb{R}}^n.$$

Note that this definition includes the case in which the random variable is almost surely equal to a constant in which case $Q_X=0$. A Gaussian random variable with these parameters is denoted by $X\in G(m_X,Q_X)$.

The effective dimension of the random variable is denoted by $dim\left(X\right)=\mathrm{rank}\left(Q_X\right)$.

Any tuple of random variables $X_1,\dots ,X_k$ is called jointly Gaussian if the vector ${(X_1^T,X_2^T,\dots ,X_k^T)}^T$ is a Gaussian random variable. A tuple of Gaussian random variables $(Y_1,Y_2)$ will be denoted this way to save space, rather than by

$$\begin{array}{c}\hfill \left(\begin{array}{c}{Y}_1\hfill \\ Y_2\hfill \end{array}\right).\end{array}$$

Then, the variance matrix of this tuple is denoted by

$$(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)}),\hspace{1em}\hspace{1em}{Q}_{(Y_1,Y_2)}=\left(\begin{array}{cc}{Q}_{Y_1}\hfill & Q_{Y_1,Y_2}\hfill \\ Q_{Y_1,Y_2}^T\hfill & Q_{Y_2}\hfill \end{array}\right)\in {\mathbb{R}}^{(p_1+p_2)\times (p_1+p_2)}.$$

The reader should distinguish the variance matrices $Q_{(Y_1,Y_2)}$ and $Q_{Y_1,Y_2}\in {\mathbb{R}}^{p_1\times p_2}$. Any such tuple of Gaussian random variables is independent if and only if $Q_{Y_1,Y_2}=0$.

Definition 1.

The canonical variable form.

Consider a tuple of Gaussian random variables $Y_i:\Omega \to {\mathbb{R}}^{p_i}$, with $Q_{Y_i}>0$, for $i=1,2$, $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)})$, $Q_{(Y_1,Y_2)}\ge 0$. Define the canonical variable form of these random variables if a basis has been chosen and a transformation of the random variables to this basis has been carried out such that with respect to the new basis, one has the representation

$$(Y_1,Y_2)\in G(0,Q_{\mathrm{cvf}}),\hspace{1em}\hspace{1em}where,$$

$$\begin{array}{c}{Q}_{\mathrm{cvf}}=\left(\begin{array}{cccccc}{I}_{p_{11}}\hfill & 0\hfill & 0& I_{p_{21}}\hfill & 0\hfill & 0\hfill \\ 0\hfill & I_{p_{12}}\hfill & 0\hfill & 0\hfill & D\hfill & 0\hfill \\ 0\hfill & 0\hfill & I_{p_{13}}\hfill & 0\hfill & 0\hfill & 0\hfill \\ I_{p_{21}}\hfill & 0\hfill & 0\hfill & I_{p_{21}}\hfill & 0\hfill & 0\hfill \\ 0\hfill & D\hfill & 0\hfill & 0\hfill & I_{p_{22}}\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0\hfill & 0\hfill & 0\hfill & I_{p_{23}}\hfill \end{array}\right)\in {\mathbb{R}}^{p\times p},\hfill \\ p,\hspace{1em}{p}_1,\hspace{1em}{p}_2,\hspace{1em}{p}_{11},\hspace{1em}{p}_{12},\hspace{1em}{p}_{13},\hspace{1em}{p}_{21},\hspace{1em}{p}_{22},\hspace{1em}{p}_{23}\in \mathbb{N},\hfill \\ p=p_1+p_2,\hspace{1em}{p}_1=p_{11}+p_{12}+p_{13},\hspace{1em}{p}_2=p_{21}+p_{22}+p_{23},\hspace{1em}{p}_{11}=p_{21},\hspace{1em}{p}_{12}=p_{22},\hfill \end{array}$$

(50)

$$D=\mathrm{Diag}(d_1,\dots ,d_{p_{12}}),\hspace{1em}\hspace{1em}1>d_1\ge d_2\ge \dots \ge d_{p_{12}}>0,$$

(51)

$$Y=\left(\begin{array}{c}{Y}_1\hfill \\ Y_2\hfill \end{array}\right)=\left(\begin{array}{c}{Y}_{11}\hfill \\ Y_{12}\hfill \\ Y_{13}\hfill \\ Y_{21}\hfill \\ Y_{22}\hfill \\ Y_{23}\hfill \end{array}\right),\hspace{1em}\hspace{1em}{Y}_{ij}:\Omega \to {\mathbb{R}}^{p_{ij}},\hspace{1em}i=1,2,\hspace{1em}j=1,2,3.$$

(52)

One then says that $(Y_{11},\dots ,Y_{1k_1})$, $(Y_{21},\dots ,Y_{2k_2})$ are the canonical variables and $(d_1,\dots ,d_{k_{12}})$ the canonical correlation coefficients.

If $Q_{(Y_1,Y_2)}>0$ then necessarily $p_{11}=p_{21}=0$.

Appendix A.1 gives Algorithm A1 to transform the variance matrix $Q_{(Y_1,Y_2)}\ge 0$ by two nonsingular transformations $S_i\in {\mathbb{R}}^{p_i\times p_i},i=1,2$, to its canonical variable form $Q_{\mathrm{cvf}}$ of Definition 1, such that

$$S_1{Y}_1=(V_1,Y_1^{\prime })=((Y_{11},Y_{12}),Y_{13}),\hspace{1em}{S}_2{Y}_2=(V_2,Y_2^{\prime })=((Y_{21},Y_{22}),Y_{23}),$$

(53)

$$Y_{11}=Y_{21}-a.s.,\hspace{1em}\mathbf{E}\left[Y_{12}{Y}_{22}^T\right]=D.$$

(54)

Proposition 1.

Properties of components of the canonical variable form.

Consider a tuple $(Y_1,Y_2)\in G(0,Q_{\mathrm{cvf}})$ of Gaussian random variables in the canonical variable form.

(a) The three components $Y_{11},Y_{12},Y_{13}$ of $Y_1$ are independent random variables. Similarly, the three components $Y_{21},Y_{22},Y_{23}$ of $Y_2$ are independent random variables.

(b) The equality $Y_{11}=Y_{21}$ of these random variables holds almost surely.

(c) The tuple of random variables $(Y_{12},Y_{22})$ is correlated as shown by the formula

$$\mathbf{E}\left[Y_{12}{Y}_{22}^T\right]=D=\mathrm{Diag}(d_1,\dots ,d_{p_{12}}).$$

(55)

Note that the different components of $Y_{12}$ and of $Y_{22}$ are independent random variables; thus, $Y_{12,i}$ and $Y_{12,j}$ are independent, and $Y_{22,i}$ and $Y_{22,j}$ are independent, and $Y_{12,i}$ and $Y_{22,j}$ are independent, for all $i\ne j$; and that $Y_{12,j}$ and $Y_{22,j}$ for $j=1,\dots ,p_{12}=p_{22}$ are correlated.

(d) The random variable $Y_{13}$ is independent of $Y_2$. Similarly, the random variable $Y_{23}$ is independent of $Y_1$

Proof. 

The results are immediately obvious from the fact that the random variables are all jointly Gaussian and from the variance formula (51) of the canonical variable form. □

Next, the interpretation of the various components of the canonical variable form is defined, as in [20].

Definition 2.

Interpretation of components of the canonical variable form.

Consider a tuple of jointly Gaussian random variables $(Y_1,Y_2)\in G(0,Q_{\mathrm{cvf}})$ in the canonical variable form of Definition 1. Call the various components as defined in the next table.

$Y_{11}=Y_{21}-\mathrm{a}.\mathrm{s}.$	identical information of $Y_1$ and $Y_2$
$Y_{12}$	correlated information of $Y_1$ with respect to $Y_2$
$Y_{13}$	private information of $Y_1$ with respect to $Y_2$
$Y_{21}=Y_{11}-\mathrm{a}.\mathrm{s}.$	identical information of $Y_1$ and $Y_2$
$Y_{22}$	correlated information of $Y_2$ with respect to $Y_1$
$Y_{23}$	private information of $Y_2$ with respect to $Y_1$

For $Y_{11}=Y_{21}-a.s.$ the term identical information is used.

Theorem 7 is a formula of mutual information $I(Y_1;Y_2)$ for a general tuple of finite-dimensional Gaussian random variables $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)})$. This formula is the subject of much discussion in Gelfand and Yaglom [19] (see Equantion (2.8’) and Chapter II).

Theorem 7.

Consider a tuple of finite-dimensional Gaussian random variables $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)})$, $Q_{Y_i}>0,i=1,2$.

Compute the canonical variable form of the tuple of Gaussian random variables according to Algorithm A1. This yields the indices $p_{11}=p_{21}$, $p_{12}=p_{22}$, $p_{13}$, $p_{23}$, and $n=p_{11}+p_{12}=p_{21}+p_{22}$ and the diagonal matrix D with canonical correlation coefficients or singular values $d_i\in (0,1)$ for $i=1,\dots ,n$.

Then, mutual information $I(Y_1;Y_2)$ is computed according to the formula

$$\begin{array}{cc}\hfill I(Y_1;Y_2)=& \left\{\begin{array}{ccc}0,\hfill & if\hfill & 0=p_{11}=p_{12}=p_{21}=p_{22},\hspace{1em}{p}_{13}>0,\hspace{1em}{p}_{23}>0,\hfill \\ -\frac{1}{2}\sum _{i=1}^{n}ln\left(1-d_i^2\right),\hfill & if\hfill & 0=p_{11}=p_{21},\hspace{1em}{p}_{12}=p_{22}>0,\hspace{1em}{p}_{13}\ge 0,\hspace{1em}{p}_{23}\ge 0,\hfill \\ \infty ,\hfill & if\hfill & p_{11}=p_{21}>0,\hspace{1em}{p}_{12}=p_{22}\ge 0,\hspace{1em}{p}_{13}\ge 0,\hspace{1em}{p}_{23}\ge 0.\hfill \end{array}\right.\hfill \end{array}$$

(56)

where $d_i$ are the canonical correlation coefficients, i.e.,

$$d_i=d_i\left(Y_{12,i}{Y}_{22,i}\right)=\frac{\mathbf{E}\left\{Y_{12,i}{Y}_{22,i}\right\}}{\sqrt{\mathbf{E}{\left\{Y_{12,i}\right\}}^2\mathbf{E}{\left\{Y_{22,i}\right\}}^2}}=\mathbf{E}\left\{Y_{12,i}{Y}_{22,i}\right\},\hspace{1em}\hspace{1em}i=1,\dots ,n.$$

(57)

Proof. 

The derivation given in Appendix A.3 of [21] (since it is not given in [19]). □

By the last entry of (57), it is appropriate to consider to only $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)})$ such that $p_{11}=p_{21}=0$, i.e., by removing the identical components prior to the analysis of mutual information problems.

Remark 3.

The material discussed in Section 1.2 makes use of the concepts of this section. The main point to be made is that, in lossy source coding problems, the source distribution is fixed, while the optimal reproduction distribution needs to be found and realized. Then, a pre-encoder can be used by invoking Algorithm A1.

2.3. Conditional Independence of a Triple of Gaussian Random Variables

The concept of conditional independence is basic to the entire paper. The definition is provided below. The characterization of a Gaussian measure on a triple of Gaussian random variables having the conditional independence property is stated.

Definition 3.

Conditional independence.

Consider a probability space $(\Omega ,F,\mathbb{P})$ and three sub-σ-algebras $F_1,F_2,G\subseteq F$. Call the sub-σ-algebras $F_1$ and $F_2$ conditionally independent given, or conditioned on, the sub-σ-algebra G if the following factorization property holds:

$$\begin{array}{c}\hfill \mathbf{E}\left[Y_1{Y}_2\right|G]=\mathbf{E}\left[Y_1\right|G]E\left[Y_2\right|G],\hspace{1em}\hspace{1em}\forall \hspace{1em}\hspace{1em}{Y}_1\in L_{+}\left(F_1\right),\hspace{1em}\hspace{1em}{Y}_2\in L_{+}\left(F_2\right).\end{array}$$

(58)

Denote this property by $(F_1,F_2|G)\in CI$.

For Gaussian random variables, the definition of minimality of a Gaussian random variable X that makes two Gaussian random variables $(Y_1,Y_2)$ conditionally independent is needed. The definition is introduced below.

Definition 4.

Minimality of conditional independence of Gaussian random variables.

Consider three random variables, $Y_i:\Omega \to {\mathbb{R}}^{p_i}$ for $i=1,\hspace{1em}2$ and $X:\Omega \to {\mathbb{R}}^n$.

Call the random variables $Y_1$ and $Y_2$ Gaussian conditionally independent conditioned on or given $F^X$ if:

(1) $(F^{Y_1},F^{Y_2}|F^X)\in \mathrm{CI}$;

(2) $(Y_1,Y_2,X)$ are jointly Gaussian random variables.

The notation $(Y_1,Y_2|X)\in \mathrm{CIG}$ is used to denote this property.

Call the random variables $(Y_1,Y_2|X)$ minimally Gaussian conditionally independent if

(1) They are Gaussian conditionally independent;

(2) There does not exist another tuple $(Y_1,Y_2|X_1)$ with $X_1:\Omega \to {\mathbb{R}}^{n_1}$ such that $(Y_1,Y_2|X_1)\in \mathrm{CIG}$ and $n_1<n$.

This property is denoted by $(Y_1,Y_2|X_1)\in {\mathrm{CIG}}_{min}$.

There exists a simple equivalent condition for the conditional independence of tuple of Gaussian random variables by a third Gaussian random variable. This condition is expressed in terms of parameterizing the variance matrix of the tuple as presented in the next proposition.

Proposition 2.

[22] (Proposition 3.4) Equivalent condition for the conditional independence of the tuple of Gaussian random variables.

Consider a triple of jointly Gaussian random variables denoted as $(Y_1,Y_2,X)\in G(0,Q)$ with $Q_X>0$. This triple is Gaussian conditionally independent if and only if

$$Q_{Y_1,Y_2}=Q_{Y_1,X}{Q}_X^{-1}{Q}_{X,Y_2}.$$

(59)

It is minimally Gaussian conditionally independent if and only if, in addition, $n=dim\left(X\right)=\mathrm{rank}\left(Q_{Y_1,Y_2}\right)$.

It will become apparent in Section 4.4 that the Gray and Wyner lossy rate region ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$ is parameterized by a triple of jointly Gaussian random variables $(Y_1,Y_2,W)$, but not necessarily such that W makes $Y_1$ and $Y_2$ conditionally independent. However, subsets of ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$, are characterized by a triple $(Y_1,Y_2,W)$, such that W makes $Y_1$ and $Y_2$ conditionally independent.

2.4. Weak Realization of a Gaussian Probability Measure on a Tuple of Random Variables

This section is motivated by Theorem 9, which states that, among all joint distributions ${\mathbf{P}}_{Y_1,Y_2,W}$ induced by a tuple of multivariate correlated Gaussian random variables $(Y_1,Y_2)$, and an arbitrary random variable $W:\Omega \to \mathbb{W}$, continuous or discrete-valued, Wyner’s common information $C(Y_1,Y_2)$, defined by (13), is minimized by a triple $(Y_1,Y_2,W)$ which induces a jointly Gaussian distribution ${\mathbf{P}}_{Y_1,Y_2,W}$, and $W:\Omega \to \mathbb{W}={\mathbb{R}}^n$ is finite-dimensional Gaussian random variable.

To develop the above results, use is made of the solution of the problem of the weak Gaussian stochastic realization of a tuple of Gaussian random variables. Specifically, to determine a Gaussian probability measure on a triple of Gaussian random variables such that:

(1)

The measure restricted to the first two Gaussian random variables is equal to the considered probability measure;

(2)

The third Gaussian random variable makes the other two random variables conditionally independent. This problem does not have a unique solution, there is a set of Gaussian probability measures which meets those conditions. Needed is the parameterization of this set of solutions.

Below, the problem is stated in more detail. Its solution is provided in the next section.

Problem 1.

Weak stochastic realization of a tuple of conditionally independent Gaussian random variables.

Weak stochastic realization problem of a Gaussian random variable. Consider a Gaussian measure ${\mathbf{P}}_0=G(0,Q_0)$ on the space $({\mathbb{R}}^{p_1+p_2},B\left({\mathbb{R}}^{p_1+p_2}\right))$. Determine the integer $n\in \mathbb{N}$ and construct all Gaussian measures on the space $({\mathbb{R}}^{p_1+p_2+n},\left(B\left({\mathbb{R}}^{p_1+p_2+n}\right)\right)$ such that, if ${\mathbf{P}}_1=G(0,Q_1)$ is such a measure with $(Y_1,Y_2,X)\in G(0,Q_1)$, then:

(1) $G(0,Q_1){|}_{{\mathbb{R}}^{p_1+p_2}}=G(0,Q_0)$;

(2) $(Y_1,Y_2|X)\in {\mathrm{CIG}}_{min}$.

Here, the indicated random variables $(Y_1,Y_2,X)$ are constructed having the measure $G(0,Q_1)$ with the dimensions $p_1,p_2,n\in {\mathbb{Z}}_{+}$, respectively.

The next definition and proposition are about the weak Gaussian stochastic realization of a tuple of jointly Gaussian multivariate random variables and its weak stochastic realization.

Definition 5.

Minimality of weak stochastic realization of a tuple of conditionally independent Gaussian random variables.

Consider a Gaussian measure ${\mathbf{P}}_0=G_0(0,Q_{(y_1,Y_2)})$ with zero mean values for a tuple $(Y_1,Y_2)$ of random variables on the product space $({\mathbb{R}}^{p_1}\times {\mathbb{R}}^{p_2},B\left({\mathbb{R}}^{p_1}\right)\otimes B\left({\mathbb{R}}^{p_2}\right)$ for $p_1,\hspace{1em}{p}_2\in {\mathbb{Z}}_{+}$ with

$$\begin{array}{ccc}\hfill Q_{(Y_1,Y_2)}& =& \left(\begin{array}{cc}{Q}_{Y_1}\hfill & Q_{Y_1,Y_2}\hfill \\ Q_{Y_1,Y_2}^T\hfill & Q_{Y_2}\hfill \end{array}\right),\hspace{1em}\hspace{1em}{Q}_{Y_1}>0,\hspace{1em}{Q}_{Y_2}>0.\hfill \end{array}$$

(a) A weak Gaussian stochastic realization of the Gaussian measure $G_0(0,Q_{(y_1,Y_2)})$ is defined to be a Gaussian measure ${\mathbf{P}}_1=G_1$ if there exists an integer $n\in {\mathbb{Z}}_{+}$ such that the Gaussian measure $G_1$ is defined on the space $({\mathbb{R}}^{p_1}\times {\mathbb{R}}^{p_1}\times {\mathbb{R}}^n,B\left({\mathbb{R}}^{p_1}\right)\otimes B\left({\mathbb{R}}^{p_2}\right)\otimes B\left({\mathbb{R}}^n\right))$ associated with random variables in the three spaces denoted, respectively, by $Y_1$, $Y_2$, and X, and such that:

(1) $G_1{|}_{{\mathbb{R}}^{p_1}\times {\mathbb{R}}^{p_2}}=G_0(0,Q_{(Y_1,Y_2)})$;

(2) $Q_X>0$;

(3) Conditional independence holds: ${\mathbf{P}}_{Y_1,Y_2|X}={\mathbf{P}}_{Y_1|X}{\mathbf{P}}_{Y_2|X},$ where these are Gaussian measures, with means which are linear functions of the random variable X and deterministic variance matrices.

(b) The weak Gaussian stochastic realization is called minimal if the dimension n of the random variable X is the smallest possible over all weak Gaussian stochastic realizations as defined in (a).

(c) A Gaussian random variable representation of a weak Gaussian stochastic realization $G_1$ is defined as a triple of random variables satisfying the following relations

$$\begin{array}{cc}\hfill & (Y_1,Y_2,X,V_1,V_2),\hspace{1em}{p}_{V_1},\hspace{1em}{p}_{V_2}\in {\mathbb{Z}}_{+},\hspace{1em}{p}_{V_1}\ge p_1,\hspace{1em}{p}_{V_2}\ge p_2,\hfill \\ \hfill & Y_1:\Omega \to {\mathbb{R}}^{p_1},\hspace{1em}{Y}_2:\Omega \to {\mathbb{R}}^{p_2},\hspace{1em}{V}_1:\Omega \to {\mathbb{R}}^{p_{v_1}},\hspace{1em}{V}_2:\Omega \to {\mathbb{R}}^{p_{v_2}},\hspace{1em}X:\Omega \to {\mathbb{R}}^n,\hfill \\ \hfill & (V_1,V_2,X)\in G,\hspace{1em}and these are zero mean independent random variables\hfill \\ \hfill & Q_{V_1}>0,\hspace{1em}{Q}_{V_2}>0,\hspace{1em}{Q}_X>0;\hfill \end{array}$$

(60)

$$C_1\in {\mathbb{R}}^{p_1\times n},\hspace{1em}{C}_2\in {\mathbb{R}}^{p_2\times n},\hspace{1em}{N}_1\in {\mathbb{R}}^{p_1\times p_{V_1}},\hspace{1em}{N}_2\in {\mathbb{R}}^{p_2\times p_{V_2}},$$

(61)

$$Y_1=C_1\hspace{1em}X+N_1\hspace{1em}{V}_1,\hspace{1em}$$

(62)

$$Y_2=C_2\hspace{1em}X+N_2\hspace{1em}{V}_2,\hspace{1em}$$

(63)

$$Q_{Y_1}=C_1{Q}_X{C}_1^T+N_1{Q}_{V_1}{N}_1^T,$$

(64)

$$Q_{Y_2}=C_2{Q}_X{C}_2^T+N_2{Q}_{V_2}{N}_2^T,$$

(65)

$$\begin{array}{cc}\hfill & Q_{Y_1,Y_2}=C_1{Q}_X{C}_2^T,\hfill \\ \hfill & G_0(0,Q_{(Y_1,Y_2)})=G_1{|}_{{\mathbb{R}}^{p_1}\times {\mathbb{R}}^{p_2}}.\hfill \end{array}$$

(66)

From the assumptions, it then follows that $(Y_1,Y_2)$ are Gaussian random variables, hence the last equality makes sense.

(d) A minimal Gaussian random variable representation of a weak Gaussian stochastic realization is defined as a triple of random variables as in (c) except that, in addition, it is required that,

$$\mathrm{rank}\left(C_1\right)=n=\mathrm{rank}\left(C_2\right).$$

(67)

The case $Q_X\ge 0$ in (a).(2) is similar.

The next proposition shows the equivalence of weak Gaussian stochastic realizations of Definition 5. (a), (b) to Definition 5. (c), (d), respectively.

Proposition 3.

Consider the setting of Definition 5 with $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)})$ with the representation of (62) and (63).

(a) A weak Gaussian stochastic realization in terms of a measure ${\mathbf{P}}_1=G_1$ as defined in Definition 5. (a) is equivalent to a Gaussian random variable representation of Definition 5. (c).

(b) The minimal weak Gaussian stochastic realization of Definition 5. (b) is equivalent to a minimal weak Gaussian random variable representation of Definition 5. (d).

Proof. 

The derivation given in Appendix A.5 of [21]. □

Consider Figure 2. The two signals $Y_1,Y_2$ are to be reproduced at the two decoders by ${\widehat{Y}}_1,{\widehat{Y}}_2$ subject to the square-error distortion functions. According to Gray and Wyner, the characterization of the lossy rate region is described by a single coding scheme that uses the auxiliary random variable W, which is common to both $Y_1,Y_2$. A subset of the rate triples on the Gray and Wyner rate region, which is achieved by a triple that satisfies $(F^{Y_1},F^{Y_2}|F^W)\in \mathrm{CIG}$. Below, this conditional independence is further detailed in terms of the mathematical framework of weak stochastic realization such that $(F^{Y_1},F^{Y_2}|F^W)\in \mathrm{CIG}$.

Definition 6.

The model for a triple of Gaussian random variables.

Consider a tuple of Gaussian random variables specified by $Y=(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)})$ with $Y_i:\Omega \to {\mathbb{R}}^{p_i}$ for $i=1,2$. Take a jointly Gaussian measure $G(0,Q_{(Y_1,Y_2,W)})$ for the triple $(Y_1,Y_2,W)$, $W:\Omega \to {\mathbb{R}}^n$, $W\in G(0,Q_W)$, such that the marginal measure on $(Y_1,Y_2)$ is equal to the considered measure, with

$$\begin{array}{c}\hfill Q_{(Y_1,Y_2,W)}=\left(\begin{array}{ccc}{Q}_{Y_1}\hfill & Q_{Y_1,Y_2}\hfill & Q_{Y_1,W}\hfill \\ Q_{Y_1,Y_2}^T\hfill & Q_{Y_2}\hfill & Q_{Y_2,W}\hfill \\ Q_{Y_1,W}^T\hfill & Q_{Y_2,W}^T\hfill & Q_W\hfill \end{array}\right).\end{array}$$

(68)

Denote the parameterized joint measure with respect to W, by $(Y_1,Y_2,W)\in G(0,Q_{(Y_1,Y_2,W)})$. This parameterized joint measure $(Y_1,Y_2,W)\in G(0,Q_{(Y_1,Y_2,W)})$ also includes the subset such that the conditional independence holds, $(F^{Y_1},F^{Y_2}|F^W)\in \mathrm{CIG}$.

In the following subsections, it will be shown how such a random variable W can be constructed in a number of cases.

Algorithm 1. (a) gives the general case, while (b) gives the special case when the joint measure by $(Y_1,Y_2,W)\in G(0,Q_{(Y_1,Y_2,W)})$ such that $(F^{Y_1},F^{Y_2}|F^W)\in \mathrm{CIG}$ via weak stochastic realization.

Aldorithm 1.

Consider the model of a tuple of Gaussian random variables of Definition 6.

(a) General case.

• At the encoder, first compute the variables,

  $$\left(\begin{array}{c}{Z}_1\hfill \\ Z_2\hfill \end{array}\right)=\left(\begin{array}{c}{Y}_1\hfill \\ Y_2\hfill \end{array}\right)-\mathbf{E}\left[\left(\begin{array}{c}{Y}_1\hfill \\ Y_2\hfill \end{array}\right)W^T\right]=\left(\begin{array}{c}{Y}_1\hfill \\ Y_2\hfill \end{array}\right)-Q_{(Y_1,Y_2),W}{Q}_W^{†}W$$

  (69)

  $$Q_{(Y_1,Y_2),W}=\mathbf{E}\left[\left(\begin{array}{c}{Y}_1\hfill \\ Y_2\hfill \end{array}\right)W^T\right]=\left(\begin{array}{c}{Q}_{Y_1,W}\hfill \\ Q_{Y_2,W}\hfill \end{array}\right);$$

  (70)

  then, the triple$(Z_1,Z_2,W)$of jointly Gaussian random variables are such that$(Z_1,Z_2)\in G(0,Q_{(Z_1,Z_2)})$and$(Z_1,Z_2)$independent of W.
• The tuple of random variables$(Y_1,Y_2)$are represented according to

  $$\left(\begin{array}{c}{Y}_1\hfill \\ Y_2\hfill \end{array}\right)=Q_{(Y_1,Y_2),W}{Q}_W^{†}W+\left(\begin{array}{c}{Z}_1\hfill \\ Z_2\hfill \end{array}\right)$$

  (71)

(b) Special case. Consider$(F^{Y_1},F^{Y_2}|F^W)\in \mathrm{CIG}$, and assume$Q_W>0$.

• At the encoder, compute first the variables,

  $$\begin{array}{cc}\hfill Z_1=& Y_1-\mathbf{E}\left[Y_1\right|F^W]=Y_1-Q_{Y_1,W}{Q}_W^{-1}W,\hfill \end{array}$$

  (72)

  $$\begin{array}{cc}\hfill Z_2=& Y_2-\mathbf{E}\left[Y_2\right|F^W]=Y_2-Q_{Y_2,W}{Q}_W^{-1}W;\hfill \end{array}$$

  (73)

  then the triple$(Z_1,Z_2,W)$of jointly Gaussian random variables are independent.
• The tuple of random variables$(Y_1,Y_2)$are represented according to,

  $$\begin{array}{c}\hfill Y_1=Q_{Y_1,W}{Q}_W^{-1}W+Z_1,\hspace{1em}\hspace{1em}{Y}_2=Q_{Y_2,W}{Q}_W^{-1}W+Z_2.\end{array}$$

  (74)

We emphasize that$Y_1$ and $Y_2$ are conditionally independent condition on W if and only if $Z_1$ and $Z_2$ are independent.

The validity of the statements of the algorithm follow from the next proposition.

Proposition 4.

Consider the model of a tuple of Gaussian random variables of Definition 6, for cases (a), (b).

(a)

At the encoder, the conditional expectations are correct and the definitions of $Z_1$ and of $Z_2$ are well defined.

(b)

The three random variables $(Z_1,Z_2,W)$ are independent. Consequently, the three sequences

$(W^N,Z_1^N,Z_2^N)$, and messages generated by the Gray–Wyner encoder,

$f^{\left(E\right)}(Y_1^N,Y_2^N)={\overline{f}}^{\left(E\right)}(W^N,Z_1^N,Z_2^N)=(S_0,S_1,S_2)$ are independent.

Proof. 

Case (a). This follows from realization theory (since no constraints are imposed). Case (b). This is a specific application of Proposition 3. □

For the definition of $C(Y_1,Y_2)$, use is made of the construction of the actual family of measures such that $(Y_1,Y_2|W)\in \mathrm{CIG}$ holds, and the weak strochastic realization. These are presented in Theorem 8 and Corollary 1.

2.5. Characterization of Minimal Conditional Independence of a Triple of Gaussian Random Variables

Introduce the notation of the parameterization of the family of Gaussian probability distributions

$$\begin{array}{cc}\hfill {\mathcal{P}}^{CIG}=& \left\{{\mathbf{P}}_{Y_1,Y_2,W}(y_1,y_2,w)\right|\hspace{1em}{\mathbf{P}}_{Y_1,Y_2|W}(y_1,y_2|w)={\mathbf{P}}_{Y_1|W}\left(y_1\right|w){\mathbf{P}}_{Y_2|W}\left(y_2\right|w),\hfill \\ \hfill & {\mathbf{P}}_{Y_1,Y_2,W}(y_1,y_2,\infty )={\mathbf{P}}_{Y_1,Y_2}(y_1,y_2),\hspace{1em}(Y_1,Y_2,W)\hspace{1em}is jointly Gaussian\}.\hfill \end{array}$$

(75)

A subset of the set $P^{CIG}$ is the set of distributions $P_{min}^{CIG}$, with the additional constraint that the dimension of the random variable W is minimal while all other conditions hold, defined by

$${\mathcal{P}}_{min}^{CIG}=\left\{{\mathbf{P}}_{Y_1,Y_2,W}(y_1,y_2,w)\in {\mathcal{P}}^{CIG}|\hspace{1em}\hspace{1em}(Y_1,Y_2|W)\in {\mathrm{CIG}}_{min}\right\}\subseteq {\mathcal{P}}^{CIG}.$$

(76)

The parameterization of the family of Gaussian probability distributions ${\mathcal{P}}^{CIG}$ and ${\mathcal{P}}_{min}^{CIG}$ require the solution of the weak stochastic realization problem of Gaussian random variables defined by Problem 1. This problem is solved in [22] (Theorem 4.2). For the readers’ convenience, it is stated below.

Theorem 8.

Ref. [22] (Theorem 4.2) Consider a tuple $(Y_1,Y_2)$ of Gaussian random variables in the canonical variable form of Definition 1. Restrict attention to the correlated parts of these random variables. Thus, the random variables $Y_1,\hspace{1em}{Y}_2$ have the same dimension $n=p_1=p_2$, and their covariance matrix $D\in {\mathbb{R}}^{n\times n}$ is a nonsingular diagonal matrix with the diagonal ordered real-numbers in the interval $(0,1)$. Hence,

$$(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)})={\mathbf{P}}_0,\hspace{1em}\hspace{1em}{Y}_1,Y_2:\Omega \to {\mathbb{R}}^n,\hspace{1em}\hspace{1em}n\in {\mathbb{Z}}_{+},$$

(77)

$$Q_{(y_1,y_2)}=\left(\begin{array}{cc}I\hfill & D\hfill \\ D\hfill & I\hfill \end{array}\right),$$

(78)

$$D=\mathrm{Diag}(d_1,d_2,\dots ,d_n)\in {\mathbb{R}}^{n\times n},\hspace{1em}\hspace{1em}1>d_1\ge d_2\ge \dots \ge d_n>0.$$

(79)

That is, $p_{11}=p_{21}=0,p_{13}=p_{23}=0.$

(a) There exists a probability measure ${\mathbf{P}}_1$, and a triple of Gaussian random variables $Y_1,Y_2,$$W:\Omega \to {\mathbb{R}}^n$ defined on it, such that (i) ${\mathbf{P}}_1{|}_{(Y_1,Y_2)}={\mathbf{P}}_0$ and (ii) $(F^{Y_1},F^{Y_2}|F^W)\in {\mathrm{CIG}}_{min}$;

(b) There exist a family of Gaussian measures denoted by ${\mathbf{P}}_{\mathbf{ci}}\subseteq {\mathcal{P}}_{min}^{CIG}$, that satisfy (i) and (ii) of (a), and moreover, this family is parameterized by the matrices and sets, as follows.

$$G(0,Q_s\left(Q_W\right)),\hspace{1em}{Q}_W\in {\mathbf{Q}}_{\mathbf{W}},$$

(80)

$$\begin{array}{cc}\hfill & Q_s=Q_s\left(Q_W\right)=\left(\begin{array}{ccc}I\hfill & D\hfill & D^{1/2}\hfill \\ D\hfill & I\hfill & D^{1/2}{Q}_W\hfill \\ D^{1/2}\hfill & Q_W{D}^{1/2}\hfill & Q_W\hfill \end{array}\right),\hfill \end{array}$$

(81)

$${\mathbf{Q}}_{\mathbf{W}}=\left\{Q_W\in {\mathbb{R}}^{n\times n}|Q_W=Q_W^T,\hspace{1em}0<D\le Q_W\le D^{-1}\right\},$$

(82)

$${\mathbf{P}}_{\mathbf{ci}}=\left\{G(0,Q_s\left(Q_W\right))\hspace{1em}on\hspace{1em}({\mathbb{R}}^{3n},B\left({\mathbb{R}}^{3n}\right))|Q_W\in {\mathbf{Q}}_{\mathbf{W}}\right\}\subseteq {\mathcal{P}}_{min}^{CIG}.$$

(83)

Furthermore, for any measure $P_1\in {\mathcal{P}}_{min}^{CIG}$, there exists a triple of state transformation of the form $(Y_1,Y_2,W)\mapsto (S_1{Y}_1,S_2{Y}_2,S_{W}W)$ for nonsingular square matrices $S_1,\hspace{1em}{S}_2,S_W$ such that the corresponding measure of the three transformed variables belongs to ${\mathbf{P}}_{\mathbf{ci}}$.

The application of Theorem 8 is discussed in the next remark, in the context of parameterizing any rate-triple on the Gray–Wyner lossy rate region $(R_0,R_1,R_2)\in {\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$ that lies on the Pangloss plane.

Remark 4.

Applications of Theorem 8.

(a) Theorem 8 is a parameterization of the family of Gaussian measures ${\mathbf{P}}_{\mathbf{ci}}\subseteq {\mathcal{P}}_{min}^{CIG}$ by the entries of the covariance matrix $Q_W$. Hence, it is at most an $n(n+1)/2-$ dimensional parameterization;

(b) It is shown in Section 4.4 that only a subset of the achievable rate region ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)={\mathcal{R}}_{GW}^{\ast }({\Delta }_1,{\Delta }_2)$ is generated from distributions ${\mathbf{P}}_{\mathbf{ci}}\subseteq {\mathcal{P}}_{min}^{CIG}\subseteq \mathcal{P}$.

The next corollary is useful to the calculation of $C(Y_1,Y_2)$, since by Theorem 9, an achievable lower bound on $I(Y_1,Y_2;W)$ is incurred by a Gaussian random variable W, such that the distribution ${\mathbf{P}}_{Y_1,Y_2,W}\in {\mathbf{P}}_{\mathbf{ci}}\subseteq {\mathcal{P}}_{min}^{CIG}$, corresponding to $W\in G(0,Q_W)$. By Theorem 9, and since $C(Y_1,Y_2)$ is invariant with respect to nonsingular transformations applied to $(Y_1,Y_2,W)$, the next corollary gives the realization of $(Y_1,Y_2)$ as defined in Theorem 8, by (77)–(79), expressed in terms of an arbitrary Gaussian random variable $W\in G(0,Q_W)$.

Corollary 1.

Consider a tuple $(Y_1,Y_2)$ of Gaussian random variables in the canonical variable form of Definition 1. Restrict the attention to the correlated parts of these random variables, as defined in Theorem 8, by (77)–(79).

Then, a realization of the random variables $(Y_1,Y_2)$ which induce the family of measures ${\mathbf{P}}_{\mathbf{ci}}\subseteq {\mathcal{P}}_{min}^{CIG}$, defined by (80)–(83), is

$$Y_1=Q_{Y_1,W}{Q}_W^{-1}W+Z_1$$

(84)

$$Q_{Y_1,W}=D^{1/2},\hspace{1em}\hspace{1em}{Z}_1\in G(0,(I-D^{1/2}{Q}_W^{-1}{D}^{1/2})),$$

(85)

$$Y_2=Q_{Y_2,W}{Q}_W^{-1}W+Z_2$$

(86)

$$Q_{Y_2,W}=D^{1/2}{Q}_W,\hspace{1em}\hspace{1em}{Z}_2\in G(0,(I-D^{1/2}{Q}_W{D}^{1/2})),$$

(87)

$$(Z_1,Z_2,W),\hspace{1em}are independent.$$

(88)

Furthermore, the mutual information $I(Y_1,Y_2;W)$ is given by

$$\begin{array}{cc}\hfill I(Y_1,Y_2;W)=& H(Y_1,Y_2)-H\left(Y_1\right|W)-H\left(Y_2\right|W)\hfill \end{array}$$

(89)

$$\begin{array}{cc}\hfill =& \frac{1}{2}\sum _{i=1}^{n}ln(1-d_i^2)-\frac{1}{2}ln(det\left([I-D^{1/2}{Q}_W^{-1}{D}^{1/2}][I-D^{1/2}{Q}_W{D}^{1/2}]\right))\hfill \end{array}$$

(90)

and it is parameterized by $Q_W\in {\mathbf{Q}}_{\mathbf{W}}$, where ${\mathbf{Q}}_{\mathbf{W}}$ is defined by the set of Equation (82).

Proof. 

The correctness of the realization is due to Proposition 2 and Theorem 8. The calculation of mutual information follows from the realization. □

3. Wyner’s Common Information

This section is devoted to the calculation of Wyner’s common information $C(Y_1,Y_2)$, defined by (13), for ${\mathbf{P}}_{Y_1,Y_2}=G(0,Q_{(Y_1,Y_2)})$, and the construction of the weak stochastic realization of $(Y_1,Y_2,W)$ that achieves this.

3.1. Reduction of the Calculation of Wyner’s Common Information

First, we show Theorem 9, which states: given a tuple of multivariate correlated Gaussian random variables $(Y_1,Y_2)$, and an arbitrary random variable W (i.e., taking continuous or discrete values), Wyner’s common information $C(Y_1,Y_2)$, defined by (13), is minimized by a triple $(Y_1,Y_2,W)$ which induces a jointly Gaussian distribution ${\mathbf{P}}_{Y_1,Y_2,W}$, and $W:\Omega \to \mathbb{W}={\mathbb{R}}^n$ is finite-dimensional Gaussian random variable.

Theorem 9.

Consider a tuple of multivariate-correlated Gaussian random variables $Y_1:\Omega \to {\mathbb{R}}^{p_1},Y_2:\Omega \to {\mathbb{R}}^{p_2}$, $p_i\in {\mathbb{Z}}_{+},i=1,2$ with the variance matrix of this tuple denoted by

$$\begin{array}{cc}\hfill & (Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)}),\hspace{1em}\hspace{1em}{Q}_{(Y_1,Y_2)}=\left(\begin{array}{cc}{Q}_{Y_1}\hfill & Q_{Y_1,Y_2}\hfill \\ Q_{Y_1,Y_2}^T\hfill & Q_{Y_2}\hfill \end{array}\right)\in {\mathbb{R}}^{(p_1+p_2)\times (p_1+p_2)}\hfill \end{array}$$

(91)

and, without loss of generality, assume that $Q_{(Y_1,Y_2)}$ is a positive definite matrix. Let $W:\Omega \to \mathbb{W}$ be any auxiliary random variable, with $\mathbb{W}$ being an arbitrary measurable space, and ${\mathbf{P}}_{Y_1,Y_2,W}$ any joint probability distribution of the triple $(Y_1,Y_2,W)$ on the product space $({\mathbb{R}}^{p_1}\times {\mathbb{R}}^{p_2}\times \mathbb{W},$ $B\left({\mathbb{R}}^{p_1}\right)\otimes B\left({\mathbb{R}}^{p_2}\right)\otimes B\left(\mathbb{W}\right))$ with $(Y_1,Y_2)-$ marginal ${\mathbf{P}}_{Y_1,Y_2}$ the Gaussian distribution ${\mathbf{P}}_{Y_1,Y_2}=G(0,Q_{(Y_1,Y_2)})$.

The following hold.

(a) Define the random variables $Z_1,Z_2$ by

$$Z_i=Y_i-\mathbf{E}\left[Y_i\right|W],\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{Z}_i:\Omega \to {\mathbb{R}}^{p_i},\hspace{1em}\hspace{1em}i=1,2.$$

(92)

The inequalities hold:

$$\begin{array}{cc}\hfill I(Y_1,Y_2;W)=& H(Y_1,Y_2)-H(Y_1,Y_2|W)\hfill \end{array}$$

(93)

$$\begin{array}{cc}\hfill =& H(Y_1,Y_2)-H\left(Y_1\right|Y_2,W)-H\left(Y_2\right|W)\hfill \end{array}$$

(94)

$$\begin{array}{cc}\hfill \ge & H(Y_1,Y_2)-H\left(Y_1\right|W)-H\left(Y_2\right|W)\hfill \end{array}$$

(95)

$$\begin{array}{cc}\hfill =& H(Y_1,Y_2)-H(Y_1-\mathbf{E}\left[Y_1\right|W]|W)-H(Y_2-\mathbf{E}\left[Y_2\right|W]|W)\hfill \end{array}$$

(96)

$$\begin{array}{cc}\hfill \ge & H(Y_1,Y_2)-H(Y_1-\mathbf{E}\left[Y_1\right|W])-H(Y_2-\mathbf{E}\left[Y_2\right|W])\hfill \end{array}$$

(97)

$$\begin{array}{cc}\hfill =& H(Y_1,Y_2)-H\left(Z_1\right)-H\left(Z_2\right)\hfill \end{array}$$

(98)

$$\begin{array}{cc}\hfill \ge & H(Y_1,Y_2)-H\left(Z_1\right)-H\left(Z_2\right)\hspace{1em}\hspace{1em}if\hspace{1em}{Z}_1,Z_2\hspace{1em}have finite variances, and Gaussian.\hfill \end{array}$$

(99)

(b) If:

(i) $W:\Omega \to \mathbb{W}={\mathbb{R}}^n$ is an $n-$ dimensional, $n\in {\mathbb{Z}}_{+}$, Gaussian random variable;

(ii) $(Z_1,Z_2,W)$ are mutually independent jointly Gaussian random variables, then all inequalities in (93)–(99) hold with equality, and $(Y_1,Y_2,W)$ induces a family of jointly probability distributions ${\mathbf{P}}_{Y_1,Y_2,W}$ with $(Y_1,Y_2)-$ marginal ${\mathbf{P}}_{Y_1,Y_2}$, such that W makes $Y_1$ and $Y_2$ conditionally independent, that is ${\mathbf{P}}_{Y_1,Y_2|W}={\mathbf{P}}_{Y_1|W}{\mathbf{P}}_{Y_2|W}$;

(c) Among all joint distributions, ${\mathbf{P}}_{Y_1,Y_2,W}$ induced by the jointly Gaussian random variables $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)})$ of (91), and an arbitrary random variable $W:\Omega \to \mathbb{W}$, such that the $(Y_1,Y_2)-$ marginal ${\mathbf{P}}_{Y_1,Y_2}$ is the Gaussian distribution ${\mathbf{P}}_{Y_1,Y_2}=G(0,Q_{(Y_1,Y_2)})$, and ${\mathbf{P}}_{Y_1,Y_2|W}={\mathbf{P}}_{Y_1|W}{\mathbf{P}}_{Y_2|W}$, a jointly Gaussian distribution achieves the lower bounds of $I(Y_1,Y_2;W)$ in part (a), i.e., achieves Wyner’s common information $C(Y_1,Y_2)$, defined by (13), and ${\mathbf{P}}_{Y_1,Y_2,W}$ is induced by an $n-$ dimensional, $n\in {\mathbb{Z}}_{+}$, Gaussian random variable $W:\Omega \to \mathbb{W}={\mathbb{R}}^n$, and $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)}$, and such a distribution is induced by the triple $(Y_1,Y_2,W)$ represented by

$$\begin{array}{cc}\hfill & Y_1=\mathbf{E}\left[Y_1\right|W]+Z_1,\hspace{1em}\hspace{1em}{Y}_2=\mathbf{E}\left[Y_2\right|W]+Z_2,\hfill \end{array}$$

(100)

$$\begin{array}{cc}\hfill & (W,Z_1,Z_2)\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}are mutually independent, Gaussian random variables.\hfill \end{array}$$

(101)

Proof. 

(a) (93) is due to an identity of mutual information, (94) is due to the chain rule of entropy, (95) due to conditioning reduces entropy, (96) due to a property of conditional entropy, (97) due to conditioning reduces entropy, (98) is due to definition (92) and (99), is due to maximum entropy principle. (b) Since, $Y_i=\mathbf{E}\left[Y_i\right|W]+Z_i,i=1,2$ and $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)})$, if (i) and (ii) hold, then all inequalities hold with equality, and the statements are easily verified. (c) Follows from part (b). □

Remark 5.

Theorem 9 shows that, among all random variables W which induce a joint distribution ${\mathbf{P}}_{Y_1,Y_2,W}$, with $(Y_1,Y_2)-$ marginal ${\mathbf{P}}_{Y_1,Y_2}$ the Gaussian distribution ${\mathbf{P}}_{Y_1,Y_2}=G(0,Q_{(Y_1,Y_2)})$, then for the Wyner’s common information $C(Y_1,Y_2)$ problem, it suffices to consider a jointly Gaussian triple $(Y_1,Y_2,W)$ such that W makes $Y_1$ and $Y_2$ conditionally independent.

3.2. Wyner’s Common Information of Correlated Random Variables

Assume that the tuple of multivariate correlated Gaussian random variables $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)})$ of Theorem 9 is already transformed to the canonical variable representation, see Definition 1, using Algorithm A1, i.e., by the nonsingular transformation, $S=\mathrm{Block}-\mathrm{diag}(S_1,S_2)$. Mutual information is invariant with respect to nonsingular transformations, and $I(Y_1,Y_2;W)=I(S_1{Y}_1,S_2{Y}_2;W)$.

By Theorem 9. (c), the joint probability distributions ${\mathbf{P}}_{Y_1,Y_2,W}(y_1,y_2,w)$ are jointly Gaussian, and parameterized by the random variable W. This family of distributions is parameterized by the multidimensional random variable W, such that $(Y_1,Y_2)$ are conditionally independent, conditioned on W, the marginal distribution ${\mathbf{P}}_{Y_1,Y_2,W}(y_1,y_2,\infty )={\mathbf{P}}_{Y_1,Y_2}(y_1,y_2)$ coincides with the distribution of $(Y_1,Y_2)$, and to represent $(Y_1,Y_2)$.

Using the above construction, one obtains the next theorem.

Theorem 10.

Consider a tuple of Gaussian random variables $(Y_1,Y_2)\in G(0,Q_{\mathrm{cvf}})$ as described and decomposed according to Algorithm A1. Restrict attention to the correlated parts of these random variables, as described in Theorem 8, (77)–(79) (i.e., only components $(Y_{12},Y_{22})$ are present).

(a) Theorem 8 holds, and in particular, the family of jointly Gaussian distributions ${\mathbf{P}}_{Y_1,Y_2,W}$ induced by $(Y_1,Y_2)\in G(0,Q_{\mathrm{cvf}}))$ and a Gaussian random variable $W:\Omega \to {\mathbb{R}}^n$, with minimum dimension n, such that the $(Y_1,Y_2)-$ marginal is ${\mathbf{P}}_{Y_1,Y_2}=G(0,G(0,Q_{\mathrm{cvf}}))$, and ${\mathbf{P}}_{Y_1,Y_2|W}={\mathbf{P}}_{Y_1|W}{\mathbf{P}}_{Y_2|W}$, is paremetrized by the family of Theorem 8. (b), i.e., (80)–(83);

(b) Corollary 1, (84)–(88) characterizes the family of realizations of $(Y_1,Y_2,W)$, parameterized by W, which induce jointly Gaussian distributions, such that $W:\Omega \to {\mathbb{R}}^n$ is a Gaussian random variable with minimum dimension n, ${\mathbf{P}}_{Y_1,Y_2|W}={\mathbf{P}}_{Y_1|W}{\mathbf{P}}_{Y_2|W}$, and $(Y_1,Y_2)\in G(0,Q_{\mathrm{cvf}}))$. Moreover, Wyner’s common information $C(Y_1,Y_2)$ is computed from the expression $I(Y_1,Y_2;W)$ of Corollary 1, (90), optimized over $Q_W\in {\mathbf{Q}}_{\mathbf{W}}$, where ${\mathbf{Q}}_{\mathbf{W}}$ is defined by the set of Equation (82).

Remark 6.

It is apparent that the proof of the formula $C(Y_1,Y_2;W)$ in [15,16] is based on rate distortion function, i.e., they do not directly address Wyner’s optimization problem (13), as in Theorem 9, which first shows, among all continuous or discrete random variables, W is Gaussian, and there is no parameterization of the set of distributions ${\mathbf{P}}_{Y_1,Y_2,W}$ achieving conditional independence ${\mathbf{P}}_{Y_1,Y_2|W}={\mathbf{P}}_{Y_1|W}{\mathbf{P}}_{Y_2|W}$, i.e., the optimization over the parameterized family of Gaussian measures of Theorem 8 is not given.

In the next theorem, the family of measures ${\mathbf{P}}_{\mathbf{ci}}\subseteq {\mathcal{P}}_{min}^{CIG}$, defined by (80)–(83), which leads to realization of $(Y_1,Y_2)$, given in Corollary 1, is ordered for the determination of a single joint distribution ${\mathbf{P}}_{Y_1,Y_2,W^{\ast }}\in {\mathbf{P}}_{\mathbf{ci}}\subseteq {\mathcal{P}}_{min}^{CIG}$, which achieves $C(Y_1,Y_2)$. This leads to the realization of $(Y_1,Y_2)$ expressed in terms of $W^{\ast }$ and vectors of independent Gaussian random variables $(Z_1,Z_2)$, one for each realization, each having independent components.

Theorem 11.

Consider a tuple $(Y_1,Y_2)$ of Gaussian random variables in the canonical variable form of Definition 1. Restrict attention to the correlated parts of these random variables, as defined in Theorem 8, defined by (77)–(79.

The following hold.

(a) The information quantity $C(Y_1,Y_2)$ is given by

$$\begin{array}{c}\hfill C(Y_1,Y_2)=\frac{1}{2}\sum _{i=1}^{n}ln\left(\frac{1+d_i}{1-d_i}\right)=\frac{1}{2}\sum _{i=1}^{n}ln\left(1+\frac{2d_i}{1-d_i}\right)\in (0,\infty ).\end{array}$$

(102)

(b) The realizations of the random variables $(Y_1,Y_2,W^{\ast })$ that achieve $C(Y_1,Y_2)$ are represented by

$$\begin{array}{cc}\hfill & V:\Omega \to {\mathbb{R}}^n,\hspace{1em}V\in G(0,I),\hspace{1em}\hspace{1em}the vector V has independent components,\hfill \\ \hfill & F^V,\hspace{1em}{F}^{Y_1}\vee F^{Y_2},\hspace{1em}\hspace{1em}are independent \sigma - algebras,\hfill \end{array}$$

$$\begin{array}{cc}\hfill & L_1=L_2=D^{1/2}{(I+D)}^{-1}\in {\mathbb{R}}^{n\times n},\hfill \end{array}$$

(103)

$$\begin{array}{cc}\hfill & L_3={(I-D)}^{1/2}{(I+D)}^{-1/2}\in {\mathbb{R}}^{n\times n},\hspace{1em}\hspace{1em}{L}_1,\hspace{1em}{L}_2,\hspace{1em}{L}_3,\hspace{1em}are diagonal matrices,\hfill \end{array}$$

(104)

$$\begin{array}{cc}\hfill & W^{\ast }=L_1{Y}_1+L_2{Y}_2+L_{3}V,\hspace{1em}\hspace{1em}{W}^{\ast }:\Omega \to {\mathbb{R}}^n,\hfill \end{array}$$

(105)

$$\begin{array}{cc}\hfill & Z_1=Y_1-D^{1/2}{W}^{\ast },\hspace{1em}\hspace{1em}{Z}_1:\Omega \to {\mathbb{R}}^n,\hfill \end{array}$$

(106)

$$\begin{array}{cc}\hfill & Z_2=Y_2-D^{1/2}{W}^{\ast },\hspace{1em}\hspace{1em}{Z}_2:\Omega \to {\mathbb{R}}^n.\hfill \end{array}$$

(107)

Then:

$$Z_1\in G(0,(I-D)),\hspace{1em}\hspace{1em}{Z}_2\in G(0,(I-D)),\hspace{1em}\hspace{1em}{W}^{\ast }\in G(0,I);$$

(108)

$$(Z_1,Z_2,W^{\ast }),\hspace{1em}are independent and$$

(109)

$$Y_1=D^{1/2}{W}^{\ast }+Z_1,\hspace{1em}\hspace{1em}{Y}_2=D^{1/2}{W}^{\ast }+Z_2$$

(110)

hence, the variables $(Y_1,Y_2,W^{\ast })$ induce a distribution ${\mathbf{P}}_{Y_1,Y_2,W^{\ast }}\in {\mathbf{P}}_{\mathbf{ci}}\subseteq {\mathcal{P}}_{min}^{CIG}$. Note that, in addition, each of the random variables $Z_1$, $Z_2$, and $W^{\ast }$ have independent components.

(c) The variables $(Y_1,Y_2,W^{\ast })$ defined in (b) induce a distribution ${\mathbf{P}}_{Y_1,Y_2,W^{\ast }}\in {\mathbf{P}}_{\mathbf{ci}}\subseteq {\mathcal{P}}_{min}^{CIG}$ which achieves $C(Y_1,Y_2)$,

$$C(Y_1,Y_2)=I(Y_1,Y_2;W^{\ast }).$$

(111)

Proof. 

By Theorem 9, the random variables $(Y_1,Y_2,W)$ are restricted to jointly Gaussian random variables. Since mutual information $I(Y_1,Y_2;W)$ is invariant with respect to nonsingular transformations $S_1,S_2$, i.e., $I(Y_1,Y_2;W)=I(S_1{Y}_1,S{Y}_2;W)$, and $(F^{Y_1},F^{Y_2}|F^W)\in \mathrm{CIG}$ is equivalent to $(F^{S_1{Y}_1},F^{S_2{Y}_2}|F^W)\in \mathrm{CIG}$, then it suffices to consider the canonical variable form of Definition 1, and to construct a measure that carries a triple of jointly Gaussian random variables $Y_1,Y_2,\hspace{1em}\hspace{1em}W:\Omega \to {\mathbb{R}}^n$ such that $(F^{Y_1},F^{Y_2}|F^W)\in \mathrm{CIG}$.

(a) (1) Take a probability measure ${\mathbf{P}}_1$ such that there exists a triple of Gaussian random variables $Y_1,Y_2,\hspace{1em}\hspace{1em}W:\Omega \to {\mathbb{R}}^n$ with ${\mathbf{P}}_1{|}_{(y_1,y_2)}={\mathbf{P}}_0$ and $(F^{Y_1},F^{Y_2}|F^W)\in \mathrm{CIG}$. It will first be proven that attention can be restricted to those state random variables W of which the dimension equals $n=p_{12}=p_{22}$.

Suppose that there exists a state random variable $W:\Omega \to {\mathbb{R}}^{n_1}$ such that $(F^{Y_1},F^{Y_2}|F^W)$$\in \mathrm{CIG}$ and $n_1>n$. Hence, W does not make $(Y_1,Y_2)$ minimally conditionally independent. Construct a minimal vector which makes the tuple minimally conditionally independent according to the procedure of [22] (Proposition 3.5). Thus,

$$\begin{array}{cc}\hfill W_1=& \mathbf{E}\left[Y_1\right|F^W]=L_{11}W,\hspace{1em}\hspace{1em}{L}_{11}\in {\mathbb{R}}^{n\times n_1},\hfill \\ \hfill W_2=& \mathbf{E}\left[Y_2\right|F^{W_1}]=L_{12}{W}_1,\hspace{1em}\hspace{1em}{L}_{12}\in {\mathbb{R}}^{n\times n}.\hfill \end{array}$$

Then, $(F^{Y_1},F^{Y_2}|F^{W_2})\in {\mathrm{CIG}}_{min}$ and the dimension of $W_2$ is $n=p_{12}=p_{22}$. Determine a linear transformation of $W_2$ by a matrix $L_{15}\in {\mathbb{R}}^{n\times n}$ such that

$$W_3=L_{15}{W}_2=L_{15}{L}_{12}{L}_{11}W=L_{13}W,\hspace{1em}\hspace{1em}{L}_{13}=L_{15}{L}_{12}{L}_{11},\hspace{1em}\hspace{1em}{W}_3\in G(0,Q_3),\hspace{1em}\hspace{1em}{Q}_3=I_n=L_{13}{Q}_W{L}_{13}^T.$$

It is then possible to construct a matrix $L_{14}\in {\mathbb{R}}^{(n_1-n)\times n_1}$ such that

$$\begin{gathered} W_4=L_{14}W,\hspace{1em}\hspace{1em}{W}_4\in G(0,Q_4),\hspace{1em}\hspace{1em}{Q}_4=I,\hspace{1em}\hspace{1em}{L}_{14}{Q}_W{L}_{13}^T=0; \\ \left(\begin{array}{c}{W}_3\hfill \\ W_4\hfill \end{array}\right)\in G(0,I_{n_1}),\hspace{1em}\mathrm{rank}\left(\left(\begin{array}{c}{L}_{13}\hfill \\ L_{14}\hfill \end{array}\right)\right)=n_1, \end{gathered}$$

and, due to $L_{14}{Q}_W{L}_{13}^T=0$, $W_3$, $W_4$ are independent random variables. See [23] [Theorem 4.9] for a theorem with which the existence of $L_4$ can be proven. Note further that $F^W=F^{W_3,W_4}$.

Hence, the random variables $W_3,W_4$ are independent, $(F^{Y_1},F^{Y_2}|F^{W_3})\in {\mathrm{CIG}}_{min}$, and $I(Y_1,Y_2;W)=I(Y_1,Y_2;W_3,W_4)$.

By properties of mutual information, it now follows that

$$\begin{array}{c}I(Y_1,Y_2;W_3,W_4)-I(Y_1,Y_2;W_3)\hfill \\ =H(Y_1,Y_2)+H(W_3,W_4)-H(Y_1,Y_2,W_3,W_4)-H(Y_1,Y_2)-H\left(W_3\right)+H(Y_1,Y_2,W_3)\hfill \\ =H(Y_1,Y_2,W_3)+H\left(W_4\right)-H(Y_1,Y_2,W_3,W_4),\hspace{1em}by independence of{W}_{3}and{W}_4;\hfill \\ =I(Y_1,Y_2,W_3;W_4)\ge 0.\hfill \end{array}$$

Thus, for the computation of $C(Y_1,Y_2)$, attention can be restricted to those state variables W which are of miminal dimension.

(2) Take a probability measure ${\mathbf{P}}_1$ such that there exists a triple of Gaussian random variables $Y_1,Y_2,W:\Omega \to {\mathbb{R}}^n$ with ${\mathbf{P}}_1{|}_{(Y_1,Y_2)}={\mathbf{P}}_0$ and $(F^{Y_1},F^{Y_2}|F^W)\in {\mathrm{CIG}}_{min}$.

According to [22] (Theorem 4.2), there exist in general many such measures which are parameterized by the matrices and the sets, as stated in Theorem 8, (b), and defined by (80)–(83).

(3) Then, the mutual information of the triple of Gaussian random variables is calculated, using Theorem 8. (b) for any choice of $Q_W\in {\mathbf{Q}}_{\mathbf{W}}$, where ${\mathbf{Q}}_{\mathbf{W}}$ is given by (82). Then

$$I(Y_1,Y_2;W)=H(Y_1,Y_2)-H\left(Y_1\right|W)-H\left(Y_2\right|W).$$

The following calculations are then obvious:

$$\begin{array}{cc}\hfill det\left(Q_{(Y_1,Y_2)}\right)=& det\left(\begin{array}{cc}I\hfill & D\hfill \\ D\hfill & I\hfill \end{array}\right)=det(I-D^2)=\prod _{i=1}^n(1-d_i^2);\hfill \\ \hfill H(Y_1,Y_2)=& \frac{1}{2}ln(det\left(Q_{(y_1,y_2)}\right))+\frac{1}{2}\left(2n\right)ln\left(2\pi e\right)=\frac{1}{2}\sum _{i=1}^{n}ln(1-d_i^2)+nln\left(2\pi e\right);\hfill \\ \hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\mathbf{P}}_{Y_1|W}\left(y_1\right|w)\in G(\mathbf{E}\left[Y_1\right|F^W],Q_{Y_1|W}),\hfill \\ \hfill \mathbf{E}\left[Y_1\right|F^W]=& Q_{Y_1,W}{Q}_W^{-1}W=D^{1/2}{Q}_W^{-1}W;\hspace{1em}\hspace{1em}by\left(81\right)\hfill \\ \hfill Q_{Y_1|W}=& I-Q_{Y_1,W}{Q}_W^{-1}{Q}_W{Q}_W^{-1}{Q}_{Y_1,W}^T=I-D^{1/2}{Q}_W^{-1}{D}^{1/2};\hspace{1em}\hspace{1em}by\left(81\right)\hfill \\ \hfill H\left(Y_1\right|W)=& \frac{1}{2}ln(det(I-D^{1/2}{Q}_W^{-1}{D}^{1/2}))+\frac{1}{2}nln\left(2\pi e\right);\hfill \\ \hfill \mathbf{E}\left[Y_2\right|F^W]=& Q_{Y_2,W}{Q}_W^{-1}W=D^{1/2}{Q}_W{Q}_W^{-1}W=D^{1/2}W;\hfill \\ \hfill Q_{Y_2|W}=& I-Q_{Y_2,W}{Q}_W^{-1}{Q}_W{Q}_W^{-1}{Q}_{Y_2,W}^T=I-D^{1/2}{Q}_W{D}^{1/2};\hfill \\ \hfill H\left(Y_2\right|W)=& \frac{1}{2}ln(det(I-D^{1/2}{Q}_W{D}^{1/2}))+\frac{1}{2}nln\left(2\pi e\right).\hfill \end{array}$$

From the above calculations, it then follows

$$\begin{array}{cc}\hfill I(Y_1,Y_2;W)=& H(Y_1,Y_2)-H\left(Y_1\right|W)-H\left(Y_2\right|W)\hfill \end{array}$$

(112)

$$\begin{array}{cc}\hfill =& \frac{1}{2}\sum _{i=1}^{n}ln(1-d_i^2)+nln\left(2\pi e\right)\hfill \end{array}$$

$$\begin{array}{cc}\hfill & -\frac{1}{2}ln(det(I-D^{1/2}{Q}_W^{-1}{D}^{1/2}))-\frac{1}{2}nln\left(2\pi e\right)\hfill \end{array}$$

$$\begin{array}{cc}\hfill & -\frac{1}{2}ln(det(I-D^{1/2}{Q}_W{D}^{1/2}))-\frac{1}{2}nln\left(2\pi e\right)\hfill \end{array}$$

(113)

$$\begin{array}{cc}\hfill =& \frac{1}{2}\sum _{i=1}^{n}ln(1-d_i^2)-\frac{1}{2}ln(det\left([I-D^{1/2}{Q}_W^{-1}{D}^{1/2}][I-D^{1/2}{Q}_W{D}^{1/2}]\right)).\hfill \end{array}$$

(114)

The above calculations verify the statements of Corollary 1.

(4) The computation of $C(Y_1,Y_2)$ requires the solution of an optimization problem.

$$\begin{array}{cc}\hfill C(Y_1,Y_2)=& \underset{P_1\in {\mathbf{P}}_{\mathbf{ci}}}{inf}I(Y_1,Y_2;W)\hfill \\ \hfill =& \underset{Q_W\in {\mathbf{Q}}_{\mathbf{W}}}{inf}\left\{\frac{1}{2}\sum _{i=1}^{n}ln(1-d_i^2)-\frac{1}{2}ln(det\left([I-D^{1/2}{Q}_W^{-1}{D}^{1/2}][I-D^{1/2}{Q}_W{D}^{1/2}]\right))\right\}.\hfill \end{array}$$

(115)

Since the first term in (115), $\frac{1}{2}{\sum }_{i=1}^{n}ln(1-d_i^2)$, does not depend on $Q_W$ and the natural logarithm is a strictly increasing function, then

$$C(Y_1,Y_2)\hspace{1em}\hspace{1em}is equivalent to:\hspace{1em}\hspace{1em}\underset{Q_W\in {\mathbf{Q}}_{\mathbf{W}}}{sup}det\left[(I-D^{1/2}{Q}_W^{-1}{D}^{1/2})(I-D^{1/2}{Q}_W{D}^{1/2})\right].$$

(116)

Define:

$$\begin{array}{cc}\hfill L_1\left(Q_W\right)=& (I-D^{1/2}{Q}_W^{-1}{D}^{1/2})(I-D^{1/2}{Q}_W{D}^{1/2}),\hfill \end{array}$$

(117)

$$\begin{array}{cc}\hfill f_1\left(Q_W\right)=& det\left(L_1\left(Q_W\right)\right).\hfill \end{array}$$

(118)

Note that the expression $L_1\left(Q_W\right)\in {\mathbb{R}}^{n\times n}$ is a non-symmetric square matrix in general.

It will be proven that

$$\begin{array}{cc}\hfill f_1\left(Q_W\right)=& det\left(L_1\left(Q_W\right)\right)\le det\left({[I-D]}^2\right),\hspace{1em}\hspace{1em}\forall \hspace{1em}{Q}_W\in {\mathbf{Q}}_{\mathbf{W}},\hfill \end{array}$$

(119)

$$\begin{array}{cc}\hfill det\left(L_1\left(Q_W\right)\right)=& det\left({[I-D]}^2\right)\hspace{1em}\hspace{1em}ifandonlyif\hspace{1em}\hspace{1em}{Q}_W=I.\hfill \end{array}$$

(120)

From these two relations, it follows that $Q_W^{\ast }=I\in {\mathbb{R}}^{n\times n}$ is the unique solution of the supremization problem.

The inequality in (119) follows from Proposition A4. The equality of (120) is proven in two steps. If $Q_W=I$, then the equality of (120) holds as follows from direct substitution in (117). The converse is proven by contradiction. Suppose that $Q_W\ne I$. Then, it again follows from Proposition A4 that strict inequality holds in (119). Hence, the equality is proven.

(5) Finally, the value of $C(Y_1,Y_2)$ is computed for $Q_W^{\ast }=I$.

$$\begin{array}{cc}\hfill C(Y_1,Y_2)=& \frac{1}{2}\sum _{i=1}^{n}ln(1-d_1^2)-\frac{1}{2}ln(det(I-D^{1/2}{\left(Q_W^{\ast }\right)}^{-1}{D}^{1/2}))-\frac{1}{2}ln(det(I-D^{1/2}\left(Q_W^{\ast }\right)D^{1/2}))\hfill \\ \hfill =& \frac{1}{2}\sum _{i=1}^{n}ln(1-d_i^2)-\frac{1}{2}2ln(det(I-D))\hfill \\ \hfill =& \frac{1}{2}\sum _{i=1}^{n}ln(1-d_i^2)-\frac{1}{2}\sum _{i=1}^{n}ln\left({(1-d_i)}^2\right)=\frac{1}{2}\sum _{i=1}^{n}ln\left(\frac{1-d_i^2}{{(1-d_i)}^2}\right)\hfill \\ \hfill =& \frac{1}{2}\sum _{i=1}^{n}ln\left(\frac{1+d_i}{1-d_i}\right)=\frac{1}{2}\sum _{i=1}^{n}ln\left(1+\frac{2d_i}{1-d_i}\right).\hfill \end{array}$$

(b) It follows from part (a) of the theorem that $C(Y_1,Y_2)$ is attained as the mutual information $I(Y_1,Y_2;W)$ for a random variable W with $Q_W=Q=I$. Consider now a triple of random variables $(Y_1,Y_2,W)\in G(0,Q_s\left(I\right))$ as defined in (80)–(83), hence, $Q_W=I$. Denote the random variable W from now on by $W^{\ast }$ to indicate that it achieves the infimum of the definition of $C(Y_1,Y_2)$. Thus, $Q_{W^{\ast }}=I$ and

$$\begin{array}{cc}\hfill & (Y_{12},Y_{22},W^{\ast })\in G(0,Q_s\left(I\right)),\hfill \\ \hfill & Q_s\left(I\right)=\left(\begin{array}{ccc}I\hfill & D\hfill & D^{1/2}\hfill \\ D\hfill & I\hfill & D^{1/2}\hfill \\ D^{1/2}\hfill & D^{1/2}\hfill & I\hfill \end{array}\right)>0.\hfill \end{array}$$

(121)

Let $V:\Omega \to {\mathbb{R}}^{n_{12}}$ be a Gaussian random variable with $V\in G(0,I)$ which is independent of $(Y_1,Y_2,W)$.

Define the new state variable $\overline{W}=L_1{Y}_1+L_2{Y}_2+L_{3}V$. Then, $(Y_1,Y_2,V,W^{\ast })$ are jointly Gaussian and it has to be shown that then $Q_{\overline{W}}=I$, $Q_{Y_1,\overline{W}}=D^{1/2}$, and $Q_{Y_2,\overline{W}}=D^{1/2}$. These equalities follow from simple calculations using the expressions of $L_1$, $L_2$, and $L_3$ which calculations are omitted. It then follows from those calculations and the definition of the Gaussian measure $G(0,Q_s\left(I\right))$ that, almost surely, $\overline{W}=W^{\ast }$.

The signals are then represented by

$$Z_1=Y_1-\mathbf{E}\left[Y_1\right|F^{W^{\ast }}]=Y_1-Q_{Y_1,W^{\ast }}{\left(Q_{W^{\ast }}\right)}^{-1}{W}^{\ast }=Y_1-D^{1/2}{W}^{\ast },$$

(122)

$$Z_2=Y_2-\mathbf{E}\left[Y_2\right|F^{W^{\ast }}]=Y_2-Q_{Y_2,W^{\ast }}{\left(Q_{W^{\ast }}\right)}^{-1}{W}^{\ast }=Y_2-D^{1/2}{W}^{\ast }.$$

(123)

It is proven that the triple of random variables $(Z_1,Z_2,W^{\ast })$ are independent.

$$\begin{array}{cc}\hfill \mathbf{E}\left[Z_1{\left(W^{\ast }\right)}^T\right]=& \mathbf{E}\left[Y_1{\left(W^{\ast }\right)}^T\right]-D^{1/2}\mathbf{E}\left[W^{\ast }{\left(W^{\ast }\right)}^T\right]=D^{1/2}-D^{1/2}=0,\hspace{1em}\hspace{1em}\mathbf{E}\left[Z_2{\left(W^{\ast }\right)}^T\right]=0,\hfill \\ \hfill \mathbf{E}\left[Z_1{Z}_2^T\right]=& \mathbf{E}\left[(Y_1-D^{1/2}{W}^{\ast }){(Y_2-D^{1/2}{W}^{\ast })}^T\right]=0.\hfill \end{array}$$

Hence, the original signals are represented as shown by the formulas,

$$\begin{array}{cc}\hfill Y_1=& Z_1+D^{1/2}{W}^{\ast },\hspace{1em}\hspace{1em}by\left(121\right),Q_{y_1,W^{\ast }}{Q}_{W^{\ast }}^{-1}=D^{1/2},and by definition of{Z}_1,\hfill \\ \hfill Y_2=& Z_2+D^{1/2}{W}^{\ast },\hspace{1em}\hspace{1em}similarly.\hfill \end{array}$$

□

3.3. Wyner’s Common Information of Arbitrary Gaussian Random Variables

First, the two special cases of (1) a tuple of independent Gaussian random variables and (2) a tuple of identical Gaussian random variables are analyzed. From those results and that of the previous subsection, one can then prove Wyner’s common information for arbitrary Gaussian random variables.

The special case of the canonical variable form with only private parts is presented below.

Proposition 5.

Consider the case of a tuple of Gaussian vectors with only private parts. Hence, the Gaussian measure is

$$(Y_{13},Y_{23})\in G(0,Q_{(Y_{13},Y_{23})}),\hspace{1em}\hspace{1em}{Q}_{(Y_{13},Y_{23})}=\left(\begin{array}{cc}I\hfill & 0\hfill \\ 0\hfill & I\hfill \end{array}\right),\hspace{1em}{Y}_{13}:\Omega \to {\mathbb{R}}^{p_{13}},\hspace{1em}{Y}_{23}:\Omega \to {\mathbb{R}}^{p_{23}}.$$

(124)

(a) 

The minimal σ-algebra $F^W$ which makes $Y_{13},Y_{23}$ conditionally independent is the trivial σ-algebra denoted by $F_0=\{\varnothing ,\Omega \}$. Thus, $(F^{Y_{13}},F^{Y_{23}}|F_0)\in \mathrm{CI}$. The random variable W, in this case, is the constant $W_3=0\in \mathbb{R}$, hence $F^{W_3}=F_0$.

(b) 

Then, $W^{\ast }=W_3$ and

$$C(Y_1,Y_2)=C(Y_{13},Y_{23})=I(Y_{13},Y_{23};W_3)=0.$$

(125)

(c) 

The weak stochastic realization that achieves $C(Y_{13},Y_{23})=0$ is

$$Z_1=Y_{13},\hspace{1em}\hspace{1em}{Z}_2=Y_{23},\hspace{1em}\hspace{1em}{W}_3=0.$$

(126)

The special case of canonical variable form with only identical parts is presented below.

Proposition 6.

Consider the case of a tuple of Gaussian vectors with only the identical part. Hence the Gaussian measure is,

$$\begin{array}{cc}\hfill & Y_{11}:\Omega \to {\mathbb{R}}^{p_{11}},\hspace{1em}\hspace{1em}{Y}_{21}:\Omega \to {\mathbb{R}}^{p_{21}},\hspace{1em}\hspace{1em}{p}_{11}=p_{21},\hfill \\ \hfill & (Y_{11},Y_{21})\in G(0,Q_{(Y_{11},Y_{21})}),\hspace{1em}\hspace{1em}{Q}_{(y_{11},y_{21})}=\left(\begin{array}{cc}I\hfill & I\hfill \\ I\hfill & I\hfill \end{array}\right),\hspace{1em}\hspace{1em}{Y}_{11}=Y_{21}\hspace{1em}\mathrm{a}.\mathrm{s}.\hfill \end{array}$$

(127)

(a) 

The only minimal σ-algebra which makes $Y_{11}$ and $Y_{21}$ Gaussian conditional-independent is $F^{Y_{11}}=F^{Y_{21}}$. The state variable is thus, $W_1=Y_{11}=Y_{21}$ and $F^W=F^{Y_{11}}=F^{Y_{21}}$.

(b) 

Then $C(Y_1,Y_2)=C(Y_{11},Y_{21})=+\infty$.

(c) 

The weak stochastic realization is again simple, the variable W equals the identical component and there is no need to use the signals $Z_1$ and $Z_2$. Thus, the representations are,

$$Z_1=0\in \mathbb{R},\hspace{1em}\hspace{1em}{Z}_2=0\in \mathbb{R},\hspace{1em}\hspace{1em}W=Y_{11}=Y_{21}.$$

(128)

Theorem 4 is now proven. Thus, the setting is that of a tuple of arbitrary Gaussian random variables, not necessarily restricted to the correlated parts of these random variables of Theorem 8, by (77)–(79). It is shown that $C(Y_1,Y_2)$ is computed by a decomposition and by the use of the formulas previously obtained in Section 3.2.

Proof of Theorem 4.

(a)

$$\begin{array}{cc}\hfill & C(Y_1,Y_2)=\underset{(Y_1,Y_2,W)\in \mathrm{CIG}}{inf}I(Y_1,Y_2;W)\hfill \\ \hfill =& inf\hspace{1em}\left\{I(Y_{11},Y_{21};W_1)+I(Y_{12},Y_{22};W_2)+I(Y_{13},Y_{23};0)\right\},\hspace{1em}\hspace{1em}by Proposition \mathrm{A}1,\hfill \\ \hfill \ge & \underset{(Y_1,Y_2,W)\in \mathrm{CIG}}{inf}I(Y_{11},Y_{21};W_1)+\underset{(Y_1,Y_2,W)\in \mathrm{CIG}}{inf}I(Y_{12},Y_{22};W_2)+\underset{(Y_1,Y_2,W)\in \mathrm{CIG}}{inf}I(Y_{13},Y_{23};0)\hfill \\ \hfill =& \underset{(Y_{11},Y_{21},W_1)\in \mathrm{CIG}}{inf}I(Y_{11},Y_{21};W_1)+\underset{(Y_{12},Y_{22},W_2)\in \mathrm{CIG}}{inf}I(Y_{12},Y_{22};W_2)+I(Y_{13},Y_{23};0)\hfill \\ \hfill =& C(Y_{11},Y_{21};W_1)+C(Y_{12},Y_{22};W_2)+C(Y_{13},Y_{23};0)\hfill \\ \hfill =& \left\{\begin{array}{ccc}0,\hfill & if\hfill & p_{13}>0,\hspace{1em}{p}_{23}>0,\hspace{1em}{p}_{11}=p_{12}=p_{21}=p_{22}=0,\hfill \\ \frac{1}{2}\sum _{i=1}^{n}ln\left(\frac{1+d_i}{1-d_i}\right),\hfill & if\hfill & p_{12}=p_{22}>0,\hspace{1em}{p}_{11}=p_{21}=0,\hspace{1em}{p}_{13}\ge 0,\hspace{1em}{p}_{23}\ge 0,\hfill \\ +\infty ,\hfill & else.\hfill \end{array}\right.\hfill \end{array}$$

(129)

The latter equality follows from, respectively, Proposition 6, Theorem 11 and Proposition 5

(a and b). It will be shown that $C(Y_1,Y_2)$ is less than or equal to the right-hand side of Equation (129). From the latter inequality and the above inequality then follows the expression according to Equation (129).

To be specific, it will be proven that $C(Y_1,Y_2)$ is less than the expression $I(Y_1,Y_2;W^{\ast })$ where $W^{\ast }$ is defined in statement (b) of the proposition. It then follows from the proof of Theorem 11 that $(F^{Y_{12}},F^{Y_{22}}|F^{W_2^{\ast }})\in {\mathrm{CIG}}_{min}$.

Then:

$$\begin{array}{cc}\hfill C(Y_1,Y_2)=& \underset{(Y_1,Y_2|W)\in \mathrm{CIG}}{inf}I(Y_1,Y_2;W)\le I(Y_1,Y_2;W^{\ast })\hfill \\ \hfill =& I(Y_{11},Y_{21};W_1^{\ast })+I(Y_{12},Y_{22};W_2^{\ast })+I(Y_{13},Y_{23};\varnothing )\hfill \\ \hfill =& \left\{\begin{array}{ccc}0,\hfill & if\hfill & p_{13}>0,\hspace{1em}{p}_{23}>0,\hspace{1em}{p}_{11}=p_{12}=p_{21}=p_{22}=0,\hfill \\ \frac{1}{2}\sum _{i=1}^{n}ln\left(\frac{1+d_i}{1-d_i}\right),\hfill & if\hfill & p_{12}=p_{22}>0,\hspace{1em}{p}_{11}=p_{21}=0,\hspace{1em}{p}_{13}\ge 0,\hspace{1em}{p}_{23}\ge 0,\hfill \\ +\infty ,\hfill & else.\hfill \end{array}\right.\hfill \end{array}$$

The latter equality is proven as follows. In the first case, when $p_{13}>0$, $p_{23}>0$, and $p_{11}=p_{12}=p_{21}=p_{22}=0$, then $Y_1=Y_{13}$ and $Y_2=Y_{23}$ are independent random variables. It then follows from Proposition 5 that $I(Y_1,Y_2;0)=I(Y_{13},Y_{23};0)=0$. In the second case, when $p_{12}=p_{22}>0$, $p_{13}\ge 0$, $p_{23}\ge 0$, and $p_{11}=p_{21}=0$, it follows from Proposition A1 and from Theorem 11 that

$$I(Y_1,Y_2;W^{\ast })=I(Y_{12},Y_{22};W_2^{\ast })+I(Y_{13},Y_{23};0)=\frac{1}{2}\sum _{i=1}^{n}ln\left(\frac{1+d_i}{1-d_i}\right).$$

In the third case, when $p_{11}=p_{21}>0$ and the other $p_{ij}$ indices are arbitrary, then $I(Y_1,Y_2;W^{\ast })=+\infty$. Hence, the inequality $C(Y_1,Y_2)\le right-handside$ is proven and hence equality holds.

(c) This directly follows from Proposition 4. See also Section 3.6 of [21]. □

A procedure for the numerical calculations of Wyner’s information common information is given in Section 3.7 of [21].

4. Parametrization of Gray and Wyner Rate Region and Wyner’s Lossy Common Information

This section is devoted to the characterizations of rates that lie in the Gray–Wyner rate region for a tuple of Gaussian random variables with square error distortion functions.

By Gray–Wyner [1] (Theorem 8), reproduced in Theorem 1 to characterize rate triples $(R_0,R_1,R_2)\in {\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$, it is necessary to:

(i) Characterize the rate distortion functions $R_{Y_i}\left({\Delta }_i\right),R_{Y_i|W}\left({\Delta }_i\right),i=1,2$ and $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$;

(ii) Construct the realizations that induce the test channels of $R_{Y_i}\left({\Delta }_i\right),R_{Y_i|W}\left({\Delta }_i\right),i=1,2$ and $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$, and understand the structural properties of the realizations.

4.1. Characterizations of Joint, Conditional and Marginal RDFs

Theorem 12 is the characterization of the joint RDF $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$ from [24].

Theorem 12.

Ref. [24] Consider a tuple of Gaussian random variables $Y_i:\Omega \to {\mathbb{R}}^{p_i}$, with $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)})$, $Q_{(Y_1,Y_2)}\ge 0$ (which implies $Q_{Y_i}>0$, for $i=1,2$). Consider the joint RDF $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$ with square error distortion functions $D_{Y_1}(y_1,{\widehat{y}}_1)=\left|\right|y_1-{\widehat{y}}_1{\left|\right|}_{{\mathbb{R}}^{p_1}}^2,$$\hspace{1em}{D}_{Y_2}(y_2,{\widehat{y}}_2)=\left|\right|y_2-{\widehat{y}}_2{\left|\right|}_{{\mathbb{R}}^{p_2}}^2$. Then, the following hold:

(a) The mutual information $I(Y_1,Y_2;{\widehat{Y}}_1,{\widehat{Y}}_2)$ satisfies

$$I(Y_1,Y_2;{\widehat{Y}}_1,{\widehat{Y}}_2)\ge I(Y_1,Y_2;\mathbf{E}\left\{Y_1|F^{{\widehat{Y}}_1,{\widehat{Y}}_2}\right\},\mathbf{E}\left\{Y_2|F^{{\widehat{Y}}_1,{\widehat{Y}}_2}\right\})$$

(130)

and the mean square error satisfies

$$\mathbf{E}\left\{\left|\right|Y_i-{\widehat{Y}}_i{\left|\right|}_{{\mathbb{R}}^{p_1}}^2\right\}\ge \mathbf{E}\left\{\left|\right|Y_i-\mathbf{E}\left\{Y_i|F^{{\widehat{Y}}_1,{\widehat{Y}}_2}\right\}{\left|\right|}_{{\mathbb{R}}^{p_i}}^2\right\},\hspace{1em}\hspace{1em}i=1,2.$$

(131)

Moreover, inequalities in (130) and (131) hold with equality if there exists a jointly Gaussian realization of $({\widehat{Y}}_1,{\widehat{Y}}_2)$ or a Gaussian test channel distribution ${\mathbf{P}}_{{\widehat{Y}}_1,{\widehat{Y}}_2|Y_1,Y_2}$ such that the joint distribution ${\mathbf{P}}_{{\widehat{Y}}_1,{\widehat{Y}}_2,Y_1,Y_2}$ is jointly Gaussian, and such that the following identities both hold;

$$\mathbf{E}\left\{Y_1|F^{{\widehat{Y}}_1,{\widehat{Y}}_2}\right\}=\mathbf{E}\left\{Y_1|F^{{\widehat{Y}}_1}\right\}={\widehat{Y}}_1,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\mathbf{E}\left\{Y_2|F^{{\widehat{Y}}_1,{\widehat{Y}}_2}\right\}=\mathbf{E}\left\{Y_2|F^{{\widehat{Y}}_2}\right\}={\widehat{Y}}_2.$$

(132)

(b) A realization that achieves the lower bounds of part (a), i.e., satisfies (132), is the Gaussian realization of $(Y_1,Y_2,{\widehat{Y}}_1,{\widehat{Y}}_2)$ given by

$$\left(\begin{array}{c}{\widehat{Y}}_1\hfill \\ {\widehat{Y}}_2\hfill \end{array}\right)=H\left(\begin{array}{c}{Y}_1\hfill \\ Y_2\hfill \end{array}\right)+\left(\begin{array}{c}{V}_1\hfill \\ V_2\hfill \end{array}\right)$$

(133)

$$(V_1,V_2)\in G(0,Q_{(V_1,V_2)}),\hspace{1em}\hspace{1em}(V_1,V_2)\hspace{1em}\hspace{1em}independent of\hspace{1em}\hspace{1em}(Y_1,Y_2),$$

(134)

$$H{Q}_{(Y_1,Y_2)}=Q_{(Y_1,Y_2)}-Q_{(E_1,E_2)}\ge 0,$$

(135)

$$\begin{array}{cc}\hfill & Q_{(V_1,V_2)}=Q_{(E_1,E_2)}{H}^T=H{Q}_{(Y_1,Y_2)}-H{Q}_{(Y_1,Y_2)}{H}^T\ge 0,\hfill \\ \hfill & if{Q}_{(Y_1,Y_2)}>0then\hfill \end{array}$$

(136)

$$\begin{array}{cc}\hfill & H=I_{p_1+p_2}-Q_{(E_1,E_2)}{Q}_{(Y_1,Y_2)}^{-1},\hspace{1em}\hspace{1em}{Q}_{(V_1,V_2)}=Q_{(E_1,E_2)}-Q_{(E_1,E_2)}{Q}_{(Y_1,Y_2)}^{-1}{Q}_{(E_1,E_2)}\ge 0,\hfill \end{array}$$

(137)

where $(E_1,E_2)$ is the error tuple, that satisfies the structural property,

$$\begin{array}{c}\hfill E_i=Y_i-\mathbf{E}\left\{Y_i|F^{{\widehat{Y}}_1,{\widehat{Y}}_2}\right\}=Y_i-\mathbf{E}\left\{Y_i|F^{{\widehat{Y}}_i}\right\}=Y_i-{\widehat{Y}}_i,\hspace{1em}\hspace{1em}i=1,2\end{array}$$

(138)

and the variance matrix of this tuple is,

$$\begin{array}{cc}\hfill & (E_1,E_2)\in G(0,Q_{(E_1,E_2)}),\hspace{1em}\hspace{1em}{Q}_{(E_1,E_2)}=\left(\begin{array}{cc}{Q}_{E_1}\hfill & Q_{E_1,E_2}\hfill \\ Q_{E_1,E_2}^T\hfill & Q_{E_2}\hfill \end{array}\right)\in {\mathbb{R}}^{(p_1+p_2)\times (p_1+p_2)}.\hfill \end{array}$$

(c) The joint RDF $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$ is characterized by

$$\begin{array}{cc}\hfill & R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)=\underset{\mathbf{E}\left\{\left|\right|E_i{\left|\right|}_{{\mathbb{R}}^{p_i}}^2\right\}=\mathrm{trace}\left(Q_{E_i}\right)\le {\Delta }_i,\hspace{1em}i=1,2}{inf}\frac{1}{2}ln{\left(\frac{det\left(Q_{(Y_1,Y_2)}\right)}{det\left(Q_{(E_1,E_2)}\right)}\right)}^{+}\in [0,\infty ],\hfill \end{array}$$

(139)

$$\begin{array}{cc}\hfill & such that\hspace{1em}\hspace{1em}{Q}_{({\widehat{Y}}_1,{\widehat{Y}}_2)}=Q_{(Y_1,Y_2)}-Q_{(E_1,E_2)}\ge 0\hfill \end{array}$$

(140)

where the test channel distribution ${\mathbf{P}}_{{\widehat{Y}}_1,{\widehat{Y}}_2|Y_1,Y_2}$ or the joint distribution ${\mathbf{P}}_{{\widehat{Y}}_1,{\widehat{Y}}_2,Y_1,Y_2}$ is induced by the realization of part (b).

The conditional rate distortion function, derived in [25,26], is also required.

Theorem 13.

Ref. [25] (Theorem 1, Thorem 4), [26] Consider a triple of random variables $Y_i:\Omega \to {\mathbb{R}}^{p_i},i=1,2$, $W:\Omega \to \mathbb{W}$, where W is continuous or finite-valued, with joint distribution ${\mathbf{P}}_{Y_1,Y_2,W}$, and marginal distributions ${\mathbf{P}}_{Y_1,Y_2}$ and ${\mathbf{P}}_{Y_i},i=1,2$, the jointly Gaussian distribution $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)}),Q_{(Y_1,Y_2)}\ge 0$ and $Y_i\in G(0,Q_{Y_i}),Q_{Y_i}>0,i=1,2$, respectively. Consider the conditional RDFs $R_{Y_i|W}\left({\Delta }_i\right),i=1,2$ with square error distortion functions $D_{Y_i}(y_i,{\widehat{y}}_i)=\left|\right|y_i-{\widehat{y}}_i{\left|\right|}_{{\mathbb{R}}^{p_i}}^2,i=1,2$. Then, the following hold.

(a) For an arbitrary random variable, $W:\Omega \to \mathbb{W}$, the mutual information $I(Y_i;{\widehat{Y}}_i|W)$ satisfies

$$\begin{array}{c}\hfill I(Y_i;{\widehat{Y}}_i|W)\ge I(Y_i;\mathbf{E}\left\{Y_i|F^{{\widehat{Y}}_i,W}\right\}|W),\hspace{1em}\hspace{1em}i=1,2\end{array}$$

(141)

and the mean square error satisfies

$$\begin{array}{c}\hfill \mathbf{E}\left\{\left|\right|Y_i-{\widehat{Y}}_i{\left|\right|}_{{\mathbb{R}}^{p_i}}^2\right\}\ge \mathbf{E}\left\{\left|\right|Y_i-\mathbf{E}\left\{Y_i|F^{{\widehat{Y}}_i,W}\right\}{\left|\right|}_{{\mathbb{R}}^{p_i}}^2\right\},\hspace{1em}\hspace{1em}i=1,2.\end{array}$$

(142)

Moreover, inequalities in (141) and (142) hold with equality, if there exists a realization of ${\widehat{Y}}_1$ of the test channel distribution ${\mathbf{P}}_{{\widehat{Y}}_i|Y_i,W}$, such that the joint distribution ${\mathbf{P}}_{{\widehat{Y}}_i,Y_i,W}$ satisfies the identity:

$$\begin{array}{c}\hfill {\widehat{X}}_i^{\mathrm{cm}}\doteq \mathbf{E}\left\{Y_i|F^{{\widehat{Y}}_i,W}\right\}=\mathbf{E}\left\{Y_i|F^{{\widehat{Y}}_i}\right\}={\widehat{Y}}_i,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}i=1,2.\end{array}$$

(143)

(b) Suppose ${\mathbf{P}}_{Y_1,Y_2,W}$ is a jointly Gaussian distribution and $W:\Omega \to {\mathbb{R}}^n$ is Gaussian. A realization that achieves the lower bounds of part (a), i.e., satisfies (143), is the Gaussian realization of $(Y_i,W,{\widehat{Y}}_i)$ given by

$$\begin{array}{cc}\hfill & {\widehat{Y}}_i=H_i{Y}_i+\left(I_{p_i}-H_i\right)Q_{Y_i,W}{Q}_W^{†}W+V_i,\hspace{1em}\hspace{1em}i=1,2,\hfill \end{array}$$

(144)

$$\begin{array}{cc}\hfill & V_i\in G(0,Q_{V_1}),\hspace{1em}\hspace{1em}{V}_i\hspace{1em}\hspace{1em}independent of\hspace{1em}\hspace{1em}{Y}_i,\hfill \end{array}$$

(145)

$$\begin{array}{cc}\hfill & H_i{Q}_{Y_i|W}=Q_{Y_i|W}-Q_{E_i}\ge 0,\hfill \end{array}$$

(146)

$$\begin{array}{cc}\hfill & Q_{V_i}=H_i{Q}_{E_i}=H_i{Q}_{Y_i|W}-H_i{Q}_{Y_i|W}{H}_i^T\ge 0,\hfill \end{array}$$

(147)

where ${}^{†}$ denotes the pseudoinverse of a matrix, $E_i$ is the error, that satisfies the structural property,

$$\begin{array}{c}\hfill E_i=Y_i-\mathbf{E}\left\{Y_i|F^{{\widehat{Y}}_i,W}\right\}=Y_i-{\widehat{Y}}_i,\hspace{1em}\hspace{1em}{E}_i\in G(0,Q_{E_i}),\hspace{1em}\hspace{1em}i=1,2.\end{array}$$

(148)

The RDF $R_{Y_i|W}\left({\Delta }_i\right)$ for jointly Gaussian $(Y_1,Y_2,W)$ is characterized by

$$\begin{array}{cc}\hfill & R_{Y_i|W}\left({\Delta }_i\right)=\underset{\mathbf{E}\left\{\left|\right|E_i{\left|\right|}_{{\mathbb{R}}^{p_i}}^2\right\}=\mathrm{trace}\left(Q_{E_i}\right)\le {\Delta }_i}{inf}\frac{1}{2}ln{\left(\frac{det\left(Q_{Y_i|W}\right)}{det\left(Q_{E_i}\right)}\right)}^{+}\in [0,\infty ],\hspace{1em}\hspace{1em}i=1,2,\hfill \end{array}$$

(149)

$$\begin{array}{cc}\hfill & suchthat\hspace{1em}\hspace{1em}{Q}_{{\widehat{Y}}_i}=Q_{Y_i|W}-Q_{E_i}\ge 0,\hfill \end{array}$$

(150)

where the test channel distribution ${\mathbf{P}}_{{\widehat{Y}}_i|Y_i,W}$ or the joint distribution ${\mathbf{P}}_{{\widehat{Y}}_i,Y_i,W}$ is induced by the above realization.

The following is stated as a conjectured, because it is not shown in this paper; it can be shown using Theorem 13.

Conjecture 1.

Conditional RDF of Gaussian sources with arbitrary conditioning RV

Consider a triple of random variables $Y_i:\Omega \to {\mathbb{R}}^{p_i},i=1,2$, $W:\Omega \to \mathbb{W}$, where W is continuous or finite-valued, with joint distribution ${\mathbf{P}}_{Y_1,Y_2,W}$, and marginal distributions ${\mathbf{P}}_{Y_1,Y_2}$ and ${\mathbf{P}}_{Y_i},i=1,2$, the jointly Gaussian distribution $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)}),Q_{(Y_1,Y_2)}\ge 0$ and $Y_i\in G(0,Q_{Y_i}),Q_{Y_i}>0,i=1,2$, respectively. Consider the conditional RDFs $R_{Y_i|W}\left({\Delta }_i\right),i=1,2$ with square error distortion functions $D_{Y_i}(y_i,{\widehat{y}}_i)=\left|\right|y_i-{\widehat{y}}_i{\left|\right|}_{{\mathbb{R}}^{p_i}}^2,i=1,2$.

Then, the following hold.

(a) For an arbitrary random variable, $W:\Omega \to \mathbb{W}$, and ${\widehat{X}}_i^{\mathrm{cm}}$ satisfying (143), the following lower bounds hold.

$$\begin{array}{cc}\hfill I(X_i;{\widehat{X}}_i|W)\ge & I(X_i;{\widehat{X}}_i^{\mathrm{cm}}|W),\hspace{1em}i=1,2\hfill \end{array}$$

(151)

$$\begin{array}{cc}\hfill =& {\int }_{\mathbb{W}}I(X_i;{\widehat{X}}_i^{\mathrm{cm}}|W=w){\mathbf{P}}_W\left(dw\right)\hfill \end{array}$$

(152)

$$\begin{array}{cc}\hfill \ge & \underset{w\in \mathbb{W}}{inf}I(X_i;{\widehat{X}}_i^{\mathrm{cm}}|W=w),\hfill \end{array}$$

(153)

$$\begin{array}{cc}\hfill \mathbf{E}\left[D_{X_i}(X_i,{\widehat{X}}_i)\right]=& {\int }_{\mathbb{W}}\left({\int }_{{\mathbb{R}}^{p_1}\times {\mathbb{R}}^{p_2}}{D}_{X_i}(x_i,{\widehat{x}}_i){\mathbf{P}}_{X_i,{\widehat{X}}_i|W}(x_i,{\widehat{x}}_i|w)\right){\mathbf{P}}_W\left(dw\right)\hfill \end{array}$$

(154)

$$\begin{array}{cc}\hfill =& {\int }_{\mathbb{W}}{\Delta }_i\left(w\right){\mathbf{P}}_W\left(dw\right),\hfill \end{array}$$

(155)

$$\begin{array}{cc}\hfill \ge & {\int }_{\mathbb{W}}{\Delta }_i^{\mathrm{cm}}\left(w\right){\mathbf{P}}_W\left(dw\right),\hfill \end{array}$$

(156)

$$\begin{array}{cc}\hfill \ge & \underset{w\in \mathbb{W}}{inf}{\Delta }_i^{\mathrm{cm}}\left(w\right),i=1,2,\hfill \end{array}$$

(157)

where

$$\begin{array}{c}\hfill {\Delta }_i\left(w\right)\doteq \mathbf{E}\left[D_{X_i}(X_i,{\widehat{X}}_i)\right|W=w],\hspace{1em}{\Delta }_i^{\mathrm{cm}}\left(w\right)\doteq \mathbf{E}\left[D_{X_i}(X_i,{\widehat{X}}_i^{\mathrm{cm}})\right|W=w],\hspace{1em}i=1,2.\end{array}$$

(158)

Moreover, the inequalities in (153), (157), are achieved if,

(i) (141) holds, and

(ii) the mutual information $I(X_i;{\widehat{X}}_i|W=w)$ and ${\Delta }_i\left(w\right)$ for $i=1,2$, are independent of $w\in \mathbb{W}$.

(b) The rate distortion function $R_{X_i|W}\left({\Delta }_i\right)$ for $W:\Omega \to \mathbb{W}$ a continuous or finite-valued, achieves a minimum value if,

(i) $W:\Omega \to {\mathbb{R}}^n$, $n\in {\mathbb{Z}}_{+}$ is Gaussian, and ${\mathbf{P}}_{Y_i,W}$ is jointly Gaussian,

(ii) $({\widehat{X}}_i,X_i,W)$ is given by the realization of Theorem 13.(b) for $i=1,2$.

The characterization of the marginal RDFs $R_{Y_i}\left({\Delta }_i\right),i=1,2$—which are well-known, and can be found in many books—is also needed; the weak realization of the test channel, which follows from Theorem 13 (see also [25]), as a degenerate case, and summarized in the next theorem, is important in this paper.

Theorem 14.

Ref. [25] (Theorem 1, Theorem 4) Consider a tuple of Gaussian random variables $Y_i:\Omega \to {\mathbb{R}}^{p_i},i=1,2$, with $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)})$, $Q_{(Y_1,Y_2)}>0$, $Q_{Y_i}>0$, for $i=1,2$.

For the marginal RDFs $R_{Y_i}\left({\Delta }_i\right),i=1,2$ with square error distortion functions $D_{Y_i}(y_i,{\widehat{y}}_i)=\left|\right|y_i-{\widehat{y}}_i{\left|\right|}_{{\mathbb{R}}^{p_i}}^2$, the statements of Theorem 13 hold with W, generating the trivial information, i.e., $F^W=\{\Omega ,\varnothing \}$. That is, the marginal RDFs $R_{Y_i}\left({\Delta }_i\right)$ are characterized by

$$\begin{array}{cc}\hfill & R_{Y_i}\left({\Delta }_i\right)=\underset{\mathbf{E}\left\{\left|\right|E_i{\left|\right|}_{{\mathbb{R}}^{p_i}}^2\right\}=\mathrm{trace}\left(Q_{E_i}\right)\le {\Delta }_i}{inf}\frac{1}{2}ln{\left(\frac{det\left(Q_{Y_i}\right)}{det\left(Q_{E_i}\right)}\right)}^{+}\in [0,\infty ],\hspace{1em}\hspace{1em}i=1,2,\hfill \end{array}$$

(159)

$$\begin{array}{cc}\hfill & suchthat\hspace{1em}\hspace{1em}{Q}_{{\widehat{Y}}_i}=Q_{Y_i}-Q_{E_i}\ge 0\hfill \end{array}$$

(160)

where the test channel distribution ${\mathbf{P}}_{{\widehat{Y}}_i|Y_i}$ or the joint distribution ${\mathbf{P}}_{{\widehat{Y}}_i,Y_i}$ is induced by the realization

$$\begin{array}{cc}\hfill & {\widehat{Y}}_i=H_i{Y}_i+V_i,\hspace{1em}\hspace{1em}i=1,2,\hfill \end{array}$$

(161)

$$\begin{array}{cc}\hfill & V_i\in G(0,Q_{V_1}),\hspace{1em}\hspace{1em}{V}_i\hspace{1em}\hspace{1em}independent of\hspace{1em}\hspace{1em}{Y}_i,\hfill \end{array}$$

(162)

$$\begin{array}{cc}\hfill & H_i{Q}_{Y_i}=Q_{Y_i}-Q_{E_i}\ge 0,\hfill \end{array}$$

(163)

$$\begin{array}{cc}\hfill & Q_{V_i}=H_i{Q}_{E_i}=H_i{Q}_{Y_i}-H_i{Q}_{Y_i}{H}_i^T\ge 0.\hfill \end{array}$$

(164)

and where $E_i$ is the error that satisfies the structural property

$$\begin{array}{c}\hfill E_i=Y_i-\mathbf{E}\left\{Y_i|F^{{\widehat{Y}}_i}\right\}=Y_i-{\widehat{Y}}_i,\hspace{1em}\hspace{1em}{E}_i\in G(0,Q_{E_i}),\hspace{1em}\hspace{1em}i=1,2.\end{array}$$

(165)

Then, we express the characterization of the joint RDF of Theorem 12, using the canonical variable form, and the canonical correlation coefficients. The special case when $Q_{(E_1,E_2)}$ is is block-diagonal is given in [27].

Theorem 15.

Consider the statement of Theorem 12. Compute the canonical variable form of the tuple of Gaussian random variables $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)}),Q_{Y_i}>0$, according to Algorithm A1. This yields the indices $p_{11}=p_{21}$, $p_{12}=p_{22}$, $p_{13}$, $p_{23}$, and $n=p_{11}+p_{12}=p_{21}+p_{22}$, the diagonal matrix $D_4$ with canonical correlation coefficients $d_{4,i}\in (0,1)$ for $i=1,\dots ,p_{12}$, and decompositions (see Algorithm A1, 1–4)

$$\begin{array}{c}\hfill Q_{Y_1}\hspace{1em}=\hspace{1em}{U}_1{D}_1{U}_1^T,\hspace{1em}\hspace{1em}{Q}_{Y_2}\hspace{1em}=\hspace{1em}{U}_2{D}_2{U}_2^T,\end{array}$$

(166)

with $U_i\in {\mathbb{R}}^{p_i\times p_i}$ orthogonal ($U_i{U}_i^T=I_{p_i}=U_i^T{U}_i$), $i=1,2$, singular-value decomposition of

$$\begin{array}{c}\hfill D_1^{-\frac{1}{2}}{U}_1^T{Q}_{Y_1{Y}_2}{U}_2{D}_2^{-\frac{1}{2}}\hspace{1em}=\hspace{1em}{U}_3{D}_3{U}_4^T,\end{array}$$

(167)

with $U_3\in {\mathbb{R}}^{p_1\times p_1}$, $U_4\in {\mathbb{R}}^{p_2\times p_2}$ orthogonal,

$$\begin{array}{cc}\hfill D_3=& \left(\begin{array}{ccc}{I}_{p_{11}}\hfill & 0\hfill & 0\hfill \\ 0\hfill & D_4\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0\hfill \end{array}\right)\in {\mathbb{R}}^{p_1\times p_2},\hfill \end{array}$$

(168)

$$\begin{array}{cc}\hfill D_4=& \mathrm{Diag}(d_{4,1},...,d_{4,p_{12}})\in {\mathbb{R}}^{p_{12}\times p_{12}},\hspace{1em}\hspace{1em}1>d_{4,1}\ge d_{4,2}\ge \dots \ge d_{4,p_{12}}>0.\hfill \end{array}$$

(169)

Define the new variance matrix of $Q_{(Y_1,Y_2)}$ according to

$$\begin{array}{c}\hfill Q_{\mathrm{cvf}}=\left(\begin{array}{cc}{I}_{p_1}\hfill & D_3\hfill \\ D_3^T\hfill & I_{p_2}\hfill \end{array}\right).\end{array}$$

(170)

Compute the canonical variable form of the tuple of Gaussian error random variables $(E_1,E_2)\in G(0,Q_{(E_1,E_2)})$ of Theorem 12.(b), according to Algorithm A1. This yields the indices ${\overline{p}}_{11}={\overline{p}}_{21}$, ${\overline{p}}_{12}={\overline{p}}_{22}$, ${\overline{p}}_{13}$, ${\overline{p}}_{23}$, and $\overline{n}={\overline{p}}_{11}+{\overline{p}}_{12}={\overline{p}}_{21}+{\overline{p}}_{22}$ and the diagonal matrix ${\overline{D}}_4$ with canonical correlation coefficients ${\overline{d}}_{4,i}\in (0,1)$ for $i=1,\dots ,{\overline{p}}_{12}$, and decompositions (see Algorithm A1, 1–4),

$$\begin{array}{cc}\hfill & Q_{E_1}\hspace{1em}=\hspace{1em}{\overline{U}}_1{\overline{D}}_1{\overline{U}}_1^T,\hspace{1em}\hspace{1em}{Q}_{E_2}\hspace{1em}=\hspace{1em}{\overline{U}}_2{\overline{D}}_2{\overline{U}}_2^T,\hfill \end{array}$$

(171)

$$\begin{array}{cc}\hfill & {\overline{D}}_i=\mathrm{Diag}({\overline{d}}_{i,1},\dots ,{\overline{d}}_{i,p_i})\in {\mathbb{R}}^{p_i\times p_i},\hspace{1em}\hspace{1em}{\overline{d}}_{i,1}\ge {\overline{d}}_{i,2}\ge \dots \ge {\overline{d}}_{i,p_i}>0,\hspace{1em}\hspace{1em}i=1,2,\hfill \end{array}$$

(172)

with ${\overline{U}}_i\in {\mathbb{R}}^{p_i\times p_i}$ orthogonal (${\overline{U}}_i{\overline{U}}_i^T=I_{p_i}={\overline{U}}_i^T{\overline{U}}_i$), $i=1,2$, singular-value decomposition of

$$\begin{array}{c}\hfill {\overline{D}}_1^{-\frac{1}{2}}{\overline{U}}_1^T{Q}_{E_1{E}_2}{\overline{U}}_2{\overline{D}}_2^{-\frac{1}{2}}\hspace{1em}=\hspace{1em}{\overline{U}}_3{\overline{D}}_3{\overline{U}}_4^T,\end{array}$$

(173)

with ${\overline{U}}_3\in {\mathbb{R}}^{p_1\times p_1}$, ${\overline{U}}_4\in {\mathbb{R}}^{p_2\times p_2}$ orthogonal,

$$\begin{array}{cc}\hfill {\overline{D}}_3=& \left(\begin{array}{ccc}{I}_{{\overline{p}}_{11}}\hfill & 0\hfill & 0\hfill \\ 0\hfill & {\overline{D}}_4\hfill & 0\hfill \\ 0\hfill & 0\hfill & 0\hfill \end{array}\right)\in {\mathbb{R}}^{p_1\times p_2},\hfill \end{array}$$

(174)

$$\begin{array}{cc}\hfill {\overline{D}}_4=& \mathrm{Diag}({\overline{d}}_{4,1},...,{\overline{d}}_{4,{\overline{p}}_{12}})\in {\mathbb{R}}^{{\overline{p}}_{12}\times {\overline{p}}_{12}},\hspace{1em}\hspace{1em}1>{\overline{d}}_{4,1}\ge {\overline{d}}_{4,2}\ge \dots \ge {\overline{d}}_{4,{\overline{p}}_{12}}>0.\hfill \end{array}$$

(175)

Define the new variance matrix of $Q_{(E_1,E_2)}$ according to,

$$\begin{array}{c}\hfill {\overline{Q}}_{\mathrm{cvf}}=\left(\begin{array}{cc}{I}_{p_1}\hfill & {\overline{D}}_3\hfill \\ {\overline{D}}_3^T\hfill & I_{p_2}\hfill \end{array}\right).\end{array}$$

(176)

The joint RDF $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$ of Theorem 12. (c) is equivalently characterized by

$$\begin{array}{cc}\hfill & R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)=\underset{Q_{(E_1,E_2)}\ge 0:\overline{n}\in {\mathbb{Z}}_{+},{\sum }_{i=1}^{p_1}{\overline{d}}_{1,i}\le {\Delta }_1,\hspace{1em}{\sum }_{i=1}^{p_2}{\overline{d}}_{2,i}\le {\Delta }_2}{inf}\frac{1}{2}ln{\left(\frac{det\left(D_1\right)det\left(D_2\right)det\left(Q_{\mathrm{cvf}}\right)}{det\left({\overline{D}}_1\right)det\left({\overline{D}}_2\right)det\left({\overline{Q}}_{\mathrm{cvf}}\right)}\right)}^{+}\in [0,\infty ],\hfill \end{array}$$

(177)

$$\begin{array}{cc}\hfill & such that\hspace{1em}\hspace{1em}{Q}_{({\widehat{Y}}_1,{\widehat{Y}}_2)}=Q_{(Y_1,Y_2)}-Q_{(E_1,E_2)}\ge 0,\hfill \end{array}$$

(178)

where

$$\begin{array}{cc}\hfill & det\left(Q_{\mathrm{cvf}}\right)=det(I_{p_1}-D_3{D}_3^T)\hfill \end{array}$$

(179)

$$\begin{array}{cc}\hfill =& \left\{\begin{array}{ccc}1,\hfill & if\hfill & p_{13}>0,\hspace{1em}{p}_{23}>0,\hspace{1em}{p}_{11}=p_{12}=p_{21}=p_{22}=0,\hfill \\ \prod _{i=1}^n\left(1-d_{4,i}^2\right),\hfill & if\hfill & p_{11}=p_{21}=0,\hspace{1em}{p}_{12}=p_{22}=n,\hspace{1em}{p}_{13}\ge 0,\hspace{1em}{p}_{23}\ge 0,\hfill \\ 0,\hfill & if\hfill & p_{11}=p_{21}>0,\hspace{1em}{p}_{12}=p_{22}\ge 0,\hspace{1em}{p}_{13}\ge 0,\hspace{1em}{p}_{23}\ge 0,\hfill \end{array}\right.\hfill \end{array}$$

(180)

$$\begin{array}{cc}\hfill & det\left({\overline{Q}}_{\mathrm{cvf}}\right)=det(I_{p_1}-{\overline{D}}_3{\overline{D}}_3^T)\hfill \end{array}$$

(181)

$$\begin{array}{cc}\hfill =& \left\{\begin{array}{ccc}1,\hfill & if\hfill & {\overline{p}}_{13}>0,\hspace{1em}{\overline{p}}_{23}>0,\hspace{1em}{\overline{p}}_{11}={\overline{p}}_{12}={\overline{p}}_{21}={\overline{p}}_{22}=0,\hfill \\ \prod _{i=1}^{\overline{n}}\left(1-{\overline{d}}_{4,i}^2\right),\hfill & if\hfill & {\overline{p}}_{11}={\overline{p}}_{21}=0,\hspace{1em}{\overline{p}}_{12}={\overline{p}}_{22}=\overline{n},\hspace{1em}{\overline{p}}_{13}\ge 0,\hspace{1em}{\overline{p}}_{23}\ge 0,\hfill \\ 0,\hfill & if\hfill & {\overline{p}}_{11}={\overline{p}}_{21}>0,\hspace{1em}{\overline{p}}_{12}={\overline{p}}_{22}\ge 0,\hspace{1em}{\overline{p}}_{13}\ge 0,\hspace{1em}{\overline{p}}_{23}\ge 0,\hfill \end{array}\right.\hfill \end{array}$$

(182)

Moreover, a necessary condition for $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)<+\infty$ is ${\overline{p}}_{11}={\overline{p}}_{21}=0$.

Proof. 

First, apply Algorithm A1 to the tuple of Gaussian random variables $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)})$, and then to the Gaussian random variables $(E_1,E_2)\in G(0,Q_{(E_1,E_2)})$ of Theorem 12. (b). This gives (166)–(176). Then, (177) follows from (139) using (166)–(176), and the standard properties of a determinant of a matrix. The remaining equations are obtained from (166)–(176). The last statement follows from the values of $det\left({\overline{Q}}_{\mathrm{cvf}}\right),det\left(Q_{\mathrm{cvf}}\right)$.

□

Remark 7.

By Theorem 15, since $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)\in [0,\infty ]$, by (180), it suffices to consider $Q_{(Y_1,Y_2)}>0$, which implies $p_{11}=p_{21}=0$, $Q_{Y_i}>0,i=1,2$. Furthermore, to ensure $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)\in [0,\infty )$, it suffices to also consider $Q_{(E_1,E_2)}>0$, which implies that ${\overline{p}}_{11}={\overline{p}}_{21}=0$, $Q_{E_i}>0,i=1,2$.

From Theorem 15, the next corollary directly follows which identifies the subset of the distortion region such that Gray’s lower bound [28], $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)\ge R_{Y_2|Y_1}\left({\Delta }_2\right)+R_{Y_1}\left({\Delta }_1\right)$ holds with equality.

Corollary 2.

Consider the statement of Theorem 15, and without loss of generality, assume $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)})$, with $Q_{(Y_1,Y_2)}>0$ (and hence $Q_{Y_i}>0,i=1,2$).

The joint RDF $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$ with $D_1,D_2,{\overline{D}}_1,{\overline{D}}_2,Q_{\mathrm{cvf}},{\overline{Q}}_{\mathrm{cvf}}$, defined in Theorem 15, and corresponding to $p_{11}=p_{21}={\overline{p}}_{11}={\overline{p}}_{21}=0$ satisfies the lower bound ($R_{Y_2|Y_1}\left({\Delta }_2\right)$ is obtained from Theorem 13 by letting $W=Y_1$.),

$$\begin{array}{cc}\hfill R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)\ge & R_{Y_2|Y_1}\left({\Delta }_2\right)+R_{Y_1}\left({\Delta }_1\right)\hfill \\ \hfill =& \underset{\mathbf{E}\left\{\left|\right|E_2{\left|\right|}_{{\mathbb{R}}^{p_2}}^2\right\}=\mathrm{trace}\left(Q_{E_2}\right)\le {\Delta }_2}{inf}\frac{1}{2}ln\left(\frac{det\left(Q_{Y_2|Y_1}\right)}{det\left(Q_{E_2}\right)}\right)\hfill \end{array}$$

(183)

$$\begin{array}{cc}\hfill & \hspace{1em}\hspace{1em}+\underset{\mathbf{E}\left\{\left|\right|E_1{\left|\right|}_{{\mathbb{R}}^{p_1}}^2\right\}=\mathrm{trace}\left(Q_{E_1}\right)\le {\Delta }_1}{inf}\frac{1}{2}ln\left(\frac{det\left(Q_{Y_1}\right)}{det\left(Q_{E_1}\right)}\right)\hfill \end{array}$$

(184)

$$\begin{array}{cc}\hfill =& \underset{{\sum }_{i=1}^{p_2}{\overline{d}}_{2,i}\le {\Delta }_2}{inf}\frac{1}{2}ln\left(\frac{det\left(D_2\right)det\left(Q_{\mathrm{cvf}}\right)}{det\left({\overline{D}}_2\right)}\right)+\underset{{\sum }_{i=1}^{p_1}{\overline{d}}_{1,i}\le {\Delta }_1}{inf}\frac{1}{2}ln\left(\frac{det\left(D_1\right)}{det\left({\overline{D}}_1\right)}\right)\hfill \end{array}$$

(185)

that is, ${\overline{p}}_{12}={\overline{p}}_{22}=0$.

Moreover, the inequalities (184) and (185) hold with the equalities, on the strictly positive surface ${\mathcal{D}}_C(Y_1,Y_2)$, defined by

$$\begin{array}{c}\hfill {\mathcal{D}}_C(Y_1,Y_2)=\left\{({\Delta }_1,{\Delta }_2)\in [0,\infty ]\times [0,\infty ]|Q_{(Y_1,Y_2)}-Q_{(E_1,E_2)}>0\right\}.\end{array}$$

(186)

Proof. 

The lower bound (183) is due to Gray [28]. The equality in (184) follows by using the values of the rate distortion functions in the right hand side of (183). Equality (185) follows from the singular value decomposition of the matrices given in Theorem 15, using $Q_{Y_2|Y_1}=Q_{Y_2}-Q_{Y_2,Y_1}{Q}_{Y_1}^{-1}{Q}_{Y_2,Y_1}^T$. To establish the equalities, note that (177) with $det\left({\overline{Q}}_{\mathrm{cvf}}\right)=1$ equivalently ${\overline{p}}_{12}={\overline{p}}_{22}=0$, is precisely (185). Moreover, it can be easily verified that ${\overline{p}}_{12}={\overline{p}}_{22}=0$ for the distortion region ${\mathcal{D}}_C(Y_1,Y_2)$. □

4.2. Wyner’s Lossy Common Information of Correlated Gaussian Vectors

Derived in this section are the characterizations of $C_{GW}(Y_1,Y_2;{\Delta }_1,{\Delta }_2)$ via Theorem 2, for jointly Gaussian random variables with square-error distortion, as well as $C_W(Y_1,Y_2)$ via Theorem 3.

Definition 7.

Wyner’s lossy common information of a tuple of Gaussian multivariate random variables. Consider a tuple of jointly Gaussian random variables $Y_1:\Omega \to {\mathbb{R}}^{p_1}\equiv {\mathbb{Y}}_1$, $Y_2:\Omega \to {\mathbb{R}}^{p_2}\equiv {\mathbb{Y}}_2$, in terms of the notation $(Y_1,Y_2)\in G(0,Q_{(Y_1,Y_2)}),Q_{Y_i}>0,i=1,2$, and square error distortion functions between $(y_1,y_2)$, and its reproduction $({\widehat{y}}_1,{\widehat{y}}_2)$, given by

$$\begin{array}{c}\hfill D_{Y_1}(y_1,{\widehat{y}}_1)=\left|\right|y_1-{\widehat{y}}_1{\left|\right|}_{{\mathbb{R}}^{p_1}}^2,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{D}_{Y_2}(y_2,{\widehat{y}}_2)=\left|\right|y_2-{\widehat{y}}_2{\left|\right|}_{{\mathbb{R}}^{p_2}}^2\end{array}$$

(187)

where $\left|\right|·{\left|\right|}_{{\mathbb{R}}^{p_i}}^2$ denotes Euclidean distances on ${\mathbb{R}}^{p_i},i=1,2$.

(a) Wyner’s common information (information definition) of the tuple of Gaussian random variables $(Y_1,Y_2)$ is defined by the expression

$$\begin{array}{c}\hfill C(Y_1,Y_2)=\underset{W:\Omega \to {\mathbb{R}}^n,\hspace{1em}(F^{Y_1},F^{Y_2}|F^W)\in \mathrm{CIG}}{inf}\hspace{1em}I(Y_1,Y_2;W)\in [0,\infty ].\end{array}$$

(188)

Call any random variable W as defined above such that $(Y_1,Y_2,W)\in G(0,Q_{(Y_1,Y_2,W)})$ and $(F^{Y_1},F^{Y_2}|F^W)\in \mathrm{CIG}$ a state of the tuple $(Y_1,Y_2)$.

If there exists a random variable $W^{\ast }:\Omega \to {\mathbb{R}}^{n^{\ast }}$ with $n^{\ast }\in {\mathbb{Z}}_{+}=\{1,2,\dots ,\}$ which attains the infimum; thus, if $C(Y_1,Y_2)=I(Y_1,Y_2;W^{\ast })$, then call that random variable a minimal information state of the tuple $(Y_1,Y_2)$.

(b) Wyner’s common information (operational definition) is defined for a tuple of strictly positive real numbers $\gamma =({\gamma }_1,{\gamma }_2)\in {\mathbb{R}}_{++}\times {\mathbb{R}}_{++}=(0,\infty )\times (0,\infty )$ such that, for all $0\le ({\Delta }_1,{\Delta }_2)\le \gamma$,

$$\begin{array}{cc}\hfill C_{GW}(Y_1,Y_2;& {\Delta }_1,{\Delta }_2)=C_W(Y_1,Y_2)=C(Y_1,Y_2),\hfill \\ \hfill & for\hspace{1em}\hspace{1em}({\Delta }_1,{\Delta }_2)\in {\mathcal{D}}_W=\left\{({\Delta }_1,{\Delta }_2)\in [0,\infty ]\times [0,\infty ]|0\le ({\Delta }_1,{\Delta }_2)\le \gamma \right\}\hfill \end{array}$$

(189)

provided identity (15) holds, i.e., $R_{Y_1|W}\left({\Delta }_1\right)+R_{Y_2|W}\left({\Delta }_2\right)+I(Y_1,Y_2;W)=R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$.

By the above definition, the problem of calculating Wyner’s lossy common information via (18) is decomposed into the characterization of $C(Y_1,Y_2)$ such that identity (15) is satisfied. This follows from the fact that the only difference between $C_W(Y_1,Y_2)$ and $C(Y_1,Y_2)$ is the specification of the region ${\mathcal{D}}_W$ such that $C_{GW}(Y_1,Y_2;{\Delta }_1,{\Delta }_2)=C_W(Y_1,Y_2)=C(Y_1,Y_2)$ is constant for $({\Delta }_1,{\Delta }_2)\in {\mathcal{D}}_W$.

In the next theorem, we make use of the characterizations of the various rate distortion functions, and the test channel realizations to identify subsets of the rate region that lie on the Pangloss plane, and are consistent with the characterization of Viswanatha, Akyol and Rose [12] (Theorem 1, Equations (19) and (20)).

Theorem 16.

Consider a tuple $(Y_1,Y_2)$ of Gaussian random variables in the canonical variable form of Definition 1. Restrict the attention to the correlated parts of these random variables, as defined in Theorem 8, by (77)–(79). Furthermore, consider a realization of the random variables $(Y_1,Y_2)$ which induces the family of measures ${\mathbf{P}}_{\mathbf{ci}}\subseteq {\mathcal{P}}_{min}^{CIG}$, as defined in Corollary 1, by (84)–(88).

Then, the following hold.

(a) The joint rate distortion function $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$ of $(Y_1,Y_2)$ with square error distortion satisfies

$$\begin{array}{cc}\hfill & R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)=\underset{{\sum }_{j=1}^n{\Delta }_{1,j}\le {\Delta }_1,{\sum }_{j=1}^n{\Delta }_{2,j}\le {\Delta }_2}{inf}\frac{1}{2}\sum _{j=1}^{n}ln\left(\frac{(1-d_j^2)}{{\Delta }_{1,j}{\Delta }_{2,j}}\right),({\Delta }_1,{\Delta }_2)\in {\mathcal{D}}_C(Y_1,Y_2),\hfill \end{array}$$

(190)

$$\begin{array}{cc}\hfill & \mathrm{trace}\left(Q_{E_1}\right)=\mathbf{E}\left|\right|Y_1-{\widehat{Y}}_1{\left|\right|}_{{\mathbb{R}}^n}^2=\sum _{j=1}^n{\Delta }_{1,j},\hspace{1em}\hspace{1em}\mathrm{trace}\left(Q_{E_2}\right)=\mathbf{E}\left|\right|Y_2-{\widehat{Y}}_2{\left|\right|}_{{\mathbb{R}}^n}^2=\sum _{j=1}^n{\Delta }_{2,j}\hfill \end{array}$$

(191)

where ${\mathcal{D}}_C(Y_1,Y_2)$ is a strictly positive surface, defined by

$$\begin{array}{c}\hfill {\mathcal{D}}_C(Y_1,Y_2)=\left\{({\Delta }_1,{\Delta }_2)\in [0,\infty ]\times [0,\infty ]|Q_{(Y_1,Y_2)}-Q_{(E_1,E_2)}>0\right\}\end{array}$$

(192)

and where $Q_{(E_1,E_2)}$ is the variance of the errors $E_i=Y_i-{\widehat{Y}}_i,i=1,2$, with parameters ${\overline{p}}_{11}={\overline{p}}_{21}={\overline{p}}_{12}={\overline{p}}_{22}=0$, and ${\overline{p}}_{13}={\overline{p}}_{23}=n$.

The conditional rate distortion functions $R_{Y_i|W}\left({\Delta }_i\right)$ of $Y_i$ conditioned on W with square error distortion, and mutual information $I(Y_1,Y_2;W)$ satisfy

$$\begin{array}{cc}\hfill & R_{Y_1|W}\left({\Delta }_1\right)=\underset{\mathrm{trace}\left(Q_{E_1}\right)\le {\Delta }_1}{inf}\frac{1}{2}ln{\left(\frac{det(I-D^{1/2}{Q}_W^{-1}{D}^{1/2})}{det\left(Q_{E_1}\right)}\right)}^{+},\hspace{1em}\hspace{1em}{\Delta }_1\in [0,\infty )\hfill \end{array}$$

(193)

$$\begin{array}{cc}\hfill & R_{Y_2|W}\left({\Delta }_2\right)=\underset{\mathrm{trace}\left(Q_{E_2}\right)\le {\Delta }_2}{inf}\frac{1}{2}ln{\left(\frac{det(I-D^{1/2}{Q}_W{D}^{1/2})}{det\left(Q_{E_2}\right)}\right)}^{+},\hspace{1em}\hspace{1em}{\Delta }_2\in [0,\infty )\hfill \end{array}$$

(194)

$$\begin{array}{cc}\hfill & I(Y_1,Y_2;W)=\frac{1}{2}ln{\left(\frac{det(I-D^2)}{det\left([I-D^{1/2}{D}_W^{-1}{D}^{1/2}][I-D^{1/2}{D}_W{D}^{1/2}]\right)}\right)}^{+}.\hfill \end{array}$$

(195)

where $\mathrm{trace}\left(Q_{E_i}\right),i=1,2$ are defined as in (191).

(b) The representations of reproductions (the reader may verify that the realization satisfies the conditions given in Viswanatha, Akyol and Rose [12], Theorem 1, Equations (19) and (20)), $({\widehat{Y}}_1,{\widehat{Y}}_2)$ of $(Y_1,Y_2)$ at the output of decoder 1 and decoder 2, which achieve the joint rate distortion functions $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2),R_{Y_i|W}\left({\Delta }_i\right),i=1,2$ of part (a), are

$$\begin{array}{cc}\hfill & Y_1=D^{1/2}{Q}_W^{-1}W+Z_1,\hfill \end{array}$$

(196)

$$\begin{array}{cc}\hfill & Y_2=D^{1/2}W+Z_2,\hfill \end{array}$$

(197)

$$\begin{array}{cc}\hfill & {\widehat{Y}}_1=Y_1-Q_{E_1}{(I-D^{1/2}{Q}_W^{-1}{D}^{1/2})}^{-1}{Z}_1+V_1,\hfill \end{array}$$

(198)

$$\begin{array}{cc}\hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=D^{1/2}{Q}_W^{-1}W+A_1{Z}_1+V_1,\hfill \end{array}$$

(199)

$$\begin{array}{cc}\hfill & {\widehat{Y}}_2=Y_2-Q_{E_2}{(I-D^{1/2}{Q}_W{D}^{1/2})}^{-1}{Z}_2+V_2,\hfill \end{array}$$

(200)

$$\begin{array}{cc}\hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=D^{1/2}W+A_2{Z}_2+V_2,\hfill \end{array}$$

(201)

$$\begin{array}{cc}\hfill & Z_1\in G(0,(I-D^{1/2}{Q}_W^{-1}{D}^{1/2})),\hspace{1em}\hspace{1em}{Z}_2\in G(0,(I-D^{1/2}{Q}_W{D}^{1/2})),\hspace{1em}\hspace{1em}W\in G(0,Q_W),\hfill \end{array}$$

(202)

$$\begin{array}{cc}\hfill & Q_{E_1}=\mathbf{E}\left\{(Y_1-{\widehat{Y}}_1){(Y_1-{\widehat{Y}}_1)}^T\right\},\hspace{1em}\hspace{1em}{Q}_{E_2}=\mathbf{E}\{(Y_2-{\widehat{Y}}_2){(Y_2-{\widehat{Y}}_2)}^T\},\hfill \end{array}$$

(203)

$$\begin{array}{cc}\hfill & V_1\in G(0,Q_{E_1}{A}_1^T),\hspace{1em}\hspace{1em}{V}_2\in G(0,Q_{E_2}{A}_2^T),\hfill \end{array}$$

(204)

$$\begin{array}{cc}\hfill & A_1=I-Q_{E_1}{(I-D^{1/2}{Q}_W^{-1}{D}^{1/2})}^{-1},\hspace{1em}\hspace{1em}{A}_2=I-Q_{E_2}{(I-D^{1/2}{Q}_W{D}^{1/2})}^{-1},\hfill \end{array}$$

(205)

$$\begin{array}{cc}\hfill & Q_{E_i}=U_i{\Lambda }_i{U}_i^T,\hspace{1em}\hspace{1em}{\Lambda }_i=\mathrm{Diag}({\Delta }_{i,1},\dots ,{\Delta }_{i,n})\in {\mathbb{R}}^{n\times n},\hspace{1em}\hspace{1em}{U}_i{U}_i^T=U_i^T{U}_i=I,\hspace{1em}\hspace{1em}i=1,2,\hfill \end{array}$$

(206)

$$\begin{array}{cc}\hfill & Q_{Y_1|W}=I-D^{1/2}{Q}_W^{-1}{D}^{1/2}=U_1{\Lambda }_{Y_1|W}{U}_1^T,\hspace{1em}\hspace{1em}{\Lambda }_{Y_1|W}=\mathrm{Diag}({\Lambda }_{Y_1|W,1},\dots ,{\Lambda }_{Y_1|W,n}),\hfill \end{array}$$

(207)

$$\begin{array}{cc}\hfill & Q_{Y_2|W}=I-D^{1/2}{Q}_W{D}^{1/2}=U_2{\Lambda }_{Y_2|W}{U}_2^T,\hspace{1em}\hspace{1em}{\Lambda }_{Y_2|W}=\mathrm{Diag}({\Lambda }_{Y_2|W,1},\dots ,{\Lambda }_{Y_2|W,n}),\hfill \end{array}$$

(208)

$$\begin{array}{cc}\hfill & (V_1,V_2,Z_1,Z_2,W),\hspace{1em}\hspace{1em}\hspace{1em}are independent\hfill \end{array}$$

(209)

and are parameterized by $Q_W\in {\mathbf{Q}}_{\mathbf{W}}$, where ${\mathbf{Q}}_{\mathbf{W}}$ is defined by the set of Equation (82).

Moreover, the joint distribution ${\mathbf{P}}_{Y_1,Y_2,{\widehat{Y}}_1,{\widehat{Y}}_2,W}$ satisfies (the reader may verify that conditions (210) are identical to Viswanatha, Akyol and Rose [12], Theorem 1, Equations (19) and (20)) for rates that lie on the Pangloss plane)

$$\begin{array}{cc}\hfill & {\mathbf{P}}_{{\widehat{Y}}_1,{\widehat{Y}}_2|W}={\mathbf{P}}_{{\widehat{Y}}_1|W}{\mathbf{P}}_{{\widehat{Y}}_2|W},\hspace{1em}\hspace{1em}{\mathbf{P}}_{Y_1,Y_2|{\widehat{Y}}_1,{\widehat{Y}}_2,W}={\mathbf{P}}_{Y_1,Y_2|{\widehat{Y}}_1,{\widehat{Y}}_2},\hfill \end{array}$$

(210)

$$\begin{array}{cc}\hfill & {\mathbf{P}}_{Y_1,Y_2,{\widehat{Y}}_1,{\widehat{Y}}_2,W}={\mathbf{P}}_{{\widehat{Y}}_1|Y_1,W}{\mathbf{P}}_{{\widehat{Y}}_2|Y_2,W}{\mathbf{P}}_{Y_1|W}{\mathbf{P}}_{Y_2|W}{\mathbf{P}}_W,\hfill \end{array}$$

(211)

$$\begin{array}{cc}\hfill & {\mathbf{P}}_{Y_1,Y_2,{\widehat{Y}}_1,{\widehat{Y}}_2,W}={\mathbf{P}}_{Y_1|{\widehat{Y}}_1}{\mathbf{P}}_{Y_2|{\widehat{Y}}_2}{\mathbf{P}}_{{\widehat{Y}}_1|W}{\mathbf{P}}_{{\widehat{Y}}_2|W}{\mathbf{P}}_W.\hfill \end{array}$$

(212)

(c) Consider part (a) and the realization of part (b). Then, $R_{Y_1|W}\left({\Delta }_1\right)+R_{Y_2|W}\left({\Delta }_2\right)+I(Y_1,Y_2;W)$$=R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$ on the subset ${\mathcal{D}}_C(Y_1,Y_2)$, defined by (192) such that (210) holds.

(d) Suppose $Q_W=Q_{W^{\ast }}\in {\mathbf{Q}}_W$ is diagonal, i.e., $Q_{W^{\ast }}=\mathrm{Diag}(Q_{W_1^{\ast }},\dots ,Q_{W_n^{\ast }}),\hspace{1em}{d}_i\le Q_{W_i^{\ast }}\le d_i^{-1},\forall i$. Then, the conditional RDFs $R_{Y_i|W^{\ast }}\left({\Delta }_i\right)$ are given by

$$\begin{array}{cc}\hfill R_{Y_1|W^{\ast }}\left({\Delta }_1\right)=& \underset{{\sum }_{j=1}^n{\Delta }_{1,j}={\Delta }_1}{inf}\frac{1}{2}\sum _{j=1}^{n}ln\left(\frac{(1-d_j/Q_{W_j^{\ast }})}{{\Delta }_{1,j}}\right),\hfill \end{array}$$

(213)

$$\begin{array}{cc}\hfill R_{Y_2|W^{\ast }}\left({\Delta }_2\right)=& \underset{{\sum }_{j=1}^n{\Delta }_{2,j}={\Delta }_2}{inf}\frac{1}{2}\sum _{j=1}^{n}ln\left(\frac{(1-d_j{Q}_{W_j^{\ast }})}{{\Delta }_{2,j}}\right),\hfill \end{array}$$

(214)

and the optimal ${\Delta }_{1,j},{\Delta }_{2,j}$ are obtained from the water-filling equations,

$$\begin{array}{cc}\hfill & {\Delta }_{1,j}=\left\{\begin{array}{cc}\lambda ,& \lambda <1-d_j/Q_{W_j^{\ast }}\hspace{1em}\\ 1-d_j,& \lambda \ge 1-d_j/Q_{W_j^{\ast }},\end{array}\right.\hspace{1em}{\Delta }_1\in [0,\infty ),\hfill \end{array}$$

(215)

$$\begin{array}{cc}\hfill & {\Delta }_{2,j}=\left\{\begin{array}{cc}\lambda ,& \lambda <1-d_j{Q}_{W_j^{\ast }}\hspace{1em}\\ 1-d_j,& \lambda \ge 1-d_j{Q}_{W_j^{\ast }},\end{array}\right.\hspace{1em}{\Delta }_2\in [0,\infty ).\hfill \end{array}$$

(216)

and the representations of part (b) hold, with $Q_{E_i},Q_{Y_i|W^{\ast }}$ diagonal matrices.

Proof. 

(a) Since the attention is restricted to the correlated parts of these random variables, as defined in Theorem 8, by (77)–(79), then the statements of joint RDF $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$ of part (a) are a special case of Theorem 12. (c), and obtained from Corollary 2. Similarly, expressions (193)–(195) follow from (13). However, as demonstrated shortly, these also follow, from the derivation of part (b). (b) Recall that the joint rate distortion function is achieved by a jointly Gaussian distribution ${\mathbf{P}}_{Y_1,Y_2,{\widehat{Y}}_1,{\widehat{Y}}_2}$ such that the average square-error distortions are satisfied. Consider the realization of the random variables $(Y_1,Y_2)$ which induce the family of measures ${\mathbf{P}}_{\mathbf{ci}}\subseteq {\mathcal{P}}_{min}^{CIG}$, as defined in Corollary 1, by (84)–(88). By properties of mutual information, then

$$\begin{array}{cc}\hfill I(Y_1,Y_2;{\widehat{Y}}_1,{\widehat{Y}}_2)=& H(Y_1,Y_2)-I(Y_1,Y_2|{\widehat{Y}}_1,{\widehat{Y}}_2)\hfill \end{array}$$

(217)

$$\begin{array}{cc}\hfill =& H(Y_1,Y_2)-H\left(Y_1\right|{\widehat{Y}}_1,{\widehat{Y}}_2,Y_2)-H\left(Y_2\right|{\widehat{Y}}_1,{\widehat{Y}}_2)\hfill \end{array}$$

(218)

$$\begin{array}{cc}\hfill =& H(Y_1,Y_2)-H\left(Y_2\right|{\widehat{Y}}_1,{\widehat{Y}}_2,Y_1)-H\left(Y_1\right|{\widehat{Y}}_1,{\widehat{Y}}_2)\hfill \end{array}$$

(219)

$$\begin{array}{cc}\hfill \ge & H(Y_1,Y_2)-H\left(Y_1\right|{\widehat{Y}}_1)-H\left(Y_2\right|{\widehat{Y}}_2),\hspace{1em}\hspace{1em}cond. reduces entropy,\hfill \end{array}$$

(220)

$$\begin{array}{cc}\hfill =& \frac{1}{2}\sum _{i=1}^{n}ln(1-d_i^2)+nln\left(2\pi e\right)-H\left(Y_1\right|{\widehat{Y}}_1)-H\left(Y_2\right|{\widehat{Y}}_2)\hfill \\ \hfill \ge & \frac{1}{2}\sum _{i=1}^{n}ln(1-d_i^2)+nln\left(2\pi e\right)-\frac{1}{2}\sum _{i=1}^{n}ln\left({\Delta }_{1,i}\right)-\frac{1}{2}nln\left(2\pi e\right)\hfill \end{array}$$

(221)

$$\begin{array}{cc}\hfill & -\frac{1}{2}\sum _{i=1}^{n}ln\left({\Delta }_{2,i}\right)-\frac{1}{2}nln\left(2\pi e\right),\hspace{1em}\hspace{1em}maximum entropy of Gaus. dist.\hfill \end{array}$$

(222)

$$\begin{array}{cc}\hfill =& \frac{1}{2}\sum _{i=1}^{n}ln\left(\frac{(1-d_i^2)}{{\Delta }_{1,i}{\Delta }_{2,i}}\right)\hfill \end{array}$$

(223)

where ${\sum }_{i=1}^n{\Delta }_{1,i}=\mathbf{E}\left[\right||Y_1-{\widehat{Y}}_1{\left|\right|}_{{\mathbb{R}}^n}^2]\le {\Delta }_1$ and ${\sum }_{i=1}^n{\Delta }_{2,i}=\mathbf{E}\left[\right||Y_2-{\widehat{Y}}_2{\left|\right|}_{{\mathbb{R}}^n}^2]\le {\Delta }_2$. The average distortion satisfies

$${\Delta }_1\ge \mathbf{E}\left[\right||Y_1-{\widehat{Y}}_1{\left|\right|}_{{\mathbb{R}}^n}^2]\ge \mathbf{E}[\left|\right|Y_1-\mathbf{E}\left[Y_1\right|F^{{\widehat{Y}}_1}]{\left|\right|}_{{\mathbb{R}}^n}^2],$$

(224)

$${\Delta }_2\ge \mathbf{E}\left[\right||Y_1-{\widehat{Y}}_1{\left|\right|}_{{\mathbb{R}}^n}^2]\ge \mathbf{E}[\left|\right|Y_1-\mathbf{E}\left[Y_1\right|F^{{\widehat{Y}}_1}]{\left|\right|}_{{\mathbb{R}}^n}^2].$$

(225)

Furthermore,

$$\begin{array}{cc}\hfill & if\hspace{1em}\hspace{1em}{\mathbf{P}}_{Y_1,Y_2|{\widehat{Y}}_1,{\widehat{Y}}_2}={\mathbf{P}}_{Y_1|{\widehat{Y}}_1}{\mathbf{P}}_{Y_2|{\widehat{Y}}_2}\hspace{1em}\hspace{1em}then inequality (220) holds with equality,\hfill \end{array}$$

(226)

$$if\hspace{1em}\hspace{1em}\mathbf{E}\left[Y_1\right|F^{{\widehat{Y}}_1}]={\widehat{Y}}_1\hspace{1em}\hspace{1em}then inequality (224) holds with equality,$$

(227)

$$\begin{array}{cc}\hfill & if\hspace{1em}\hspace{1em}\mathbf{E}\left[Y_2\right|F^{{\widehat{Y}}_2}]={\widehat{Y}}_2\hspace{1em}\hspace{1em}then inequality (225) holds with equality.\hfill \end{array}$$

(228)

It can be verified that the representations (196)–(209) satisfy ${\mathbf{P}}_{Y_1,Y_2|{\widehat{Y}}_1,{\widehat{Y}}_2}={\mathbf{P}}_{Y_1|{\widehat{Y}}_1}{\mathbf{P}}_{Y_2|{\widehat{Y}}_2}$, $\mathbf{E}\left[Y_1\right|F^{{\widehat{Y}}_1}]={\widehat{Y}}_1$, $\mathbf{E}\left[Y_2\right|F^{{\widehat{Y}}_2}]={\widehat{Y}}_2$, and that all inequalities become equalities. The decomposition of the joint distribution according to (210) follows from the representations of $({\widehat{Y}}_1,{\widehat{Y}}_2)$, and similarly for (211) and (112). The conditional RDFs $R_{Y_i|W}\left({\Delta }_i\right),i=1,2$ are shown as above. (c) This is easily verified, because (210) holds and hence rates lie on the Pangloss plane, for the strictly positive surface, ${\mathcal{D}}_C(Y_1,Y_2)$. (d) This follows directly from parts (a)–(c). □

Remark 8.

We should emphasize that Theorem 16 does not fully characterize the Pangloss plane, i.e., the subset of the rate regions such that $R_{Y_1|W}\left({\Delta }_1\right)+R_{Y_2|W}\left({\Delta }_2\right)+I(Y_1,Y_2;W)=R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$ is larger than ${\mathcal{D}}_C(Y_1,Y_2)$. To determine the entire set that characterizes the Pangloss plane, we need to consider the rate distortion function (177) with (178), and the general realization (40). We do not pursue this further, because it requires the closed-form solution of $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$, which is currently an open and challenging problem, and beyond the scope of this paper. We should mention that the analysis of the scalar-valued Gaussian example in [12,13], i.e., when $p_1=p_2=1$, made use of closed-form expression of $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$ due to [13].

Proof 

(Proof of Theorem 6). One way to prove the statement is to compute the characterizations of the rate distortion functions $R_{Y_i}\left({\Delta }_i\right),R_{Y_i|W}\left({\Delta }_i\right),i=1,2$ and $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$, using the realization of the random variables $(Y_1,Y_2)$ which induce the family of measures ${\mathbf{P}}_{\mathbf{ci}}\subseteq {\mathcal{P}}_{min}^{CIG}$, as defined in Corollary 1, by (84)–(88). In view of Definition 7. (b), it suffices to verify that identity (15) holds, i.e., $R_{Y_1|W}\left({\Delta }_1\right)+R_{Y_2|W}\left({\Delta }_2\right)+I(Y_1,Y_2;W)=R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$ for $({\Delta }_1,{\Delta }_2)\in {\mathcal{D}}_W$, for the choice $W=W^{\ast }\in G(0,I)$ which achieves the minimum in (188) (i.e., due to Theorem 11. (b)).

Similar to Theorem 16, it can be shown that the conditional RDFs $R_{Y_i|W^{\ast }}\left({\Delta }_i\right),i=1,2$ are given by

$$R_{Y_1|W^{\ast }}\left({\Delta }_1\right)=\underset{{\sum }_{j=1}^n{\Delta }_{1,j}\le {\Delta }_1}{inf}\frac{1}{2}\sum _{j=1}^{n}ln\left(\frac{(1-d_j)}{{\Delta }_{1,j}}\right),\hspace{1em}\hspace{1em}{W}^{\ast }\in G(0,I)$$

(229)

$$R_{Y_2|W^{\ast }}\left({\Delta }_2\right)=\underset{{\sum }_{j=1}^n{\Delta }_{2,j}\le {\Delta }_1}{inf}\frac{1}{2}\sum _{j=1}^{n}ln\left(\frac{(1-d_j)}{{\Delta }_{2,j}}\right),\hspace{1em}\hspace{1em}{W}^{\ast }\in G(0,I),$$

(230)

$$\begin{array}{cc}\hfill & \mathbf{E}\left|\right|Y_1-{\widehat{Y}}_1{\left|\right|}_{{\mathbb{R}}^n}^2=\sum _{j=1}^n{\Delta }_{1,j},\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\mathbf{E}\left|\right|Y_2-{\widehat{Y}}_2{\left|\right|}_{{\mathbb{R}}^n}^2=\sum _{j=1}^n{\Delta }_{2,j}.\hfill \end{array}$$

(231)

The pay-off of the joint RDF $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$ in (190) is related to the pay-offs of the conditional RDFs $R_{Y_1|W}\left({\Delta }_1\right),R_{Y_2|W}\left({\Delta }_2\right)$, and $C(Y_1,Y_2)=I(Y_1,Y_2;W^{\ast })$ in (102), via the identity

$$\frac{1}{2}\sum _{j=1}^{n}ln\left(\frac{(1-d_j^2)}{{\Delta }_{1,j}{\Delta }_{2,j}}\right)=\frac{1}{2}\sum _{j=1}^{n}ln\left(\frac{(1-d_j)}{{\Delta }_{1,j}}\right)+\frac{1}{2}\sum _{j=1}^{n}ln\left(\frac{(1-d_j)}{{\Delta }_{2,j}}\right)+\frac{1}{2}\sum _{i=1}^{n}ln\left(\frac{1+d_i}{1-d_i}\right).$$

(232)

For $({\Delta }_1,{\Delta }_2)\in {\mathcal{D}}_W$ defined by (47), it then follows from (232), the identity

$$R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)=\underset{{\sum }_{j=1}^n{\Delta }_{1,j}\le {\Delta }_1,{\sum }_{j=1}^n{\Delta }_{2,j}\le {\Delta }_2}{inf}\frac{1}{2}\sum _{j=1}^{n}ln\left(\frac{(1-d_j^2)}{{\Delta }_{1,j}{\Delta }_{2,j}}\right)$$

(233)

$$=R_{Y_1|W^{\ast }}\left({\Delta }_1\right)+R_{Y_2|W^{\ast }}\left({\Delta }_2\right)+I(Y_1,Y_2;W^{\ast }),\hspace{1em}\hspace{1em}({\Delta }_1,{\Delta }_2)\in {\mathcal{D}}_W\hspace{1em}\hspace{1em}defined by (47).$$

(234)

This completes the proof. □

4.3. Applications to Problems of the Literature [15,16,17]

The next two corollaries illustrate the application of the results developed in this paper to the optimization problems analyzed in [15,16,17].

Corollary 3.

Applications to problems in [15]

Consider the Gaussian secure source coding and Wyner’s common information [15], defined by the optimization problem [15] (see Equation (18), Section IV.B),

$$\begin{array}{c}\hfill arg\underset{{\mathbf{P}}_{Y_1,Y_2,W}:{\mathbf{P}}_{Y_1,Y_2|W}={\mathbf{P}}_{Y_1|W}{\mathbf{P}}_{Y_2|W}}{min}\left\{\lambda I(Y_1;W)+I(Y_1,Y_2;W)\right\},\hspace{1em}\hspace{1em}\lambda \in [0,\infty )\end{array}$$

(235)

where the tuple $(Y_1,Y_2)$ are the zero mean jointly Gaussian, and $W:\Omega \to \mathbb{W}$ is continuous or discrete-valued random variable (The derivation of the formula for (235) in [15] makes use of rate distortion functions, [15] (Equation (47))).

Then, the following hold.

For any jointly distributed random variables $(Y_1,Y_2,W)$ that minimize the expression in (235), there exists a jointly Gaussian triple $(Y_1,Y_2,W)$ such that $W:\to {\mathbb{R}}^n$ is a Gaussian random variable, which achieves the same minimum value.

Moreover, the following characterization of (235) holds.

$$\begin{array}{cc}\hfill & arg\underset{{\mathbf{P}}_{Y_1,Y_2,W}:{\mathbf{P}}_{Y_1,Y_2|W}={\mathbf{P}}_{Y_1|W}{\mathbf{P}}_{Y_2|W}}{min}\left\{\lambda I(Y_1;W)+I(Y_1,Y_2;W)\right\}\hfill \\ \hfill & =\hspace{1em}\hspace{1em}arg\underset{Q_W\in {\mathbf{Q}}_{\mathbf{W}}:{\mathbf{Q}}_{\mathbf{W}}defined by (82)}{min}\{\frac{\lambda }{2}ln\frac{1}{det(I-D^{1/2}{Q}_W{D}^{1/2})}+\frac{1}{2}\sum _{i=1}^{n}ln(1-d_i^2)\hfill \end{array}$$

(236)

$$\begin{array}{c}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-\frac{1}{2}ln(det\left([I-D^{1/2}{D}_W^{-1}{D}^{1/2}][I-D^{1/2}{D}_W{D}^{1/2}]\right))\}\hfill \\ =\hspace{1em}\hspace{1em}arg\underset{Q_W=D_W\in {\mathbf{Q}}_{\mathbf{W}}:{\mathbf{Q}}_{\mathbf{W}}defined by (82)}{min}\{\frac{\lambda }{2}\sum _{i=1}^{n}ln\left({[1-\frac{d_i}{q_i}]}^{-1}\right)+\frac{1}{2}\sum _{i=1}^{n}ln(1-d_i^2)\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-\frac{1}{2}ln\left([1-\frac{d_i}{q_i}][1-q_i{d}_i]\right)\}\hfill \end{array}$$

(237)

where

$$\begin{array}{c}\hfill Q_W=D_W=\mathrm{Diag}(q_1,q_2,\dots ,q_n)\in {\mathbf{Q}}_{\mathbf{W}},\hspace{1em}\hspace{1em}{q}_1\ge q_2\ge \dots \ge q_n>0,\hspace{1em}\hspace{1em}{\mathbf{Q}}_{\mathbf{W}}=\left(82\right).\end{array}$$

(238)

Proof. 

By the use Theorem 9. (c), it suffices to restrict attention to jointly Gaussian random variables $(Y_1,Y_2,W)$. Transform the tuple $(Y_1,Y_2)$ in the canonical variable form of Definition 1. Restrict attention to the correlated parts of these random variables, as defined in Theorem 8, by (77)–(79), and consider the realization of the transformed random variables of Corollary 1. Then, the value of $I(Y_1;W)+I(Y_2,Y_2;W)$ is identical to the value of the same expression, evaluated using the realization of Corollary 1. By simple evaluation, using the realization of Corollary 1,

$$\begin{array}{cc}\hfill \lambda I(Y_1;W)& +I(Y_2,Y_2;W)=\frac{\lambda }{2}ln\frac{1}{det(I-D^{1/2}{Q}_W^{-1}{D}^{1/2})}\hfill \\ \hfill & +\frac{1}{2}\sum _{i=1}^{n}ln(1-d_i^2)-\frac{1}{2}ln(det\left([I-D^{1/2}{Q}_W^{-1}{D}^{1/2}][I-D^{1/2}{Q}_W{D}^{1/2}]\right))\hfill \end{array}$$

(239)

and it is parameterized by $Q_W\in {\mathbf{Q}}_{\mathbf{W}}$, where ${\mathbf{Q}}_{\mathbf{W}}$ is defined by the set of Equation (82). By Hadamard’s determinant inequality, an achievable lower bound on the first right-hand side term of (239), holds if $Q_W\in {\mathbf{Q}}_{\mathbf{W}}$ and $(I-D^{1/2}{Q}_W^{-1}{D}^{1/2})$ is a diagonal matrix, and this lower bound is achieved by a diagonal $Q_W\in {\mathbf{Q}}_{\mathbf{W}}$. Furthermore, by recalling the derivation of Theorem 11, an achievable lower bound on the second right-hand side term of (239) holds, i.e., of $I(Y_1,Y_2;W)$, when $Q_W\in {\mathbf{Q}}_{\mathbf{W}}$ is diagonal. Hence, both lower bounds are achieved simultaneously, by $Q_W\in {\mathbf{Q}}_{\mathbf{W}}$ and $Q_W$ a diagonal matrix. Then, an achievable lower bound on (239) is obtained, if $Q_W$ is specified by (238). □

The remaining optimization problem in (237) is easily carried out, and hence omitted.

Corollary 4 illustrates the application of the results developed in this paper to the Gaussian relaxed Wyner’s common information [16,17] (Definition 2 and Section III).

Corollary 4.

Applications to problems in [16,17]

Consider the Gaussian relaxed Wyner’s common information considered in [16,17] (see Definition 2 and Section III of [17])

$$\begin{array}{cc}\hfill C_{\gamma }(Y_1,Y_2)=& \underset{{\mathbf{P}}_{W|Y_1,Y_2}:\hspace{1em}I(Y_1;Y_2|W)\le \gamma }{min}I(Y_1,Y_2;W)\hfill \end{array}$$

(240)

where the tuple $(Y_1,Y_2)$ are zero mean jointly Gaussian, and $W:\Omega \to \mathbb{W}$ is continuous or discrete-valued random variable (the value of (240) computed in [16,17], Theorem 4, is different from (241); moreover, the derivation in [16,17], Section III.A, is different from the derivation presented below). Then

$$\begin{array}{c}\hfill C_{\gamma }(Y_1,Y_2)=C(Y_1,Y_2)=\left(19\right),\hspace{1em}\hspace{1em}\hspace{1em}\forall \gamma \in (0,\infty ).\end{array}$$

(241)

Proof. 

By the use Theorem 9. (c), it suffices to restrict the attention to jointly Gaussian random variables $(Y_1,Y_2,W)$. By Proposition 3 or Corollary 1, there exists a family of realizations of $(Y_1,Y_2)$ parameterized by a Gaussian random variable W, which induces conditional independence ${\mathbf{P}}_{Y_1,Y_2|W}={\mathbf{P}}_{Y_1|W}{\mathbf{P}}_{Y_2|W}$, and hence the lower bound $I(Y_1,Y_2;W)\ge H(Y_1,Y_2)-H\left(Y_1\right|W)-H\left(Y_2\right|W)$ is achieved, i.e., the constraint in (240) is always satisfied, because the minimizer is such that $I(Y_1;Y_2|W)=0$, i.e., the constraint is not active. Hence, the general solution of (240) is the one given in Theorem 4. □

Remark 9.

Corollary 4 implies that the definition of the relaxed Gaussian Wyner’s common information considered in [16,17] (see Definition 2 and Section III of [17]) should be replaced by ${min}_{{\mathbf{P}}_{W|Y_1,Y_2}:\hspace{1em}I(Y_1;Y_2|W)=\gamma }I(Y_1,Y_2;W)$, i.e., the inequality is replaced by an equality, so that the constraint is active for all $\gamma \in (0,\infty )$.

4.4. Characterization and Parameterization of the Gray and Wyner Rate Region by Jointly Gaussian RVs

Derived in this section, for jointly Gaussian random variables with square-error distortion, using [1] ((4) of page 1703, Equation (42)), i.e., (12), and the RDFs, $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2),$$R_{Y_i|W}\left({\Delta }_i\right)$, $R_{Y_i}\left({\Delta }_i\right),i=1,2$, of Theorems 12, 13, 14, and Theorem 9, are:

(1)

Theorem 5—the characterizations of the rate region ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$, and

(2)

The characterization of rates that lie on Pangloss Plane.

Proof 

(Proof of Theorem 5). (a) This follows from the realization of random variables that induce Gaussian measures, by repeating Theorem 9, without requiring W makes $Y_1$ and $Y_2$ conditionally independent, i.e., the stated realization with random variables $Z_1$ and $Z_2$ correlated, achieves a lower bound on $I(Y_1,Y_2;W)$, among all random variables W. (b) The stated characterization (42) follows from the discussion prior to the Theorem, i.e., by an application of [1] ((4) of page 1703, Equation (42)), i.e., $T({\alpha }_1,{\alpha }_2)$, and Theorem 9, Theorem 13, which imply that the infimum in $T({\alpha }_1,{\alpha }_2)$ is over the parameterized set of jointly Gaussian random variables $(Y_1,Y_2,W)\in G(0,Q_{(Y_1,Y_2,W)})$ with joint distribution (68). From part (a), then (43) follows. (44) is due to identity $I(Y_1,Y_2;W)=H(Y_1,Y_2)-H\left(Y_1\right|Y_2,W)-H\left(Y_2\right|W)$ and that the values $R_{Y_1|W}\left({\Delta }_1\right)$ and $R_{Y_2|W}\left({\Delta }_2\right)$ depend only on $Q_{Y_1|W}$ and $Q_{Y_2|W}$, and the errors (see Theorem 13). □

Theorem 17 gives the parameterization of a subset of the Pangloss Plane, as a degenerate case of Theorem 5.

Theorem 17.

Consider the statement of Theorem 5.

(a) Rate triples $(R_0,R_1,R_2)$ that lie on the Pangloss Plane, is determined by the subset of the rate region ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$ of Theorem 5. (b), such that the joint distribution ${\mathbf{P}}_{W,Y_1,Y_2,{\widehat{Y}}_1,{\widehat{Y}}_2}$ satisfies the conditions,

$$\begin{array}{c}\hfill {\mathbf{P}}_{{\widehat{Y}}_1,{\widehat{Y}}_2|W}={\mathbf{P}}_{{\widehat{Y}}_1|W}{\mathbf{P}}_{{\widehat{Y}}_2|W},\hspace{1em}\hspace{1em}{\mathbf{P}}_{Y_1,Y_2|{\widehat{Y}}_1,{\widehat{Y}}_2,W}={\mathbf{P}}_{Y_1,Y_2|{\widehat{Y}}_1,{\widehat{Y}}_2}.\end{array}$$

(242)

Specifically, the Pangloss Plane is characterized by

$$\begin{array}{cc}\hfill T^G({\alpha }_1,{\alpha }_2)=& \underset{(Y_1,Y_2,W)\in G(0,Q_{(Y_1,Y_2,W)}of\left(30\right),\left(40\right)}{inf}\{I(Y_1,Y_2;W)\hfill \end{array}$$

$$\begin{array}{cc}\hfill & +{\alpha }_1{R}_{Y_1|W}\left({\Delta }_1\right)+{\alpha }_2{R}_{Y_2|W}\left({\Delta }_2\right)\}\hfill \\ \hfill =& \underset{Q_{Y_1|Y_2,W},Q_{Y_2|W}}{inf}\{\frac{1}{2}ln\frac{det\left(Q_{(Y_1,Y_2)}\right)}{det\left(Q_{Y_1|Y_2,W}\right)det\left(Q_{Y_2|W}\right)}\hfill \end{array}$$

(243)

$$+{\alpha }_1{R}_{Y_1|W}\left({\Delta }_1\right)+{\alpha }_2{R}_{Y_2|W}\left({\Delta }_2\right)\}$$

(244)

such that

$$\begin{array}{c}{\mathbf{P}}_{W,Y_1,Y_2,{\widehat{Y}}_1,{\widehat{Y}}_2}satisfies\left(242\right),and its marginals induce the test channels\hfill \\ oftheRDFs,R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2),R_{Y_i|W}\left({\Delta }_i\right),R_{Y_i}\left({\Delta }_i\right),i=1,2,of Theorems 12, 13, 14.\hfill \end{array}$$

(245)

(b) A subset of the rate triple $(R_0,R_1,R_2)$ that lie on the Pangloss Plane is determined by the restriction of part (a) to $(Y_1,Y_2|W)\in \mathrm{CIG}$, i.e.,

$$\begin{array}{cc}\hfill T^{GCI}({\alpha }_1,{\alpha }_2)=& \underset{(Y_1,Y_2,W)\in G(0,Q_{(Y_1,Y_2,W)},\hspace{1em}(Y_1,Y_2|W)\in \mathrm{CIG}}{inf}\{I(Y_1,Y_2;W)\hfill \end{array}$$

$$\begin{array}{cc}\hfill & +{\alpha }_1{R}_{Y_1|W}\left({\Delta }_1\right)+{\alpha }_2{R}_{Y_2|W}\left({\Delta }_2\right)\}\hfill \\ \hfill =& \underset{Q_{Y_1|W},Q_{Y_2|W}}{inf}\{\frac{1}{2}ln\frac{det\left(Q_{(Y_1,Y_2)}\right)}{det\left(Q_{Y_1|W}\right)det\left(Q_{Y_2|W}\right)}\hfill \end{array}$$

(246)

$$+{\alpha }_1{R}_{Y_1|W}\left({\Delta }_1\right)+{\alpha }_2{R}_{Y_2|W}\left({\Delta }_2\right)\}$$

(247)

such that (245) holds, where $I(Y_1,Y_2;W)=H(Y_1,Y_2)-H\left(Y_1\right|W)-H\left(Y_2\right|W)$, and $R_{Y_i|W}\left({\Delta }_i\right),$$i=1,2$ are given in Theorem 13. (c).

Proof. 

(a) Condition (242) characterizes the rates $(R_0,R_1,R_2)\in {\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$ that lie on the Pangloss plane, and these are derived in [12] (Theorem 1, Equations (19) and (20)). Hence, the statement follows from Theorem 5.

(b) That (246) defines a subset of ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$, follows from the fact that the set of joint distributions $(Y_1,Y_2,W)\in G(0,Q_{(Y_1,Y_2,W)}$ and $(Y_1,Y_2|W)\in \mathrm{CIG}$ is a subset of the set of joint distributions $(Y_1,Y_2,W)\in G(0,Q_{(Y_1,Y_2,W)})$. Moreover, by part (a) and $(Y_1,Y_2|W)\in \mathrm{CIG}$ implies $I(Y_1,Y_2;W)=H(Y_1,Y_2)-H\left(Y_1\right|Y_2,W)-H\left(Y_2\right|W)=H(Y_1,Y_2)-H\left(Y_1\right|W)-H\left(Y_2\right|W)$ and that the values $R_{Y_1|W}\left({\Delta }_1\right)$ and $R_{Y_2|W}\left({\Delta }_2\right)$ only depend on $Q_{Y_1|W}$ and $Q_{Y_2|W}$, and the errors, as shown in Theorem 13. (c). Hence, the statement holds. □

From Theorem 16 and Theorem 17. (b) follows a simpler parameterization of rates that lie on the Pangloss Plane of the Gray–Wyner rate region ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$, when $(Y_1,Y_2)$ are in canonical variable form.

Corollary 5.

Consider the statement of Theorem 16 with square error distortion functions, i.e., a tuple $(Y_1,Y_2)$ is the canonical variable form.

A subset of the rate triple $(R_0,R_1,R_2)$ that lie on the Pangloss Plane corresponding to the restriction $(Y_1,Y_2|W)\in \mathrm{CIG}$, is determined from

$$\begin{array}{cc}\hfill T_{\mathrm{cvf}}^{CIG}({\alpha }_1,{\alpha }_2)=& \underset{{\mathbf{Q}}_{\mathbf{W}}}{inf}\left\{I(Y_1,Y_2;W)+{\alpha }_1{R}_{Y_1|W}\left({\Delta }_1\right)+{\alpha }_2{R}_{Y_2|W}\left({\Delta }_2\right)\right\}\hfill \\ \hfill =& \underset{{\mathbf{Q}}_{\mathbf{W}}}{inf}\{\frac{1}{2}\sum _{i=1}^{n}ln(1-d_i^2)-\frac{1}{2}ln(det\left([I-D^{1/2}{Q}_W^{-1}{D}^{1/2}][I-D^{1/2}{Q}_W{D}^{1/2}]\right))\hfill \end{array}$$

(248)

$$+{\alpha }_1{R}_{Y_1|W}\left({\Delta }_1\right)+{\alpha }_2{R}_{Y_2|W}\left({\Delta }_2\right)\}$$

(249)

such that (245) holds, where $0\le {\alpha }_i\le 1,i=1,2,{\alpha }_1+{\alpha }_2\ge 1$, and where $R_{Y_i|W}\left({\Delta }_i\right),i=1,2$ are given in Theorem 16, and the infimum is taken over $Q_W\in {\mathbf{Q}}_{\mathbf{W}}$, defined by the set of Equation (82).

Proof. 

The stated characterization (248) is the application of [1] ((4) of page 1703, Equation (42)) and the results of Theorem 16 and Theorem 17. (b). □

In view of Theorem 1 (i.e., Theorem 8 in [1]), additional parameterizations of the Gray–Wyner rate region ${\mathcal{R}}_{GW}({\Delta }_1,{\Delta }_2)$ directly follow from the expressions already derived, i.e., the joint RDF $R_{Y_1,Y_2}({\Delta }_1,{\Delta }_2)$ of Theorem 12, the conditional RDFs $R_{Y_1|W}\left({\Delta }_1\right),R_{Y_2|W}\left({\Delta }_2\right)$ of Theorem 13, the maginal RDFs $R_{Y_1}\left({\Delta }_1\right),R_{Y_2}\left({\Delta }_2\right)$ of Theorem 14, and the values of $I(Y_1,Y_2;W)$, based on Gaussian realization of Theorem 5. (a).

5. Conclusions

This paper formulates the classical Gray and Wyner source coding for a simple network with a tuple of multivariate, correlated Gaussian random variables, with square-error fidelity at the two decoders, from the geometric approach of a Gaussian random variables and the weak stochastic realization of correlated Gaussian random variables. This approach leads to a parameterization of the Gray–Wyner rate region, with respect to variance matrix of the jointly Gaussian triple $(Y_1,Y_2,W)$, where W a Gaussian auxiliary random variable. However, much remains to be achieved, from the computation point of view, for this problem, and to exploit the new approach to other multi-user problems of information theory.