%% Conditional uncorrelation (normal distribution), chained linear predictions: examples
% © 2022, Antonio Sala Piqueras. All rights reserved.
% Video-presentation: http://personales.upv.es/asala/YT/V/condnocoEN.html
% This code ran without errors in Matlab R2022a
% Table of Contents
% Conditional Independence (revision)
% Condicionally uncorrelated Variables (normal case)
% Example 1: sum of two random variables
% Example 2
%     2.1: Incorrelated, but "condicionally" correlated
%     2.2: correlated but condicionally uncorrelated
%         Bayesian network representation
%
%
%% Conditional Independence (revision)
%
% Suppose we have three random variables a, b, c.
% \bullet b and c are said to be "conditionally independent when a=a_m " if
%    p(b|c=c_m,a=a_m)=p(b|a=a_m)
% or
%  p(c|b=b_m,a=a_m)=p(c|a=a_m).
% Equivalently, if p(b,c|a=a_m)=p(b|a=a_m)\cdot p(c|a=a_m).
% \bullet If b and c and are conditionally independent given any possible observation of a, then we say that they are "conditionally independent given a".
%
% This relationship allows expressing probabilistic models in graphical diagrams such as:
%
% which are called "Bayesian networks" in a general case.
%
%% Condicionally uncorrelated Variables (normal case)
% Statistical independence is harder to prove than incorrelation (at least, approximately, given a sufficient amount of data). No correlation is equivalent to inability to set up any useful "linear" predictor.
% If we have a symmetric covariance matrix:
% \Sigma:=\left(\begin{array}{ccc}\Sigma_{aa}&\Sigma_{ab}&\Sigma_{ac}\\\Sigma_{ba}&\Sigma_{bb}&\Sigma_{bc}\\\Sigma_{ca}&\Sigma_{cb}&\Sigma_{cc}\end{array}\right)
% then, the best (minimum error variance) linear prediction of (b,c) given a is:
% \left(\begin{array}{c}\hat b_{|a}\\\hat c_{|a}\end{array}\right)=\left(\begin{array}{c}\Sigma_{ba}\\\Sigma_{ca}\end{array}\right)\Sigma_{aa}^{-1} \ \cdot \ a
% with a prediction error variance given by:
% \Sigma_{err}:=\left(\begin{array}{cc}\Sigma_{bb}&\Sigma_{bc}\\ \Sigma_{cb}&\Sigma_{cc}\end{array}\right)-\left(\begin{array}{c}\Sigma_{ba}\\\Sigma_{ca}\end{array}\right)\Sigma_{aa}^{-1}\left(\begin{array}{cc}\Sigma_{ba}^T&\Sigma_{ca}^T\end{array}\right) = \left(\begin{array}{cc}\Sigma_{bb}-\Sigma_{ba}\Sigma_{aa}^{-1}\Sigma_{ba}^T&\Sigma_{bc}-\Sigma_{ba}\Sigma_{aa}^{-1}\Sigma_{ca}^T\\ \Sigma_{cb}-\Sigma_{ca}\Sigma_{aa}^{-1}\Sigma_{ba}^T&\Sigma_{cc}-\Sigma_{ca}\Sigma_{aa}^{-1}\Sigma_{ca}^T\end{array}\right)
% Hence, if \Sigma_{err\,(1,2)}=\Sigma_{bc}-\Sigma_{ba}\Sigma_{aa}^{-1}\Sigma_{ca}^T=0, i.e., \Sigma_{bc}=\Sigma_{ba}\Sigma_{aa}^{-1}\Sigma_{ca}^T, then \Sigma_{err} is diagonal, so extra information about e:=c-\hat c_{|a} is of no use to predict (linearly) b with greater precision than the already available \hat b_{|a}, and likewise happens with predictions of c with additional info on b. They are conditionally incorrelated.
% In the case of normal distribution, this would serve to "exactly"  determine the conditional distribution in mean and variance, so we will call them "conditionally uncorrelated", but, obviously, it is only true in certain cases, for example with the extra assumption of normality. Otherwise, the true "conditional" probability distribution may not have the mean and variance from the best linear prediction formulae.
% In fact, in the multivariate normal distribution case, uncorrelation implies independence, so these variables would also be conditionally independent. So, the focus on "uncorrelation" in this material is just for numerical/illustrative reasons.
%
%% Example 1: sum of two random variables
% The "sum" a=b+c, assuming variance of b=2, variance of c=1, b y c independent, has a covariance matrix:
% E[ab]=E[(b+c)b]=E[b^2]+E[bc]=E[b^2],
% E[ac]=E[(b+c)c]=E[bc]+E[c^2]=E[c^2],
% E[a^2]=E[(b+c)^2]=E[b^2+c^2+2bc]=E[b^2]+E[c^2]

VCabc=[3 2 1;2 2 0;1 0 1]
Resid=VCabc(2:3,2:3)-VCabc(2:3,1)*inv(VCabc(1,1))*VCabc(1,2:3)
[V,D]=eig(Resid)

% The null eigenvalue indicates that there is a "deterministic" relationship in there (clearly, if I know the sum and one of the variables, the remaining one can be deterministically obtained).
%
%
%% Example 2

Sigma=@(s_bc) [2 3 4;3 10 s_bc;4 s_bc 20]; %cov(b,c) adjustable parameter
S0=Sigma(0) %b and c are uncorrelated.
eig(Sigma(0)) %must be positive def.

%% 2.1: Incorrelated, but "condicionally" correlated

S0(2:3,2:3)% diagonal
vzaerrbc=S0(2:3,2:3)-S0(2:3,1)*inv(S0(1,1))*S0(1,2:3) %NOT diagonal
eig(vzaerrbc)% some randomness remains, not like in the "sum" example.

% Given the conditional correlation, the best prediction of "c", given "a" and "b", will have a residual prediction error variance given by:

S0(3,3)-S0(3,1:2)*inv(S0(1:2,1:2))*S0(1:2,3) 

% which is better than "20" (equal to the marginal) with only the knowledge of "b", and better than the variance of the best prediction knowing only "a", which is:

S0(3,3)-S0(3,1)*inv(S0(1,1))*S0(1,3) 

% *Note that the result is the same if I used the information on "b" with the residual covariance arising from a previous use of "a":

vzaerrbc(2,2)-vzaerrbc(2,1)*inv(vzaerrbc(1,1))*vzaerrbc(1,2)

%
%% 2.2: correlated but condicionally uncorrelated

bc_ci=S0(2,1)*inv(S0(1,1))*S0(1,3)
Sci=Sigma(bc_ci)
eig(Sci) %es def+
Resid_given_a=Sci(2:3,2:3)-Sci(2:3,1)*inv(Sci(1,1))*Sci(1,2:3) %sale diagonal, cond. incorrelada

%
%% Bayesian network representation
% \bulletOption 1:

bestpred_given_a=Sci(2:3,1)*inv(Sci(1,1))

%
% \bulletOption 2:

bestpred_given_b=Sci([1 3],2)*inv(Sci(2,2))
Resid_given_b=Sci([1 3],[1 3])-Sci([1 3],2)*inv(Sci(2,2))*Sci(2,[1 3]) 

%
% \bulletOption 3:

bestpred_given_c=Sci([1 2],3)*inv(Sci(3,3))
Resid_given_c=Sci([1 2],[1 2])-Sci([1 2],3)*inv(Sci(3,3))*Sci(3,[1 2]) 

%
%
%
%
%
%
%
%
%
%