Discussion between user351579 and sagivd

2:22 PM

A: Representing pairwise-independent but not independent occurrences with venn diagram

Your Venn diagram is correct. $A$ is separate from the others, and because $B$ and $C$ are not independent of each other, they intersect.

sagivd

as i said, it is possible that $I(A;B,C) \ne 0$ since i only have pair-wise independency, and this cannot be seen from the diagram, so how can this be correct?

user351579

Please confirm, A is independent from both B and C, right? Then it's impossible for anything in A to be in either or both of B and C.

sagivd

As far as venn diagrams go - i think you are correct. But the tricky part is the pairwise independency. check out wiki-link for more info.

user351579

I think I misunderstood. Please wait.

@sagivd If you really meant $A,B$ and $A,C$ are pairwise independent as defined in the wiki article, then the notations $I(A;B)=0$ and $I(A;C)=0$ are incorrect.

If the notation is correct and they are pairwise independent, then either A or either of B and C is 0.

@sagivd I think the problem is in your notation.

sagivd

I am sure that if A and B are independent then $I(A;B)=0$. I am not sure about the converse, but i had the feeling that it is true as well. why do you think this is not true? I use the first definition of mutual information from here, and you can see that it is zero only if X,Y are independent.

user351579

2:23 PM

@sagivd A,B being pairwise independent means P(A)P(B)=P(intersection(A,B))

A,B being independent means I(A;B)=0, they are almost completely different

In fact A and B can only be both independent and pairwise independent if either or both of them has probability 0.

I think you meant independent, not pairwise independent, so just use my answer.

@sagivd Either your terminology or notation is incorrect.

1 hour later…

sagivd

3:41 PM

i dont understand this sentence: In fact A and B can only be both independent and pairwise independent if either or both of them has probability 0.

user351579

If A and B are independent, their intersection has probability 0. The rest is obvious.

sagivd

i think you misunderstood

I(A;B) is not intersect

it stands for mutual information

can i add an image to the chat?

user351579

I mean, why did you write pairwise independent if your notation means independent

yes, you can

sagivd

how?

user351579

upload...

sagivd

3:47 PM

cant seem to find it

i use this definitionhttps://en.wikipedia.org/wiki/Mutual_information#Definition_of_mutual_‌information

Mutual information

In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables. More specifically, it quantifies the "amount of information" (in units such as bits) obtained about one random variable, through the other random variable. The concept of mutual information is intricately linked to that of entropy of a random variable, a fundamental notion in information theory, that defines the "amount of information" held in a random variable. Not limited to real-valued random variables like the correlation coefficient...

user351579

so I(A;B) means the probabiity of the intersection of A and B?

sagivd

and you can see in the expression that if X, Y are independent (meaning $P(X,Y)=P(X)P(Y)$

no, it is a deterministic expression

user351579

that's pairwise independent

sagivd

let me explain it whole:

lets assume 3 occurences - A, B, C

if P(A,B)=P(A)P(B), P(B,C)=P(B)P(C) and P(A,C)=P(A)P(C) than all occurences are pairwise independent

BUT

it does not mean that P(A,B,C)=P(A)P(B)P(C)

user351579

Mutual information is intersect.

sagivd

3:51 PM

it can be seen as the intersect of the entropy, but this is not the same

are you familiar with entropy and mutual information?

user351579

I'm familiar with probability theory

I assume mutual information here is mutual occurence in prob. theory

right?

sagivd

not quite. i am not that fluent in information theory to give a good explenation.

but if you have A,B, each with their own entropy, which is a measure for "in-order" of an RV

than the mutual information is how much information one provides on the other

in-order is not a good expression

disorder is better

i dont mean to disrespect, but if you are not familiar with the subject i cannot explain it thoroughly.. i dont have time and i will probably mislead you a lot :)

user351579

wait, just define pairwise independent, is it I(..)=0? I'll try to figure it out

sagivd

this is the definition i believe:

lets assume 3 occurences - A, B, C
if P(A,B)=P(A)P(B), P(B,C)=P(B)P(C) and P(A,C)=P(A)P(C) than all occurences are pairwise independent

user351579

what is P(..)? is it probability?

if it's P, not I, then you should write P in the question, as not to confuse.

sagivd

4:04 PM

P stands for probability, I stands for mutual information

if A, B are independent than P(A,B)=P(A)P(B) and also I(A;B)=0