Discussion between user351579 and sagivd - 2016-07-15

« first day last day (14 days later) »

11:51 AM

en.wikipedia.org/wiki/Conditional_mutual_information check here

and here is better

Interaction information

The interaction information (McGill 1954), or amounts of information (Hu Kuo Ting, 1962) or co-information (Bell 2003) is one of several generalizations of the mutual information, and expresses the amount information (redundancy or synergy) bound up in a set of variables, beyond that which is present in any subset of those variables. Unlike the mutual information, the interaction information can be either positive or negative. This confusing property has likely retarded its wider adoption as an information measure in machine learning and cognitive science. These functions, their negativity and...

Unlike the mutual information, the interaction information can be either positive or negative

But you said mutual information.

If you meant interaction information, your terminology is invalid.

interaction information is defined by mutual information, check the definitino

But you are asking about mutual information, right?

yes, i did

Then it can't be negative.

11:55 AM

i have not seen this term before (interation information)

Then why are you bringing up interaction information in a question about mutual information?

since this is related

If the $I(...)$s in your question represents mutual information, your question has already been answered.

I also haven't seen the term before.

these are the facts that i have, if you disagree - tell me:

1. I(A;B)=I(A;B|C)+II(A;B;C) (double I stands for interaction information)

2. I(A;B;C) can be positive or negative

3. since we know I(A;B)=0 (in the quesion) we get I(A;B|C)= -I(A;B;C)

my bad

II(A;B;C) can be negative

and II again here: I(A;B|C)= -II(A;B;C)

4. and finally, we know that mutual information is positive, so II(A;B;C) must be negative in this case

(or zero)

Please wait, I'm trying to understand.

12:14 PM

I'd suggest an intuitive Venn diagram (which only includes mutual information) like the one in your question, or the ones in my answer, or the one in the Wikipedia article on conditional mutual information. But in the intersection between three RVs you can indicate that the II is negative (image upcoming)

user image

This is not the scenario in the question.

It is an example for demonstrative purposes only.

The numbers in the areas represent the number of bits in that area.

So basically it is a normal Venn diagram with "negative II indicators".

If your calculation only involves MI then ignore the indicators.

In the scenario depicted in the question, the MI of A, B, and C is 0, so their II is also 0.

In your scenario, I(A;B|C) is zero. (We are talking about MIs here.)

To compare, here is the diagram of your scenario:

user image

@sagivd Hello?

hi, sorry, i was solving some other questions

one second, ill read

why do you say the the MI of A,B and C is zero?

we only "know" that the MI of A-B, and A-C is zero.

12:31 PM

In the diagram.

quoting: "In the scenario depicted in the question, the MI of A, B, and C is 0, so their II is also 0."

why do you say the MI of A,B and C is zero?

Venn diagrams are completely valid if we are talking about MI.

i think you have a very good solution

Just think of it like this: (image upcoming, might take a while)

how do you upload an image?

12:33 PM

"upload..." beside "send"

i dont have it.. maybe chrome issue?

I use Firefox.

edge has no button as well..

oh well

I didn't know Chrome had an issue about that.

what i meant to say is that i think your solution is good

in my question we only know that I(A;B)=0 and I(A;C)=0

if we say that I(A;B|C) != 0

than we must say that II(A;B;C) <0

12:38 PM

user image

this was I(A;B)=I(A;B|C)+II(A;B;C)=0

I just want to clarify.

The coins each represent a bit.

yes

The number near each letter is the number of bits in the corresponding RV.

i understand this

12:39 PM

We see, each area has a non-negative number of bits.

So in the question scenario, (image upcoming)

what im saying if we make the middle coin red (for example), for signifying its negative, than we get I(A;B)=0

but I(A;B|C)!=0

which is definately a possibility

user image

The indicator only applies for II.

If you are dealing with MI, you ignore its negative II indicator (shall we call it NI3?)

So in the 2nd demonstrative diagram above I(A;B) is still 2.

And I(A;B|C) is 1.

true

In the question scenario, I(A;B;C) is very clearly 0.

I is II?

12:46 PM

We don't need an NI3 in the question scenario.

I'm sure I wrote correctly.

you cannot ignore it

why

you ignore the mutual dependency of A-B-C

lets say A=B XOR C

B ~ ber(1/2)

The only dependency between A, B, and C is between B and C.

C~ber(1/2)

12:47 PM

you can continue.

What's ber?

first option - lets look at A

bernouli

it means 0.5 chance its 1 and 0.5 chance its 0

both B and C, ok?

first option - lets look at A

ok

it is also 0.5 chance 1 and 0.5 chance 0

right?

ok

you can draw it in a prob. table

now lets look at A|B.

12:49 PM

wait

it is still 0.5 1 and 0.5 0

no mutual information here, only probability

waiting..

ok

what are you waiting for?

you said wait :)
nevermind im continuing

if we also look at A|C, it is still random with ber(1/2)

just like A|B

BUT

if we look at A|B,C, A is already known, right?

it is deterministic

up until know - were good?

ok

now im getting back to MI

12:54 PM

continue

since we "learned" nothing with the conditioning A|B and A|C

(this is not the proper way, but it is true) they are each independent

meaning p(A,B)=p(A)p(B) and also for A-C

what's independent?

B and C?

A-B and A-C

ok

(in this case i think B-C also)

yes, they are because we defined them as independent random variables

if they are independent, then the MI is I(A;B)=0 and I(A;C)=0

true?

12:58 PM

yes

but we know that I(A;B,C) is not zero

since if we know B,C - it gives us information about A

yes

now i open up: I(A;B,C)=I(A;B)+I(A;C|B) > 0

and substituting I(A;B)=0 i get I(A;C|B) > 0

which is kind of counter-intuitive

wait

?

1:02 PM

continue

A here is not a random variable.

We can only determine the MIs of random variable.

Here only B and C is random.

A is random, we dont use "given B,C"

we compare the information they provide on each other

if it was somthing like this:

I(A;X|B,C) than yes, A is not an RV

ok

continue

one second

ok

i was re-looking at the definition of II(A;B;C)

and i think its the opposite of what i would expect

from here, again en.wikipedia.org/wiki/Interaction_information

i would expect II(A;B;C)=I(A;B)-I(A;B|C) but its the other way around

1:13 PM

Wait a second

You conjecture that if A and B are independent, I(A;B)=0, right?

I noticed that A is a random variable containing 2 bits

But it can only take 2 values.

why 2 bits?

A random variable only being able to take 2 values doesn't mean it only has 1 bits of entropy.

Entropy is a measure of unpredictability.

A has 2 bits if unpredictability.

i think it has one

This is because both B and C determine A.

B and C each have 1 bit.

but the outcome has only 1 bit

for XOR operation

1:17 PM

So they together make 2 bit.

Image upcoming...

user image

Here A contains 2 coins.

This is because both coins determine A.

Without either, the value of A cannot be determined.

So, A contains 2 bits of entropy.

I'll give you another example of the number of bits in the outcome not equaling the number of bits of entropy.

im sorry but this is just not true

Let V be a random variable with 1000 coins.

A is either 0 or 1

Let the value of V be the number of coins with value 1.

So there ara 1000 possible values of V, which is just under 10 bits.

But we know that V has 1000 bits of entropy.

this is not how entropy is calculated

the entropy for a RV is definded as E[-log(p(x))], where E is the expected value and p(x) is the probability function

so in our case: H(A)=-0.5*log(0.5)-0.5*log(0.5)=1

which means we need 1 bit to represent A

1:27 PM

But 2 bits to determine A.

it can also be 1000, A still has only 2 possible values

What about V?

A can be {0,1,...,1000}

so we need an upperbound of log2(1000), which is 10 bits

With that definition, how much entropy does it have?

that depends if the coin tosses are IID

1:31 PM

Let's assume each coin is fair and independent.

A has bimomial distribution

since it sums up IID bernoulli variables

from wiki: en.wikipedia.org/wiki/Binomial_distribution

entropy is 1/2*log2(...... (long)

ok

we strayed a bit, and i have to go

are you leaving?

i am not sure if i will study more today

1:36 PM

are you leaving?

yeah, lets talk later\tomorrow if you want

im still convinced i am correct though

bye for now

ok, bye

« first day last day (14 days later) »

Transcript for

Jul '1615

$Mathematics$ Discussion between user351579 and sagivd

Imported from a comment discussion on math.stackexchange.com/q...

information-theory

join this room
about this room

00:00

06:00

12:00

18:00

all times are UTC

$Mathematics$

site design / logo © 2024 Stack Exchange Inc; legal

mobile