so, in $p(\mathbf{y} | \mathbf{x}, \mathbf{w})$, both $\mathbf{y}$ and $\mathbf{x}$ should be fixed. I think this is the explanation

Yes, all of this is "for a given y,x,w" in 3.5 and 3.6

right, and the confusing is that we use the notation p(. | .), i.e. the notation to denote a conditional, to denote a likelihood actually, i.e. some function of w

w is the only thing that can vary here

sorry, the notation p(. | .) is a little bit unfortunate if you are a pervert :P

Btw, do you think that the following diagram well represent the supervised learning setting?

This should be the PGM that represents the underlying process of a typical supervised learning problem

X_i and Y_i are the random variables associated with the dataset (observables, that's why the grey color) and W is r.v. associated with the parameters (hidden variables)

Seems pretty standard to what I've seen in the past. It's not my exact area of expertise but it seems fine

So, the likelihood, even in the case of Bayes theorem, also has the data fixed, right?

the likelihood is never a density or mass, right?

or, at least, IN GENERAL, it's never a density or mass, right

Well, I think again that distinction is important

The likelihood is always a function of the parameters you want to find actually

You can have the likelihood at a point x,y,w

Or you can have a likelihood function of variables x,y,w

@Drathora Right, in that case, it will be a number

Where the first is the result of applying the second to the point

@Drathora But, in this case, it will be a function of w, which is the only one that can vary, although x and y can be arbitrary, but they should be fixed, right?

Ah yes, of course you can also fix one or more of the variables and have it be a function of w, or a function of x or a function of x and y

That kind of thing

That's just currying and uncurrying really

But when you optimize the likelihood you fix x and y

e.g. with MLE

you fix x and y

Then yes, you have a function of a variable w

you find w that maximises the likelihood given those fixed x and y

but the likelihood is always of the parameters according to Neal and MacKay

you cannot say "likelihood of the data"

but "likelihood of the parameters (given the data)"

yeah it's the likelihood that the parameters were "these values" given that this is the dataset you've observed

right, that makes senses

another thing. when we set p(y|x, w) = N(mu, sigma), p will be a Gaussian function

but is there any probability distribution here? no, right?

The integral of this thing is a conditional probability distribution

It's the distribution of y give x,w

I think you meant we set p(y:x,w) to the density of N(mu,sigma) right? Or am I mistaken?

what do you mean by ":" ? do you mean "|" ?

ye, I just don't have a key bound to it and I'm being lazy

No, I actually really meant p(y | x, w)

why do you put y twice there, in p(y | x, y)?

oh oops, that was a typo

Fixed, now look again

ha, so were you asking if I meant that p(y | x, w) is assigned to a NORMAL DENSITY?

yes

Well, I didn't want to use the word DENSITY there because the integral of it when y and x are fixed and as of function of w may not be 1

the integrals of densities are 1, right?

also, integral with respect to what? that's what we also need to clarify (probably!)

Only normalised densities

All of this generalises to measures with Radon-Nikodym derivatives

well, anyway, if I say "normal/Gaussian exponential function", there is not ambiguity

Probability measures are just a particular case where the total measure is 1

Yeah, in most literature the assumed base measure is the lebesgue measure for continuous distributions and the counting measure for discrete distributions

But it's good to be specific

ok, so you also use the word density to refer to functions that don't necessarily integrate to 1

I do, I'm not sure if it's really the best practice to do so

another thing, if you have two r.v.s, one is continuous and the other is discrete, the joint density or distribution must be continuous, right?

it must because there's no way you can cover the values of the continuous variable with a discrete distribution

I would describe it as a "mixture"

Since you can kind of thing of it as a collection of continuous distributions parameterised by the discrete value

There are of course lots of different ways to think of it

but I mean something like p(x, y), where X is continuous and Y is discrete

Is this possible?

I never thought about it

But my intuition tells me that the joint must be continuous

I am not sure why you're talking about parameterization here

That is possible, and yes

And the reason I mentioned that is that a common way of considering product measures m(x,y) is by considering them as a family of measures $m_x(y)$, parameterised by this x

well, that's already too many details because, although the word measure doesn't scare me, I don't know much about measure theory

Let's instead recap the stuff about realizations, r.v.s and variables by looking at the paper arxiv.org/pdf/1601.00670.pdf that I mentioned at the beginning

and connecting it to the discussion about the likelihood

Ah I see

In section 1 (page 2) of the paper, the author says:

What's your background may I ask? What angle are you coming at this topic from?

> Consider a joint density of latent variables z = z1:m and observations x = x1:n

> p(z, x) = p(z)p(x | z).

z in z = z1:m should be arbitrary realizations of the random variable Z (and similarly for x)

but, in the case of p(z, x) = p(z)p(x | z), is z the dummy variable or the realization mentioned before?

In other words, are we evaluating the density at the realizations z = z1:m and x = x1:n?

we can, the equation holds in both cases

In that equation both are dummy variables, as that relation holds for all x and z in the space

$f(p^{j}) \ll_{k} j^{\mathcal{O}(1}$ implies that $\lim_{p \to infty} \frac

$f(p^{j}) \ll_{k} j^{\mathcal{O}(1)}$ implies that $\lim_{p \to \infty} \frac{\log(f(p^{j})))}{\log(j)} = \mathcal{O}(1)$

did i miss anything?

@Drathora If I say "probabilistic interpretation", what does that mean for you?

Hmm. I can't say I've really heard the term thrown about. If I had to guess I'd say something along the lines of the probability distribution we use to model whatever it is we're trying to interpret,

Right, I want to say that training the neural networks with MSE for regression has a probabilistic interpretation and it's the derivation shared above (plus other parts). Does that make sense?

I guess that's fine. I'd argue it's more that the training is an application of this probability theory, but they're essentially the same thing

@Drathora Can we say that the likelihood that we optimize in MLE is not a conditional density?

assuming we have x and y fixed and let w vary, i.e. we max wrt w

Hmm, the thing you're optimising is a log likelihood right?

So the log of a conditional density?

So you're optimising a conditional density p(y:x,w) for a fixed y and x

Ok... but the weird thing is that people like to say that likelihoods are not probability distributions, which seems to go against what you have just said

can u clarify that?

I mean, we already talked about that, but I am a little slow :P

Likelihoods aren't distributions they're densities

right

A distribution on X takes a subset of X to a non-negative real number between 0 and 1

A density on X takes a single element of X to a non-negative real number

another thing that confuses me a little bit is the following. When I see a conditional density, e.g. p(y|x), it's hard for me to understand that y can be fixed (because it's on the left) and x can actually vary, because, usually, y is the one allowed to vary and x is fixed. In general, can the right side vary and the left side be fixed? I guess so, otherwise, we couldn't define the likelihood, right?

Yes, you can just think of p(y:x) as a function f

That takes two arguments (x and y) and returns a non-negative real number

Fixing either one gives us a new function

ok, thanks

Either the density of y with respect to a fixed x

Or the "likelihood"

another thing: I want to say that I define the likelihood as the composite function BETWEEN the Gaussian (density) function and the neural network (in my case, the model I am using). Does this make sense and, specifically, is the "between" the actual right word to use there, i.e. do we say "compositive function between function x and function y"?

Maybe compositive function OF?

I would say "composition of [function 1] and [function 2]

"

Ok

thanks

If I use the notation "$\mathcal{N}(\mu, \sigma)$", does that necessarily imply that it's a prob. dist. or can it also mean that it's Gaussian density? What's the convention in statistics?

I would always assume it's a distribution myself, but I've definitely seen some people use it to mean density in certain contexts

I would never do that myself though, as I find it confusing

Ok, then I will not use that notation here. In fact, I thought that would imply I mean a distribution and that can confuse people

another thing. I want to say that this formulate is analogous to the typical formulate of a linear regression model

Can I say that $\mathbf{y} = f_\mathbf{w}(\mathbf{x}) + \epsilon$, where $\epsilon$ is sampled from $N(0, \sigma)$ is equivalent to the following

here $f_w$ is the neural net

*I want to say that this formulation

I think these two views are equivalent, but I am not fully sure

y (the correct label) is a non-linear function of the net (when given x as input) + some noise from a Gaussian

that's what the definition of the likelihood should mean

I think that's okay. Getting a bit late for me though so I might not be thinking properly

ok, thanks a lot, man!! You really clarified a lot of things!

good night