as far as I know $\sigma\overset{P}{=}\sigma'$ implies $\sigma=\sigma'$. They're constant, and if they're not equal you would land on a contradiction. And as much as that's the definition, I beg to disagree. The whole point of identifiability is to show that in the presence of large enough sample size the parameters of the true model are retrievable, and that in essence follows from identifiability. What we call a "data point" is technically a random variable so there is not much difference between the likelihood for that and the true distribution.

Well, I can explain in more detail if you'd like, but first consider this: if $x \stackrel{P}{=} y$ implied that $x = y$, why would we ever bother writing $x \stackrel{P}{=} y$? Why not just skip writing this and write $x = y$ instead if we can?

That notation is made for sequences. Here I am already dealing with constants. That's why I can remove the $P$.

@Cupitor I hope I've answered this in the addendum to my answer. The (weak) law of large numbers does not say that $\lim 1/N \sum (X_i - \mu)^2$ is the constant $\sigma^2$. This is still a random variable, i.e. a non-constant function of $\omega$.

\begin{align} \left(\frac{1}{N}\right) \sum_{i=1}^N (X_i (\omega) - \mu)^2 & \stackrel{P}{\rightarrow} E[(X_i (\omega) - \mu)^2] = \sigma^2 \\ \left(\frac{1}{N}\right) \sum_{i=1}^N (Y_i (\omega) - \mu')^2 & \stackrel{P}{\rightarrow} E[(Y_i (\omega) - \mu')^2] = \sigma'^2. \end{align} Writing above limits in $\epsilon$/$\delta$ form (sum lhs of both and rhs of both) we can get $P\{\omega||(X_n(\omega)-\mu)^2-(X'_n(\omega)-\mu')+\sigma-\sigma'|>\epsilon\}\to 0=P\{\omega||\sigma-\sigma'|>\epsilon\}\to 0$, which implies the latter should be equal as there is no $N$ involved.

Also thanks for pointing out the L vs. S difference. My bad for confusing them. And thanks for the effort. I just upvoted your answer. I am not sure if I made a mistake but I think my reasoning is right.

Consider this: There is some value $\omega$ where $X_i (\omega) = 0$ for all $i$ and thus the sample variance is always exactly 0. If the the $Y_i (\omega)$ "do something normal" that leads to that sample variance converging to $\sigma'^2$, then we have a value of $\omega$ where the sample variances are equal even in the limit, and don't cancel out. This doesn't contradict the WLLN, because $P(\{\omega\}) = 0$, but we can't cancel the sample variances and thus we can't prove that $\sigma = \sigma'$ for this $\omega$.

I am not following how with i.i.d. Assumption you have $X_i(\omega)=0$ all the time??

It's possible for a normally distributed random variable $X_1$ to be 0, because $p(0) \ne 0$, so the i.i.d. assumption implies that it is also possible for $X_2 = 0$. You can repeat this argument to prove that there is a sequence $X_1, X_2, \dots$ that is always $0$. Call this outcome $\omega$. What is confusing is that this is a probability 0 event, but that does not mean that this is impossible.

Technically, the outcome is $\omega$ and the event is the set $\{ \omega \}$. I should have been a little clearer. In particular, the existence of a single $\omega$ where the sample variance does not converge to $\sigma^2$ is enough to prevent you from claiming that $\sigma = \sigma '$.

That's my bad and I agree with you, but I don't get what you mean by " If the the 𝑌𝑖(𝜔) "do something normal" that leads to that sample variance converging to $\sigma'^2$, then we have a value of 𝜔 where the sample variances are equal even in the limit, and don't cancel out. ". Either way, I stil think the reasoning I made 6 comments ago stands. The summation I presented there literally implies $$P\{\omega:|\sum(X_n(\omega)-\mu)^2 - \sum(X_n'(\omega)-\mu')^2+\sigma-\sigma'|>\epsilon\}=P\{\omega:|\sigma-\sigma'|>\epsilon\}\to 0 .$$ In fact let's assume that's not the case...

Choose $\epsilon<|\omega-\omega'|/2$. Then $P\{\omega:|\sigma-\sigma'|<\epsilon/2\}=1$, which is a contradiction as much as I can see.

The expression $|\omega - \omega'|$ doesn't make any sense. The sample space is not assumed to be a vector space, so you can't add or subtract the outcomes $\omega$. In fact, it's not assumed to be a metric space, so you can't even define the more general notion of distance using a metric $d(\omega, \omega')$, so I'm not sure what you mean by this.

Hello! I wasn't expecting you to be awake! Thanks for taking the time to address this!

Here's what's going on: convergence in probability does not imply convergence in the usual sense. What convergence in probability is a weaker condition than almost sure convergence, which is in turn weaker than convergence. Even if "most" outcomes $\omega$ have the desired convergence, there is a small set of outcomes where the convergence does not hold.

I wrote $\sigma-\sigma'$ not $\omega-\omega'$. If I did then I made a typo.

Believe it or not I know the definitions of convergence in probability, in distribution and a.s.

I just disagree in this case one cannot make such a conclusion

So let me explain my example a bit more. I claim that we can find a particular outcome $\omega$ with the following properties:

1. The sequence $X_i (\omega)$ is constant, and thus has 0 sample variance.

2. The sequence $Y_i (\omega)$ behaves "normally" in the sense that its sample variance converges in the limit to $\sigma'$.

I'll let you write until you say you're done. Btw we can also zoom or something. I am the same person who wrote about your CV and the website HTTPS issue.

Thus for this particular $\omega$, the sample variance of the $X_i$s is not $\sigma$ like you would expect.

Ah, thanks again for that!

No problem! Thanks for being so active in open source community and helping people!

Let me process what you wrote.

No problem at all

Yes but the point is that in my case I know for every $n$ and for any $\omega$ the said equality holds!

And in that sense I can get rid of $N$ in the final limiting relationship I have.

So I understand your example. I just don't think my reasoning suffers from this counterexample. It drops N at some point of reasoning, as there is no $N$ involved.

But it's not for any $\omega$, because I just constructed an example where it isn't.

As you said yourself!

(Why LaTeX is not parsed here?? :|)

I'm not sure. I was not expecting that when I moved to chat.

This equation is 100% valid: $$N \log(\sigma) + \frac{N}{2\sigma^2} \left(\frac{1}{N}\right) \sum_{i=1}^N (X_i (\omega) - \mu)^2 & = N \log(\sigma') + \frac{N}{2\sigma'^2} \left(\frac{1}{N}\right) \sum_{i=1}^N (Y_i (\omega) - \mu')^2$$

But when we apply the WLLN, it does not let us take the limit for all $\omega$, only "most."

So you can't claim that the limit will hold for all $/omega$.

With my particular $\omega^*$, say, what we have is the sample variance on the left side of the equation is 0 instead of \sigma

That's not the part that I am applying things. When I have things in paranthesis and I am calculating limits, I use triangle inequality to get rid of anything dependant on $N$ does it make sense until that part?

so the equation would become: $$N \log(\sigma) = N \log(\sigma') + \frac{N}{2}$$

Which is no longer true

No no, I am talking about how I get from your equation above to 𝑃{𝜔:|𝜎−𝜎′|>𝜖}→0

Does that make sense to you or should I explain how I get that?

Wait, how did you get that LaTeX to be parsed?

lol, I just copied an pasted from the original comment

Oh, okay

I'm trying to find where you used the triangle inequality

It is where I say Writing above limits in 𝜖/𝛿 form (sum lhs of both and rhs of both) we can get

I think this will actually be easier to explain if we switch to using the strong law of large numbers, which still applies in this case.

If we apply the SLLN instead, we can vastly simply things here and say that each sample variance converges to their respective 𝜎s with probability 1.

Thus, with probability 1, we would have the equality 𝜎=𝜎′. This is actually how I was interpreting your notation of $\sigma \stackrel{P}{=} \sigma'$, which is nonstandard notation

In this case, the equation actually depends on a term that equals zero with probability 1

so it's more like 𝜎=𝜎′ + (term that cancels for almost all 𝜔)

And the SLLN proves this much. The problem is that it still only implies that P{𝜔 : 𝜎=𝜎′} = 1

Yes the way I wrote it is meaningless, and quite handwavy, but I think I can formalize it. I know your phd is in math and my bachelors is in math, so I have a lot to catch up on formalism. But either way may argument all happens before the final limit is approached.

This does not prove the equality for all 𝜔, only a set of 𝜔s with probability 1. This is why it's possible for me to find an example 𝜔* where the equality does not hold

No no, hold on

how are you writing in latex?

Haha, copy and paste for the win

lol ok

So to recap: N log(𝜎) + N/(2𝜎^2) S = N log(𝜎') + N/(2𝜎'^2) S' is valid for all 𝜔, where S is the sample variance of the Xs and S' is the sample variance of the Ys

𝑃{lim𝑋𝑛=𝜎2}=1 and 𝑃{lim𝑌𝑛=𝜎′2}=1. But additionally I know 𝑋𝑛=𝑌𝑛 and that's where I can do everything before the final value of the limit is realized to free up myself from dependence on 𝜔.

We want to apply the SLLN to both sides of the equation, which implies that S → 𝜎 almost surely, and S' → 𝜎' almost surely.

Yes, but I am summing those sequences basically to cancel out S and S' because I can!

It is not but it is actually equal to \log(\sigma)-\log(sigma') which is still fine

whoops, wrong thing

Hang on, let me finish writing this out

But you can't cancel out (1 a.s.) - (1 a.s.) even though they look the same, because they aren't guaranteed to be 1 on the same set of 𝜔s

You can only know that (1 a.s.) - (1 a.s.) = (0 a.s.)

But you can't claim that taking "one such 𝜔 that equality holds" then implies that what you proved holds for all 𝜔

You have to prove that equality holds for all 𝜔

And I can still make a counter example where S = 0 and S' -> 𝜎'

Exactly because they're constant I am only willing to take one such 𝜔. Nothing at this point depends on 𝜔.

S-S' is not constant in what you're implying

It is in my case.

But it does, you just didn't write it. You're not writing the log(𝜎/𝜎') = (0 a.s.) part

which then implies that 𝜎/𝜎' = (1 a.s.)

and then that 𝜎 = 𝜎' a.s.

Even though both 𝜎 and 𝜎' are constants, the equality still depends on 𝜔

well 𝜎 = 𝜎' a.s. as you say. Take one 𝜔 that this holds. It is measure 1 right?

Then it follows that 𝜎 = 𝜎'

But that's the main point. 𝜎 = 𝜎' a.s. does not imply that 𝜎 = 𝜎' for all 𝜔

That's what I meant by the first line of my answer

what does this mean "𝜎 = 𝜎' a.s."?

It means that the set A of 𝜔 for which we are able to prove that 𝜎 = 𝜎' has probability P(A) = 1.

Or that the equation holds "almost surely"

So we agree that there is a 𝜔 where this holds

right?

Oh yes, almost all of them will have the equality hold

But you can't drop the a.s. unless it holds for all of them

But 𝜎 and 𝜎' are constant. So if it holds for one omega it needs to hodl all the time.

𝜎 and 𝜎' are cosntant independent of the sample space

But this equation is more like 𝜎 = 𝜎' + error(𝜔)

and we're just not writing the error(𝜔) term because P(𝜔 : error(𝜔) = 0) = 1

Does it really matter? Write the error term. I am saying that there is one such omega where the equality holds. It should therefore hold for any omega because sigma and sigma' do not depend on omega any way!

But I could choose 𝜎 = 1 and 𝜎' = 2, so that they aren't equal, and 𝜎 = 𝜎' + error(𝜔) is still a true equation, even though it will hold for at least one 𝜔

in other words, error(𝜔) = -1 for some 𝜔, which makes the equation true

even though the constants 𝜎 = 1 and 𝜎' = 2 are not equal

No you cant is in that case that set would be of measure 1! because then it should hold for any omega with the same reasoning but we know the other set is of measure zero!

Holding for a set of measure 1 doesn't imply that it should hold for any omega

Holding for a single omega is enough to ensure it holds all the time, because sigma and sigma' are not relevant to omega.

Or better said \Omega

"Holding for a single omega is enough to ensure it holds all the time" is not true though, because error(𝜔) ≠ 0 for some values of 𝜔. If that logic worked, you could prove simultaneously that 𝜎 = 𝜎' and that 𝜎 = 𝜎' + 1 by picking two values of 𝜔, call them 𝜔* and 𝜔', where error(𝜔*) = 0 and error(𝜔') = 1. Then we can prove that 0 = 1 which is a contradiction.

No you can't because the set that 𝜎 = 𝜎' + 1 is of measure zero. The only thing that proves is that such a set, i.e. where 𝜎 = 𝜎' + 1 holds is exactly the empty set.

Also, technically since we exponentiated both sides I should be writing 𝜎 = 𝜎'*error(𝜔) instead of 𝜎 = 𝜎' + error(𝜔), but that doesn't really matter

"𝜎 = 𝜎' + 1 is of measure zero" still holds for at least one value of 𝜔

There are plenty of sets with measure 0 other than the empty set

So what you said first is true, the set of 𝜔 where 𝜎 = 𝜎' + 1 is 0, but it's not empty

I've enjoyed this discussion, Cupitor, but I am about to go to bed. I'm happy to continue discussing this problem tomorrow though.

Yes in general it can have elemtns and that's why it is called measure zero. I understand. Now I additionally know (based on my reasoning) that 𝜎 = 𝜎' all the time, i.e. for all the omega. Therefore that particular measure zero set we are talking about needs to be empty.

Sounds good!

Thank you so much for taking the time!

Really!

Yeah sure. I am on [email protected]

Or we can find some other way to chat. I am not sure if this will be available tomorrow, but we can try this as well.

Have a good night. And my name is Naji.