the no-normie zone - 2017-10-13

Anonymous

16:15

@BalarkaSen Do you know any place where the exact proof of the second derivative test (Hessian test) for multivariable functions is given ?

Anonymous

The $AC-B^2$ stuff

Balarka Sen

18:03

@Blue It's in Ted's book but I can tell you the proof

Anonymous

18:26

@BalarkaSen Thanks, that would be very helpful

Balarka Sen

@Blue Perhaps not right now though. How about tomorrow?

Anonymous

@BalarkaSen Yep, that's fine :)

Balarka Sen

OK

Balarka Sen

18:56

@Blue It's tomorrow already; do you want to know the proof? :P

Anonymous

Yes!

Anonymous

I found a proof using quadratic forms

Balarka Sen

do you know the multivariable Taylor's theorem?

Anonymous

@BalarkaSen Yup

Balarka Sen

tell me about it

by that i mean just state the theorem

Anonymous

19:05

$$f(\mathbf{x})=f(\mathbf{x_0})+\nabla f(\mathbf{x_0})(\mathbf{x}-\mathbf{x_0})+\frac{1}{2}(\mathbf{x}-\mathbf{x_0})^{T‌}Hf(\mathbf{x_0})(\mathbf{x}-\mathbf{x_0})+...$$

Anonymous

$x$ and $x_0$ are vectors

Balarka Sen

Where $f : \Bbb R^n \to \Bbb R$ is a $C^\infty$ function near $x_0$, right.

Ok, good

Anonymous

@BalarkaSen Yup!

Balarka Sen

You can just think about $C^2$ functions and consider the second order Taylor's theorem $$f(x + h) = f(x) + \nabla f(x) h + \frac12 h^\top Hf(a) h + o(\|h\|^2)$$

Where $o(\|h\|^2)$ means that term goes to zero after dividing by $\|h\|^2$ and letting $h \to 0$.

Anonymous

Yes, got it so far

Anonymous

19:15

Let's do it for two variables first perhaps

Balarka Sen

Nah you don't have to.

Anonymous

Okay?

Balarka Sen

Suppose $a$ is a critical point of $f$.

Then $\nabla f(a)= 0$.

Anonymous

Yup

Balarka Sen

Then near that point we have $f(a + h) - f(a) = 1/2 h^\top Hf(a) h + o(\|h\|^2)$

Let's call $Q(x) = x^\top Hf(a) x$

Anonymous

19:19

Alrighty

Balarka Sen

This is, as you said, a quadratic form. Why? Well, write down the matrix of $Hf(a)$ and let $x = (x_1, \cdots, x_n)$ be a vector and expand it out

Anonymous

Oh, yes. I did that derivation sometime back.

Balarka Sen

You'll get $Q(x) = \sum_{i, j = 1}^n \partial^2 f/\partial x_i \partial x_j(a) \cdot x_i x_j$

Anonymous

Right

Balarka Sen

The Hessian matrix is called positive definite if $Q(x) > 0$ for all $x \neq 0$

negative definite if $Q(x) < 0$ for all $x \neq 0$

and indefinite if $Q(x) < 0$ for some $x$ and $Q(x) > 0$ for some $x$

Anonymous

19:24

Gotcha

Anonymous

Go on

Anonymous

I think that says something about sign of f(a+h)-f(a)

Balarka Sen

Yeppers

Anonymous

Saddle point for indefinite?

Anonymous

Minima for positive definite?

Anonymous

19:28

Maxima for negative definite?

Balarka Sen

@Blue Huh?

The first two are true. You wrote positive definite for both maxima and minima

Yes, good

Anonymous

Wait, what about when along one direction is is a minima but along another perpendicular direction it remains 0 throughout ?

Balarka Sen

Well, what about it? All the second derivative test says is the following: If $Hf(a)$ is +ve/-ve definite, $f$ has a minimal/maxima at $x = a$ and if $Hf(a)$ is indefinite, $f$ has a saddle at $x = a$.

The kind of case you are describing happens when $Hf(a)$ is "semidefinite". $Q(x)$ is sometimes $0$ even for nonzero $x$'s.

In those cases you can draw no conclusion about the critical point $x = a$

It can be really complicated, like Monkey saddles

Anonymous

@BalarkaSen Huh, right. That's what I mean!

Balarka Sen

But yeah that's it

There's a neat description of all the (in)definiteness for two variables like you were saying

Anonymous

19:34

@BalarkaSen Uh, like?

Anonymous

AC-B^2 ?

Balarka Sen

Write $Hf(a)$ as the matrix $[A, B; B, C]$ and compute $Q(x)$ for me

Anonymous

One sec

Anonymous

$Ax^2+2Bxy+Cy^2$

Balarka Sen

Great

What happens if you complete le square?

Like, $Q(x, y) = Ax^2 + 2Bxy + Cy^2$

Complete the square of $A\cdot Q(x, y)$

Anonymous

19:40

I'm forgetting how to do that. I'll use the quadratic formula perhaps

Balarka Sen

Nah, just like this

$A^2 x^2 + 2AB xy + AC y^2$

$= (Ax + By)^2 + ACy^2 - B^2y^2$

$=(Ax + By)^2 + (AC - B^2)y^2$

Anonymous

Uh, oh. Lol

Anonymous

Got it

Balarka Sen

So what you basically did was introduced new variables $u$ and $v$ by a linear change of variables from $x$ and $y$ such that $Q(x, y) = u^2 + (AC - B^2)v^2$

The point being, it's easy to identify when this is +ve/-ve definite or indefinite

Anonymous

Oh, I see

Anonymous

19:44

Only $AC-B^2$ is not enough to tell the sign of that thing probably

Balarka Sen

no i made a dumb algebra mistake

$A \cdot Q(x, y) = (Ax + By)^2 + (AC - B^2) y^2$

So $Q(x, y) = A(x + B/Ay)^2 + (AC - B^2)/A y^2$

i.e., $Q(x, y) = Au^2 + (AC - B^2)/A v^2$

Anonymous

Okay, yes

Anonymous

:P

Anonymous

Good so far

Anonymous

Let's take $A>0$

Anonymous

19:50

If $AC-B^2>0$, then $Q$ is positive definite

Balarka Sen

Sorry I was off

@Blue Right

Anonymous

If $AC-B^2<0$ we can't say unless we know the value of $u^2$

Anonymous

Sometimes positive sometimes negative

Balarka Sen

No, if $AC - B^2 < 0$ it's indefinite

Regardless of what $A$ is\

Anonymous

Yes. But what is the condition for it to be positive, given that $AC-B^2<0$? Indefinite means just that. Q is sometimes positive and sometimes negative

Balarka Sen

19:54

Sorry, huh? $Q$ is positive definite if and only if $A > 0$ and $AC - B^2 > 0$

That's the condition

Anonymous

31 mins ago, by Balarka Sen

and indefinite if $Q(x) < 0$ for some $x$ and $Q(x) > 0$ for some $x$

Anonymous

@BalarkaSen I didn't mention "positive definite" in my previous sentence

Anonymous

I just said "positive"

Anonymous

and "negative"

Balarka Sen

What does that mean? What is positive? I don't understand the question

Anonymous

19:57

Given, $AC-B^2<0$ what is the condition for $Q(x,y)$'s value to be "positive"? (I am not saying "positive definite")

Balarka Sen

Oh, you're asking for which $(x, y)$ is $Q(x, y) > 0$? That depends entirely on the quadratic form $Q$

Anonymous

@BalarkaSen I think we can generalize the result

Balarka Sen

You can just work it out. Plug $(x, y)$ in $A(x + B/A y)^2 + (AC - B^2)/A y^2$

@Blue Generalize what result? Look, what you are asking is a dumb question. Where $Q$ is positive or not is of no interest to us

If $Q$ is indefinite, regardless of where it is positive or negative, we have that $f$ has a saddle point at $x = a$.

That's all that is needed. Hate to break it down to ya.

Anonymous

@BalarkaSen Lol, I said the same thing when our prof asked us that question :'D. Yeah, I'll rather leave that question

Anonymous

I think it would just involve some algebraic manipulation

Balarka Sen

20:01

Sure, it's just algebraic messing around. I don't even see why that's interesting

in this context, at least

Anonymous

I don't know but our prof told us to thing about it. Weird

Anonymous

I'll try sometime

Anonymous

Anyhow, I think I get the basic concept now

Balarka Sen

Ok, perhaps. But for now "somewhere positive, elsewhere negative" is all that is necessary. I don't really want to think about a tangential question

idt it's very interesting

What can be generalized, and is 100x interesting that any of this, is generalizing this to higher dimensions

Anonymous

@BalarkaSen Yep, I'd like to know that

Balarka Sen

20:05

The whole process of this centers around trying to turn the quadratic form $Q(x) = x^\top Hf(a) x$ into the form $a_1 U_1^2 + a_2 U_2^2 + \cdots + a_n U_n^2$ where $U_i$'s are obtained by a linear change of variables from $x_i$'s where $x = (x_1, x_2, \cdots, x_n)$

You just saw this for $n = 2$

Which is just "completing the square"

Anonymous

Diagonalization

Anonymous

?

Balarka Sen

Exactly.

Take a general quadratic form. That's of the form $Q(x) = x^\top A x$ where $A$ is a symmetric $n$-by-$n$ matrix.

If you can diagonalize $A$, then that means there is a diagonal matrix $D = P^{-1}AP$ similar to $A$ by some basechange matrix $P$.

Anonymous

@BalarkaSen Right!

Balarka Sen

Plug this shit in. $Q(x) = x^\top P D P^{-1} x = (P^\top x)^\top D (P^{-1} x)$, here I am just using $(AB)^\top = B^\top A^\top$

Anonymous

20:09

mmhmm

Balarka Sen

Sorry, I had to plug in $A = PDP^{-1}$

Fixed now

Is this Ok?

Anonymous

Yes, looks fine

Balarka Sen

Cool

Now, turns out you can not only diagonalize symmetric $n\times n$ matrices like $A$, you can also orthogonally diagonalize them

What this means is that you can choose $P$ to be an orthogonal matrix

Aka, $P^\top = P^{-1}$

This is a remarkable result known as the "spectral theorem"

Anonymous

@BalarkaSen Interesting!

Anonymous

@BalarkaSen There seems to be whole books on "spectral theorem" =P

Anonymous

20:15

Balarka Sen

There are a lot of spectral theorems out there. But this finite dimensional result for real symmetric matrices is the start of the story, yes

Anonymous

Isee :)

Balarka Sen

But when you do "orthogonally diagonalize", what happens is this

$Q(x) = (P^{-1} x)^\top D (P^{-1} x)$, because $P^{-1} = P^\top$

Write $y = P^{-1} x$ (linear change of coordinates!!). Then $Q(x) = y^\top D y$

Since $D$ is diagonal, write the diagonal entries as $d_1, \cdots, d_n$ and expand (where $y = (y_1, \cdots, y_n)$)

You get $Q(x) = d_1 y_1^2 + d_2 y_2^2 + \cdots + d_n y_n^2$

Done.

Anonymous

Oh. Wow. But the conditions on $d_1,d_2,d_3,...$ would probably be complicated. Perhaps we won't even bother to find such a condition. Just find Q after plugging in values into the $Q(x) = d_1 y_1^2 + d_2 y_2^2 + \cdots + d_n y_n^2$ form

Balarka Sen

$d_1, \cdots, d_n$ are actually the eigenvalues of $A$ :)

Anonymous

20:23

@BalarkaSen Oh, right. Yes

Anonymous

That makes it simpler

Balarka Sen

'Cuz $A(Pe_i) = PDP^{-1} (Pe_i) = PD(e_i) = Pd_i e_i = d_i (Pe_i)$

So $v_i = Pe_i$ is an eigenvector of $A$ with eigenvalue $d_i$

Anonymous

Right, right. Gotcha!

Anonymous

I should go to sleep now. Too sleepy :P

Anonymous

See you tomorrow!

Anonymous

20:26

Thanks a lot

Balarka Sen

See ya

Transcript for

♦ $Mathematics$ the no-normie zone

Transcript for

♦ the no-normie zone

♦ $Mathematics$ the no-normie zone