The h Bar

6:00 PM

o.o

Is locality related to higher order derivatives bringing in adjacent terms? Like going from first to second derivative is $\frac{f(x + h) - f(x)}{h} \to \frac{f(x + h) + f(x - h) - 2 f(x)}{h^2}$?

As an extension, would a smooth function essentially require being specified everywhere from the beginning?

Well, specified up to conditions on each derivative

enumaris

locality is a physical concept, what u got there is a pure math one. An everywhere smooth function must be defined everywhere since a discontinuity somewhere would preclude that function from being smooth everywhere.

ACuriousMind

@danielunderwood "locality" in what context?

And when you say "from the beginning", the beginning of what do you mean?

danielunderwood

I think as in the order of derivatives in the action. I'm still trying to wrap my head around the locality concept, but to me it seems like those adjacent terms make something "not local" in the sense that it depends on a larger region than just $h$ (or $\epsilon$ if you prefer) away from $f(x)$

And in the beginning as in you can define all smooth functions by $\frac{d^\infty f}{dx^\infty} = 0$ along with conditions on each derivative. Though I'm not sure an infinite differential operator really makes sense

I suppose that could also be true with $C^n$ with $\frac{d^n f}{dx^n}$?

Though I feel I'm getting dangerously far away from defining things rigorously

ACuriousMind

6:15 PM

I'm not sure what you mean. Derivatives of all orders are local in the sense that they just depend on an infinitesimal neighbourhood

enumaris

In a derivative, you take the limit $h\rightarrow 0$ bro

ACuriousMind

However, an analytic function is uniquely determined by all its derivatives at a point, but not all real smooth functions are analytic

Semiclassical

Some pertinent discussion here: physics.stackexchange.com/q/13624/55641

enumaris

how many of them are? Is it almost none of them?

ACuriousMind

@enumaris what measure do you want to use on the space of smooth functions? :P Without the jargon, I'm just saying there are functions that are not equal to their Taylor series

@Semiclassical Ahhh, if that's what @danielunderwood means, then Arnold Neumaier's answer there is probably what he's looking for

enumaris

6:22 PM

Usually mathematicians like to say stuff like "almost none of them are X"...or "almost all of them are X"...like "almost all real numbers are irrational" or "almost no real numbers are rational" or w/e

I've heard like "almost all functions are non-analytical" (pretty sure I've heard this, but not 100%)

danielunderwood

But say we look at second derivative quantifying the curvature. We need a bigger region to talk about the curvature than slope, no?

I'll have to take a look at that question

enumaris

so is there "almost all smooth functions are not analytical"

@danielunderwood ain't no region

All orders of derivatives involve a limit as the region size in consideration goes to 0

danielunderwood

Or say we looked at arbitrarily high derivatives that bring in terms like $f(x + nh)$. Could we have a situation where $nh \not\to 0$?

thats...ugly

enumaris

wot

ACuriousMind

@danielunderwood No,we don't. Regardless of order, you only need to know the function in an infinitesimal neighbourhood.

danielunderwood

6:25 PM

hmmm

guess I should study some analysis?

enumaris

$dx^2$ ain't bigger than $dx$

waits for mathematicians to crucify me

danielunderwood

If mathematicians aren't coming out to attack what I say, I think you're safe

enumaris

they are wily

beware of ninja mathematicians at night

danielunderwood

@ACuriousMind but if that's the case, you'd only need to know the function in an infinitesimal neighborhood to look at smoothness?

Is is it a standard to use neighborhood/neighbourhood in math or is it still a regional thing?

ACuriousMind

@danielunderwood Yes. Smoothness is a local property.

enumaris

6:31 PM

you need to be able to take the limit

but u are taking a limit

danielunderwood

Is there some sort of link between smoothness being a local property and needing derivatives defined everywhere in the domain? The latter sounds like a property of the function as a whole

enumaris

everywhere smooth vs smooth at x

ACM is talkin about smooth at x for some given x

Everywhere smooth means smooth at all x in the domain

danielunderwood

ahhh

enumaris

ahhhhhh

danielunderwood

Hopefully I'll know enough to tell other people where they're wrong one day :D

enumaris

6:42 PM

you don't need to know anything to tell people they are wrong over the internet

have you learned nothing from youtube comments?

literally just yell "WRONGGGGGGGGGGG'

danielunderwood

Well that's where you're wrong!

enumaris

getting the hang of it

ordered some bulgolgi for lunch today...should be good...mmm

getting hungry...

Avnish Kabaj

Bulgogi

Bulgogi ( buul-GOH-ghee; from Korean bul-gogi [pul.ɡo.ɡi]), literally "fire meat", is a gui (Korean-style grilled or roasted dish) made of thin, marinated slices of beef or pork grilled on a barbecue or on a stove-top griddle. It is also often stir-fried in a pan in home cooking. Sirloin, rib eye or brisket are frequently used cuts of beef for the dish. The dish originated from the north area of the Korean Peninsula and is very popular in South Korea. In fact, bulgogi is widespread in South Korea, such as at fancy restaurants and at local supermarkets selling pan-ready kits. == Etymology... ==

vzn

← would insert xkcd on "being wrong on internet" here if not so many had seen it + was drawn better

danielunderwood

that sounds pretty delivious

delicious

enumaris

6:49 PM

should be :D

generally I prefer the short rib

but...today, bulgogi

danielunderwood

I don't think there's even a place to get Korean food around me

enumaris

I'm a little dissatisfied with my food choices here tho...I gotta order out every day...or else there's only like 2 sandwich shops nearby

danielunderwood

Though there are 10 Chinese places with the exact same menu

enumaris

lol

where are you?

danielunderwood

And there are ones 2 hours away with the same menu too

enumaris

6:52 PM

east coast?

west coast?

danielunderwood

NC. Kind of near Charlotte

enumaris

Any relation to Charlottesville?

oh, that's in Virginia

danielunderwood

Not that I know of. I have no idea where Charlotte even got its name though

enumaris

lol

Anonymous

@enumaris And it's worse when you have to see the same shopkeeper's face everday and you have to ask for the same food item everytime ;)

Anonymous

6:53 PM

Assuming it's not a doorstep delivery service

danielunderwood

We did have a Mexican restaurant in college that supposedly came from the west coast and that was interesting

enumaris

NC is the state I learned about in HS that's like...always getting bad influence from SC...getting dragged into the Civil war by SC or something...

iirc

SC was always the "problem child" state

@Blue it's a delivery service so every day it's a new delivery person lol

@Blue though, getting to know the shopkeeper ain't bad. I got to know an Iranian place's shopkeeper during my time in San Diego...I didn't have to order food anymore after a while, he'd just know what I wanted lol

fooood timeeee

danielunderwood

Yeah I'm not really sure of our Civil War history. The people I know of seem to either think we shouldn't have been in the Confederacy or wish we still were

ACuriousMind

I just had some sushi

Avnish Kabaj

I don't like sushi

Maybe it's just that I had terrible sushi the one and only time I tried it

But it's raw fish yo

ACuriousMind

6:57 PM

Getting to know all the delivery services around here since I can't stand in front of the oven to cook with this ankle :P

danielunderwood

sushi can vary a lot depending on where you get it from and what it is

I did used to get premade sushi from one of our dining halls and I guess it wasn't too bad

Everyone thought I was crazy for doing it though

Anonymous

Yeah, very few places can make sushi well

Anonymous

I had good sushi only twice in my life. Other times they were either too salty or too bland

Anonymous

The sauce matters too

Avnish Kabaj

Steaks are overrated

danielunderwood

7:03 PM

Yeah my doctor put me on a diet a couple months ago and I couldn't have soy sauce...it made sushi miserable

Anonymous

@AvnishKabaj Depends. I do remember having some great steak in some restaurants

Avnish Kabaj

@Blue no flavor

Anonymous

Almost all kinds of edible items can if prepared well, can be tasty ;)

Anonymous

Ask the michelin guys

Avnish Kabaj

I mean the texture's like amazing

Sometimes

Mostly I get chewy pieces of meat

Anonymous

7:05 PM

@AvnishKabaj Eh, that's strange

danielunderwood

Seasoning matters a lot in steaks. Or burgers. Or chicken. Anything really

I'd probably choose a good burger over a good steak most days though

Avnish Kabaj

@danielunderwood yas

Anonymous

I had some great crab meat last Friday in a Bengali restaurant

Anonymous

after a long time

Anonymous

I don't visit Bengali restaurants often, but I guess I will visit them more often from now on :P

Avnish Kabaj

7:08 PM

You're bangali no?

Anonymous

@AvnishKabaj You have tasted Nolen Gur ice cream sometime I guess?

Anonymous

@AvnishKabaj Rule of thumb is: you don't visit restaurants which have the same cuisine as your home :P

Avnish Kabaj

@Blue I've had nolen gur rasmalai

@Blue true that

Anonymous

@AvnishKabaj Nice. I love it :D

Avnish Kabaj

Me 2

It's something else

Anonymous

7:16 PM

@enumaris Did you have Iranian food there?

Anonymous

Among middle east cuisines I only have had experience of tasting Lebanese food. And some kinds of meat and seafood at Abu Dhabi. But the problem with those rich middle eastern tourist states is that they're almost Americanized these days and traditional food items are harder to find

Avnish Kabaj

Hmmm

When I went to Norway

I asked for some curry

And it was spicy by my spice standards

Never had spicy stuff out of india

It wasn't americanized

Anonymous

@AvnishKabaj Really? I thought Indians are the only ones infamous for over-spicy food ;)

Anonymous

I am a complete noob about Norway though

Avnish Kabaj

@Blue prolly they over did it

enumaris

7:26 PM

@Blue indeed, it was good Iranian food :D

Avnish Kabaj

We were the only Indians at the restaurant

Anonymous

One of my seniors went to China this summer and mentioned that he used to have fried insects and worms almost everyday. That sounds puke-worthy XD

Anonymous

Japan and China are two places I surely want to visit solely due to their peculiar cuisines

Semiclassical

I can understand fried insects better than fried worms

I find them both pretty weird, but somehow the latter is harder to wrap my head around

Anonymous

I found a related video:

Anonymous

7:34 PM

Chinese Street Food - Live Scorpians, Insects, Tiger Claws

►

Avnish Kabaj

Wak wak

enumaris

I don't eat any of that...but I've seen it while in China :D

danielunderwood

I went to China and didn't have any of that. Though we were told not to ever eat any of the street food

Anonymous

Oh, and fried maggots are also popular there

Anonymous

Apparently

danielunderwood

7:36 PM

ew

enumaris

"popular" - not really

Avnish Kabaj

The hit and trial process must have been quite daring

Anonymous

@enumaris That's better :P

enumaris

At least not very popular with the current generation Cityfolk

So, Kaluza-Klein theory...if another person with surname beginning with K modifies it...would we have a KKK theory...

Does this disqualify anybody with surname beginning with K from working on Kaluza-Klein theory?

Avnish Kabaj

Was that an elaborate klu klux Klan joke

Anonymous

7:39 PM

@danielunderwood Good advice. If you ever visit India don't touch the street food here either. Almost certainly you'll get diarrhoea XD

Anonymous

They re-use the oil in which they fry street food several times over

enumaris

In China, they use "di-gou-you"

danielunderwood

Doesn't everywhere re-use the oil?

Semiclassical

"Kramers-Kronig relations" are another near miss

Anonymous

If you look into the cooking pan, it's almost always black and sticky

enumaris

7:40 PM

Gutter oil

Gutter oil (Chinese: 地沟油; pinyin: dìgōu yóu, or 餿水油; sōushuǐ yóu) is a term used in mainland China, Hong Kong, Macau and Taiwan to describe illicit cooking oil which has been recycled from waste oil collected from sources such as restaurant fryers, grease traps, slaughterhouse waste and sewage from sewer drains. == Description == Reprocessing of used cooking oil is often very rudimentary; techniques include filtration, boiling, refining, and the removal of some adulterants. It is then packaged and resold as a cheaper alternative to normal cooking oil. Another version of gutter oil uses discarded...

Avnish Kabaj

They say that the dough is knead using bare feet

danielunderwood

And I've been relatively fortunate in my travels. One of the people I went to Costa Rica with got a pretty bad infection and from what I hear the doctor trip wasn't fun. And one of my friends and his mom got food poisoning on a trip to Mexico while either his dad or brother was bitten by a stray dog.

Anonymous

And I've seen the street food vendors sometimes even using tap water (in extreme cases gutter water) directly to cook

danielunderwood

> illicit cooking oil

Anonymous

@AvnishKabaj True, lol

Semiclassical

7:42 PM

huh:

"In the second part of my report, we deal with the Kähler varieties known as K3, named in honor of Kummer, Kähler, Kodaira and of the beautiful mountain K2 in Kashmir."

Anonymous

I've seen that once

Semiclassical

Now -that- is a near miss

Lozansky

Lol nice work-around

Avnish Kabaj

@Blue gutter water?

enumaris

@Ke

@Semiclassical so they chose K3 rather than KKK...hmmm

Avnish Kabaj

7:43 PM

Do you mean that one specific WhatsApp video that went viral

Semiclassical

apparently

I mean, this was Andre Weil writing in 1958

Anonymous

@danielunderwood I can't say about other countries, so... :P

Semiclassical

so it's possible he wasn't aware of what KKK would mean in a US context

Anonymous

@AvnishKabaj I meant the water which is flowing into the street drains from roadside taps

Anonymous

I've seen them collecting it directly using mugs

Semiclassical

7:44 PM

but it was fortunate, regardless of whether it was intentional

Anonymous

To wash the plates

Avnish Kabaj

Ok that's enough

Thank you bye bye

enumaris

does anyone here currently live in India...

Anonymous

Lots do :P

Anonymous

Including me and Avnish

Anonymous

7:46 PM

@AvnishKabaj Don't underestimate the strong immune system we develop as a result though :P I haven't had food poisoning in years

Anonymous

Although I have eaten street food a few times

Anonymous

The street urchins almost have to live on that food throughout the year

Avnish Kabaj

@Blue I would rather not drink gutter water to develop immunity

Anonymous

And they survive!

enumaris

it's like Princess Bride!

danielunderwood

7:47 PM

Do a lot of people in India go elsewhere for college? It seems like we had a lot of exchange students from India (also a good number from China)

Anonymous

@AvnishKabaj Well, we have that "privilege" :P

Anonymous

@danielunderwood Yup, they do. For grad school

ACuriousMind

@Blue you can get fried locusts at a booth on the Christmas Market here. The taste is pretty bland

Avnish Kabaj

@danielunderwood yup

Rich indian kids do

enumaris

@danielunderwood China and India account for like 1/3 of all the people in the world

danielunderwood

7:49 PM

Ahh whether they were typically rich or not was my next question

Anonymous

@AvnishKabaj Not really. Even middle class students go for their PhDs

enumaris

so it's more of a total population kind of thing

danielunderwood

And yeah, but going to a foreign college seems like a strange concept to me. A lot of people debated whether or not to even go out of state

You might call it a foreign concept to me

heh

Anonymous

@danielunderwood Well, we don't have good research universities here, so going abroad becomes a necessity for higher studies

enumaris

well, the top Uni in China is ranked like 50 worldwide...

Belgium Leuven
70.5
Global Score
#64
Tsinghua University
China Beijing
70.4
Global Score
#65
Peking University
China Beijing

danielunderwood

7:51 PM

Although what I hear is that NC and CA have some of the best college systems in the states. I don't know if that's biased since I'm in NC or not

Avnish Kabaj

The current trend in India is that the competition for good indian institutes is too high. Like if you're not studying for 10 hours a day then some other bloke is.SAT/ACT whatnot is easy peasy. The difficult bit is the the extracurricular. Need money for that or a lot of talent

enumaris

UCLA ranked 13 :P

go bruins!

Anonymous

@AvnishKabaj You're talking about undergrad admissions

enumaris

unfortunately I will still forever be a shame to my dad for not getting into Harvard lol

Anonymous

Money doesn't matter much for PhD admissions

Avnish Kabaj

7:52 PM

@Blue yup

enumaris

internal resentment grows

Avnish Kabaj

@Blue dunno about that

Anonymous

@AvnishKabaj Well, PhD students are supposed to be paid by the university for their research work, not the other way round :P

enumaris

TA and research work

danielunderwood

I looked at higher tier schools, but my high school self wasn't about to write an essay to get in lol

Anonymous

7:54 PM

So if you're having to pay from your pocket for a PhD they're cheating you

enumaris

Generally TA work the first year or 2 unless you start off having an advisor already

Anonymous

@enumaris Yeah, that too

Avnish Kabaj

@Blue aaah

enumaris

I got an extra scholarship my first year

unfortunately it was only the first year

lol

Anonymous

I heard ETH Zurich is renowned for the high stipends they pay their grad students though. It's like 2x-3x the normal rates

Avnish Kabaj

7:56 PM

@Blue the cost of living is the highest there

Anonymous

But well, most European unis pay a decent amount

Avnish Kabaj

I think

danielunderwood

Speaking of schools, were you guys' physics courses mostly just derivations and working out samples?

enumaris

still...phd students get paid peanuts on average

my current pay is so much higher lol

Anonymous

@AvnishKabaj That's true too!

enumaris

7:56 PM

but I guess that's cus I'm not factoring in my tuition...

Avnish Kabaj

@danielunderwood couple of experiments

Too

enumaris

My first astro course

consisted of debate about why Pluto ain't no longer a planet

tells you how old I am lol

Anonymous

@enumaris Well, those 4 years of sacrifice do pay off I guess? Also you're working on something you love :)

danielunderwood

When I was looking at jobs, I realized that I'd get paid more in software than a postdoc in physics would. It was a bit sad, though I think I'd prefer the physics world

enumaris

@Blue 7 years

danielunderwood

7:57 PM

@AvnishKabaj ahh we had those as well and lab-only courses

Anonymous

@enumaris Damn, not going to the US then XD

enumaris

lol

u can do it in 5

Anonymous

In Europe it's like 4 years avg it seems

enumaris

if you're fast

Anonymous

But you need a masters degree

enumaris

7:58 PM

includes the MS

Anonymous

Yeah :P

Anonymous

But 7 years in one place is a bit too much

Anonymous

Id like to break it into parts

danielunderwood

My first astro course was not at all what I had hoped. It was pretty boring, but I think a lot of if was the professor. I took a "Spacetime Physics" course later that was pretty awesome

ACuriousMind

@Blue The difference is largely explained by Europeans usually seeing a master's degree as prerequisite for a PhD, which USAians don't

Anonymous

8:02 PM

@ACuriousMind Yeah, but still a masters degree is like 1 year and then 4 years of PhD. So 1+4 still less than 7. In Germany and some other places I think MS takes 2 years though ?

ACuriousMind

Yes, 2 years master is usual here

Anonymous

Ah

danielunderwood

Does European PhD have required classes? Or do you just go in and do research?

ACuriousMind

Also don't forget that US undergraduate education is often less specialised than European versions (we don't have "majors", you only study a single subject to begin with)

Semiclassical

@ACuriousMind i sorta wonder why the European system is comparatively so narrow vs. the US system

ACuriousMind

8:05 PM

@danielunderwood Depends on the institution, but if there are required classes they're usually few and rather specialized

Anonymous

Less specialization sounds good in some ways. I had no idea what I was getting myself into when I joined college

Anonymous

Would have benefitted if I could take some more formal math classes

Semiclassical

i imagine there's some history there

danielunderwood

The amount of education people get to college with in the US varies a lot. Maybe it's to round out general education?

ACuriousMind

@Semiclassical No idea, but I think it probably has to do with European universities being much older and feeling more bound to certain traditions

danielunderwood

8:08 PM

Some of the requirements I had really just seemed like excuses to charge people tuition though

Semiclassical

hmm.

i could also see it being a reaction on the US side, e.g. "we don't want to be like these hidebound European universities"

i'm sure there's a thesis on this somewhere

ACuriousMind

@danielunderwood Interesting point. Of course one has to look at the US system through the lens of tuitions

Anonymous

@danielunderwood The tuition fee problem seems real there. Here our tuition fee is practically 0.

Semiclassical

And through the lens of "how much support does the state give"

I mean, tuition for a public uni vs. a private college is naturally going to be different

Anonymous

But it's not true for all Indian unis. My uni is a bit special in that regard since it is very very highly subsidized by the state government. You can have a wholesome breakfast+lunch+dinner in 5$ at the uni canteens.

Anonymous

8:13 PM

Even then, the US tuition fees are like 10x times the average fees here

danielunderwood

Yeah I still don't understand part of the public/private debate. Like there are a handful of private colleges that have more prestige. But it doesn't say much about the quality of education. And I know a large number of people who went to small private colleges that have neither prestige nor good education.

Anonymous

Are the US community colleges good?

Anonymous

They're state run I suppose?

danielunderwood

A number of people use them to get the non-major courses out of the way or learning trades

They don't really have anything advanced from what I've seen. Though I'm kind of in middle of nowhere and that's my local one. I think my state has one in each county, but don't think they're as common elsewhere

Anonymous

@danielunderwood Say, how would you compare the Ivy Leagues with the UCs in terms of education quality?

Anonymous

8:21 PM

Apparently the Ivy League undergrad admissions are extremely biased. But not sure if that affects education quality.

danielunderwood

I'm not sure since I haven't been at an Ivy League school. Some of the lectures that I've seen online suggest that at least some of the classes are better though. Like I've liked a lot of the MIT physics lectures

I kind of wonder how consistency varies in other schools. Like I know in mine, the grade and understanding you came out of a course was very dependent on the professor. It seems like the Ivy Leagues may be better about that, but I'm not sure

Anonymous

I've seen terrible professors on the MIT OCW too, with a severe lack of interaction with the students

Anonymous

I wouldn't really bet on the teaching quality of all professors though :P

Anonymous

But of course most of them are established researchers

danielunderwood

Yeah there are both those and the ones that will pass anyone

I had a diff eq class where something like 40% of the class got a D or F and another 30% got a C, but I came out with more understanding than a friend that was in a class where 50% got an A...it's kind of weird

I also knew people that were able to get a perfect GPA in their program without going to most of their classes

Anonymous

8:29 PM

@danielunderwood Happens pretty often. It's almost always a matter of exam temperament and luck

danielunderwood

Yeah going into my second semester I realized that everyone was looking up the grade distributions to choose the professor they took a class with

I wasn't a huge fan of that mindset, but I guess I did get lucky with a couple of those courses

I had a classmate that would ask how to get an A in every class at the beginning of the semester and would do anything but actually try to learn the material

Anonymous

I really sort of wish grades were kept a bit more personal and not made a public thing for people to stare at. Otherwise, if you get a bad grade once in a certain subject, you tend to suffer from imposter syndrome. I've had situations like that before where I managed to improve from a D to a A in around 6 months, but I really wish the peer pressure was a bit less.

Anonymous

Looking back, that D was mainly because I didn't manage to get hold of good sources to study from and the lecturer was terrible at explaining things

enumaris

whomp

Anonymous

But then again, there's an argument that peer pressure helps you to improve

Anonymous

8:37 PM

@danielunderwood Heh

danielunderwood

I think grades shouldn't really be a thing and there should be a standard way to determine understanding. Grades vary too much between schools or between professors in the same school. I also think that we should have oral exams and professors would pass people or fail them with a list of things to keep studying. I guess that's not really doable in schools with 50k students and bias could easily come in.

enumaris

grades r weird

danielunderwood

@enumaris is loss sometimes expected to increase with sgd? I'd think it was learning rate, but it's happening far away from the optimum loss. Dropout seems to have settled it down a good bit, but it's still happening from time to time

Anonymous

Yeah. I support the idea of regular assignments and oral tests instead of end of semester exams.

danielunderwood

Yeah I'm a fan of take-home tests that are more difficult as well

Though some of my professors seemed to be paranoid that people were going to cheat by going to the bathroom...like how is someone going to cheat on those questions?

Semiclassical

8:42 PM

the bigger problem is people cheating on their phones

danielunderwood

I don't think we had anyone bold enough to try that. But I still don't really understand how someone would do that on our typical tests. Like if you're integrating to find an electric field or something and don't get credit without showing work, what are you going to do?

Anonymous

I don't really care about those edge cases. Those who really want to learn the material will learn it anyway and full use of those take home tests. They're not kids and they should be free to do whatever they want with their lives. But yes, if caught cheating, they should be punished - like deducting their marks.

danielunderwood

I suppose they could look up the integral

enumaris

@danielunderwood loss increasing is not really supposed to happen..but it can happen cus of high dropout or high learning rate...

sgd is pretty basic optimizer tho

generally Adam does wayyy better

danielunderwood

switching to adam makes my training loss and validation loss way different o.O

enumaris

8:46 PM

overfit?

I find in my experience Adam and Adamax are the two best optimizers by far

generally far superior to all others incl. rmsprop and adadelta

danielunderwood

@Blue yeah I agree with that. I had the unfortunate experience of wanting to learn physics, but not really wanting to learn the things I was required to. I definitely think they should be punished for cheating though

Shouldn't validation loss be higher for overfit? And I've wondered about the different optimizers, but haven't seen anything about their differences

I've just used grad descent since I actually know what's going on there

enumaris

U see a lower validation loss than training loss during training?

danielunderwood

I have heard data scientists say gradients and derivatives are scary though lol

and yeah

enumaris

That's possible cus validation loss is calculated at the end of each epoch, whereas training loss is an average over the epoch

danielunderwood

loss: 90.9244 - val_loss: 4.1736 but with sgd they were both similar until I started overfitting

enumaris

8:49 PM

SGD is guaranteed to converge to a local optimum given decaying alpha

but it just might take forever to do so

and it might be stuck in a saddlepoint forever

is that only for the first epoch?

danielunderwood

And I'm able to do the whole training set as my batch size here, so they should both be calculated per epoch I think

Nah that was on epoch 617

they're both dropping, but I don't know why they're so different

enumaris

in that case you're not using SGD lol, you're using just GD

If your whole dataset is one batch, then it's just gradient descent - there's no stochastic

danielunderwood

ahh

enumaris

technically "stochastic gradient descent" should be when you have 1 data point per update

and when using mini-batches it would be mini-batch gradient descent

danielunderwood

But it says SGDOptimizer lol

enumaris

8:52 PM

yeah cus it assumes you're using minibatches

but the "S" part of that is "stochastic" which only applies if you're using mini-batches

danielunderwood

For some reason I thought it meant that it put random inputs into the gradient

enumaris

Are you passing the right alpha to your optimizer?

danielunderwood

well not random inputs, but random nudges

enumaris

no, it's just stochastic because it's not able to see your whole data at a time

so the directions it takes w.r.t. to the loss function is random depending on which data point it's seeing

danielunderwood

ahh that makes more sense

enumaris

8:54 PM

indeed

danielunderwood

And I'm just using the default learning rate (assuming that's what you mean by alpha) at the moment

enumaris

hmmm

90 and 4 is indeed quite weird lol

I wonder if that's some error in implementation

danielunderwood

I suppose it could be. It started at like 100/10 or something

enumaris

yeah that doesn't sound right lol

And all you're doing is swapping out optimizer.SGD to optimizer.Adam?

Transcript for