yep

I did swap to a network with dropout, but I used grad descent once before swapping

I guess I can do some tweaking after this run to see if I can fix it

Also convolutions are for relations between features and recurrences are for relations between training samples, right?

not necessarily

convolutions and recurrence is quite different

you can build a RNN that spans past multiple training examples, but actually the default behavior of most frameworks is to still treat the training examples as separate

this is to increase computational efficiency

keeping everything in nice square arrays

But they still kind of retain information from the previous training example or something? I haven't really learned the technical side of RNNs yet though

no, not by default generally

a training batch input to a RNN is generally of type (N,T,D)

N=batch_size, T=time_steps, D=dimension of input

the recurrence is on the T dimension only

But the time steps could just as well be a sequence of any type of training data couldn't it?

Generally N is assumed to be independent

I can give an example

say you are training your data on Shakespeare

and you have Romeo and Juliet as your training set

Though I suppose it probably wouldn't work well if the model was built on the assumption of a sequence of events and your input wasn't

You can theoretically input the training set as shape (1,T=len(Romeo+Juliet),D=size_of_vocab)

but generally you don't do this

generally you break up romeo and juliet into mini-batches

and you input (N=batch_size,T=predefined max time_steps,D=size_of_vocab)

if you do it that way, the RNN will not propagate internal states along the "N" dimension

only along the "T" dimension

you will lose those super long term dependencies

Ahh I see

You generally do the latter because it's computationally MUUUUCH faster

Are they typically done with a sliding window or do you just sequentially do batches?

sequentially do batches

Cus LSTMs are good for "long term dependencies" but there's still a limit

if your sequence is like 1000 time-steps long, it will still be difficult

so mostly you split your sequence down to like...between 64-256 time-steps or something like that

for extremely long sequences (like say you are doing machine translation), it might be necessary to modify the RNN model to include things like attention.

So this is true of LSTMs as well? Like they don't "remember" batches

The "memory" (internal state) goes along the T dimension

so when it sees (N,24,D) it remembers, for each N separately, (N,1-23,D)

but there's no connection between (1,24,D) and (2,24,D)

when you move to the next training batch, it starts over

hopefully that's clear o.o

Well I certainly understand more than I did 10 minutes ago. Thanks!

lol np

there is a way to keep the internal state

I think

in Keras

but it's not the default behavior

I haven't had to work with that too much

I'm kind of caught in a limbo between learning the fundamentals and trying to build networks that do something so I'm probably trying to learn things that I need more background for

here: stateful: Boolean (default False). If True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch.

If you set this to true, you can go in between batches

@Blue In case you're interested in another opinion on that: it probably varies somewhat from university to university and also from department to department within the university. Assuming by UC you mean the University of California system, the better-regarded UCs like Berkeley are probably overall comparable with the middle Ivies like Brown and Columbia. I'd expect Berkeley to be better than average in physics though because of its close relationship with the adjacent national lab.

I did start reading the Colah blog and it's been pretty interesting so far though I wish they went into more detail sometimes. Like with this one colah.github.io/posts/2014-03-NN-Manifolds-Topology

that's like batch 2 (1,23,D) has "memory" of batch 1 (1,1-T,D) and batch 2 (1,1-22,D)

still there's no link along the N dimension in a single batch

I think I'm still missing something now that you say that. When you say that, it sounds like it would just have the weights of the last batch, but a normal NN would have that

hmmm

not the weights

the internal memory state

if you have a RNN you are feeding the output into the input. A normal NN doesn't do that.

I was thinking the internal memory state was just a set of weights on what's more or less a logic gate

no

Ahh yeah there's that part too

so there's a slight abuse of terminology on my part

technically a vanilla RNN has no "memory state"

a memory state is unique to LSTM

so a RNN feeds output O into the input

a LSTM feeds output O and a memory state M into the input

both O and M persist between batches if you set "stateful" to True

weights W are a separate issue

All of my "memory" talk previously are with regards to O amd M

a normal NN has an O, but O is only the output, it is never reinserted and so you don't have any of these considerations

Normal NN: Input -> magic box -> Output

RNN: Input+Internal State O -> magic box -> New Internal State O' + Input -> magic box -> New Internal State O''...etc.

So you move to a batch and M is either disconnected from the new input or zeroed?

LSTM you do the same for RNN but instead of just one O, you have O and M

it's zero'd if you set stateful to False

Both O and M are connected through the T-dimension only - in other words, you have a different O and M (read: O and M are vectors) for each different N.

hmmm I feel like this is the type of thing that would be helped by an animation

so a RNN process T=1 as a vector operation, then sequentially T=2, T=3, etc...until it reaches T=T. If stateful = False, it rezeros everything (except weights) and does it all over again.

Indeed

So you keep M in a vector (presumably for inference) and reset but don't throw it away on the next batch?

You throw it away for the next batch if stateful=False

otherwise you keep it for the next batch

The backpropagation will be truncated in batches though. THere's no way to backpropagate through time past a batch boundary.

This is called "truncated back propagation through time"

If you want backprop to go back farther, you will have to just input a longer T

but like I mentioned before, even LSTMs have limits on what T can be before they start performing poorly

performing poorly...you mean extra coffee break!

lol

And so M isn't kept for inference?

neither O nor M are kept

that would be like wanting to keep O in a normal NN

makes no sense

Your training is only to train the weights W

O and M should change depending on the inputs of course! :)

So the RNN part is completely gone at inference time?

And it just shifts where the optimum is at training?

And thanks for your patience. I'll learn more before asking you more RNN questions

Upon review, it makes sense, but

no bro

The RNN part stays, you just have different O and M with new inputs

you don't save O and M from your training set

Oh right. That would be like throwing away O in a regular NN

the fact that the conventions you use for the Fourier transform in QM forces there to be an extra $\sqrt{2\pi}$ in the convolution theorem is sorta yuck

curse you conventions

Procedure is Start with O=0, M=0. TRAIN: M+O+I -> O'+M' (+I') -> O''+M''...

INFERENCE: M=O=0 THEN M+O+I -> O'+M'...

(unless you take $h=1$ rather than $\hbar = 1$ but who'd ever do that)

if Stateful=False then M=O=0 happens between each batch

if it's True then M=O=0 is not done between batches

but it will always be set to 0 when you go to inference time

hopefully that's clear now :D

it would certainly be easier if I could draw a picture lol

@Semiclassical I do that and also set $2\pi=1$ so that $h=\hbar=1$

"sounds legit"

It's kind of one of those things that it's less clear than I thought it was, but at least it's clear that I didn't have everything down before lol

XD

I may make it a weekend project to learn more about RNNs and come up with an animation that shows flow through one

sounds good

Andrew does a good job teaching them

in course 5

of the specialization

I do agree with the $2 \pi = 1$ units

Well I'm in the course 1 non-specialization at the moment...though I think I can get through a week of that course in a night if I put a few hours into it. At least with what he's going over now

connecting to a different server on SQL is such a pain

Sounds like a good pace

Shouldn't you just have to swap the host/user/password?

well I don't have to do user/pw

it's all single-sign-on

but

I gotta change the server and then if I want to run a query on the old server I gotta change again

pain

Put in lines for both and comment out the one you don't need lol

Or stick it in an env var or something, but the comment one is usually easier for me

it'll give me an error if I'm on the wrong server

give it an error back

wooo I've almost reached my tpot score with a nn

nice

these tables I'm working with now are so big...90 million rows...

I keep running into memory exhausted errors -__-

I'm working on a kaggle competition and one of the kernels referred to the data as a "huge dataset"...it's like 4000x4000

And the whole thing fits on my GPU at once

lol

i've built a 79000x79000 matrix before

good thing it was a sparse one

100mb space instead of 15gigs or something lol

Yeah this one is sparse from what I've seen. I haven't actually told keras that though

is it using a sparse representation?

I think numpy doesn't actually have a sparse class in it, you will have to move to scipy

Ahh probably not then. I'm just loading everything in through pandas, so I think it's numpy on the backend?

yeah probably

if you get into sparse matrices, you'll have to worry about how that matrix is actually represented

and then a lot of operations are difficult on them...

Yeah I remember wanting to use a sparse matrix in my numerical analysis course and it being a bit of a pain

big pain

The data for this competition is so weird though. Like it's really clean, but they've given all the columns random names and don't really tell you what anything is

lol

10 years of clinical data...count the rows!

let's see if SQL doesn't spaz out

Isn't that kind of getting into hadoop/spark territory?

74 million rows

I don't work w/ hadoop or spark

ha do do do do doop it's Datman!

lol my workplace is gonna ban linkedin now

cus some guy went around pretending to be an employee and adding other (6) employees as connections....

how tf does banning linkedin solve that problem

person did bad thing -> platform for thing is bad

-> must ban platform

makes 0 sense

Maybe they banned images because someone went around taking pictures lmao

↑ breakthru ML advance for the ML geeks... what do you think? :)

not familiar with how "curiosity" is implemented in ML beyond the curiosity parameter in UCB algorithms lol

@enumaris its very new technology with many ramifications/ angles but its basically (from math/ algorithmic pov) quite simply minimizing "prediction error"!

sounds interesting

yeah think its way cool & am all over it (blogged on it heavily this yr), think it will be revolutionary/ paradigm shifting in intermediate future... :)

there is only so many paradigm shifts one can handle

the mind is willing

but the body is weak

lol youre a phd think you can stand it... maybe :P

I am in love with Ornstein-Uhlenbeck process.

@EmilioPisanty Congrats. What does it mean?

eww y tho

Langevin fetish?

Fokker-Planck fetish?