hmmm

Anyone here proficient in LSTMS as they are implemented on Tensorflow or Pytorch?

@enumaris: There are definitely a few on the site (not me, yet - I've booked a course for RNNs, so maybe in a few weeks . . . .) You won't find many experts logged in to chat at any time on Data Science currently, the site doesn't have the critical mass for it. So if you are looking for help you may have to go to the trouble of creating a question.

Someone here, even if they don't know about LSTMs in depth, might be able to discuss how to put together a question about them

I see, I didn't really want to make a question because 1) I have multiple questions and they are mostly kind of boring technical questions about the implementation of LSTMs in Tensorflow and Pytorch and 2) I'm not sure how to best formulate the questions

Actually my question would apply to any RNN in these frameworks as they are implemented I think, as long as the frameworks implement the different RNNs in a consistent fashion.

OK, so throwing out a single question with lots of separate technical details probably won't go down well. If they can be split up and separate, that sort of thing is an OK questiom in my book: e.g. How to implement backpropagation through time in PyTorch? . . . With at least a start at your code, showing where you get stuck because the concept or how to use the library is not clear . . .

. . . but I would agree that "How to . . ." questions are a lot of work. If you are not sure and a bit vague about how things work, best to keep it short. But then the question may get down-voted for not having enough detail to isolate what the real issue it

Where are you starting from? have you looked at the TF and PyTorch examples from their library github pages?

A quick google finds things like this: towardsdatascience.com/… . . . but not sure if that helps.

Or this: tensorflow.org/tutorials/recurrent

Or for PyTorch: pytorch.org/tutorials/intermediate/…

Yeah

I've followed several tutorials

and looked at the documentation in both

So now you're looking to branch out and do your own thing, and it turns out the tutorials didn't really explain everything :-)

I think I would be able to build a RNN or LSTM using these frameworks, but some of the details of their implementation confuses me

My questions mostly revolve around the fact that RNNs and LSTMs as implemented in these frameworks take in data in batches, and I don't see a clear way that the internal state or cell memory (as for LSTMs) can propagate from the end of one batch to the beginning of the next batch, assuming I'm breaking down a very long sequence into batches.

Or maybe the internal state simply does not propagate and the frameworks are relying on the fact that your sequence length is long enough to compensate for the dis-continuity between batches.

but it's hard for me to figure out how to formulate this question well lol

I don't think the batches do cross-link like that. Each batch is a separate sequence

Yeah, it appears to me that that is the case

So the RNN never learns continuously. There's a cutoff, a standard or maximum sequence length

Right, that appears to be the case

but in that case, if you are using a RNN to write a novel, for example, it could never do it

since if it introduced a character in one batch, it would have forgotten about that character in the next batch

The sequences in batches can overlap I think. But still I'd need to do the course, so I better shut up in case I advise you something that is horribly wrong :-/

If you have seen the output of character-based RNNs e.g. karpathy.github.io/2015/05/21/rnn-effectiveness - it's pretty clear that RNNs are a long way away from writing a novel.

Yeah...but I mean, one of the fundamental reason of using RNNs over other networks that I've been told is "they can take sequences of arbitrary length"

if you have to break up a long sequence into sequences of exactly 100 units each...what's the advantage of using a RNN over say a 1xN convolution...

1x100*

Well, you don't have to have fixed lengths. You would not for machine translation for example.

I guess at test time it matters...

I suspect a lot of the character language models do use exact sequence lengths to train, because they can be put into raw tensors

hmmm...

whilst ragged data cannot, so it would slow things down

but it should still be possible in principle to use ragged data to train

PyTorch will probably be better at handling that than TensorFlow

I suppose it wouldn't be a huge deal if the state also can't go past the barrier that truncated back prop sets up...

what do you mean by ragged data?

Array of arrays where the inner array can be different lengths

I see

cannot be expressed as a tensor

I don't think Pytorch Tensors can be like that either tho?

Oh yeah, that's another issue I have with how to implement RNN's

how can I use ragged data lol

pytorch.org/tutorials/beginner/nlp/…

I think TensorFlow has some horrible loop-until operation construct in order to do this.

at the end of that tutorial it asks you to augment the word-level LSTM with a character-level LSTM that gives the character embeddings $c_w$ as the last output of the c-LSTM

OK, so from your link "Pytorch’s LSTM expects all of its inputs to be 3D tensors." - I didn't know that

yeah...

If you have a sentence like "that dog over there ate my ball"

the word level inputs is fine

. . . so probably the advantage for PyTorch is in not needing TensorFlow's looping operation construct

it's a length 7 vector

but the character level inputs is like...[[4-dim],[3-dim],[4-dim],[5-dim],[3-dim],[2-dim],[4-dim]]

corresponding to the length of the words

and I can't turn that into a tensor

I had it as a list of tensors, but then I need to write some for-loop in the definition of the LSTM class so that the char-level LSTM can step N-times for every 1-time the word-level LSTM makes a step

I guess pad, and use whatever loop/stop condition is appropriate. IMO, this isn't a limitation of LSTMs, it's a limitation of the libraries.

I would have expected Pytorch to be better at this since it's specifically written to be dynamic

They do this for a good reason though - to get vectorisation on a GPU . . .

I thought the whole point of being dynamic was to be able to create different sized graphs on the fly...

seems like a highly non-trivial exercise to implement that character-level augmentation

mostly due to difficulties in implementation, and not theoretical difficulties with how the LSTM should be set up

I mean conceptually, I have a very clear idea of how that LSTM should look like

But you probably don't want to be changing shape of the graph in-between words . . . there's "on the fly" and there's interrupting the GPU

I see...

hmm

Are you familiar with the Pytorch backend in C by any chance?

I have a totally unrelated question that I posted to the pytorch forum but didn't get an answer for

Not at all. I've yet to write anything of my own in PyTorch. I've looked briefly at some simple examples. On my bucket list for next year . . .

I see

discuss.pytorch.org/t/… <--- well that's the question anyways

I guess it would be good to know if there's danger in using their CrossEntropyLoss class for problems with a large (>100) number of classes.

With regards to the "interrupting the GPU" comment, I feel like me writing a for-loop to deal with the ragged data at the character level is doing exactly that isn't it...interrupting the GPU by forcing it to explicitly loop over sequences of varying lengths?

Reading PyTorch code, I interpret ignore_index as being a target value to ignore, so can have ignored ground truth data, for instance if you had a class hierarchy and didn't care about mis-classifications in sub-parts of the hierarchy unrelated to the ground truth. So you might set your ground truth to [1, 0, 1, 0, 0, -100, -100, -100] if first values were class 1, 1a, 1b, 1c, 2, 2a, 2b, 2c, and your target class is 1b

. . . although with my level of knowledge of Torch, I could just be making that all up . . .

Actually scratch that, I think it is similar concept, but my example is wrong

hmm

Yeah, my knowledge of torch is not strong enough to really know...

Regard GPU, I think there are ways to write the LSTM processing so that it ignores or doesn't process beyond a certain point

allowing you to use tensors to represent the data for speed up, at the expense of some ugly and not very intuitive code

hmmm

well at this point, the only data I am trying to process is already truncated to be uniform in length, I did that during data pre-processing so that I could use a convnet on it.

I guess the ragged data won't matter for me too much until I move on to different projects

would be nice to be able to know how to do that though

I don't think I follow your statements about the ignore_index issue

This is TensorFlow loop construct I was referring: tensorflow.org/api_docs/python/tf/while_loop

Ignore me on ignore_index. I was just guessing

ok

I feel like Tensorflow is a bit...frankenstein ish lol

Yes. I think of it more like building a ship in a bottle . . . you write code that defines some maths that executes some other code later

Got to go

They have a tf.contrib with like experimental code, and then they move the well tested versions over to to tf.nn while keeping a version in tf.contrib...so any one function could have like 3 different versions that all do something perhaps very slightly differently...

Alrighty, nice talking with you :)