The data is a real time series not artificial it is unfortunately limited in size to the number of points that I have. I don't think that it is the best solution to the problem but right now I am trying to pick a suitable model from many (ARIMA, ETS, LSTMs etc) and at the moment the validation accuracy is comparable to that of simpler models. So I'd like to just confirm the model is working correctly before I evaluate it on the test set, so that I can correctly compare how good it is w.r.t the other models I have tried.

I also think it is weird my validation accuracy typically is better than my training accuracy... I have built the model using keras and it is quite simple. To my knowledge it doesn't use any dropout (or at least I have not configured any)

The main problem for me is that you are using a deep learning model with very few data... My personal opinion is that ittle can be inferred about the model with this small data-set. For me, is not worth working on this model with this data... I strongly recommend you try it with some other data-set

Yes I am inclined to agree with you, but I need to evaluate the feasibility of such models with these kind of (unfortunately small) data sets. I therefore want to make sure before proceeding that I have done everything I can correctly in this tuning phase, regardless of whether I use the final model or a simpler one. Ideally after this step I will work on a model which instead of forecasting one time series at a time, will produce many forecasts for all the time series at once, hopefully learning the correlations between the individual time series.

But I thought it would be better instead of jumping straight into that more complicated problem to see which methods work well on a single time series first, to see if I can learn anything that might help me design a more complicated model. Do you think the approach is wrong?

The complexity of the problem is not necessary related to the size of your data: working with a simple problem at the very beginning is not a bad approach to get insights about how different model work, but, you still need huge data. If you don't provide lots of data to a Deep Neural Model, you are breaking one of the most important pillars of Deep Learning... It is not the same, but, in an hypothetical scenario, will you work on a linear model though you know your data shows high non-linearities?

With this small data-set, what does happen with the generalization capabilities?, beyond the fact that with your tiny validation data it works...

So as the data set is small are the hyper parameters largely irrelevant? And the the model will typically always perform well against the validation data set?

Not necessary... but with a small data-set, I feel it is hard to interpret results

Let's the community give more insights on this... I also gave you some other hints, beyond from saying you have few data :)

I agree they're hard to interpret probably because of the lack of data as you suggest. But they do seem to suggest that at least the model generalises well and as time progresses more data would become available to train and validate with. True I will also think on what else you suggested.

Yes, for sure. It is the first thing I said... I won't be specially concerned about the fact your model is generalizing well... But you still get the feel that something is wrong, maybe because you don't expected having good accuracy so early. Your model may be ok! But you want to relate the fact that hyper-parameter tuning does not affect the results, but indeed, they affect... as you can see when you increase the number of neurons...

Yes very true I think the hyperparameters do have some effect on the validation acurracy of the model as you said and the plots show

I am just surprised that the ones I picked seemed to work kind of ok first time without realy thinking about it

They affect, quite a lot... number of neurons is one the most import hyper-parameter....

which made me think something must be wrong. and indeed when I use any kind of sensible configuration i.e. using more than 5 units in the LSTM layer I seem to get kind of good performance

I am just hesitant to proceed from this point onwards as I will then expose my model to the test data

for example if I decide to take the parameters shown as the most accurate for the validation set, re-train my whole model on all of the data I set aside for training and validation

then run it against the test sets, I can't "undo" that

so I want to make sure that my tuning here is as good as it can be before I move on to the test evaluation.

This are the decisions a ML engineer should make every day... My first decision with this problem is not to test it with this data

Is there another way to test it?

Well... I'm not saying your approach is bad in the sense that you want first try simple models

but the approach is bad in the sense that you are not fulfilling one of the most important premises of Deep Learning

anyway, you have to take your own decisions

Good luck my friend!

thanks for your time

For sure! These are my personal considerations, have a nice day

you too