Greetings

Hi Cagdas!

Let me warn you first of all that I am nowhere an expert in the topic.

That's alright.

I am focusing my learning on these matters these days as well.

I know somethings for sure, and some things vaguely, and somethings not at all.

Could we start from what what this was referring to in your last comment?

I was referring to the other stackexchange question.

It wasn't clear in the comment I agree.

OK. But I can't do much about that since cross validation for time series is perhaps most natural where only one-step-ahead forecasts are used, so the validation sub-samples are made of one point each.

Yes, but training samples are always overlapping.

True. But they are overlapping in regular cross validation (for cross sectional data), too.

In K-Fold?

Yes, isn't that the case? Since you use all but a small part of data for validation and do that multiple times. But perhaps it's worse in time series as the ordering in time plays a role.

Standard definitions of CV is related to regression not time series. That's the confusing part.

As you increase K in K-fold overlap increases, hence the variance of the estimation increases as far as I know.

Anyhow, I don't think one can do much about the overlap in the time series setting since doing, say, 10 partitions of data would mean using time series 10 times shorter than the original one, and that is a big problem in small samples.

True

I would use same length training samples however instead of increasing the length everytime.

So even though it is a valid critique, I will have to leave it as is, I guess.

I agree with your last idea, and actually I have done it that way.

So what was the question again?

I don't know why but I thought it makes sense to have a rolling window, not an expanding window. Expanding window must really suffer due to the overlap.

Expanding window I think will have some information theoretic problems.

For the validation samples towards the end you will be providing more evidence to the model.

The questions were about I didn't understand the sample size discussion above and about what are you referring to in Cross Validation in small number of samples also has a large variance?

Nothing special. Small number of samples mean bigger overlap hence larger variance.

When number of samples are small I prefer very small K in K-fold for regression for example.

OK. What about the first one? Is it about the same thing?

First one?

I didn't understand the sample size discussion above -- this one.

I didn't understand the sample size mismatch and parsimonious model discussion.

Oh, I think this is an important one.

Let me explain.

Suppose we have one population, and we have two samples from that same population.

One sample is small, call it s. Another is big, call it B.

In s, AIC would select a small model relative to B.

one minute I am interrupted by somebody

That is, even though the data generating process is the same, AIC will select a model which will be more parsimonious in a small sample.

Conversely, AIC will select a less parsimonious model in B.

So the problem is, if I use small training samples to select a model, the selected model will likely be too parsimonious. Because training samples are by construction smaller than the original sample.

This is not much of a problem in cross validation for cross-sectional data because there the training samples are of nearly the same size as the original sample (either leave one out or leave K out is not a problem as long as K is small, and it typically is).

Meanwhile, in time series cross validation the training samples can be twice as small as the original sample. That can matter a lot, IMHO.

Got the main point?

back. one moment reading.

Ok I think I understand what you mean.

The issue is the following. Even if the data generating process is the same, what's important is the evidence you receive,

It's IMHO information theoretically correct to offer a smaller model when the sample size is lower.

Even if the underlying process is the same. It generates better out of sample prediction.

True, I agree. But it causes a problem since you actually have a larger sample, only your training samples are small.

I think you are carried away with the definition of "sample".

Let's call it "evidence".

So you make a choice based on a small sample while you need to use the model that you can estimate on the large, original sample.

I think I totally follow what you are saying, I just cannot express myself well enough to convey the message.

when you mean "estimate" you mean "forecast" right?

No, estimate.

Then I didn't understand that sentence.

Think of an extreme example. Suppose the training sample is 50 obs. while the original sample is 1000 obs.

Of course, ARMA(5,5) will not work on 50 obs. But it may work pretty well on 1000 obs.

So if you only use the training samples, you will always get pretty bad validation results for ARMA(5,5).

But if you used training samples of size 800, then you would see that ARMA(5,5) is actually a good model.

The problem is, the training samples are too small to allow the actually best model to be selected.

In forecasting it is irrelevant. AIC won't select the true model even if you have many samples as it is incosistent.

Where the actually best model is a model that would give the lowest validation error for training samples of the size equal to the original sample size.

Agree. But what I say is that it will select too small a model if you use too small a sample relative to your actual problem.

Since you do have the large original sample and estimating a large model is feasible in the original sample.

But that's not the point. Don't confuse yourself with "sample size" or "subsample".

OK, feasible may not be the right word...

The idea here is that IC will help you work with what you have in your hand as training sample.

Whether you are keeping some more data in your pocket is not relevant.

Agree again. But unfortunately it does not solve the original problem. In the original problem you have a large sample relative to the training samples.

Let me put it in another perspective. I say that AIC on original sample will select a richer model than the one obtained from time series cross validation.

I don't understand. You are doing the partitioning. You can partition however you like.

You mean if you use AIC on the whole sample vs CV? That's an advantage of using AIC over CV.

Oooh, that's what I was trying to clarify :)

Altough not precisely that thing. I was trying to see if my argument of systematic bias towards smaller models is true in case of time series cross validation.

So your question is only applying to CV?

My question that I was trying to clarify over the last 10 minutes is: is it true that time series cross validation favours smaller models than AIC, where AIC is applied on the full original sample.

Probably it would. But it will not be a just competition I suppose.

I think I don't know will be a better answer on my part.

On the other hand asymptotically leave one out CV is claimed to be equivalent to AIC.

If the answer was YES, that would mean AIC is clearly more relevant for someone who wants to select a model in this kind of time series setting.

That asymptotic equivalence is for cross-sectional cross validation.

Yes it applies to regression.

I don't know if it applies to time series.

For time series cross validation, it sounds like this: ...But asymp­tot­i­cally, min­i­miz­ing the AIC is equiv­a­lent to min­i­miz­ing the leave-​​one-​​out cross-​​validation MSE for cross-​​sectional data, and equiv­a­lent to min­i­miz­ing the out-​​of-​​sample one-​​step fore­cast MSE for time series mod­els. This prop­erty is what makes it such an attrac­tive cri­te­rion for use in select­ing mod­els for forecasting.

Taken from Rob J. Hyndman's blog.

It makes sense.

So you see that AIC is fine, but time series cross validation need not be fine...

(Although I cannot strongly claim otherwise.)

Why did you conclude that from the above paragraph?

I only conclude about AIC. Time series cross validation is not mentioned there.

Yes it is mentioned. Read again.

"out-​​of-​​sample one-​​step fore­cast MSE for time series mod­els"

It is far from obvious that this should coincide with time series cross validation.

Or maybe I am too shortsighted :)

No I think he states rolling window LOOCV will work just as fine as cross-section.

Where do you find that?

robjhyndman.com/hyndsight/crossvalidation he says how to do LOOCV for time series but he does not say that it will be asymptotically equivalent to AIC.

Whatever, I think I need a break to digest all this information. Could you post another comment under any of my posts at Cross Validated if you get some good ideas on this question?

And for the future, here is another thing that bothers me. Maybe it will be interesting for you, too.

OK. See you later.

Some guys on Cross Validated tend to criticize model selection by AIC or BIC but they do not seem to provide a better alternative. I feel that cross validation is not their favourite either. (At least since leave-one-out cross validation is asymptotically equivalent to AIC and leave-v-out to BIC where v is defined by some formula which I forgot.)

So what is the way to go? I thought maybe there are some answers in Frank Harrell's textbook but I don't have a copy. I suspect it has to do with bootstrapping...

All these are approximations to an otherwise incomputable complexity. The method who is tailored towards a specific model will work better.

Basically it is known that there no single method that will give you the best result under any given dataset.

But sometimes you really don't know what kind of relationships to expect in the data. Then I thought AIC, BIC or cross validation is a decent way to get somewhere.

Thanks for your time! You really spent quite a lot of it helping me. I appreciate that.

There are some advanced methodologies covered in the field of Artificial General Intelligence. Marcus Hutter has very difficult papers.

I don't understand them myself yet. But the idea is to approximate the model complexity as accurately as possible given any situation.

Haha, "very difficult papers" is very encouraging :)

But thanks for the reference, I'll check them out.

bye

Bye! Have a good rest of the day!