I am very pleased I am mistaken :) Thank you. Maybe "model selection" has a technical meaning that is not exactly what I am looking for. In my case I am trying to find a good model that accurately describes the data I have. Is there a better way to do this?

What do you mean by "a good model" and why would a hypothesis test be a good way to achieve it?

I am in no way a statistician so I may not be using the right words. I want a model for my data so I can, for example, make predictions or look for anomalies. I just mean a good model in the normal non-formal sense of fitting the data reasonably well. I was using the hypothesis test to see if one model was at least better than another. Any suggestions for better ways are gratefully received.

Yes, that's fine. "Fitting the data reasonably well" doesn't really change much with sample size, but our ability to detect even trivial deviations from a specific model (like an exponential model, against a gamma model) does change with sample size. At larger sample sizes, you become practically certain to reject an exponential model against a more general alternative like the gamma (and then reject a gamma against a generalized gamma, and on an on in turn), but this doesn't imply that an exponential model would be perfectly fine for everything you want to use it for. ...(ctd)

(ctd)... Your data will almost certainly be neither exponential nor gamma (George Box's epigram is relevant); the relevant question is not whether you can detect a difference (as sample size grows, this becomes a certainty) - it's how much that matters to your inference. That's not a question answered by hypothesis tests; it's better addressed by diagnostic tools like Q-Q plots and simulations.

Thanks for this. Do you have a link for George Box's epigram? Your point about sample sizes is very interesting but I don't fully understand it. If you use the LRT you get a $\chi^2$ distribution with one degree of freedom I think here. Why would you expect to reject at the 5% level almost all of the time? It would seem you would reject 5% of the time if the data were really exponentially distributed, for example.

hi... if you don't mind chatting here I would love to hear more of your views on this.

Can't stay long. George Box famously said "All models are wrong, but some are useful." -- real data will not be either exactly exponential nor exactly gamma. This doesn't prevent one or both from being a very useful approximation.

@Glen_b oh

so the argument is that if the models are wrong then you are very likely to reject in favour of a more general model

The asymptotic results you have for the LRT are conditional on the model holding. But the null cannot be exactly true. At sufficiently large sample sizes, you will always detect even the most tiny (trivial) differences from it. Then - because the alternative is also not exactly true - if you tested it against a reasonable, wider alternative (any sensible generalization), it too would be rejected at a sufficiently large sample size. You'll reject any small model.

that's intereing

well in my case the models are all going to be imperfect

so you are extra right :)

so what to do? I mean qq-plots are just a visualisation

it's not very helpful for analysing thousands of point processes

The hypothesis test doesn't tell you how big the deviation from exponential is. Nor does it tell you what the consequences of the size of the deviation is.

But those are (at least to some extent) answerable.

A QQ plot does let you see (at least in larger samples) how close to exponential you are. Simulation methods and other sampling or resampling techniques let you get some sense of how much difference it might make.

(To assume exponential rather than gamma, or to assume exponential rather than a population distributed exactly like your sample, for example.)

You can even posit a variety of situations similar to your data (or that would be plausibly consistent with it) and see the consequences of using a given model in each of them (against using the "exact" presumed situation for that case).

can you get anything numerical from the qq-plot?

rather than just looking at it?

I need something automatable :)

or would AIC help?

actually I don't understand.. doesn't a large ratio imply a big deviation?

Well, sure, but you need to beware anything that will head you back towards hypothesis tests. You can measure discrepancy, for example, such as (1-R^2) in a Q-Q plot. For a single parameter, a decision based on AIC is no different from a hypothesis test with alpha = 15% (approximately)

So again, you're back in the same problem

If you have some sense of the effect of a given amount of deviation, and know how much you are prepared to tolerate, you might be able to check whether you have less than that.

No, a large likelihood ratio doesn't imply a big deviation, since you can get it with a small deviation and a large sample size.

this is interesting and tricky :)

are you a statistician out of interest?

Consider a case where you have data that is nearly Gamma(0.99, beta). Then with a large sample, the likelihood ratio will be large and you'll detect that it's not exponential. But an exponential model might work fine.

Well, that's what my PhD is in. What makes a statistician a statistician?

ok so I get what you are saying.. basically I need not only the data but some truthed data relating to any inference I want to make from it. I suppose I could just try to predict interarrival times

that is train on the first half and test on the second

or is there some general cross-validation method for goodness of fit tests?

If the aim is to get interarrival times, that would be useful for measuring consequences of the amount of deviation you have.

ok.. so let's go with that :)

which I sort of thought was being measured in the LRT in the first place

I mean this is what I have in reality... I have a point process, I measure the interarrival times and then want to model them

But it depends on which aspects of interarrival times are most critical for you

I would like to know a) if a gamma distribution is a better fit than poisson and b) is a gamma distribution a good fit fulls top

Which comes down to "what kinds of things do I need to say about the data"

Whoah. Where did the Poisson come from. It's for counts, not times.

how about predicting when the next event will be?

poisson process sorry

which gives exponential interarrival times

Right, no worries.

But when you say "predicting when the next event will be" -- what do you mean? It has a distribution. Do you mean just predicting the average? Some quantile? What is a measure of how well you do?

If you have enough data you can of course just use the sample itself.

The main thing is to make sure you're answering a question that's relevant for you (such as how badly non-exponential it might be or how much it might change your answers). One option is to see how much difference it makes to what you do when you assume exponential and then assume gamma.

right

thanks for this

It's all very informative

oh and it's clear what makes a statistician a statistician I though

you have your hair on fire and your feet in a bucket of ice...

and you say on average you are fine :)

The problem is answering the right question is a lot harder than answering a question that's not especially related to the issue at hand. (Tukey's epigram is relevant there.)

Well, he had many. But I mean this one: "Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise."

The problem in small samples may be even worse. Then even a disastrously large deviation from the exponential might be missed. If you're really lucky, you might just end up with a sample size where the size and type of deviation you can detect lines up well with the size and type of deviation that matters.

Getting even an approximate answer to a vague question will require some thought about what you're trying to achieve.

all true!