@JimB can you help me understand something? I'm trying to help someone fit some data of ~3 mil points via either 2, 3 or 4 Gaussians and would like to be able to use the AIC to figure out which number of Gaussians provides the best model. Unfortunately which number it tells us is optimal changes as we add more and more of our data, because the likelihood (and difference in likelihood between models) changes non-linearly with the sample size.

That is making us very uncomfortable, as we'd like some measure to help us figure out how best to model our data, but preferably one that is not so sample size dependent.

@b3m2a1 It turns out to be a simple answer. As you noticed the likelihood (and therefore log likelihood and AIC) changes with sample size: AIC can only compare models with the same exact data (not only the same sample size but the same exact data).

And two other things that you probably already know: (1) AIC will tend to choose the more complex model as sample size increases. In other words when comparing say 2, 3, or 4 Gaussians, a sample size of 1,000 might state that just two Gaussians are necessary while 100,000,000 samples might state that 4 Gaussians are best. At some point "practical significance" might come into play in that while 4 Gaussians are deemed best, maybe just 2 Gaussians are adequate for practical purposes.

(2) AIC only provides a relative ranking and not an absolute measure of goodness-of-fit. The worst model with respect to AIC might be more than adequate for practical purposes. The best model with respect to AIC might be horrible.

@JimB Thanks a bunch. That was my intuition, which is why we felt uneasy. It's good to hear it from a Statistician® though.

So would you agree that we'll need to also invoke intuition in choosing the best model?

The two-or-three Gaussian models seem to be telling us something physically, but it was hard to justify the four Gaussian one

I think I've set up prettify correctly to use your lexer tokens and things

As for something that is not sample size dependent...If it makes sense, then the area between a candidate distribution and a nonparametric density estimate might be a metric to use. Again, it's one of those "It depends..." Suppose it's the tail regions that are more important to you. Then what I just proposed might not be what you want.

But I clearly am missing something

@JimB What is a nonparametric density estimate? :) I am very much so not up on my statistics

Mathematica's SmoothKernelDistribution function. (i.e., a smoothed histogram).

Ahh. I see. I will suggest that on Monday and my coworker and I will try it out.

@b3m2a1 Looks like a bug. For whatever reason ListQ@$$tagStack is one token.

Thanks for all the advice!

@halirutan it also doesn't seem to be picking up the local variable declarations with _

I am configuring like this:

  <link href="https://www.wolframcloud.com/objects/b3m2a1/tutorial/theme/prettify/styles/prettify-mma.min.css" rel="stylesheet">
  <script src="https://www.wolframcloud.com/objects/b3m2a1/tutorial/theme/prettify/src/prettify.js"></script>
  <script src="https://www.wolframcloud.com/objects/b3m2a1/tutorial/theme/prettify/src/lang-mma.min.js"></script>

Not sure if that's the right order, though

@b3m2a1 Something is weird. If you copy this snippet to SE, which also uses the google-prettify highlighter, things seem correct.

@b3m2a1 You css class seems wrong.

<pre class="mma-input prettyprint prettyprinted" id="pre27" style="">

it should be

class="lang-mma prettyprint prettyprinted"

@b3m2a1 As a statistician I'm not wedded to needing models that make complete physical sense. I view fits (either regressions or sampling distributions) as "descriptions" of the data or something that mimics the data generation process. For example, a normal distribution (which goes from -infinity to +infinity) might very well describe the distribution of a discrete and non-negative distribution.

If there are physical justifications for a model, that's great. So it depends on whether you need to "describe" something or "explain" something.

So I'm attempting to make a case for "reasonableness" rather than "intuition".

@halirutan ah crud I left out that step of the post processing

Thanks

@b3m2a1 No problem.

@Szabolcs This is so annoying. A GraphicsGrid I ran out of idle curiosity just lost me two hours of work because it must have tried to use more than 2GB of memory.

is this AI posting? community.wolfram.com/groups/-/m/t/1088533

If so, these things are getting good...