Hi Szabolcs

made it

In stats I'd complain that you don't quite mean what you wrote; the law doesn't apply to observations, which is all you have, but only to some unobservable quantity. " y = log x - a x" might refer to an expectation, or perhaps something else. If you contemplate least squares, you're looking at expectations, but OLS implies a bunch of assumptions, at least if you want to apply some inference (e.g. additive, constant variance error term, independence, and so on).

Generally also an assumption that x is fixed or observed without error (or at least that the error is so small the resulting bias is of little importance).

Are you missing something here: " x - \ln a,"

That would be a straight line

I understand the aim not to distract with irrelevant detail; it's a good impulse. On the other hand, often the detail is kind of important to the kind of advice that might be relevant (and I have no way to tell, so I'll rely on your assurance).

Hi @Glen_b

You're trying to estimate the slope of the line at the far right?

@Glen_b yes, I meant ln x - x, sorry

No problem.

@Glen_b Yes. And I do not know anything about how it differs from a straght line on the left.

Ah. you just answered the question I was typing.

In your real problem, the right hand side is not linear?

@Glen_b Not quite, but I can make it linear.

By transformation?

Yes.

Note that if you were happy to use unweighted least square before transformation, you should not be happy to do so after it.

and vice-versa

Exactly.

What if it were okay to use it after transformation?

Do you know of any methods applicable to the linear case?

Ack.

Okay, first Q first

If it were okay after, it won't be before. (there are issues of both bias and heteroskedasticity).

@Glen_b Actually, from this discussion I get the impression that there isn't an obvious straightforward method for these kinds of things that statisticians commonly know. Which is useful information. I wasn't sure if there were such methods. So I might just go back to the problem and play with it for a while.

The bias may be small enough you don't care, the heteroskedasticity will matter if you care about the uncertainty in your estimates.

Not so fast, for goodness sake, you asked a bunch of questions and we're still talking very generally.

:-)

I just looked up "heteroscedastic" --- so that means that the noise is not of the same magnitude for all x?

We haven't quite yet got to the point that I am confident about what problem I am even trying to solve.

Yes. But I haven't even got to the second of the two questions you asked me earlier. If you ask more I'm going to miss things.

I won't :-)

If you haven't got to the point where you're saying in desperation "why won't he stop asking dumb questions?" then I probably haven't finished figuring out what the problem is yet, let alone how to answer it.

On the question of applicable methods, I see many potential ways of approaching it.

One way would be to fit some generically smooth function that only impacts the left hand side (i.e. is zero after a while) -- or rather a sequence of them - plus a linear term that picks up the right rand side. The issue is "where to stop".... which implies some form of variable selection.

The traditional 'generically smooth' function would be cubic regression splines.

Since you're dealing with variable selection and estimation simultaneously, you'd need something to deal with the 'estimation on the same data as selection' issue -- i.e. to obtain honest inference

Such as cross-validation, perhaps.

@Glen_b Do you mean to "make up" something for the left part that goes to zero, and fit all points, including on the left, instead of throwing away some points from the left?

Yes.

But the line would only be impacted by the right

@Glen_b Why is that a better approach than throwing away points from the left?

You could figure out what it was "throwing away" by looking at the influence function.

It's not 'better' than, it's a form of it.

Just in a different guise.

OK.

If the left side has no impact on the estimate of the slope, it's throwing it out in that estimate.

There are other possibilities.

Sorry for being slow, but how would I decide "where to stop", as you said? I.e. how fast the left side goes to zero?

Damn, I forgot what I was going to talk about next when I was typing about cubic splines up there.

Do you mean that in effect this method would include an extra fitting parameter which controls how fast the left side goes to zero?

You don't need to decide. You add terms until you're confident it must be linear (preferably without having to refer to the data), and let the variable selection pick it up.

Sorry, I don't know what variable selection is, and it's not immediately obvious from the wikipedia article, so I have to make note of this to read it later.

Well, in a sense. If you use cross-validation, it's using part of the sample to predict the rest, across a bunch of holdouts. This could even be done as a Bayesian problem and do model averaging, if you are happy with that.

Unfortunately I don't know a lot of statistics, so we might need to stop soon, as I'll need to read up on some of the keywords you mentioned to be able to continue the discussion.

Imagine I have a bunch of "generic nonlinear" bits over in the left half. As we approach the right, they should 'die out'. Their coefficients should eventually look like just the effect of the noise in the data.

CHoosing to stop when that happens would be selecting how many terms to include.

Each such term is a variable. Ergo, 'variable selection'

I think I got it.

Another approach - if the right half is more than half the points - is a robust linear fit, one that can throw out half the data. Indeed if you can bound it smaller than a half you can have more choice about how you do it.

That will literally throw out half the data, but you don't control which half, so if the right side has more curve than the middle it would throw out the right side.

The robust approach is very easy (and it cares less about issues like heteroskedasticity).

If we don't have good ideas about error distributions we can consider something like bootstrapping, if there's enough data.

The more notion you have about the left half, the more likely it is that some additional approaches may be possible.

There's also smoothing, of various kinds. Perhaps a variable-span kernel smoother, where the right side is forced to have a really big span, and the left has a narrower one

Some of these approaches may be adapted to nonlinear functions on the right, some not so much.

I should go, but I can probably talk some more another time. You can always ping me from here and if I am around I'll pop in.

I'm starting to think that it would really be good to be able to figure out (theoretically) how the law starts to break down on the left.

@Glen_b Thank you for the discussion, I won't hold you any longer!

The more you can say about it, generally the better options you'll have.

Okay, bye

@quit

Hmm, wrong place for that command

Back for one second: if you post a problem like that, a diagram like teh one you had here would be very useful

Bye again

bye!