Is there a standard package to get a monotone C2 interpolant in R?

Anyone else notice Gordon Smyth has been answering questions here recently? ...

This guy...

scholar.google.com/citations?user=k1z6axEAAAAJ

@LittleAlien it's actually closely related to an often-asked question about variances of combined samples from individual sample means and variances. If you search using some of those terms you should turn one of them up. The difference is easy to explain.

@GeoMatt I imagine there are (with over 7000 packages on CRAN and many, many thousands more in other places, you'd be inclined to bet that way) but I can't say I know any off the top of my head

@Glen_b I figured as much. This relates to my answer on this question stats.stackexchange.com/q/237040/127790 (essentially "resample a histogram") But I am a Matlab person (bit of Python), so my C1 monotone R solution was not fully satisfying :)

@Glen_b (Re: new guy) Sometimes you get to that sweet spot where the Stackexchange rep is just so much easier to build than the h-index ...

@GeoMatt I doubt he'd doing it for the reputation. (If there's anyone doesn't need StackExchange reputation, it's Gordon.) I expect he's doing it out of the goodness of his heart.

(Like a lot of the well known statisticians that also answer questiions on site -- they donate some of their spare time when they can, to the benefit of the rest of us.)

@GeoMatt I did fit a monotonic spline using an R package once but I don't think it was an interpolating spline

Have also fit a unimodal spline. Might have even been using the same package, I don't recall.

StackExchange reputation on its own isn't especially meaningful (answer enough questions - as long as they're not terrible answers - and you'll accumulate reputation). Reputation per answer is some indication of how valuable the average contribution is (though there's some sources of both noise and bias there too).

For a new user the average will tend to be lower because there's less time for answers to accumulate votes over time.

A fair fraction of my daily reputation comes from old answers. New users don't get that benefit.

At least not nearly to the same extent.

@Glen_b forgot my <sarcasm> tag!

For the monotone spline, I have done similar in Matlab. I run into that math problem nearly daily at work. There it is "resample cell-centered concentration field to a different grid", and the "resample the CDF, not the PDF" trick seems to be not as well known as I would have thought.

Oh, sorry about missing that. I should have realized

(the sarcasm I mean)

Median upvotes per answer is 3 I think. Mode is 1

@GeoMatt The Carveth Read quote in your profile is very similar to a John Tukey quote in mine (and we both quote Box, though in slightly different versions)

Tukey's quote:

> Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise.

I imagine Tukey was knowingly/deliberately paraphrasing when he said it, and probably imagined people would already know the original

I usually paraphrase as "I'd rather be vaguely right than precisely wrong"

I find it interesting the frequency with which people apply transformations to their data but really have no sense of the basic properties of the transformations they apply and are shocked by the consequences (and in particular are astonished by the nonlinearity of the nonlinear transformations they choose to apply).

It's like functions are eldritch things whose effects must remain a mystery until some magic pixie tells you the secret word, rather than something for which you can take some numbers, apply the transformation in question in a variety of cases and literally see for yourself what it does in a systematic way.

It's not like you even need to be able to differentiate. You can calculate and plot. Even Excel suffices to look at what these things do.

A lot of my answers to questions along those lines seem to be slightly fancier versions of "yes that thing you can see happening is actually happening, that's how it works"

Well, I am being too cynical today. Things to do

I wonder, You can see here that the distance between two sample means 5783 and 5159 is 624. But, I have checked that the midpoint between (1341 and -93) is 717. So, why the confidence interval midpoint is different from the sample midpoint?

Shouldn't it, the confidence interval, be the (avg(X)-avg(Y) ± borders)? so that distance between the averages is in the middle?

@LittleAlien Do ask these statistical questions on the main site. That's what it is for. Chat isn't a low key (let alone lazy) way to get statistical support for little questions, if only because the chance of someone else benefiting from your question is negligible that way. (On a different note, your image is barely (in my case not) readable.)

@Glen_b I agree, but it is hard to teach transformations well at an introductory or even intermediate level. An easy example is logarithms when the response is zero or negative sometimes. It's quite a long discussion on whether any kind of transformation makes sense and on ways of doing it when it does. I've seen solemn recommendations on CV of x <= 0 ? -log(-x) : log(x) which might even "work" with luck!

Do you know any really good textbook discussions that discuss pros and cons of transformations thoroughly and carefully?

@Nick in the case of the present complaint I think the central problem is mathematical rather than statistical -- it is that people don't understand even how the "standard" functions (exp, log, square root etc) actually operate (such as what it means for a function to be linear or nonlinear).

However, you raise the very interesting notion that since people come to statistics without that understanding there's definitely room for something that discusses properties of transformations at the simplest level consistent with conveying a clear understanding of the issues.

@Nick Much of Tukey's writing on transformation is very clear (for such a high powered mathematician he was very adept at avoiding the mathematical background) but his work is somewhat idiosyncratic. There may be a good book (aside from Tukey's works) that covers it at a similar level but I don't know what it is

@Oxinabox it's for "routine bookwork" type problems. Have you read the tag wiki? You may also take a look at the discussion of homework in the [help/on-topic]. Self study is broader than just homework but the comments there apply to it.

Yeah, I'm reading the tag wiki and meta closer now.

It seems like a meta-tag, and I'm not used to those.

If someone else proposed it in an edit it should be rejected.

Ok thanks

@Ox strictly it is a meta tag

I'll reject it

... just did. I was able to explain why I rejected it.

Ok, thanks.

While it is essentially a meta tag we retain it primarily in order to warn people not to give complete answers when guidance is called for.

The only person that should add it is the OP.

Fair enough. I think my question looks like homework/something from a text book.

However, users are free to request that the OP add it.

And they're also free to close questions that look like homework but don't follow the guidelines, whether or not the tag is there.

So the actions should be the same whether or not the tag is there, it just helps find it.

It also helps people looking to avoid explicit solutions to similar problems

So if you're a student looking for a little guidance, you can use the tag in a search query to hopefully avoid seeing explicit solutions

Even if you don't post a question.

It's a total pain to maintain to be honest (especially the "don't add the tag to someone else's post), but the benefits are substantial

I would love to have been set a homework problem about two blokes with drink dispensing machines.

If you explain the source of the problem that should help to pacify the people who are worried it should be self-study

Is it worth doing? This problem is written as a extended narrarive about a more abstract problem I have in my research. I rewrote it in realworld terms to try and get some insight. (which I think I have)

Something as simple as "this is an abstract research problem recast as a ..." would do.

Ok cool.

However, I would warn you that a substantial majority of attempts to recast problems into simpler/real world terms end up leaving out some crucial aspect of the original problem that impacts the solution in ways that render the efforts to answer it useless.

So you need to be very sure that there's not something that will be important that you've abstracted out. (It's often like a disguised version of an X-Y problem -- but the choice of solution is partial and implicit)

That said, when it is done well, recasting a problem can be really helpful, so I'd rather not say not to do it.

Until I recast it like this, I didn't realise I was dealing with an estimation problem.

I thought I was dealing with a "apply Bayes Theorem" type problem.

thanks

@Glen_b There are many factors here. One might be a dissociation whereby graphing functions is regarded as the territory of a calculus or pre-calculus course that is assumed to have been taken and remembered when people study statistics. Another is a common dissociation (e.g. in social sciences) between modelling anything and keeping track of what it means graphically.

@Glen_b Tukey's wonderful books indeed don't qualify as texts strict sense. Andrew Siegel's Statistics and data analysis: An introduction (1988) is the text which I think best does transformations early and often; its 2nd edition was watered down in this and other respects and it has faded away otherwise.

Thanks @Nick That one sounds familiar but I don't recall it clearly. Our library has both 1st and 2nd edition. I'll see if I can get a look at the 1988 one.

The little snippets I can see in google books are indeed explaining things very well.

Often books don't talk about how important the questions you want to ask of the data are in figuring out both whether and how to transform.

It's similar to the issue I have with the lists of what analysis is permitted with a given level of Stevens' typology of level of measurement -- it puts everything on the data and nothing on the questions you want to ask.

which sometimes leads to forcing people into counterproductive choices.

@Glen_b I was not aware of the "don't add the tag to someone else's post" rule for self-study.

I don't understand it very well. Sometimes the "self-study" nature of the Q is obvious. E.g. the question contains a quote or a screenshot of a textbook problem. Why can't other people add self-study tag then?

I think I have done it in the past, perhaps once or twice, and don't remember being told off.

@amoeba It's right in the tag wiki

But sometimes I agree it's obvious, and I feel it ought to be okay then, but that's not what the wiki says

Interesting. It's not in the excerpt, and I never read its full wiki.

I thought it was all right when it's very clearly homework: meta.stats.stackexchange.com/questions/1172/…

@Scortchi I admit that in cases where the self-study nature of the problem is blatantly obvious (such as questions put in the imperative voice), I have unilaterally applied the self-study tag. That's mainly to give the respondents some credit towards earning the tag badge. cc@amoeba @Glen_b

"Classification" refers to supervised learning in statistics, but tends to refer to unsupervised learning (i.e. clustering) in everyday English. What a source of confusion. E.g. when Lamarck was doing his classification of animals, he was doing clustering and not classification.

@Scortchi @amoeba okay, clearly that represents a consensus.

With self-study questions I would prefer to ask the OP to include the self-study tag (and read the wiki) even if the question was asked in an otherwise acceptable format. But if the OP seems to be a "drive-by" poster, or simply doesn't respond to the request to include the self-study tag for some other reason, I'd be happy to add the tag manually.

I might also add it, in blatant cases, on a "stale" question with inactive OP that I've just happened across ... but probably only as part of a more substantive edit, if there was something else that could do with fixing.

@hxd1011 yes, I changed my name

@amoeba he was "applying class labels" ;)