amoeba

If it's possible to construct such an example, it would clarify the issue for me, I guess.

I tried to come up with an example let's say with a single predictor x where a cutoff very different from 0.5 would maximize the accuracy on the training data, and failed.

Okay sure. Thanks gung!

Hi @gung! I came across an old answer of yours stats.stackexchange.com/a/225935/28666 that I was a bit confused about. I posted a comment there, but would also be happy to briefly discuss here in chat if you have a moment. Cheers!

@Tim haha, yes indeed. Consider upvoting Mark Amery's comment to increase visibility of this gem :-)

I thought it must be a typo in the paper so I emailed the first author, but he replied that the p-value as reported is correct. He wrote that the chi-square statistic was converted into a p-value via pchisq(..., log.p=TRUE) and then the outcome was exponentiated.

@Glen_b @Tim -- a new entry in the hall of fame of smallest reported p-values.

@Sycorax I see only one Legendary. I think @whuber has Epic but not Legendary. Only @Glen_b has both.

I agree that your Q is off-topic here. The upvotes etc. show that it is very useful. We have a number of questions like that; these are usually old threads that were tolerated when they were asked but are off-topic according to today's interpretation of the rules. They get closed but stay on our site.

@smci It is technically not possible to migrate a question that is >1 week old.

On the usefulness of "trivial" edits: this unanswered Q from three years ago stats.stackexchange.com/questions/145612 had 1 upvote and no comments until yesterday when @kjetil edited it to add [probability] tag. I noticed it on the front page, thought the question was interesting, and edited the title to be more understandable and catchy. A couple of edits later, I put a bounty on it, and now it has 10 upvotes and turns out to be an interesting open problem.

Also, @Glen_b is on the verge of 200k, just a couple of days to go probably.

Hey @gung, congratulations on 100k rep! I did not notice when it happened.

OK I have to stop here :-) Have a nice time in August!

(which maybe can be shortened a bit, I'm not sure. E.g. do you need both figures for ZY? I don't even understand the second, without looking into the paper. What's on the x axis?

And then you can talk about Kohavi and ZY

I'd maybe start the lit. review with this paper because it's so related to your simulations.

And (2) don't you want to add B&G to the literature review? I'm thinking of putting their Figure from Experiment 2. You could say that this essentially replicates your experiments presented above.

@XavierBourretSicotte I'm away from next week myself, and have some urgent stuff to take care of now, so I don't even have time to discuss this really. I thought mostly of two things: (1) I'd maybe put some main results from your outliers simulation into the main answer. Basically after you present all the results from the main model, you can say that with outliers it can looks differently, blablabla, see more details in my other answer.

@XavierBourretSicotte The first bounty awarded :) But may I suggest some modifications to that answer?

not sure reg. regression is any different in principle

regularized regression with any given reg. coefficient can also be completely changed by a sufficiently strong outlier

yes, because outliers is not a "small" change

in the sense that if data changes small, then the coefficients change small

intuitively linear regression is "stable"

i don't know

while saying that this happens with algorithm is unstable

My major quibble is that you explicitly list linear regression as a stable algorithm and then go on presenting a simulation using linear regression that has larger variance with LOOCV

i'd suggest to contrast it with "no outliers" directly: have two panels, on the left without ourliers, on the right with outliers. like in B&G

i agree

Are you planning to update your answer with something like this?

I have to say that I find EPE a more natural quantity to think about. I am never in a situation when I train a model on a given dataset and then am planning to "deploy" it, meaning run it on actual test data without modifying the model. This sounds like something that could happen "in production". There one could be interested in PE.

Their paper is about EPE. You say you are interested in PE. I am not quite sure if one can "adapt" their three-component variance decomposition to PE; intuitively it would seem so but I am not sure it works out.

Regarding variance, I find B&G paper very clear about this: there is expected error of the model trained on this dataset (expectation over all possible test cases), and expected error of this algorithm trained on a dataset of this size (expectation over all possible training datasets and test cases). B&G call it PE and EPE.

Hi @cbeleites! I do speak/read German but I'm not very accustomed to the scientific writing in German. Regarding writing a paper, I'm not very sure what exactly you have in mind, but am happy to discuss. I have to say though that I have quite a lot of stuff in the pipeline that I need to finish asap so I'm not sure if I can meaningfully contribute here.

bon jour @xavier! the last figure you posted is interesting: looks like a clear minimum at k=20 and higher variance for larger k. It looks quite noisy though, especially for 1000 reps! How did you generate it? What's the difference from the previous one?

above i meant, how different from one sample of 48 points is from another sample of 48 points

because it controls the amount of correlation between errors from different folds

my intuition is that it somehow depends on how different n=48 samples are from each other

i have no idea what would lead to a stronger effect :-)

in the second most do

in the first case most samples do not have any outliers

so you have n=48 now. so having 1% of outliers and 5% would be a big difference

i think the trick might be to get the right amount of outliers in your n (on average)

btw, this stats.stackexchange.com/questions/325123/… is a confusing conversation

maybe you can play a bit around with the % and the magnitude of outliers...

But I think Experiment 2 without outliers is pretty close to your setup

that part I did not understand

it's just that in Experiment 1 they use 10 datasets on each iteration for some reason