Thanks. Suppose my entire dataset is called $Z$ (1.9m rows). What about the approach of splitting $Z$ into two sets $A$ (training) and $B$ (test). Here I fit a model on $A$ and perform k-fold validation on $A$ as well. The validation will split $A$ up into $k$ groups (validation and training) and give me my error estimates. After I have a completed model, I can then apply this model on $B$ to assess its accuracy. In other words, $B$ doesn't even come into play until I have a fit model on hand and I pretend the only dataset I have is $A$. Would this be a suitable (and better approach)?

It's hard to say. I've heard @FrankHarrell say that you need an astronomical number of points for a pure train/test split to work well, but 950,000 points is probably enough. That said, people don't often report totally held-out results, so you might be better off increasing $k$ instead. The cross-validated results are nearly unbiased but with high variance, so a larger $k$ than the typical $k=10$ could help you drive down the variance. That's probably what I would do.

Thanks. So in other words, if it were you, you'd just drive $k$ up and perform model fitting on the entire 1.9million row dataset? If so, how would I test its accuracy then? My goal is to create ROC curves.

My first version might have been a little unclear. Although you have to fit the entire model first, the subsequent cross-validation is actually valid; the first step is just an artifact of how the authors of 'boot' wrote the code. I suggested trying a much larger $k$ since 10-fold CV estimates often have pretty large variance (as you'd expect from averaging together only 10 numbers). Jackknife/n-fold cross-validation has its own problems, but I think you'd be on pretty safe ground with k=100 (if you wanted smaller standard errors).

Holding the data completely out is nice, but 1) you don't get any measure of uncertainty with a single test set, though. I suppose you could bootstrap from it or something and 2) It's most convincing if you can claim that the experimenters had absolutely no access to it before testing.

Thanks Matt. I am still a little bit confused. If I use my entire dataset to train the model (and perform CV which only gives me a delta value, so I don't even know HOW to properly utilize CV), at what point do I test the model? In other words, I'd like to assess the performance of this model by seeing if it can indeed predict what is of interest. So would it be sufficient to just split my entire dataset $Z = A + B$ and set $B$ as my validation set? So I'd train my model and perform CV only on $A$.

We posted almost at the same time. I guess your comment answered my question, but I don't really grasp the concept of it. If you don't mind, could you break it down for me? (maybe we can continue this in chat). I am actually more confused about what CV does.. All it returns (in R) is the delta vector which is some error estimate. I am not sure how this helps me

Ah, I sorta missed the ROC part. Logistic regression has two parts: glm spits out P(class). Next, you need some sort of decision rule to turn P(class) into an actual classification. P(class>0.5) is the obvious one.

Now, to make the ROC curve, you need to vary the false positive rate (the x-axis), which you do by adjusting the decision rule.

P(class>=0) will give you 100% false positives (and true positives), which is the top-right corner of the ROC curve

P(class>1) will give you NO false positives (and also no true positives), which is the bottom-left corner

cv.glm lets you cross-validate the first part, regressing out P(class), but it doesn't have a mechanism for you to adjust the decision rule

Actually, that's not quite true

you can change the "cost" function to do it

instead of doing the average squared error, or whatever, you'd replace it with 0/1 loss

or actually, just the true-positive rate

and then vary the false positive rate by adjusting the threshold

syntax here: stackoverflow.com/questions/16781551/…

If you can get your cost function to provide the true positive rate at a specified false-positive rate, then each point on your ROC curve will be at (chosen false positive rate, that.output$delta[1])

Alternately, I am told this is very easy with the caret package.

Thanks, so what about my datasets?

@MattKrause I mean I am still trying to understand whether or not I should split my dataset $Z = A + B$.

Once I have the glm trained on $A$, I can use the inverse function to score $B$ and manually create a ROC curve (since I know the true classes in $B$ as it was part of the original dataset)

Actually, nevermind. If I understand you correctly, I should just use the entire data set to create my glm. Then I should CV this (which by the R code, runs the glm again $k$ times on $k$ subgroups of the dataset). And to create the ROC curve, all I need to do is vary the threshold defined in the cost part of the CV function in R

Did I understand you correctly?

Yup!

The first "whole data set" model is a red herring though because you're just using it to configure the cross-validation code

@MattKrause Thanks, so the threshold needed for the ROC curve can be implemented in the cost function of the CV function?

I think so

Suppose I wanted to do this step manually. Would it make sense to do split my dataset $Z = A + B$ and then fit the model on A and assess performance on B? To me this seems like a k = 1 CV and probably a very naïve way of doing it

Yeah

that might not be an awful way to start

but the drawback is that you won't have any sort of errorbars