« first day    last day (18 days later) » 

Matt Krause
Matt Krause
00:19
Agh, that's a good point
You can get them from cross-validation but not cv.glm
masfenix
masfenix
00:42
After thinking about it, I am not sure if the cost function can predict ROC curves. Because everytime I change the threshold, it will run my glm again K times
@MattKrause Unless that's a better thing. I was under the impression that to get accuracy of a model, one should fix that model
Also, the delta (which is the MSE) from the CV function represents the misclassifications. Is this the false positive rate (ie, the x axis on the ROC curve)? It really dosnt give me the TPR so it would be hard to draw the curve
Matt Krause
Matt Krause
I think it probably could be done, but it might be annoying.
That first part (running k times) is a feature though, not a bug
It ensures that each point shows up in the test set once
I took a quick look at caret and that might be more in line with waht oyu want
well, in that it looks like you can get an ROC curve with only a few lines of code
masfenix
masfenix
01:06
Thanks. I also found the package ROCR which can give me ROC curves
but I cant find any information combining CV and ROC curves anywhere, nor even a succinct description of what CV does
masfenix
masfenix
01:27
@MattKrause So based on what I've research so far, this is how I am going go to about it:

1) I will first use the entire dataset to fit a model. The package ROCR allows me to assess the performance of this model, and plot the goodness-of-fit by a ROC graph. Here, I was hoping to split my data to test and training, but all examples I look at consider the entire dataset. (I will be posting another question on this on the main SE).
2) I will also use cv.glm() to do cross validation. This gives me a delta value which is the percentage of misclassified. I will replicate this 10 times (since each time cv.glm samples different groups of data, giving me a different delta value).
I will modify the cost function of cv.glm from 0 to 1 (my threshold) so that the delta values I get for each threshold, should match my FPR.

« first day    last day (18 days later) »