what do you mean by accuracy? How do you compute it?

@EngrStudent I mean overall accuracy, Kappa, or F_Score estimated by caret in R in bootstrapping. So, one of the out-of-bag performance metrics mentioned.

I like the kappa because it is about agreement between raters. Personally I like to use the Boruta package because I can say "permutation importance says these are important". You need to make sure to have a train-test split where the model never sees "test" at all during training, and it is used to estimate performance in the presence of new data. You can repeat 100 times for random splits in the training then first look at variation in non-trained fold, then second test against the, pre-separated and never-touched, test to see how it fits in the distribution.

@EngrStudent Thank you very much for your answer. I have a question. Is the pre-separated and never-touched test set really helping in this case? I am considering two things: 1-my sample size is not very big, 2- in every 100 iterations, the dataset is split. As a result, for a given iteration, the model never sees the test-set in that iteration. Therefore, does it make sense to not make the "second" test you mention and benefit from using the full-sample size in each iteration? Thank you.

In the theory there is "in the limit of infinite samples" but the real world imposes constraints that are more aggressive. I would look at leave-k-out cross-validation, but I would also be very very cautious of overfitting. I like somewhere between 5 and 35 times as many samples as I do parameters in the fit as a first rule of thumb. I would randomly split into train/test at 90/10, fit the model using train, test using test, store test value, and then repeat something like 100 times. I would look at the variation in the score across replicates as well as its mean.

@EngrStudent Thank you. I have two questions: 1- Why do you prefer repeated train/test, over bootstrap in this case? Is there a specific reason? 2- Do you mean 5 and 35 times as many samples as I do "features" in the fit, instead of parameters?

I don't know your model. Some folks like OOB for RF but there are nuances (aka error to be had) there. In my personal opinion the split-then-fit makes it clearer that there is no bleed of test into training to corrupt your fit summary statistics.

If I have a 2-term linear model then I want to have something between 10 and 70 samples to fit. Personally I like to start around 300 but there are thresholds of uncertainty when you go from above 35 to below 35 and then when you go from above 5 to below 5.

The bootstrap sample is the same size as the original data set. As a result, some samples will be represented multiple times in the bootstrap sample while others will not be selected at all. The samples not selected are usually referred to as the “out-of-bag” samples. For a given iteration of bootstrap resampling, a model is built on the selected samples and is used to predict the out-of-bag samples

I copied the quote from the Applied Predictive Modelling. The author is the contributer of the package I use. In this process, I think there should not be bleeding of test into the training set.

I repeat the process 2000 times, and get distributions of each metrics for two different approaches.

Here is a more recent reference: link.springer.com/chapter/10.1007/978-3-030-56485-8_3

Interesting, thanks :)
But, if I am not mistaken, the Random Forest has bootstrapping process inside the algorithm. However, this is an additional bootstrap process outside of it.

Not the same. There are models in the RF ensemble that have seen the training data.

So, I provide a bootstrap sample to Random Forest, and keep out-of-bag sample. Random Forest takes the given sample, and apply additional bootstrap in it.

yes.

However, I do not know how to test if the first step does not have a bleed test into training effect.

I want things to be perfectly perfectly accurate, and I think you might too. The more samples you put into train means you get that. But I do not want to fool myself into thinking I have a decent or great model when it is junk on new data.

I see

Actually, to use GPU, I will code the bootstrapping process from scratch

why do you need gpu with that small of data??

I certainly consider your advices when doing that.

which RF package are you using?

Actually, this dataset is the average of pixel values in agriculture fields.

However, when I have observations at a pixel level, it can easily go up to 5 million rows.

So, I switch to GPU via h2o package.

Currently, I am using caret and randomForest.

h2o has randomforest but I don't know if it is on gpu

convolution can allow subpixel accuracy

all pixels are average counts of viable photons incident on the detector in the camera. Often the hardware uses weighted values in a neighborhood of a few pixels (~5) to smooth the pictures and make it less noisy.

Sounds like you are having my kind of fun! I hope it is a blast.

:)

I think it is on GPU, it works pretty fast.

Exactly, deep learning is another option I have. Actually, we are using it for different models.

However, the accuracy difference between classic ML tools (RF, xgBoost) and deep learning is not that much different in this problem. At least according to the literature :)

I guess it is the same in the satellite images. I am coming from an entirely different background, economics. However, my intuition tells me that we can continue with the field-level values. We will see. :)

By the way, do you know is there any way to tell the RF (or xgBoost, SVM, etc.) that the pixel values in a field are inter-related or there is a kind of neighborhood relationship? I think it assumes every row independently. But, in reality, it is not. Do I have to use deep learning to convey that information into the model?

You can nearly always win by starting simple and working toward greater complexity. You can understand easier. They run faster. They provide decent checks for higher-complexity systems. There is broader literature. The software is more mature, more global, and more likely able to be understood by someone you need to communicate to.

Sure, put the xy coords of the pixels into the inputs.

You could also put an independent index (factor, not numeric) showing they have the same field

best of luck.

Thanks, but I don't get how it helps. Suppose, I put xy coords and independent field index.

For instance, on the training set I have field id from 1 to 675. On the test set, I have field id from 676 to 750.

What model learns from the field id feature that helps to better predict on the test set?

So, does it really helps when the factor levels are entirely different?

I guess the same thing holds for xy coords as well. Do I missed anything?

So you can augment the input so that it is [numsamples, numfeatures+2] in size, and the 2 augment columns are GPS coordinates.

If the "field index" is too unique then it becomes a falsely informative variable. If you put row index in, and there was a clean hash to convert rownumber to label then a RF could memorize it. This is why I like Boruta, because it can tell you importances, and then the person with domain expertise can determine whether the values are reasonable and can detect falsely informative variables.

Boruta uses ranger, so its pretty fast. It can also handle very very big arrays using the extra-trees, but it requires extratrees package and abotu 5x multiplier on number of trees.