what is "score", which metric do you measure ? It is a binary classification problem ? I should have asked that earlier ;). What is the minimum number of samples tested ? The exact overlap at the far left looks fishy. Maybe you also want to add the code (yeah, we do not have access to the data (do we ?), but let's check anyways)

Thanks for your reply. I have added the code and uploaded the train and target vector as pickle files in the question. I knew the graphs looked fishy, but I am really unable to figure out why :(

I had to restart the download of the train vector, now the link does not work anymore ({"success":false,"error":404,"message":"Not Found"}). Is this a one-time link ?

Hi, I have changed the link. It is not a one-time link anymore. You will now be able to download it successfully

@steffen, just making sure - was the download was successful?

yes, thank you. I had some trouble reading main_train, but this solved it. I am using python 3.6.3.

I have opened this chat room because something still seems to be missing.

I was not able to reproduce the graph. I could not run it for training size proportion 0 to 1, just because the time complexity of SVC is between O(n^2*m) to O(n^3*m), where n is the number of training instances. So either you have a very very powerful machine, or something is off here

Hence I restricted the learning curve calculation from roughly 1000 to 10000 training instances. Here is the code

print(__doc__)

import numpy as np
import matplotlib.pyplot as plt
from sklearn.svm import SVC
from sklearn.model_selection import learning_curve
from sklearn.model_selection import ShuffleSplit
import pickle

def plot_learning_curve(estimator, title, X, y, ylim=None, cv=None,
                        n_jobs=1, train_sizes=np.linspace(.01, 1.0, 5)):
    # calc curve first to avoid premature opening of figure
    train_sizes, train_scores, test_scores = learning_curve(
        estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)

What I have changed: Open figure after learning curve calculation (should not matter for the result); set train sizes explicitly in the call to plot_learning_curve. The resulting graph looks like this

which is not near the result you have posted here: i.stack.imgur.com/FxgGQ.png

Maybe I have made a mistake during the frequent code edits, otherwise something is off

You could rerun my code

Maybe the workaround to load the pickle export has changed the data ?

or shudder the issue is differing python or scikit learn versions

I am currently running your code to check the resulting graph

ok

Ok, it finished running. The graph now is exactly the one you got as your result.

So, what could have gone wrong with my initial implementation?

Also, what can I infer from this graph? Because the CV score does not seem to be that great

Regarding the first question: What is your initial implementation ? An you provide the full executable snippet you have used ?

Yes I have provided the full snippet. The code in the question was everything on the learning curve

the loading and (preprocessing) of X and y is missing, so I guess the difference is there ?

It was just a pickle load - but maybe.

So what can I infer from this graph?

Btw, thank you so much for the help. I really appreciate it

It is a little unsatisfying to settle the overall question of overlapping learning curve on non-reproducible code issues ;) ... but I will add the code used here and so be it

regarding your second question

yes

the learning curve difference between train and test is ok , train is optimistic at the start (overfitting), test is lower, but the gap is closing the more samples are added

but of course the accuracy score is low

the issues with SVMs is, that you have to find the best parameter C and gamma. See e.g. stats.stackexchange.com/questions/43943/…

and for understanding stats.stackexchange.com/questions/31066/…

but as said, the SVC is maybe not the best algorithm here, due to its time complexity

as the "no free lunch theorem" states, there is no best algorithm for every classification problem, one has to find the best one by experimentation

I do not know your state of knowledge in this area, but I can recommend "Hands-On Machine Learning with Scikit-Learn and Tensorflow" from O'Reilly

But the problem is, even if I do a grid search to get the optimal parameters, I believe I should pass in all the data in to the library. Am I right? Then real problem happens after - I would just like to get your opinion on it - because the grid search returns me the best parameter it found with an accuracy of 75%. But then when I run it on my test data, it classifies most of the observations as the majority class. Is there anything I could do to get a better model?

there are options to deal with class imbalance ... SVM has one builtin (by the class_weight) parameter, other learners have different options, general solution is resampling (i.e. just changing the distribution of the classes by removing certain rows). There is a tag here on cv: stats.stackexchange.com/questions/tagged/unbalanced-classes

if you are not satisfied, maybe use a different metric like precision, instead of accuracy

OK, I will take a look. Again, thank you so much!

you are welcome

good bye