The problem is that in a non-conservative field you can move a charge round a circle and back to its starting point and have to do work on it the whole way.
That means using our definition of the voltage change there must be a potential difference between the strt and end points, but the start and end points are the same point.
So you end up concluding that the same point must have two different voltages.
But then you can go round the loop again, and now you find the same point must have three different voltages. And so on round the loop until you get bored.
The conclusion is that you cannot assign a potential to any point in space because the potential can have an infinite number of different values there.