In order to make sure people don't think I'm an MD I prefer to say "I have a PhD in X"

It depends on the context. It is unethical if people are calling a doctor for help, but not on other situations where you have time to clarify. As mentioned by @nengel in their answer, doctor usually refers to a degree and not an occupation.

There are large differences between countries. I am Dutch, and in the Netherlands I would very rarely use my title in academia and not at all (or maybe as a joke) outside academia. I currently live in Germany and the use of the doctor title, also outside academia, is much much more common. As a consequence, in Germany calling myself "Herr Doktor Buis" would be understood as referring to my educational degree, but calling myself "Doctor Buis" in the Netherlands would probably lead to confusion. (Strictly speaking the occupation is called "dokter", but the words sound very similar)

Why not a "Math Doctor", that would confuse people so much :D

People think incorrectly from stuff other people say all the time, you can't make it your responsibility to make them draw the correct conclusions of what you say. That would easily be a full time job in itself!

Most people where I am from would think you are a medical professional if you even use the word "doctor", because they have no idea what a PhD is, but they know who to call when they are sick.

I used to go by "The Philosopher."

Over a very long life I don't recall anyone holding a Ph.D. referring to themselves as "doctor". (I'm in Canada, should it matter.) Perhaps diploma-mill Ph.D.'s are more inclined to do that.

In reference to your specific example, Sam Harris has done research work in neuroscience. He might not have been particularly prolific (still, three publications for a PhD isn’t that bad either), nor does he still seem to work as a research scientist. That doesn’t detract from the fact that he’s worked as a neuroscientist. There’s no deceit. Neither his website nor his Twitter profile claim that he’s currently working as a neuroscientist.

I am surprised at so many of the comments/answers here. I am in northeast U.S., and here the title "doctor" is thrown around like candy. When I was a university student, most of the professors preferred (or even insisted) on being called "Doctor" no matter their field. I made the mistake of referring to a physicist as "Mr. X" once and received a stern warning about title, that I was always to refer to him as "Dr. X." Even outside academia, many people around here insist on being referred to with their "Doctor" title. This doesn't really answer, hence comment, but should be enlightening to you.

@Aaron Calling someone Dr. [Name] is quite a bit different than saying "They are a doctor." in that context. In my school we generally called the professors "Dr. [Last name]", but would still refer to them as professors, not doctors. The origin of the word is actually from "to teach" so Philosophical Doctorates are actually very relevant to calling one "Doctor"; moreso than medical doctors I would argue.

Harris has written a number of books in his field. Even if they were publisher-reviewed and not peer-reviewed, writing is a scientific practice.

@JMac I understand how some people feel that way, but I had that feeling knocked out of me when I spent my time in academia. Especially once you realize that a math doctor or a computer doctor are doctors, just as much as medical doctors are, then you realize that the feeling stems from the fact that children and lay people interact mostly with medical doctors, so many of them get the feeling that a medical person (irrelevant of actual title) is a doctor. However, a doctor of mathematics is a doctor. I agree with you that it "is quite a bit different," but only by perception.

@Aaron I think the meaning of the word "doctor" has shifted almost to the point where it only means medical doctor. The title (Dr.) I still see used for phds, but I've never heard doctor as an adjective refer to anything but an MD. As an aside, I had a few professors who insisted their correct title is "Prof." and that "Dr." would be disrespectful title for them.

Einstein was fond of telling a story about a persistent caller who was put off by his housekeeper. Apparently (and possibly apocryphally) she told the respondent that he should go away because her employer "wasn't the kind of doctor who helps people"

My father has a PhD and taught at the university level (now retired). The other professors and students always called him Dr. X. In the current generation, professor seems like the more comfortable title, but Dr is still used. Not as common outside of academia.

A doctor is someone adept at producing hip hop albums. Physicians and academics have no business impersonating Dr. Dre. It is misleading to the public.

I don’t know about someone with a degree of any level who’s no longer in school, but it seems common (and reasonable) for students actively engaged in research, particularly graduate students, to say “I’m a physicist/psychologist/mathematician.” They are, after all, doing research in a field, and usually getting paid for doing so.

@Aaron : Actually most licenced medical doctors aren't "doctors". They have to do research for some more years while practicing medicine to get a PhD degree. But since their ordinary job being a doctor is so demanding, they usually do the PhD required work on not much more than 20% of their work time which makes the PhD take longer to finish than say, for example a technical PhD.

@astronat Work with some of your fellow PhD students for a few years and I'm sure you'll have a very different opinion.