@sorontar Please stop commenting on my answer. There's a chat room, use it if you still think you have something to say on this topic.

Not that I think you do have anything further to contribute on the topic. I recommend that you stop trying to teach your grandmother to suck eggs.

It's uncommon to treat the words “word” and “token” as synonyms and it's obvious that I don't do so in my answer. You're trying to foster unusual terminology on others and calling everybody wrong when they don't. This is not constructive and I will not reply to you any further.

@Wildcard The relevant books would be about compilers. One or two chapters at the beginning should deal with parsing.

Formal language theory would go straight into a more mathematical analysis of programming languages, without spending much time on parsing if any.

@Gilles Okay, thank you!

@Gilles It must be clear to all that you have much experience in "How to suck". You don't have to reveal all your private "personal preferences". All I can add is that, if you ask me to do it, I have absolutely no problem in calling you "A Sucker".

@Gilles Uncommon huh? Perl "breaks those lines up into a list of words" Tcl "The result of the command is a list of words in the string." and that even has a command token_shell "The result of the command is a list of words in the string.". Yes, sure, very uncommon.

Even this highly voted answer in the subject from an expert on the matter does it: "And these "English Words" could be tokens … …".

@Gilles What is really very UN-common is to treat Chomsky level 3 problems (lexer in computer science) as a Chomsky level 0 (general "free" spoken language) problem. Most description you may come to build up for a level-0 language will be impossible to translate to a level-3 (regular language), and that is what a lexer is. Most "computer languages" are level 2 (or close to it).

Yes, my grandmother also would be very confused about the Chomsky hierarchy.