Usually you have a list of states and each states has rules. A rule is a tuple (regex, tokens, nextState), where tokens are the highlighting tokens issues for the matching groups in regex and nextState the next state. However, nextState is also allowed to push and pop multiple states.