@Manuel Essentially "the knowledge about the problems of \scantokens comes from experiences."
I don't know about such "experiences", IMHO it is clear: suppose the file aa.tex with one line:
`foo\noexpand`. Then `\edef\test{\input aa \empty xx}` does exactly the same work like
`\edef\test{\scantokens{foo\noexpand}\empty xx}`. The only difference is that the first example is possible in classical TeX too. And the result `\test` is a macro with body `foo\empty xx`. Note, that
`\empty` is not expanded despite inside `\edef`.