@Rainbolt Let me put it this way: I view chess as a game of strategy, and playing by memory hardly counts as strategy part, so it just takes away from "the main meal". YMMV.
@JoeW as I said, I don't have examples from personal experience, but it's not very hard to find some. I have experience with Eldritch Horror, another story-driven game with a lot of random elements, and all those unpredictable moving parts produce some bizzare, deadly or too easy situations. From the looks of it, it's even moreso in Betrayal.
Also, EH is a coop, occasional difficulty quirks doesn't bother me that much. But when playing against other person gaining random game-deciding advantage or disadvantage feels much more unfair.