I downvoted maybe several hundreds low quality answers and pain and effort involved in making a difference goes far beyond tiny puny -1 rep penalty for the answer downvote.
Don't expect to make a difference if you just drop downvote and run away. Chances for it to work are less than 50/50, probably something like 30 against 70 or even 20:80. Yes, and that's fair, low effort makes low impact.
When other readers look at the answer having negative score without an explanation, they tend to think (unless it's obviously horrible) WTH guy invested an effort to write something and got downvoted …