It's because you are assuming that the wall is fixed and cannot move.
So you have an external force keeping the wall in place.
That external force acts where the wall is attached to the Earth i.e. at the base of the wall.
In principle when you bounce a ball off the Earth both the ball and the Earth move after the collision, and this conserves the momentum. But in practice the Earth is so much heavier than the ball that we consider it fixed.
The price we pay for this is that with this assumption momentum is no longer conserved.