You can remove a question from the bumpable set by...
- Upvoting a zero-score answer, if it's decent (or editing it until it's upvote-worthy)
- Downvoting all zero-score answers, if they're all bad
- Adding a new answer of higher quality to attract new upvotes
- Voting to close the question if it's off-topic or can't be answered well in its current form
- Down-voting the question to a negative score
- Flagging zero-score "answers" that are not answers, to be deleted or converted to comments by the mods