People don't always number the clues in order, to attempt to avoid duplicating someone else's number. (If we all started with 1, we'd end up with several 1s at the same time.)
(a lot of this game runs on the "honor system", in a sense - you could keep clues around forever, but it'd make a boring game, and you'd be kind of a jerk)