@hichris123 Just add all the comment IDs you fed into the API to an array, loop through the returned comments and remove their ID's from the array. Bam, you have an array of ID's that should be marked valid.
@Undo I usually keep them if they are very recent. Because it is helpful to know why someone downvoted a perfectly fine answer. Although those comments are rarely replied to..