@ACuriousMind After asking, I stumbled upon
this answer by you. This seems to agree with my thought process but I don't understand the part in brackets "Note that whether you say the vacuum state or the representation of the field as an operator is different doesn't matter, you get "different" Hilbert spaces (perturbatively) built on the vacuum state in both cases."