Actually, for the 4-only case, the system actually has a gauge degree of freedom, or redundancy, a bit similar to the Z2 gauge theory. So I think the 4-only case should definitely be slower than the 2-only case.

@Jun_Gitef17 Does that mean you've figured out the answer? Maybe you can write a self-answer then?

It would be nice to have this question cleared out of the unanswered queue! It would also be helpful for future people who might have the same question.