Here's a fun proof that took me way too long to work out again:
There's a homomorphism $(\mathbb{Z}/p\mathbb{Z})^{\times}\rightarrow\mathbb{Z}^{\times}$ given by sending $x\mapsto x^{(p-1)/2}$. The domain is a cyclic group of even order, from which it follows that the morphism is non-trivial, hence has kernel of index $2$, and that the set of squares is a subgroup of index $2$. The squares, however, are obviously contained in the kernel, so we have equality. Thus, $-1$ is a square mod $p$ iff $(-1)^{(p-1)/2}=1$ iff $p\equiv1\mod 4$.