I have so far been unable to get ChatGPT to correctly prove $\operatorname{Tr}(AB) = \operatorname{Tr}(BA)$, whether or not I use LaTeX or add suggestive extra questions. It either forgets about non-diagonal entries, or turns an $a_{i,j}$ into $a_{i,i}$ somewhere. When I tell it about its errors, it "recognizes" them, sometimes elaborating on a counterexample I provide, but then it repeats the same errors again. I have a hunch that it treats lexically similar "words" as semantically similar, which dooms it to uselessness in mathematics. — darij grinberg 5 hours ago

An alternative conclusion is that rhetoric and poetry are significantly easier than (even quite straightforward) mathematics. Flattering, but not very convincing. I'm skeptical about general AI, but the prospect of having an AI help out with mathematical proofs actually sounds quite plausible to me! — darij grinberg 5 hours ago