Added a less extreme example that could plausibly be generated by a non-native speaker (I've seen hundreds of texts like this during my undergrad in Europe) and a GPT's correction that didn't add or remove any content.

@GregBurghardt: 100% this. ChatGPT invents false statements presented in well-written ways that make them look plausible if you aren't an expert reading carefully. In Stack Overflow's [cpu-architecture] / [simd] tags, every ChatGPT answer I've seen has been useless, or worse: misleading and wrong, asserting false statements as the premise for its argument (detailed example). (Of course those are low-effort trolls that asked it to answer the question itself, not improve existing "expert" text, but those errors could creep in anywhere).

@JonathanReez: The volume of bad answers has dropped a lot in the past half year; I don't remember the last such answer I've seen about explaining a performance effect or about how to manually vectorize something. But I know I haven't seen a useful one from the naive low-effort low-understanding way people use it to churn out obviously-AI SO answers. Always just generalities and/or misunderstanding the question, or making stuff up. Especially on "tricky" questions that remained unanswered because the question and/or answer is complicated.

@PeterCordes care to share an example of such an unanswered question which you think cannot be answered by AI? Happy to take up the challenge.

@JonathanReez: I checked my flags raised to find some AI answered badly; C# and SIMD: High and low speedups. What is happening? attracted a dozen bad AI answers until a semi-related answer to one aspect took it off the list of high-score-unanswered questions that AI exploiters target. I posted some ideas in comments, but never wrote up a full answer. Also Why does a NOP (as a 5th uop) speed up a 4 uop loop on Ice Lake?. I don't know the answer, and wasn't able to repro on a different microarchitecture.

@JonathanReez: Stringify template type for inline assembler is a syntax / toolchain problem which AFAIK is impossible to solve the way the OP was hoping. The deleted AI-generated answer has showstopper problems. Just for the record, I'm not claiming the performance questions can't be answered by AI; probably someone who understood the question and the general shape the answer should have could give the right prompts to get an answer. But whatever low-effort users do produces horrible results, probably just pasting the whole SO question as prompt?

@PeterCordes here’s the NOP Speed-up answer. Grade it on a 10 scale?

@JonathanReez: 0/10. Maybe 1/10 at best. Full of tangential facts, and some non-facts misinformation (like that aligning the top of a loop by 16 is relevant for macro-fusion of dec/jnz at the bottom - actually what matters is that the dec/jnz itself isn't split across a cache-line boundary, and loop size is arbitrary.) Also, it claims that Ice Lake does mov-elimination. For GPRs, this was disabled by a microcode update in 2020-nov (realworldtech.com/forum/?threadid=200635), and might be the relevant difference between a 3 uop loop on Skylake vs. a 4 uop loop on Ice Lake.

... Or maybe mov-elim isn't relevant; later tests reproduced an effect with just inc and nop uops. Anyway, ChatGPT wanted to optimize for the wrong things, like having the ROB not get full or not running out of physical registers. We'd normally want the front-end to run ahead of the back end; there's no reason to expect the back-end to run slower just because its buffers are full of work. ChatGPT seems to make this implicit assumption and never explains itself. It keeps talking about utilization of the front-end (5 slots), but that doesn't matter if the front-end isn't the bottleneck

@JonathanReez: In summary, an incoherent repetitive jumble of facts and mis-stated non-facts. And it never says anything that would explain the performance effect even if it were true! It keeps wanting to talk about how to get more useful work done (by loop unrolling), rather than come closer to 1 iteration per core clock cycle for the actual dec/jnz loop branch plus whatever loop body. (update: It does at eventually mention the limitation of 1 loop branch executed per cycle.)

Also totally misses the point for Skylake; there, the relevant test is a 3-uop loop padded up to 4 or not. But it's still talking about a 4 uop loop padded up to 5, running at 1.25 cycles per iteration (with optimal uop-cache throughput, or I think 1.33 c/iter on Broadwell or Haswell due to an LSD bottleneck with loops that aren't multiples of the pipeline width. ChatGPT might be right about that, but that's the wrong question.) Again still trying to give general optimization advice like unrolling loops, not explain this microbenchmark effect.

Anyway, 1 or 2/10 for some semi-related correct facts. 0/10 for anything resembling a plausible explanation of the performance effect. For someone who didn't know better, they could be pointed in totally the wrong direction if they didn't realize ChatGPT was spewing nonsense. So still a perfect example of something ChatGPT totally faceplants on.

@PeterCordes thank you, set a reminder, will come back to this in 1 year

@PeterCordes I know this is cheating and doesn’t count but here’s the updated answer taking your feedback into account: chatgpt.com/share/67b815e6-e4ac-8013-b849-f3696429b419

curious if at least this version makes sense?

e.g. it's proud of itself for "focuses purely on front-end delivery stalls in the LSD when the loop µop count isn’t a multiple of 5.". I suspect the difference is in uop allocation to execution ports when the loop size matches the pipeline width vs. when each issue/rename/alloc group contains about 1.25 iterations, so we get different allocations for different uops, and sometimes miss a cycle of executing one of the loop-carried dep chains, from which we can't catch up.

@PeterCordes thanks. If ChatGPT o5 is able to one-shot this question by producing a 10/10 answer, what do you think would be the impacts on the job of performance engineers?

@JonathanReez It would save time in understanding why something is running slower than you expect based on the normal simple parameters like uop throughput bottleencks and best-case latency bottlenecks. It doesn't shed any light on whether an AI would be helpful at knowing how to change a function (or whole program) to make it more efficient; it could perhaps do that without formulating an English answer to questions like this.

(Since this is a microbenchmark for understanding CPUs, not a program we want to tune. That's something that kept tripping up ChatGPT I think.)

@PeterCordes yes, I suspect it might already be able to handle a lot of trivial code changes well