I was expecting it to only work on string instructions. But I guess I was wrong :)

Well, what is the meaning of 'works' in that context? It does work with string instructions, as it makes them repeat - much like a STC (set carry) before and ADC (add with carry) increases the result by one. Doing a REP somewhere else will still work, as the Opcode gets read and executed by the CPU. Same way a NOP does.

But that would also mean that rep inc cx could create an infinite loop, right?

Why? INC isn't a string instruction. It doesn't get repeated.

This is what I wanted to know. Only string instructions get repeated. Other instructions behave normally when prefixed with rep. It's not like anything gets repeated.

@DarkAtom Yes. I'm not 100% certain on the details, but it's something like that REP sets a bit internally that's only used by a few instructions that check that flag like string instructions. It's cleared at the end of an instruction. If the instruction doesn't check the flag being set during execution (like most instructions) then REP becomes an effective NOP. (This is cheaper than treating it as an invalid instruction sequence and issuing an exception, or whatever. The 8086 still had a pretty limited transistor budget.)

@DarkAtom It's an opcode extension to string instructions. Doesn't do anything on any other instruction. There is neither a reason to flag it during assembly - as the assembler can't decideon it, nor is there a reason to trap it during instruction. It simply gets ignored.

The answer is wrong as it stands. REP is a prefix, not an opcode. It doesn't repeat the following instruction, it just sets a bit that is tested by some instructions and ignored by others.

@benrg By that definition Set Carry is as well just a prefix, right? It sets some bit that is testet later on by some instruction, doesn't it? But this question is easy solved by looking at the queue status signalled by the CPU via status lines QS0/QS1. They show a value of 01 with every first byte of an instruction. And that's exactly what they do when a REP comes along - and again when a string instruction follows (which is BTW much like the Z80 handles their Prefixes as well) so from the CPU's view it's an instruction - in addition, Intel lists it as Instruction in the original 1979 manual.

When the TF is set in FLAGS, the trap interrupt can never trigger in between a REP/LOCK prefix and an instruction. Which shows that a prefixed instruction is treated an an indivisible whole as far as the instruction decoder is concerned, not as two separate instructions. (Though on that basis alone, one may argue that mov ss, ? is a prefix as well…)

@user3840170 yeah, the prefix instructions are kind of special - inadvisable but still separate - and they get stranger with every x86 iteration ... as your writeup does quite underline. With all of that unruly behaviour the queue signals are the only definite signal, aren't they?

@benrg is correct: you can get the CPU's idea of an instruction vs prefix by single-stepping (using the TF trap flag). Single-stepping traps after executing stc because it's an instruction, but after rep before nop. On rep movs, each repeat-count traps individually, but at the rep, not at the movs (otherwise resuming would not decode the rep). Related fun fact: 8086 had a design bug where interrupting cs rep movs would only keep the last prefix, so resuming would make it run as rep movs without the CS segment override;rep cs movs could be used in a retry loop to check cx==0.

@PeterCordes True and fine whan viewed on a software level using a software trap. But isn't the decision what is an instruction or not one about the hardware? The 8086 offers a very fine mechanic telling when it perceives something read as the first byte of an instruction, aka the opcode: It's queue status signals. And here the Prefixes get signalled as separate opcodes. This is well documented at page A-16 of the October 1979 8086 Family User's Manual.

@PeterCordes Now, your example using a CMP is quite fine, but looking at it in detail, it contradicts the position you want to defend. Of course is CMP not a prefix, but an instruction - which is exactly my point - same way as REP is an instruction and both are signalled by the 8086 as such. You might want to look again into the positions taken to avoid it backfireing again :))

That's an interesting implementation detail, but not architecturally visible to software. I don't think it's helpful in this context to talk about prefixes as being x86 "instructions". Certainly not in 2020, with a modern perspective on how x86 later evolved (with later CPUs decoding prefixes in the same cycle as the instruction), and in terms of later CPUs which will #UD or #PF if you point them at a few bytes before the end of a page.

Re: cmp vs. stc: yes those are both clearly instructions. You seemed to be trying to show a grey area between prefix and instruction by saying that stc just "sets a bit", and I was pointing out that setting a bit in visible architectural state is fundamentally different from an internal decoder-state bit that's automatically cleared again at the end of the instruction. That's why stc and cmp are different from rep; that's my argument and I'm not contradicting myself.

Anyway, that's not the major deficiency of this answer. Mostly it's that you don't really say much about what REP actually does to various instructions. You do answer the question of why the assembler doesn't reject this, by arguing that it's not illegal (but you haven't said why). The "just blindly generating bytes" argument might imply that lgdt si should assemble with the opcode for LGDT and a ModRM that encodes SI as the r/m register, like add si, 1. That's not the case because that is illegal (must be a memory operand). So your asm argument hinges on calling it a separate insn. :/

@PeterCordes What is an instruction or not isn't a software definition, but hardware. The same way as a flag visible in some simple storage structure or not does not change it's operation. Take the 8008 flags (and in many ways the 8080 as well). They are not direct accessible to software. They get set by some instructions and when set or not influencing others. But otherwise not accessible. So where's now the difference between a CMP/JE and REP/MOVSB - other than the duration the fags are set that is?

@PeterCordes Arguing that 'viewed from 2020' things should be seen different than the original developers did, wen it's about exactly their product, is rather frivolous. Who if not following what the hardware shows and the developers documented in their manuals? Take a look at the mentioned 1979 manual and you'll see the prefixes listed as instructions in their own right, not as modifications as they did for example when it was about other areas. While I would say that the creators are always right, this is especially rue when it's about historical questions ... which RC.SE is about, or?

@PeterCordes For the Assembler argument you try to construct, "by arguing that it's not illegal (but you haven't said why)" if the text below the divider don't describes this, I wouldn't know what does. Your LGTD example does not fit, as it is an invalid addressing mode to an instruction, not two instruction.

@PeterCordes Finally your point of "Mostly it's that you don't really say much about what REP actually does to various instructions.": Maybe because that wasn't asked in the first place? [user3840170 does a good job in going into details here - except, all of that is off topic, as the question is about the 8086, not any later CPU.] The OP want's to know why the assembler (or other tools, he's not really clear about that) accepts the that line with generation set for 8086, not what it may mean with other instructions, or what later CPU's than the 8086 would do about. Wouldn't you agree?

I think the strongest argument is that TF exposes the hardware's idea of what software should consider an instruction in an architecturally-visible way, as part of the ISA which is targeted by assemblers. Anything else is just nitpicking over terminology, or 8086 hardware implementation details. (The fact that it aligns with modern notions of the fact that "an instruction" includes the prefixes is certainly biasing me to pick that instead of the external hardware signals you mentioned.)

The title question is "what 8086 instructions accept REP". In my own answer, I made sure to cover that with a whole section that listed every string instruction documented to work with F3 (and F2), including noting that ins / outs were new in 186. The actual question body is about a more modern assembler, otherwise the bits 16 wouldn't have been accepted, I don't think.

IDK, your answer isn't really bad, and I agree it's not necessary to cover what REP actually does to have answered the question. There were hints in comments that the OP was interested in other x86 CPUs, so answers that address that are also highly relevant.

@PeterCordes Erm, so far we are not arguing about your answer, or are we? In this case,we should move over there - but I'd have to add only minor elements, like less than great notation and a dubious interpretation of instruction. Nothing I would add a comment. Except you moved over here. For the question and RC.SE it doesn't matter how a modern x86 interprets it. This not about any later but explicit about 8086 and more over it's how the assembler sees it, so any answer should focus on that, not later developments in Hardware, Software or Philosophy unless one wants to rewrite history.

@PeterCordes As mentioned, I did enjoy user3840170 research, as well as parts of yours. Still I thing we should stick to the core first, which is, as I see it, the 'why is it not flagged' part. Everything else is bonus content.

Yeah, agreed about answering why the assembler doesn't complain. That's why my answer started with that, and why I felt the need to write a new answer (since user3840170's answer didn't cover that well).