There are definitely processors that can hit around 5 GHz and have more than 16 PCI-e lanes. I'm curious why you're targeting 5 GHz specifically though. In theory, the bandwidth for the GPU shouldn't be super duper important, since you really want to be minimizing the amount of swapping between system RAM and VRAM while training.

It's not 5GHz specifically, but it looks to me as though anything from Intel with >= 16 PCIe lanes drops down to 4.2GHz boost clock, and even then it costs $2000. The point is I want a PC which performs as a very responsive development desktop, while also being able to prototype deep learning models. I'd be open to AMD, but I'm not sure CPUs from AMD are advisable for scientific computation, their Threadripper options seem a bit quirky.

4.2 isn't exactly going to be an issue for almost any development work. Not really sure what you mean by "quirky."

It's hard to know without having the processor, all I know is that my current Xeon at 2.8GHz is showing its age, and I want something which is incredibly snappy. I can't foresee all the workloads I may do in the future, but there have been plenty of operations I have run currently where the CPU is pegged at 100% across all cores. As for what I mean by AMD being quirky, I mean (for example) how the 2990wx is actually worse than the 2950x for memory bandwidth sensitive workloads. I'm also wondering if getting an AMD means that I have to often compile libraries instead of using binaries, either..

... out of necessity or in order to get comparable performance. I know AMD is worse at AVX2, for example.

Binaries are going to be compatible in almost every case; the only thing I can think of is the rare program that uses AVX-512. Ryzen 7 just seems to make so much more sense than anything else in your case. For reference, what Xeon do you have right now?

I can't recall the model, but it is probably ~ 6 years old and is quad core with HT.

Given that AMD is coming out with Zen 2 shortly, and it looks like this is going to be a big boost especially for scientific computing, maybe it makes sense to wait then?

OK, I switched to my desktop

Intel Xeon X3450

I use it as a server and a workstation

I have a 1060 GPU, I'm considering getting a 2080ti once prices come down

I want to have a desktop system where I can rapidly prototype models and try things out

I'll actually train elsewhere

I'm not sure upgrading just the CPU makes sense, the board still uses DDR3 RAM

Obviously AMD is the more economical option

but, taking into account that the 9900k can easily be overclocked to 5GHz on all cores, and the IPC of the 9900k is already quite a bit higher than the AMD (plus it has AVX512) I'm wondering if I'd be better off spending the extra money after all