If you program through ICSP can you get any serial communication working through the USB?

I didn't ask about USB uploading. Install the USB->UART code on the 16U2. Install, on the Mega2560, some sketch that prints something to serial, and also maybe echoes what it receives on serial. That will confirm if serial is working or not.

Try this test sketch on the Mega2560: majenko.co.uk/blog/serial-test-sketch

So I have the Arduino-COMBINED-dfu-usbserial-atmega16u2-Mega2560-Rev3 hex file on the 16u2 and a blink sketch with printing in each loop. The serial monitor does report the printing. I also put a "Serial.println(Serial.read());" after the Serial.println(); and the window shows a -1. I will try your sketch

Ok. Try shorting the TX and RX pins together with nothing running on the Mega2560 (hold it in reset). Then try sending stuff over the serial port and see if it's being echoed back (basic loopback test).

I tried your sketch and I get "Recv: 0xFFFFFFF0 ⸮ Recv: 0xFFFFFFF0 ⸮ 14000 15000 16000 17000 18000 19000 20000 21000 22000 23000 24000" on the serial monitor

The loopback test spits out whatever I put in

Which end of the traces are you shorting for the loopback?

The two header pins Rx0 and Tx1

Ok. Check the soldering on your ATMega2560's RX pin.

Soldering was okay. I added a little more solder and I still get the same issue

Ok, write a small sketch to blink an LED on pin 0, and attach an LED and resistor from that pin to ground. Check it blinks. Move the LED to pin 1. Check it doesn't blink. Change the sketch to blink pin 1, and check the LED now blinks. Or better still use an oscilloscope if you have one :)

a basic question. did you flash the bootloader? or only a sketch?

On the pin 0 sketch, the led blinks on pin 0 and is off on pin 1. For the pin 1 sketch, the led blinks on pin 1 and stays on in pin 0

@juraj the bootloader is installed on the Mega, but after this ICSP programming, i will have to re-upload it ill try that now

but you did flash a sketch over icsp? it erases the bootloader. perhaps the issue is solved but now the bootloader is missing

Can you show an image of the whole board PCB layout?

Okay so I reuploaded the bootloader and I am able to upload a sketch via the usb, but only once. After that, I have to reupload the bootloader again

Where did you buy your ATMega2560?

Whole layout

I bought an Elegoo Mega board from Amazon and depopulated all of the components and placed them on my board

So there's a chance that the ATMega2560 may have been damaged during the depopulation...

I dont think so, because I have removed it from my board and put it back on the original board. I can do as many uploads as I want without restoring the bootloader.

Any chance of a higher resolution image? Only need the section to the left of the SPI/ICSP header

Not really, would it help if I sent the board and schematic files?

Not me, no - I don't use your software.

better picture how's this?

I need to see to the right of the chip. To follow the traces from the chip to pins 0/1. With enough detail to see track spacing etc.

maybe this is better

Hmmm... that looks OK, though the clearance with PA2 looks a little tight.

That via is pretty close, but I dont think it would have an issue. Like I mentioned earlier, when i re-upload the bootloader over icsp, i can upload a sketch via usb, but only once, then I have to re-upload the bootloader. Is there a fuse bit set somehow that prevents permanent storage of the bootloader, or perhaps the boot-into-bootloader bit is not set?

I wouldn't suspect that, since you change it to another board and it works fine. That wouldn't change the fuses ;)

It'll be something silly. I am suspecting some pin on the 2560 is affecting the RX pin after upload stopping it from working until you hard reset the whole thing.

Okay i misspoke. There are two 2560 chips in play. The first chip I had soldered on my pcb is now on the board that the second 2560 came from. The second 2560 is now on my pcb and is giving errors. here is a picture of the bytes of both chips excuse the rotated image. the NG stands for uploading the bootloader using Nick Gammon's sketch.

How about posting the schematic - that is more helpful to start with than looking at a layout. Make it decent sized too, not some low resolution thing that can't be zoomed in on.

I am not sure how to post the schematic on here, but I can on the arduino post I linked to in the original question above. (I'm new here) I also believe the problem to be solved as I can now upload multiple times without failures

My next step would be examining signals with oscope and LA, and injecting test signals.

In your Arduino Forum post you say you have solved it. Could you post the solution as an answer here please?

Absolutely. I have done so. Now that I have you here, I have been told to mention the library you made. Can you elaborate?

I didn't say that. In any case perhaps they are referring to: gammon.com.au/bootloader