Why do think it's a ground loop and not some other source of interference, like radiated (through the air) or conducted (on the power lines)?

@SteveSh Because of the high resistance between battery ground and RCA's ground + the voltage difference between them relative to the +12v on battery. But maybe it is something else, I don't know.

Related/duplicate: electronics.stackexchange.com/q/398745/2028

So the amplifier has RCA inputs. Does the head unit also have RCA outputs or something else?

are you listening to a radio that is tuned off station?

@Justme No, it has DIN-6 pin connector. Looks like this. Notice one of the pins is labeled as ground. The casing is also ground. I tried switching the ground of the wires to both the case ground and the ground on the pin but the result was the same. encrypted-tbn0.gstatic.com/…

@jsotola No, Im testing it on AUX input without anything playing or on CD without any CD in the radio.

Using shields of cables to carry power return currents is not a good idea, and, in general, is not recommended.

How is your head unit attached to the battery? The noise is most likely caused by the power supply (the alternator). There is probably some filtering built in to the head unit, which you bypass if you connect the battery ground directly to the connector, hence more noise. The noise might as well be caused by the amplifier, not the head unit nor the wires. You can try to attach some ferrite beads to the positive and negative wire of the power supply for the head unit and amplifier. Or wrap the individual wires around a ferrite toroid.

@SteveSh as far as I understand the shields are used to carry unamplified signal to the amplifier, not any serious power. MyCarGoesHisssssss can you confirm this?

Yes. Maybe I was mis-interpreting the "as long as it can handle enough current" part of your comment.

@DELTA12 Yes, the signal coming from the head unit through the shielded wires is unamplified.

So you go from a DIN connector at the head unit to 4 RCA connectors into the amp (in the trunk), right? Try unplugging the RCA connector from your amplifier. That is, leave the amp inputs open and see if that affects your symptoms. If it does, connect the RCA plugs to the amp, but leave the DIN connector unplugged and see what that does. For the later test, you may want to connect the shield to GND/Chassis at the DIN connector.

@SteveSh I tried disconnecting the RCA's from AMP. The whine was gone. Haven't tried it the other way (disconnecting it on the head unit end). Thank you I will try it later today.

I'm thinking that you don't have ground loop, but rather there may be a problem with your cables that are allowing radiated emissions (RE) from the ignition system to couple onto the cables and then into the amp. That would be consistent with your observation that when the speakers are hooked up directly to the head unit, you do NOT hear the whine.

@SteveSh But what about the high resistance between grounds ?

That may just be a red herring. This is where a detailed diagram of your setup, with all the interconnects and grounding, would help a lot. Every one of us here probably has a different mental image of what your setup is.

And I would also look closely at your DIN to 4 RCA plug cable. It's really tricky to maintain good shielding integrity with a 4-way Y connection like that. A break or opening in the shield can render the entire cable useless from a shielding standpoint.

@SteveSh So, I tried disconnecting the DIN connector from head unit as you proposed. The noise was gone. So the problem part is somewhere along the line from head unit to amplifier. Maybe it is the DIN connector itself. I made it myself and soldering it was very hard so the solder joints probably aren't perfect. I will try redoing the connector in a few days and if that doesn't fix it I will sketch some wiring diagram of the setup.