Actually, the Five elders are the ones that control the World government, The Gorosei(5 elders) established the warlord system. can't you remember in one of those episodes where they even gave commands to the Sengoku (Who was admiral at the time) to try and look for a warlord to replace Gecko Moriah. Also, we know that the Gorosei hate pirates because they use the Marines to lock pirates who are captured in Impel down, which is owned by the world government, (and the elder stars are the heads and highest ranking nobles of the world government) —
The Anime Scientist 2 days ago