I believe that almost every problem for non-native speakers can be traced back to only a few things.
And speaking-listening is one of them.
(I really like to emphasize more on listening; and I think you already know that.)
Speaking is really secondary, imo.