I have a feeling that these "answers on the comments" might be cleaned up later since we shouldn't suppose to "answer" on the comment, but I think one of the reasons is that the direction of the eyes is usually linked to the attention (just as you said about driving car), and people have a tendency to feel that they are listened if they are both at the same eye-level compared to... looking at mouth (still possible as an aid for those with hearing difficulties), hands (I dunno, but may be possible as an aid/visual cue if the hands are active/miming the speech), body (why???). —
Andrew T. 1 min ago