okay, I tried making SSH very verbose. I can see where the problem is, but not what it is.
I wanna ask a pro unix dude like @tchrist
before going to an SE site
huh. I think I fixed it.
and found the problem.
if you're curious @Cerberus, the ssh server was trying to look up my DNS when I connected to it. every place where it worked fine had DNS resolve fine, my home internet looks to not have DNS.
I disabled the DNS looking up option on sshd and it's fine now