Thanks for all the stats dumps here - our web sockets are spread across 9 servers, so with one being wonky we'd expect to see a ~11% bad behavior rate. Given the above dumps, that looks to be exactly the case. But, this is something
very hard to repro, they've been running for weeks. So instead of a quick fix, we want to narrow it down and figure out what's wrong where so we can fix the root. Is anyone in chat we could run tests with today? We can try rotating out each node to find the culprit faster. We can do this without, but your test suite already going would be very useful and welcome. —
Nick Craver ♦ 2 mins ago