Nick Craver

@Makyen ACK - I have memory dumps but I'm not going to be able to go spelunking for a bit, so cycled the primary tier for now until I get a chance to dig likely next week at this point

@Makyen we're testing pub/sub across 3 servers and all working here...is there any description of what's not working here?

@Makyen @rene We've gotten dumps and run a full socket server deploy - when time allows, can you please let me know if you're still seeing any issues?

kk

hmm k - it'll be a bit as I dig through logs. Is there any idea when this started?

I'll be back on after their first session to dig more on it

okay I unfortunately have to run (kids school day here) but this gives me where to dig, thank you!!

well that's fubar

So you don't get a heartbeat either, from any connection?

yeah we heartbeat every 5m generally

@rene fwiw wouldn't expect a heartbeat inside 8s, or am I misreading that?

Sorry, they won't move - they'll reconnect

If we actually nuke the socket server process (e.g. a build), then yes that'll happen

There are 2 options: you can do that, but we've cooked a NIC doing that due to heat - we use DRIAN which rejects new connections to that backend node

=P

Luckily we're down at 670k this time of morning before US is really up

Running 600k-1M concurrent sockets has some interesting fun associated with it

hmmm, that may be TLS connection stalls during the rotation (we have a lot of sockets)

Alrighty 01/02 back in, 03/04 out

Thanks again for this - huge help

@rene in 3 minutes I'll rotate to 03/04

k, let's give it some time - then we can put them back in and rotate 03/04 out for new connections...that should up the chances

k, they're out

Taking out 1 and 2 now

okay cool - let's take out 2 of 9 at a time (there are issues with doing more) to see if we get a bite

I'm assume you're looking for a publish that doesn't happen, but that's just me assuming - that correct?

Can you tell me briefly about the suite - what's the runtime to detect failures there? Minutes, hours? Devising a strategy :)

Thanks!

Hey all, any chance anyone running test suites for the websocket issue is about and has some debug time today? It'd be very much appreciated :)

we can just swap it quick every decade or so

o right

@Taryn ...which table was that?

aye - I've about replaced everything in the house + Z-Wave controls here, if only they'd let me into the data center ceiling

QTS

@SeanGallardy temp light on a string and a few burns taught me to really, really love LED bulbs as soon as we found a home depot

Not the move, but same cage album here from an upgrade trip for anyone into hardware porn: imgur.com/a/X1HoY

Fun fact: the Jersey City data center we're in today doesn't have issues with cooling...it's lighting. You know how fluorescent lights are usually in a line in drop ceiling? 2 square light, some panels, 2 square light, repeating? Turns out we always toured the data center during the day, on move in day/night it started getting dark and it turns out that data center was light, light, light, light, air vent that looked like a light, light, light... - guess which one our entire cage was under

We're moving away from it...but to U.2 on the front, so not far =P

<3

@SeanGallardy I have a spec somewhere that could run a Stack Overflow read-only node out of Europe for a lot less latency all in the 2U chassis: web, SQL, redis, elastic, and shared PCIe storage on the backplane...there are uses :)

We have some FX2S chassis for VMs that take half or quarter but you're quite limited on housepower options and only 12 DIMMs instead of 18 or 24

the hell would you cool that with

width, yeah, even quarter, but...1/2U high??

wait, they have half height blades?

that's an expensive pizza

Well hell, we should just spend $392k per 1U box and see how it goes :)

*per socket

I'm really curious to see what happens with licensing as the 56-cores per socket rolls out this year, that'll affect some things around threading and what's worth it where for sure

k

want to?