The DMZ

MechMK1

11:04

Me: [upvotes a post]
Nobody: "NOOOOO! The number displayed next to the post is now not the result of an arbitrarily chosen pattern anymore!!!!"

@FireQuacker You're not a gamer, are you? :D

djsmiley2kStaysInside

12:25

I for one have definately never upvoted a terrible question/answer that's on 68 votes.

reed

============= SEPARATOR ==============

first there are 13 equal signs, then 14. I've found myself checking that, and adding an equal sign to make it symmetric and to avoid unlucky numbers (13 or 17, depends on your culture and country)

that's pretty pathetic because I usually try to avoid stupid OCD behavior. I think I'd have no problem upvoting a question to 13 or 17. Maybe because I think that is going to change at the next upvote

nobody

@MechMK1 Arbitrarily chosen, you say?! So powers of two are now arbitrarily chosen numbers?!! What blasphemy!

If I knew enough people would be annoyed by scores of 13 or 17, I'd deliberately upvote (even bad) posts to 13 or 17 :)

2

MechMK1

12:55

A. Hersean

13:12

You can make an userscript that rounds up or down the scores to "nice" numbers, just for you, without upvoting or downvoting. This way your OCD won't be triggered.

nobody

That's... cheating. Besides I'd rather just vote anyways than write a whole userscript.

A. Hersean

No, that's script-assisted mind hacking.

nobody

Same thing

A. Hersean

Hacking is not cheating

There's 1 letter less in "hacking"

nobody

@A.Hersean Quoting MechMK1: "You're not a gamer, are you?"

A. Hersean

13:21

A gamer would say that "cheating" is when you get caught.

MechMK1

13:35

Oh god, people who call their aimbot "legit" because it's not extreme

"Look, I'm not cheating, I'm just giving myself a competitive edge"

ThoriumBR

13:50

I once was challenged by a friend to beat his record on an old iphone game called geodefense, so I decompiled the game and changed my towers parameters to be stronger and the creeps health to not increase on each level...

he told me I cheated, and I said I used my best ability: programming...

nobody

14:32

I'm lazy so I would just have changed my high score instead of modifying the game.

MechMK1

15:10

An old iPhone game a lot of people in my class used to play just stored scores in an xml file

So I edited it and set my score as "More than you"

Nobody could beat that

God damnit :D

nobody

I guess that joke is getting a bit boring now. I'll try not to use it anymore :)

reed

I didn't see what you wrote, but I guess you wrote that nobody was better than him

but now that it's been removed, nobody knows what was written

nobody

Well thoriumbr probably knows too

ThoriumBR

15:32

not anymore... I am not running my modified chat client right now... I was even thinking about it this weekend

I could just change the high score, but it was more impressive to show him how I could defeat a stream of creeps easily

reed

@ThoriumBR, what was you modified client? Script written in what, running where?

ThoriumBR

it was running on my browser, I just patched the javascript variables to ignore the "delete" event

reed

@ThoriumBR cool. But how did you do that? Plugin like GreaseMonkey or something like that (never used it), or did you write your own plugin (doesn't seem difficult to write a Firefox plugin, but only just experimented with it, never wrote a complete one)

ThoriumBR

there's an enum that creates a variable named deleteMessage (or something) with the value 10. When a new event arrives via websocket, a function parses it and identifies the event type, and if it's a 10, it sends it to the deleteMessage() function

I put a breakpoint on the line after all event types got defined, and ran "deleteMessage = 999" on the dev console...

reed

I've always wanted to write a plugin for Firefox to have more "control" over web pages, what they do and what they load, but then it'd look too much like NoScript I guess, and it would be hard to write an efficient one, so I haven't written anything yet

ThoriumBR

15:39

so this code does not get executed anymore:
if (eventType == msgDelete) deleteMessage(messageID)

because eventType will be 10 (from the websocket) but there's nothing with a 10 on the event type array...

reed

but you had to paste that code every time, didn't you? Or would it be saved by the debugger between sessions?

ThoriumBR

I have to paste it every time... but it's one line: eventTypes.deleteMessage = 999
so it does not bother me... and my browsers usually keep running for a week or more

reed

ok, I see

ThoriumBR

but I could create a nodejs client, or a python one... but this works, and takes 10 seconds

Fire Quacker

GreaseMonkey/TamperMonkey is simpler...

ThoriumBR

15:44

sure it is

back on my college days, people spent time on webchats, A LOT of time...

but the websites back in 2000's where not that user friendly, so there were a popular program down here called Loquax that was a client for web chats and I admired the guy who wrote it

I always planned to do something similar when I got better at programming, until I got better at programming and now I don't have free time for that, and webchats aren't much a thing anymore

and my wife wouldn't be thrilled with the idea:
"what are you doing until 2am?"
"I am talking with random girls online..."

reed

webchats aren't a thing anymore? Every chat protocol has a web interface today I guess

this chat only has a web interface in fact

ThoriumBR

now we have facebook/whatsapp/telegram, and smartphones

back then there was irc for some nerds, and html/javascript webchats, and desktop computers

reed

maybe I am misunderstanding what Loquax was

ThoriumBR

and they were all full of ads, with lots of functions lacking (like proper blocking, easier image sending...)

@reed imagine being able to telnet to port 6667 on a irc server and have IRC by console... and the connection fails from time to time, and you have to reconnect. and you may lose track of the messages because when it disconnects, there's no scrollback... then someone writes a graphical client that handles disconnections and recconects, saves the messages to a file, and you don't lose anything

nobody

@reed You can load the modified script everytime using chrome's devtools by setting up a local override. Caveat is the modified script only loads if devtools are open while the page is loading

ThoriumBR

15:58

that was loquax: a client that listened on the html traffic (yep, html), parsed it, and formatted on a nice window only with the text, filtered the ads away, re-joined the channel for you

and you could join more than one channel at the same time without needing two different browsers. seems absurd now, but not 20 years ago

reed

@ThoriumBR, I see. But I guess something like that would still make sense today, a desktop client that can connect to several web interfaces for chat (IRC, StackExchange, whatever). However since most people now have browsers open all the time with tabs, maybe it's not something we really need

nobody

@ThoriumBR And then some asshole comes around, sends a message that triggers some AV signature, and you lose all your chat logs

reed

in fact I'm planning to stop using IRC desktop clients, and just use the web interface (I only use Freenode, which seems to have a decent interface)

ThoriumBR

there are too many things today: slack, discord, irc, whatsapp, telegram, signal, facebook messenger...

by the news, freenode is currently melting down

I spent some time on freenode a decade ago

A. Hersean

Freenode is dead. Long live Libera.
https://www.theregister.com/2021/05/19/freenode_staff_resigns/

ThoriumBR

16:10

freenode staff are taking over some very old and popular channels, and that's not cool

they took over ubuntu (iirc) channel because some mod said something about libera.chat

reed

what does "take over" mean, and is that a problem?

as long as they are not banning users, it might not be a real problem after all

ThoriumBR

take over means: current mods aren't mods anymore, they can change the channel text/title, kick anyone, or put the channel on invisible mode, or voice only mode (only users with the "voice" flag can write)

it's like someone from another forum comes, strips the mod status from our mods, strips all our privileges, and mods the questions so only questions they agree with are posted, the others are hidden and the ones complaining are banned

A. Hersean

And the new mods found it nice to edit the channel title of some chans, removing useful information for newcomers, because it "did not comply with their company policy" or whatever

ThoriumBR

exact

reed

but why are they doing it? I mean, I haven't followed the entire story which seems far too complicated for me (and I really don't care about every detail), but I think I understood that the new "boss" is not really new, they bought or took control of Freenode anyway years ago

why is this all happening now? Is there a plan to transform Freenode into something different, like a paid platform, add ads, etc.?

ThoriumBR

16:18

basically the former owner didn't owned freenode and sold it to the current one, that on the first day on helm put forward some changes without even asking the former staff

Journeyman Geek

@reed essentially the whole of the old staff was bundled out

And they're trying to stop people fro advertising/moving out to the alternative

ThoriumBR

and every time someone talks about libera, they got banned if a common user, or they take over the channel if a mod said something

nobody

19:20

0

Q: ASP.NET or PHP for an online market?

Hello i wanted to create a little online market and i need to learn a language for that. I looked at C# and PHP and wanted to ask wich of these two are better? (Security ect.) Probaly there is no real answer but maybe i can get some tips from you. Or are there better languages for that?

Who wants to go rant about PHP?

Maybe @MechMK1

ThoriumBR

19:34

PHP is good, really... the problem is a lot of PHP programmers that don't know anything about secure coding

and all PHP books that says "security is left as an exercise for the reader" when the reader does not know how why if $x = 1 always returns true...

most tutorials are "copy and paste this, change the password and done" but have no error handling, absolutely no input sanitization, no output escaping, nothing!

nobody

Well, that is why ASP.Net is better security wise

The easy way to do things is the secure way

ThoriumBR

when I was learning PHP, you could EASILY take over any ASP site because escaping wasn't well known to ASP developers

it was so easy it wasn't even funny

nobody

I guess that has changed. Just a few days ago Mech was complaining about how secure ASP was.

ThoriumBR

a login with admin and using 'or 1=1 or' would land you on the admin page of almost every site

good frameworks can help... PHP have so many frameworks that usually people don't use them

well, I don't use them because I write smallish sites, and I can secure code (supposedly)...

nobody

I hate frameworks with a burning passion. (Although they are useful for keeping things secure with devs that don't know about security)

You know javascript? Oh that means nothing. You need to learn nodeJS, reactJS, blablablaJS....

ThoriumBR

19:42

the good thing about frameworks is that most stupid errors are absent, so novice programmers can write less blaringly insecure software

the bad thing is that one error on a popular framework means mass exploitation with a click

nobody

Yes that is right

@ThoriumBR But that is the same with any other software too

ThoriumBR

yeah, javascript frameworks are a disease... you learn one and a thousand more appears overnight

anything ending with _script is probably a javascript one... cakescript, borgscript, foobarscript, noobscript... it's madness

I don't like them either, I usually go to the "you don't need jquery" site and copy code from there

and for PHP I use a very small library called TinyButStrong (and the name is very accurate) to handle things on the backend

nobody

hunh, that you might not need jquery thing is interesting, I didn't know about it. Looks very useful

ThoriumBR

and ActiveRecord for data persistence... less layers means more performance, less resource-usage, and less phantasmagorical errors out of nowhere

@nobody it's very useful! I saw sites embedding the entire jquery script (the full version) only for show/hide a div... what a waste

nobody

What? Phantasmagorical? That's actually a real word?

ThoriumBR

19:50

and if you code anything on PHP, take a loot at TinyButStrong... the day I discovered it I never ever started anything that used a database without it

@nobody I typed it and the corrector didn't complained... so maybe it is

nobody

@ThoriumBR I will, if I ever have the misfortune of needing to do web stuff again

reed

I hate frameworks because it's an abstraction layer that forces you to see things the way someone else sees them, and meanwhile you don't learn what's really going on

ThoriumBR

hahaha

so do like every sane programmer should do and code your own framework!

nobody

@reed So you write your webapps in assembly? :)

reed

and since there are lots of people who see things differently, you end up with too many frameworks, every day there's the new cool framework to learn that is going to be the "future" (except it goes out of fashion in less than a year and is replaced by the next "vision")

ThoriumBR

19:53

Current status: there are 1738 javascript frameworks
- what? that's ridiculous! let's create one that solves all problems for all cases.

Current status: there are 1739 javascript frameworks

reed

yeah, there's an xkcd for that about standards

ThoriumBR

yep, I was thinking about it...

I read a funny quote about javascript frameworks but I cannot remember the punchline... I remember it was very funny but telling only half will make no sense

reed

on the other hand frameworks can be useful in several cases, so it's just a matter of... learning curve. If there is a de-facto standard framework, pretty stable, relatively easy to understand, doesn't add too many layers of abstraction, then it's ok

one example I guess is Bootstrap for CSS. Without it, it's just too much work for nothing, also trying to support and test different browsers

ThoriumBR

that's the premise of more than half existing frameworks...

if you google "css framework" and compare their proposed strengths, it's basically the same for every one

reed

- This code is difficult to use and understand! Let's write a better project that solves all these bugs and is also easier to understand!
RESULT: yet another software is released who is actually easy to understand only by the programmer who wrote it

ThoriumBR

20:04

Bootstrap: Quickly design and customize responsive mobile-first sites, featuring Sass variables and mixins, responsive grid system, extensive prebuilt components, and powerful JavaScript plugins.
Semantic: allows developers to build beautiful websites fast, with concise HTML, intuitive javascript, and simplified debugging, helping make front-end development a delightful experience.
Foundation: is a responsive, mobile-first framework that has completely semantic markup and works particularly well for Ruby on Rails projects. It comes with a responsive grid and HTML and CSS UI components, temp…

and the list goes on...

reed

I am going to create a better framework that will be even easier and useful. To build a website just write: createWebsite();

that's all!

nobody

Well, since people love making frameworks, let's make a framework to make frameworks

Each one of us should make a separate one. That way the framework developers will get a taste of their own medicine

ThoriumBR

you create your own framework this year and create a job opening requiring "3+ years of experience on ReedScript"

Journeyman Geek

20:20

@ThoriumBR then someone will reject you for lack of experience

reed

@ThoriumBR, people lie all the time in CVs/resumes, so when there's a 1-year-old framework, people will claim they have 2 years of experience with it. So that's why companies will require 3 years or experience or more

in turn people will claim 4 years of experience

and so on

ThoriumBR

I saw recently an opening asking for 10+ years on Swift...

reed

I know somebody who has more than 10 years of experience with PHP and still always writes "echo $variable" without any kind of output sanitization anywhere. So yeah, he has 10 years of bad experience, LOL

ThoriumBR

I saw a senior VisualBasic programmer using threads and no semaphores

when requests got over certain threshold the application would fail and crash, and people didn't knew why... I asked them about threads and they said "yes". I asked about semaphores and they said "what?"

we started calling the guy 'Mr Thread"

the other was a senior systems admin, and he had to put an IP on a server on the 10.20.30 network... he saw 10.20.30.255 available and tried to allocate it on the server... he was nicknamed "Mr Broadcast"

nobody

20:41

@ThoriumBR Wait, what's a semaphore?

forest

@ThoriumBR Are semaphores the only synchronization primitive in VB?

@nobody It's a way to synchronize activities. Like a mutex.

ThoriumBR

it's the easiest one

nobody

And what is mutex?

Sorry, I'm a terrible noob

forest

Mutual Exclusion. It's a "lock" that a threaded application can use.

It locks a resource so the other thread can't access it until the first thread unlocks it.

nobody

Oh lock

ThoriumBR

20:42

they are kinda the same thing... mutex means MUTually EXclusive, and a semaphone means "wait for your turn"

nobody

I see

forest

So for example, you could have the following function in a thread:

void do_thing(void)
{
    mutex_lock(&mtx);
    slowly_modify_structure(&big_struct);
    mutex_unlock(&mtx);
}

ThoriumBR

that was for a bank... now just imagine a race condition on the "transfer money" function for a bank.

nobody

So (reading from google) a semaphore is a lock that may allow more than one threads to enter?

forest

In that function, if two processes running it at the same time don't use the mutex, then if both enter slowly_modify_structure(&big_struct) at the same time, big_struct will be corrupted.

Semaphores are just another way of synchronizing activities.

nobody

20:45

So what is the difference between mutex and semaphores?

forest

Mutexes just atomically lock regions of code so the other process blocks (pauses) when it reaches mutex_lock(&mtx) if another thread has already locked it, and it waits until that other thread unlocks it with mutex_unlock(&mtx) before continuing. Semaphores are a bit more complex and can allow multiple processes to access a resource, up to a certain limit: stackoverflow.com/questions/34519/what-is-a-semaphore

nobody

Right, I guess I understand now. Thanks for the explanation!

reed

a race condition happens when a white programmer wants to modify a file at the same time as a black programmer

forest

hah

nobody

@reed So given that I can cause race conditions all by myself, I must consist of two two races.

forest

20:50

Unless the compiler generates code in different races.

ThoriumBR

imagine the withdraw function:
    withdraw(amount) {
        if (balance > amount) {
               balance = balance - amount
               send_money_to_user(amount)
            }
     }

forest

@ThoriumBR You can click "edit" and then "fixed font" and "send" to format the code.

ThoriumBR

I was about to ask

reed

what is the simplest example you would came up with, if you had to explain the concept of "race condition" in general to somebody who knew nothing about computers?

I mean, an example of race condition in real life, outside the world of computers

forest

@ThoriumBR It's not freenode staff doing it! It's the "new" staff, not the old staff.

ThoriumBR

20:53

@reed daylight savings clock adjustment

one sees the clock "ops, it's time to advance the clock one hour". but before he changes the clock the next person sees the clock and thinks the same and does the same

forest

Or two children vying for the same toy and not knowing that the other wants it (e.g. video games, screwing with each other's saves).

ThoriumBR

person A goes to advance the clock one hour, but person B won the race and advanced the clock one hour, so person A does the same and now the clock is 2 hours ahead

nobody

Or adding salt to food when there are multiple cooks in the kitchen. I've personally experienced the result

ThoriumBR

@nobody that's more like it... two kids wanting the same toy is a deadlock

forest

@nobody Oh that's a good one.

ThoriumBR

20:56

@nobody it's only a race condition if Cook A tastes the food and realizes it needs more salt, but Cook B does the same BEFORE Cook A adds salt to the food.

nobody

Funny thing is, both cooks were on a "diet" so they never had to suffer the taste

forest

@ThoriumBR That's a specific kind of race condition: A TOCTOU.

But there are more than one kinds of race conditions.

nobody

@ThoriumBR Yeah, how else would it happen?

reed

I was thinking of receiving a message on whatsapp or an email. You check if there are new messages, there are none, but right when you are closing the application a new message arrives, the app is still open so it considers it as "seen/read", but you haven't really seen it. The result is that you don't get any notification for the new message, but you haven't seen it

however that involves tech/phones/computers, so it's not really a great example

forest

8

Q: How would you explain what is a race condition to a five years old

TL;DR: How would you explain what is a race condition to a five years old. Which analogy to concepts he is familiar with would you use? Long version: I am not a teacher but a software engineer. Yesterday evening, while discussing what the day was like with my five years old son I told him that I...

lesson-ideas

reed

20:58

nobody's example, or variations of it, are probably much better

That answer is cool too, any example that involves "fetching" something might be good.

forest

Race conditions include a lot of different events, from TOCTOUs to deadlocks/livelocks.

A race condition is nothing more than a phenomenon that occurs when two concurrent tasks share a non-atomic resource and don't properly synchronize behavior with each other.

ThoriumBR

I have a joke on deadlock:
the director of a company says to his secretary: "I will go to a conference for 3 days and you will go with me"
the secretary calls her husband and says: "my boss wants me with him on a conference for 3 days, so I'm not home"
husband calls his mistress: "my wife is away from the city for 3 days, so come over!"
the mistress calls the boy she is teaching: "I won't be able to teach you the next 3 days, so you are free"
the boy calls his grandfather: "grandpa, my teacher said I won't have any classes the next 3 days"

forest

Isn't that a livelock?

reed

Friend A says: I'm at home right now. Friend B says: I'm at home too right now. Friend A thinks: I'll go visit my friend. Friend B thinks the same. Both leave their houses, and when either of them arrives at the friend's house, nobody's at home

forest

> A livelock is similar to a deadlock, except that the states of the processes involved in the livelock constantly change with regard to one another, none progressing. Livelock is a special case of resource starvation; the general definition only states that a specific process is not progressing.

(From Wikipedia)

ThoriumBR

21:07

good catch! it's a livelock

reed

@ThoriumBR, has anything like that ever happened in real life? Because it would be cool. What happens then? LOL

forest

Probably for a brief amount of time, but unlike programs, humans tend not to like going in loops forever.

reed

probably somebody is going to say "fuck you" at a certain point, so the dreadlock or livelock ends

forest

"Stop copying me!" "Stop copying me!" "Stop copying me!"

However... yes. Sometimes it happens with packages in the mail.

'They can get stuck in round trips, and unlike TCP packets, they have no TTL to decrement. The only time the package is finally discarded is when it's so beaten up that people realize it's stuck in a loop.

abc7news.com/news/…

40,000 miles of looping for a birthday package.

> The package was a birthday gift Jacoby sent to her god-daughter in Brooklyn. True to its Priority Mail promise, it got there in two days.

However, the package went unclaimed so the post office shipped it back to San Francisco and that's when it entered the postal vortex.

As soon as the package arrived back to San Francisco, a machine read the address and shipped it to Brooklyn again. "And when it was in Brooklyn they would say, no we already tried to deliver it. It has to go back to San Francisco and then back and then back and then back," Jacoby said.

ThoriumBR

I once got a mail for the previous guy that lived on the house I lived, I dropped it on the post office before going to work, got home and the letter was home again. I did the same next day, and guess what?

forest

21:15

heh

Race conditions are the bane of fuzzers everywhere.

It's next to impossible to pin down a bug when you can't reproduce it with AFL.

And so fuzzers tend to just ignore those bugs, leaving them open to exploitation by people doing manual analysis or with deterministic fuzzers (like those made by Brian Falk).

ThoriumBR

race conditions are terrible to troubleshoot
"it was not working a second ago! you must have changed something on your way here"

forest

That's why deterministic fuzzers are so good. They snapshot the system's state before the event in such a way that you can always reproduce it.

ThoriumBR

and more challenging are those that depend on a heavy load on the system to occur...

forest

Yup, since then the race conditions aren't as easy as "it happens once ever n instances on average", but "it happens only when the system is in a very specific state". It's yet another reason that deterministic fuzzers are so important, and why we need to move to them.

ThoriumBR

7 years ago I got hired by a large bank, and on my first day on the job (no joking) we got bitten by a race condition on the Fibre Channel kernel module...

it was VERY specific, depending on several variables, and made the throughput go from 200mbps to 100kbps...

forest

21:25

Is that kernel module in tree?

ThoriumBR

yep...

the High Performance Ficon kernel module

forest

Did you get the bug fixed upstream or is it still reproducible?

ThoriumBR

it got fixed, SuSE guys upstreamed the fix

ficon is a nice low-level disk protocol... it can batch a lot of operations and send them all at once using several channels at once.

but when this bug happened, it went from "batch everything and send a fat transaction" to "send bype per byte and acknowledge every one"

forest

Oh wow.

Now imagine how slow that'd be if the NIC also switched from DMA to PIO!

ThoriumBR

the workaround was reboot, but you don't reboot a bank database during the day and without knowing what went wrong and making sure it wouldn't happen again

forest

21:32

So this was for the DB backend?

ThoriumBR

so on my previous job I went home late night and told my wife this job was a vanilla sysadmin job, not a troubleshooting analyst, and on the first day I left my desk 11pm...

yep, on a DB... when the transactions per second got over a certain threshold, one buffer got overwritten and there were no acknowledgements of previous transactions, so the driver reduced the send window, and reduced again, and again, until it got to a single byte...

like the exponential backoff congestion on ethernet

forest

wow

@ThoriumBR No sysadmin job is a vanilla sysadmin job. :P

ThoriumBR

the previous job I was on a "disaster recovery delta force team", so when a disaster happened I was flown there to do whatever needed to be done. no clock-in time, no clock-out time...

forest

aah

So I take it you spent a lot of time with ftrace and perf tools? :P

ThoriumBR

we got a baby and I wanted to be home on a previsible schedule, so I got a sysadmin job... usually it was 9am-6pm, but this day was like the older hob

*job

@forest strace, ltrace, less, more, tail, grep/awk/cut, tcpdump...

forest

21:38

Ah

ThoriumBR

we were the last resort... after the onsite team and first and second level gave up, and the lab got involved, and couldn't find the issue we got involved

forest

Yeah sounds like my general toolkit, although I rarely have such high-stakes as a major bank.

ThoriumBR

because we were fricking expensive... out team charged $250 to $450 USD per hour, plus plane ticket and accomodations, and transportation...

so some clients got billed $20k for a week, and it was a bargain because they lost twice that per hour of downtime

forest

I read a report somewhere about a tech support guy who worked with a company that would lose half a million dollars per hour of downtime.

ThoriumBR

the most interesting part (or least interesting) is that sometimes my boss had no idea what was wrong and we got briefed on the situation between the security gate and the desk

forest

21:42

How long did it typically take you to fix problems you ran into?

ThoriumBR

@forest I have a colleague that did the same. the company called him, asked how fast he could hit the datacenter door, he said "in this traffic, 45 minutes", and they asked him to go to the roof because an helicopter would get him in 3 minutes

forest

Damn

ThoriumBR

usually 4-6 hours. I once got on a problem at that bank that took the onsite team, 1st and 2nd levels plus lab 10 days to diagnose, and they were clueless on that, and I solved it on 15 minutes... it was a VERY LUCKY shot, but was very, very nice to solve... imagine all VMs missing pings, ssh connections dropping, db and app server losing communication.

lab got send several GB's of traffic, people were sleeping on the floor, piles of pizza boxes all around, the IT DIRECTOR was there for 2 days without going home, ATMs all over the country losing communication, was a disaster..

forest

Yeah, I far prefer jobs where I can spend my sweet time finding a solution... heh

ThoriumBR

I got there, director said I would have every access I wanted, and I run 3-4 commands, and say: "your VMs have too much memory". everyone laughs, they say it's not the time for jokes, I say I'm serious, IT boss disagrees and says I'm crazy because there's no way it's memory interfering with the network, it's clearly a net issue

so I say to just for a test to shutdown a 32gb server and pings will stabilize... they do that, network works fine, boss says that server is the culprit, they got the server back, network down... I say to shutdown another random server and the same happens.

forest

21:49

How bad was the memory management that the VMs could eat up enough memory to cripple the host? O_O

ThoriumBR

what happened is that they were about the cross into virtual memory trashing and created one large server, so the hipervisor lost more time allocating memory than processing. so when the server went into processing time, its time was over and it got deallocated again, and it could not receive any traffic

when you got one server off, there was enough memory to the hipervisor allocate the next server memory before the server went in processing status and everything worked fine

wasn't bad at all, they were bad at math... they got an increase of say 64GB physical memory on the host, and gave extra memory to the VMs, but there was more VMs than they realized

physical memory was 256gb and virtual was well over 1.5TB

forest

Overcommit strikes again! It's one of the things I dislike about Linux mm...

ThoriumBR

host was fine, the VMs were suffering...

hipervisor was zVM, and it can easily manage a 6:1 overcommit... I managed a server with 8:1 overcommit, but you have to fine tune the disk subsystem to account for that. they didn't

they had a couple very large disks on the same storage array, using the same 2 channels

my 8:1 had several small disks, allocated on several storage arrays, and several pairs of channels

forest

zVM... I've never used that. Only KVM.

ThoriumBR

zVM is the father of virtualization... first version came out in 1967

vmware was created by zVM engineers that left IBM

forest

21:55

I always thought Sun came out with the first virtualization.

Is it any good as a hypervisor?

ThoriumBR

no, zVM was called VM-67 and got way before

forest

ah

ThoriumBR

I worked with KVM, VMware, Hyper-V, and I can say you cannot compare them... zvm management is clunky, the interface is from the last century, there's no "are you sure?" prompts anywhere, but it's very fast, stable, very secure, and the overhead is tiny

forest

Secure in what sense?

ThoriumBR

I once crammed 120 virtual processors and 360GB of virtual memory on a server with 15 processors and 64gb of memory

forest

21:59

I usually find that statements about a hypervisor's security are misleading, since people so very rarely actually try to break its security. Does it use any kind of interesting hardening?

ThoriumBR

it got EAL5+ certification, for example

forest

oh

I wonder what aspects of the hypervisor that covers.

Transcript for