« first day (1412 days earlier)      last day (3855 days later) » 

09:00
@MichelZ Or maybe unix.se (just remembered)
!!
maybe, yes.
but serverfault.com is for professional system administrators with professional setups
a 4G WAN gateway does not strike me as overly professional
Actually I had a similar task and wasn't able to solve it solely with ipfw. I employed a third party billing solution to do the task.
Thoughts aloud. Maybe it doesn't even matter where internet (IP packets) come from -- from 4G WAG gateway or Google Fiber ;-)
Dan
Dan
@YasirArsanukaev It may not matter technically, but it matters in terms of professionalism
Never thought about this in that way actually. Probably it can be inferred from these sites' faq or something.
to me it looks like a home-dude getting faster 4G connection than DSL....
the help center is pretty clear about stuff like that
and we take it seriously
09:05
sigh...
Server Fault is a site for system and network administrators needing expert answers related to managing computer systems in a professional capacity.
and it is not about…
•Anything in a home or development environment
so just because it's a "server operating system" does not make it on-topic here automatically
Dan
Dan
The real question is why am I let in!
@Dan pity
@Dan ?
@MichelZ Agree. Though still it fits better somewhere else than on SO, even on SU :-)
09:08
yes, that is true. but just migrating it to another site without knowing whats on-topic and what's not is not really to goal either :)
Dan
Dan
@ewwhite Probably :(
I'm in a storage bind
Dan
Dan
Too much HD porn again?
@ewwhite Sounds more like a DNS issue ;)
@Dan As a consultant, you know how you end up taking responsibility for solution recommendations
Dan
Dan
09:12
@ewwhite Sadly, yes
@Chopper3 kinkier than that
@ewwhite Was thinking the other night, how many times has it been Chicago that's the ruined city in Hollywood films recently? time was when it was always NYC getting destroyed but these days it's always Chi
Customer needs shared storage for a small 3-host vSphere cluster...
@ewwhite MSA2040
Well, they were quoted a $38k EMC VNXe...
09:13
@ewwhite StarWind
@ewwhite seems overkill perhaps
and I could sell them a $20k MSA 2040
or I could sell them a pair of ZFS-based DL380p Gen8's with replication for ~$12k
they're anticipating < 20 virtual machines.
Dan
Dan
@ewwhite I'd go for the MSA, but then I'm no fan of the home grown stuff
so maybe I could even suggest VSAN
Dan
Dan
@ewwhite Why do they even need three hosts?
09:14
They don't need 3 hosts.
Dan
Dan
:D
They were going to buy 6 CPUs of vSphere standard licensing for $14k
I'm dialing it back to Essentials Plus for $4500
@ewwhite yeeeah, keep the difference for yourself
@Dan So my homegrown stuff can be just Linux, Nexenta, ZetaVault, QuantaStor... the latter three are supported and VMware certified
and two storage servers is better than one VNXe, no?
Dan
Dan
@ewwhite I know, I appreciate that.
@ewwhite Well, surely the VNXe is fully redundant in itself? Multiple controllers etc?
09:17
and of course, the ZFS stuff will outperform... but I'm torn because I shouldn't impose my will...
@Dan no more so than an HP MSA 2040 box. The VNXe also has bad controller failover times.
and my other customers with them have definitely had problems
Dan
Dan
@ewwhite I think if you're sincerely confident in the "fit and forget" nature of your ZFS, then go for it
@ewwhite I like those 2040's, thinking of getting one for home in fact, as you can mix and match FC and iSCSI SFPs on a port-by-port basis
@Dan I've done it many times. So there's a big install base... and I eat my own dog food..
@Dan I'd agree, I don't have the confidence with the ZFS stuff that Ed does but it does sound cheaper
But I could also slap a dumb HP MSA 2040 unit in there... $7k without disks (FC, 10GbE, SAS)
@Chopper3 basically a pair of DL380p's... hardware that I trust
09:21
@ewwhite We're using the same kit for our labs, using Windows Storage Server 2012R2 and a D2700 shelf between them
Dan
Dan
@ewwhite Turning it around, you seem to like ZFS but you're asking us so what's your concern?
@Dan that I'm blinded.
I know it works... and most of my clients just go with what I say.
Dan
Dan
@ewwhite I think every consultant in the world who "cares" will always try and use things they know as a first option
but this is a slightly different customer, and I think it's hard for people to understand how a solution that hosts 1/4 the price of the EMC is viable
@ewwhite well I'm always blinded by 'ease of ownership' as I don't like supporting - so for me picking a single SKU that's supported by someone else and is based on bits I know and trust is all I care about - but you're used to these 'roll your own' setups that make my teeth itch - 'horses for courses' (this may be an English expression sorry)
Dan
Dan
09:22
My job isn't really the same as yours and, because I don't act alone, I have to watch out for the others reputation too. However, if it were me, I'd go in with the options and talk them through it. Basically, let them decide with your advice.
But then, I also know that the VNXe is kinda bad news
@Dan see as a consultant I consider it my role TO decide for them
@ewwhite Oh I agree, anything but the VNX
and ALL of my clients with commercial SANs have problems...
Dan
Dan
@Chopper3 Really?
why? because they're buying what they THINK is good stuff, but is always at the bottom of the product lines
09:24
@Dan yeah, they don't want options, I stopped giving them a long while back - every so often someone specifically asks for them but even then they get two to choose from and I've made up their mind for them before we start talking
So entry-level Netapps, VNXe, crappy Equalogic
@Chopper3 this is a good approach... but sometimes I'm not sure myself
Dan
Dan
@Chopper3 Fair enough - though, again, our role differs as I'm 'selling' to an IT Team. They always want the detail
@ewwhite yeah - I can see why they would, it sounds to me like you've made up your own mind and it's a solution you have faith in
e.g. have I just been lucky with my installs?
@Dan yeah, mine is to just design and build stuff - nobody really second-guesses me
@ewwhite what's the success/failure rate?
09:26
or maybe this is a chance to just roll out VMware's VSAN?
Dan
Dan
@ewwhite Dunno, I don't hang around long enough to ever find out about mine :D
@Chopper3 I've only had ZFS problems on my most complex installation... MINE.
@ewwhite haha
all Nexenta issues... which I really don't deploy anymore
never lost data... done a lot of stupid things... performance still beats these entry SANs
@Chopper3 No VSAN love?
@Chopper3 at clients, zero failures, though
(checks wife's ZFS "SAN")
root@san1:/volumes# uptime
  4:29am  up 712 day(s), 15:59,  1 user,  load average: 0.24, 0.16, 0.14
hmmm
@ewwhite Not yet, I've seen the next version and it makes a lot more sense, I might use that in lab kit or whatever in the future but the version of today isn't something I need or trust
@ewwhite cool uptime, no updates?
09:30
It's appliance software from Nexenta. Reboots weren't needed
(Solaris, really)
@ewwhite I know it (a bit)
Running on a DL380 G7
@ewwhite fair enough
Dan
Dan
My wife is perusing christianlouboutin.com ...
09:33
@Chopper3 so wait for VSAN
@Dan u fancy
@ewwhite Lab it, see what you think, I don't think it's ready for production yet no, but then perhaps I'm more cautious than most - also the benefit to us is lesser as we get 'proper SANs' for so little
Dan
Dan
@ewwhite I'll buy her some for her birthday probably
@Dan oh god, my wife likes that sort of stuff too - and kate spade too for some reason
@Chopper3 but at the same time, you trust those DL380's for storage, right?
@Chopper3 Not to mention my Jack Spade addiction.
@ewwhite yes, the tin, no experience of the whole zfs stuff though really
Dan
Dan
09:35
@Chopper3 Interestingly, she's never really shown that much interest in shoes before.
@Dan my wife likes 'birkin bags' - way too much
Dan
Dan
I need some new glasses, actually. I'll be going for Oakley as ever
@Dan :(
Dan
Dan
@Chopper3 Now they're not cheap
@ewwhite At the glasses, or Oakley
Ray Ban... Warby Parker...
09:36
@Dan nope, luckily she's not poor herself so it's less of an issue
I use Oakley for cycling... have for 20 years...
but just had a terrible experience at their store last week
@ewwhite I was talking to someone wearing some WP glasses just last week, I'd asked where they were from, good value I thought
@Chopper3 they tried to hire me... next door to my old firm in NY
@ewwhite son got some of those custom ones made for him when we were in hawaii last, wears them all the time
but I did get my wife to buy three pairs
09:37
@ewwhite oh cool, these were round tortoiseshell ones
Dan
Dan
@ewwhite Always liked them, have a couple of pairs of sunglasses and these cost me about £600 (Incl lenses etc), but have lasted easily 5 years
Being nagged to do some packing (heading off in a few hours) - back later chaps
Dan
Dan
@Chopper3 Enjoy
@Dan as regular glasses?
Dan
Dan
@ewwhite Yeah
They're fucked now, though, tbh!
09:39
and metal frames (Titanium?)
Dan
Dan
@ewwhite Metal framed, but I don't recall the type
I'm in a "changing my look" kind of mood. Very much contemplating going completely bald to see how it looks
Get fancy... check out Warby Parker :)
@Dan Manscaping can be a good thing!
Dan
Dan
@ewwhite I agree, and historically I've not been that concerned by my looks beyond generally staying healthy - but, y'know, just sometimes
@ewwhite Retro!
@Dan I think you'd look good in The Sinclair
Dan
Dan
Oh man, I'm not feeling that!
09:44
(picked at random)
Dan
Dan
:D
hey I'm billing right now...
copying AppData to some windows file share
Dan
Dan
@ewwhite Me too, and I'm doing nothing!
09:46
@Dan those are tech geek glasses! Titanium and carbon fiber
(fibre to you)
Dan
Dan
@ewwhite I am a geek :(
understood
>_>
those are really wide arms
Sandra is back
0
Q: Possible to put larger disks in LSI arrays?

Jasmine LognnesI have a couple of these LSI arrays LSI Model: 0834 LSI Class: 3600 LSI Name: LSI Engenio LSI product name: 1932 LSI Product codename: Mary Jane Enclosure name: Shea (DM1300) End of Life: 31-Dec-2010 which was bought with 2TB disks. My plan is to use either XFS on Linux or ZFS on OmniOS. Que...

iis that Sandra?
09:56
yeah
ah, you already said that above
I think I need double-glasses :D
Dan
Dan
So is Sandra even a girl?
Or is just a ploy to get us silly IT Men to click and answer?
(Works, obv)
I suspect no... not a woman - This is the likely scene... NSFW
oops
Dan
Dan
:D
(but is that SFW?)
Dan
Dan
10:00
No.
@Dan I'm exercising restraint and NOT answering
10:18
@ewwhite aww, c'mon. You know you want to
10:44
lol
11:01
Hi All, does anyone recognise this program?
looks like it is running from a soft reset (no iso mounted) just wondering what it is
Dan
Dan
BIOS disk wiper?
Don't think so, it would be a vmware bios, I' haven't seen a disk wipe option (but I will now look for one :))
Dan
Dan
@Rqomey Oh, it's a VM?
I'm pretty sure the vmware emulated BIOS doesn't have a disk wiper
ahh yea, sorry
Poor customer is loosing a number of VMs, trying to figure out who or what is wiping his disks on him
Dan
Dan
@Rqomey Wow, holy shit
yep
Dan
Dan
As that machine is now goosed, what happens if you reboot it? Whatever application it is may have a splash screen or an initialisation screen
That dosen't look like either disk wiper I have seen
Dan
Dan
@JourneymanGeek Weird for it not to have its name anywhere
11:15
and multiple passes?
yeah
no os found
What OS are the VMs running?
Yea, thats the fucker- there is no iso mounted, there is not even a hard reset logged
w2k8r2
Dan
Dan
@JourneymanGeek *were!
11:17
point?
they arn't running anything
Dan
Dan
Have you tried mounting a now-broken VMDK to a (non-network attached!) VM to see what's on there?
yep, no partition table. havent been able to hex dump it yet
If its a disk wiping tool... and two passes...
Asked him to advise his AD team to lock out the boxes
poor bastard
11:19
I'm suspecting dban
I can't figure out what tool that is
maybe you should post a question on the site
but if it's an actual disk wiper and it's written two passess of 1's and 0's then the customer must get backups ready..
@Rqomey: do you have a PXE server anywhere on the network?
Dan
Dan
@JourneymanGeek Oh that would be an incredible fuck up. Surely you can't automate it to a degree where it's no touch....can you? Can you?
11:21
@Dan: dban, yeah
this isn't it
Dan
Dan
@JourneymanGeek That's so dangerous it's not even funny
You must be monumentally stupid though
@Dan: or running an illegal operation
I'd note, I'm a forensics guy. I assume the worst.
@JourneymanGeek no, but we checked by mounting the disk on another vm -- no partition table. Seems that all VMs are on a particular storage array, so suspecting a borked backup run, they using netapp so we'll engaged them and see if they have any handy disk wiping tools bundled with their software :)
Dan
Dan
@pauska Even a single pass will see him fucked.
with modern hard drives, yeah
11:23
NFS array, hes fucked
god. this is such an 'interesting' case.
Dan
Dan
It is, and it doesn't "feel" malicious unless it's someone on the inside
@Rqomey: eh. In this specific scenario... he's fucked no matter what
EVEN if he has backups
@Dan: if its malicious, its beautiful
:) I'll update later on anyway, let yea know what happens, thanks for the opinions!
Dan
Dan
11:24
@Rqomey Post it on the main site - I think many would be interested and smoeone may recognise the screen
@Dan will do
Right, I'm off see you all in a week or so, enjoy the sun
Dan
Dan
@Chopper3 Have a good one
@Chopper3 Where you off nice?
zenshaze.com/wp/?p=371 dosen't look like netapp either.
@Dan states - daughter's birthday
Dan
Dan
11:27
@Chopper3 Excellent
this guy is mental
Dan
Dan
@Rqomey Made some brief edits :)
tanks a millinoin :)
added NFS storage information
Dan
Dan
How many VM's has it happened to?
11:43
about 10 so far
netapp say nothing to do with them, they use vsphere api's only
@Rqomey do you guys have SCCM?
I'll check,
@Colyn1337 I'll have to get back to you, they are busy pulling logs at the moment
I ask because I wonder if someone advertised a foobar package
Dan
Dan
11:48
@Colyn1337 The DestroyerOfWorlds.msi
LOL basically
like if someone built a package to decom a server and advertised it to the wrong collection
Worth checking, thanks for the hint (and jesus I hope not)
lol
@Rqomey: honestly, if you work out what it is, its a good start.
so hope so!
ahh its fuckin panic stations at the moment there anyway, all their disks are full, I think the wiper is filling their backend array or something. Wont get any useful information for a while I think.
@Rqomey just for giggles I took that image and plugged it into google image search. There aren't any other images like it on google.
11:52
yep
@Colyn1337: I tried that and a few search terms ;p
First thing I did was check google images. Even searching for the string "Operation 2 of 2" "Wiping disk" Comes back with nothing... well it did anyway :) google.com/…
and the first two hits are.... your question
such an awefully generic thing too
I'll find out that some joker has it as their screensaver now or something
11:54
any indication that's going to happen to a server before it's rebooted?
rofl
@Rqomey: that wouldn't explain the disks being wiped
Oh my sweet jesus
3
Q: An unknown tool is wiping our Virtual Machines and we can't ID it

RqomeyA console view of a Windows 2008 R2 VM, on vSphere is showing the following screen: "Operation 2 of 2" "Wiping disk" Can someone advise on what this program is? Some information on this mystery: A number of VMs are now effected. The symptom is after reboot "OS not found" message is appearin...

2
:) @Colyn1337 no, we are only seeing the tail end at the moment
Looks like it's already being discussed
as in "no os found"
Dan
Dan
11:56
@MDMarra Welcome to the room :)
@Rqomey A virtual rootkit maybe? :-)
there's always the possibility you've been hacked and this is how the hacker covers his/her tracks
@Colyn1337 yep
do you have an IDS?
Ideally identifying the program would help.
Ahh to clarify, is isn't my servers :)
Infra guys have lead now, I'll have a chat with them about engaging security.... again
11:59
part of the problem in identifying the program is knowing from where it's executing. It could be outside windows, but it looks just MS enough to make me thing it's running just after the HAL
oi I don't envy you on this @Rqomey
but at the same time, the mystery is intriguing
@Rqomey I have seen windows getting really weird when people tried to use the same disks for two separate windows server. And I've also seen windows clear out disks when it saw disks without NTFS on it.
Yea, the fact no ISO was mounted, VM was not reset, and no console access was logged makes me think it is an internal program
but I'm not a windows admin so all I've got are horror stories by hearsay
@JennyD YEa if the NFS was presented to a windows server it can decide to auto format the LUN, this displays in a different way, in that case the esxi complains about invalid partition table
In this case the esxi is happy, and even some other VMs are running on the datastore
yeah, this looks like it's something done deliberately. Have you fired any sysadmins recently?
12:04
I've never been in google before, exciting :)
I don't suppose there's anybody on your site that's been working on a script to clear out old VMs?
Bob
Bob
@JennyD If so, they're never going to own up now!
Dan
Dan
I can't fathom the idea of doing a disk overwrite on a VM!
Another horror story: guy made a reimaging disk which he built on his own desktop system, and tested it out on a remote system. Worked perfectly. Then he left the disk in the drive and rebooted his system when he got back in the next morning... and he'd of course made sure he'd never backed up his local drive :-)
@JennyD its not my site exactly, but that may be a possibility. The fact they are localised to one of two datastores is still a bit suspicious. But are you thinking custom script?
12:11
@Rqomey Yes. Or possibly some sort of provisioning software.
@Dan Especially as I suspect they are doing a write-out operation only (hence the space issues). A bad idea (TM) for thin provisioned disks
@Dan me neither, but I have no problem thinking that someone brought a program meant for physical boxen and used it on a VM without thinking first
Bob
Bob
The only wiper I can find that looks similar is HDDErase.
Hm, nope.
Sup
one of the VMs is 200GB, will take an age to restore, bleh
12:20
@JennyD people who do not back up their stuff deserve that to happen :)
@MichelZ indeed, how else will they learn?
Dan
Dan
Automated phone systems are FUCKING WANK
@Dan they are?
Dan
Dan
@MichelZ Diabolically rubbish
And hugely irritating
12:25
:)
were you calling DELL?
Dan
Dan
I was calling Equifax and then a bank
@Rqomey Have you tried looking at the domain logs on who logged on to the machines last?
@Rqomey Are only Windows VM's affected?
@MichelZ I just asked the same in a comment, didn't know he was in here
that does appear to be a Windows operation, but it's hard to tell
@BigHomie apparently, there are no VMware logs of anyone doing anything (mounting ISO, looking @ console)
very cruel virus maybe :D
yeah, so it is probably Windows, unmistakable font, I'm thining virus too
12:29
I should program such a thing for all the fucking security nightmares and windows xp's out there
@BigHomie Or someone who's been fired setting up the windows equiv of an at-job
@dan Equifax is evil
@JennyD maybe @HopelessN00b works there?!
@JennyD yeah, now I'm curious as to how that might be done...
Dan
Dan
@Colyn1337 +!
@MichelZ Every employers nightmare!
12:31
@Dan yep. that's what backups are for. Wait... the sysadmin controls the backups too? you're screwed :D
@MichelZ Funny stuff. :p
Dan
Dan
It's 1:30pm - maybe I should start some work now
@HopelessN00b I can see your wheels turning and preparing the scheduled tasks...
@MichelZ Please, why bother? Console access to SAN and ESXi host servers. And Avamar boxes. Why bother wiping from inside the OS, when you can just wipe all the storage?
Dan
Dan
12:37
@MichelZ To be honest, I see Hopeless more of a napalm and AK47 kind of guy
@Dan idk....
(Not that I'm the type to do that, mind you, but still... do it right and nuke the disks, don't wipe from within the OS.)
Jul 11 at 19:54, by HopelessN00b
Wait, nevermind. It spanked the server so hard the screen didn't even refresh. Just got the RDP connection time out box.
Forkbomb extraordinaire
@BigHomie Thanks for that. Made for a completely uneventful on-call weekend.
On a side note, refrain from searching this room for spanked
np ;p
12:39
lol
you guys are so filthy
Dan
Dan
whip noise
@MichelZ Hey, I ain't running the call at the moment, so the focus is still on the storage side (they are trying to figure out if the other VMs were corrupted or not also)
looks to me like a windows only environment also
I'll update commetns
You guys aren't a non-paying client of mine, are you? =D
@HopelessN00b and the circle of life is complete
Where I work, we don't have a circle of life. Just a circle-jerk of insanity.
12:44
@Rqomey Why the storage side? How could that do something like that?
They wanted to check the VAAI stats to see if there was a write-out getting called. They have now ruled that out, so I am waiting for engagement from the windows team. Not sure what the time frame of this will be #
storage array filling up is not a big surprise if the disks are being randomized with junk - probably thin provisioned through NFS
anyone know how to fix raspicamera
and/or dedup
mmal_vc_component_enable: failed to enable component: ENOSPC
12:49
@user44517 look at the top right corner of this page...
@user44517 Hi user44517, did you look at the top right corner of this window?
@pauska thin on the back yes, I'd say that's what filled the disk
ahah sorry
ok bye :D
@MichelZ A number of years ago I was the mail sysadmin at the same company where my husband managed NFS and backups. The company got bought out and started treating its employees like crap. Husband and I spent a lot of time designing ways to destroy the company without getting detected.
One of the favourites was encrypting all backups, having the decryption key in the restoration software, but set a job to destroy the key some time after we'd quit. Then set another job to start randomly deleting stuff later on.
as much as I hate some of my previous employers/bosses, I would never ever do something like that..
12:52
@JennyD remind me never to fuck with yea lopsa.org/CodeOfEthics
@pauska We never did it, it was cathartic enough to sit and design the destruction.
:)))
I gotta go now, but thanks for all the ideas, I'll update when I get more info on what happend
@Rqomey good luck, I really look forward to hearing the end of this story
@Rqomey btw, you guys should probably suspend one of those VM's running disk wipe so that you have some kind of evidence for forensics research
@JennyD my favourite is to use our job scheduling mechanism to schedule a reboot of all the things simultaneously. Should cause a nice brownout :)
12:54
@pauska very good point
@DennisKaarsemaker nod
should be fairly straightforward to dump the RAM contents and find out exactly what software it is
2
@pauska good idea, unfortunately disk wipe was completed a while ago.
oh :(
yep
@Rqomey if it happens again though... maybe set the VMware to make snapshots regularly?
Dan
Dan
12:57
@JennyD Noo!
Not unless it's combined with a proper backup solution which will take the snapshots elsewhere and then delete them
@Dan aye
Dan
Dan
@pauska Is that simple? Wouldn't have a clue how to do or analyse that
@JennyD I'll ask the admin to suspend, not snap. It is is writing rnadom data would fill the whole array (again)
also have lots of logging, to a separate server that is not on the VM
Dan
Dan
12:59
@Rqomey Cool
yea, life saver

« first day (1412 days earlier)      last day (3855 days later) »