Stack Exchange
log in

Rqomey

 The Comms Room

ServerFault's lobby
2
Rqomey
Jul 22, 2015 10:02
my god what happened here? Has everyone left?
Rqomey
Jul 22, 2015 09:57
Hi All, I'm on a 4 cpu server (ubuntu) running more or less idle, and is has a load average of 103! Hss anyone seen this before? I assume it is a bug or something, as the server isn't on fire and begging for death
Rqomey
Aug 4, 2014 11:06
Unfortunately the investigation was removed from my "domain" so I don't know what happened in the end... All I heard was a "user initiated disk wipe" but fuck it I threw together a lessons learned for the question in case anyone else hits something like that. But wanted to say thanks again :) @Dan @JennyD @pauska @Colyn1337
@MichelZ
9
Rqomey
Aug 4, 2014 11:01
Hi All, a while back you helped me out with a horrible issue: serverfault.com/q/615130/118470
Rqomey
Jul 24, 2014 12:59
yea, life saver
Rqomey
Jul 24, 2014 12:59
@Dan kb.vmware.com/kb/2003941
Rqomey
Jul 24, 2014 12:58
@JennyD I'll ask the admin to suspend, not snap. It is is writing rnadom data would fill the whole array (again)
Rqomey
Jul 24, 2014 12:55
yep
Rqomey
Jul 24, 2014 12:55
@pauska good idea, unfortunately disk wipe was completed a while ago.
Rqomey
Jul 24, 2014 12:54
I gotta go now, but thanks for all the ideas, I'll update when I get more info on what happend
Rqomey
Jul 24, 2014 12:53
:)))
Rqomey
Jul 24, 2014 12:52
@JennyD remind me never to fuck with yea lopsa.org/CodeOfEthics
Rqomey
Jul 24, 2014 12:49
@pauska thin on the back yes, I'd say that's what filled the disk
Rqomey
Jul 24, 2014 12:46
They wanted to check the VAAI stats to see if there was a write-out getting called. They have now ruled that out, so I am waiting for engagement from the windows team. Not sure what the time frame of this will be #
Rqomey
Jul 24, 2014 12:42
@HopelessN00b and the circle of life is complete
Rqomey
Jul 24, 2014 12:41
I'll update commetns
Rqomey
Jul 24, 2014 12:41
looks to me like a windows only environment also
Rqomey
Jul 24, 2014 12:41
@MichelZ Hey, I ain't running the call at the moment, so the focus is still on the storage side (they are trying to figure out if the other VMs were corrupted or not also)
Rqomey
Jul 24, 2014 12:18
one of the VMs is 200GB, will take an age to restore, bleh
Rqomey
Jul 24, 2014 12:11
@Dan Especially as I suspect they are doing a write-out operation only (hence the space issues). A bad idea (TM) for thin provisioned disks
Rqomey
Jul 24, 2014 12:10
@JennyD its not my site exactly, but that may be a possibility. The fact they are localised to one of two datastores is still a bit suspicious. But are you thinking custom script?
Rqomey
Jul 24, 2014 12:04
I've never been in google before, exciting :)
Rqomey
Jul 24, 2014 12:03
In this case the esxi is happy, and even some other VMs are running on the datastore
Rqomey
Jul 24, 2014 12:03
@JennyD YEa if the NFS was presented to a windows server it can decide to auto format the LUN, this displays in a different way, in that case the esxi complains about invalid partition table
Rqomey
Jul 24, 2014 12:02
Yea, the fact no ISO was mounted, VM was not reset, and no console access was logged makes me think it is an internal program
Rqomey
Jul 24, 2014 11:58
Infra guys have lead now, I'll have a chat with them about engaging security.... again
Rqomey
Jul 24, 2014 11:58
Ahh to clarify, is isn't my servers :)
Rqomey
Jul 24, 2014 11:57
Ideally identifying the program would help.
Rqomey
Jul 24, 2014 11:57
@Colyn1337 yep
Rqomey
Jul 24, 2014 11:56
as in "no os found"
Rqomey
Jul 24, 2014 11:56
:) @Colyn1337 no, we are only seeing the tail end at the moment
Rqomey
Jul 24, 2014 11:54
I'll find out that some joker has it as their screensaver now or something
Rqomey
Jul 24, 2014 11:53
First thing I did was check google images. Even searching for the string "Operation 2 of 2" "Wiping disk" Comes back with nothing... well it did anyway :) google.com/…
Rqomey
Jul 24, 2014 11:52
yep
Rqomey
Jul 24, 2014 11:52
ahh its fuckin panic stations at the moment there anyway, all their disks are full, I think the wiper is filling their backend array or something. Wont get any useful information for a while I think.
Rqomey
Jul 24, 2014 11:50
Worth checking, thanks for the hint (and jesus I hope not)
Rqomey
Jul 24, 2014 11:46
@Colyn1337 I'll have to get back to you, they are busy pulling logs at the moment
Rqomey
Jul 24, 2014 11:44
I'll check,
Rqomey
Jul 24, 2014 11:44
netapp say nothing to do with them, they use vsphere api's only
Rqomey
Jul 24, 2014 11:43
about 10 so far
Rqomey
Jul 24, 2014 11:41
added NFS storage information
Rqomey
Jul 24, 2014 11:38
tanks a millinoin :)
Rqomey
Jul 24, 2014 11:36
@Dan as requested :) serverfault.com/questions/615130/…
 

 Ask Ubuntu General Room

Normally: General discussion around Ask Ubuntu, Ubuntu & offic...
3
Rqomey
Apr 29, 2015 08:47
hi all: askubuntu.com/questions/414783/… top answer breaks install (and doesn't specify take a backup). I commented a link to the correct answer so people will hopefully read this before trying the first answer. Not sure of the correct procedure for old high search ranking questions such as this!
 

 /dev/chat

General discussion for unix.stackexchange.com. If you have a q...
2
Rqomey
Dec 4, 2014 16:42
@Braiam You have to define a fixed number of lines there, and there is a variable amount prior to the timestamp
Rqomey
Dec 4, 2014 16:34
Please :)
Rqomey
Dec 4, 2014 16:31
Just want the first timestamp above the keyword. I'm sure this has been asked but don;t know how to phrase the search
Rqomey
Dec 4, 2014 16:31
[11/30/2014 5:13:30 PM] [Debug] blah
[11/30/2014 5:13:34 PM] [Debug] blah
[11/30/2014 5:16:46 PM] [Error] blah
stack trace starts (can be a variable number of lines)
keyword
Rqomey
Dec 4, 2014 16:28
I'm sure I have seen this question already a few times in the weekly digest... but I don't know how to search for it :) I am trying to search for the first instance of a time stamp in the lines preceding a match (across multiple files). something like:
Rqomey
Dec 4, 2014 16:25
Hi all!