« first day (1100 days earlier)      last day (3881 days later) » 

12:20 AM
@ewwhite DOOOOOMED!!
 
12:32 AM
 
1:01 AM
@FalconMomot You may find this interesting. I just ran across it while reading something else. people.umass.edu/gbecker/BeckerChes13.pdf
 
1:21 AM
@MichaelHampton That is exactly why I asked the question I asked yesterday. If you look closely you can determine my proposed mitigation.
 
Why am I not surprised?
 
I have another mitigation too, but it's much harder, takes days, and is less reliable
@JourneymanGeek Beautiful.
 
Hey, I have a Linux multipath question
and Linux nerds on?
@MichaelHampton You can come out of hibernation...
 
1:48 AM
:(
 
2:00 AM
@ewwhite I'm a linux nerd, and I've touched multipath, but it's been a while since I actually had to do anything with it (it keeps just working in the few places we use it)
Is this another crazy LVM on top of multipath on top of iscsi thing?
 
2:24 AM
@freiheit Yes.
[root@prodmessage1v ~]# pvs
  /dev/vg2/data: read failed after 0 of 4096 at 214744104960: Input/output error
  /dev/vg2/data: read failed after 0 of 4096 at 214744162304: Input/output error
  /dev/vg2/data: read failed after 0 of 4096 at 0: Input/output error
  /dev/vg2/data: read failed after 0 of 4096 at 4096: Input/output error
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda2  vg1  lvm2 a--   49.88g    0
  /dev/sdd1  vg2  lvm2 a--  200.00g    0
 
@ewwhite multipath -ll? (or maybe mpath -ll)
 
[root@prodmessage1v ~]# multipath -ll
Sep 14 22:28:34 | DM multipath kernel driver not loaded
Sep 14 22:28:34 | /etc/multipath.conf does not exist, blacklisting all devices.
Sep 14 22:28:34 | A sample multipath.conf file is located at
Sep 14 22:28:34 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf
Sep 14 22:28:34 | You can run /sbin/mpathconf to create or modify /etc/multipath.conf
Sep 14 22:28:34 | DM multipath kernel driver not loaded
gulp...
 
@ewwhite fdisk -l /dev/sdd?
 
well, it used to be sdb and sdc...
now it's sdd and sde
and I can't tell what's handling multipath
 
you sure this is a multipath setup? /dev/sdd doesn't look particularly multipath...
 
2:30 AM
RHEL6
 
uhm
 
I can't tell what they did.
 
I'd do fdisk -l on /dev/sd[abcdef] and see if you have what looks like duplicates.
 
sd 7:0:0:0: [sdd] Attached SCSI disk
sd 8:0:0:0: [sde] Attached SCSI disk
__ratelimit: 106 callbacks suppressed
Buffer I/O error on device dm-2, logical block 0
Buffer I/O error on device dm-2, logical block 1
Buffer I/O error on device dm-2, logical block 2
Buffer I/O error on device dm-2, logical block 3
Disk /dev/sdd: 214.7 GB, 214748364800 bytes
255 heads, 63 sectors/track, 26108 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xc585e0a5

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1       26108   209712478+  8e  Linux LVM

Disk /dev/sde: 429.5 GB, 429496729600 bytes
255 heads, 63 sectors/track, 52216 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
they were trying to expand a SAN LUN from 200GB to 400GB
and only one device took the rescan
 
In my experience, multipath and LVM really don't play well together, because LVM scans for devices at boot time, and with multipath the same device can show up 3 or more times.
 
2:34 AM
so yes, this is a VMware VM with what appears to be a direct connection to the SAN...
with LVM..
and possibly multipath
 
Is it possible that they configured the system to have multiple paths to the LUN, and just happened to get lucky that LVM worked with it?
 
and ext3... on top
misaligned partitions...
and the iSCSI LUN is shared between two servers.
 
I mean, I think it's really really likely with LVM that if you lose a path, you could reboot and LVM would find the other path...
ext3 instead of ext4 on RHEL6? huh.
 
I'm ashamed that this trash is running in production
the client is trying to do their own HA.
 
Does lvm.conf include any filters?
Is there anything in /dev/mapper that looks like multipath?
 
2:36 AM
no filters.
 
having to reboot
ugh
It's a disincentive to help any of these situations...
because they're so fragile that I'll end up drowning in paperwork
@freiheit server is back up
so I think I need to reassemble the lvm
or reactivate.
sd 4:0:0:0: [sdb] 838860800 512-byte logical blocks: (429 GB/400 GiB)
sd 3:0:0:0: [sdc] 419430400 512-byte logical blocks: (214 GB/200 GiB)
 
Maybe pvscan -v to get more idea what's going on with the LVM?
 
@ewwhite Working late on a Saturday night. =(
 
@ewwhite You rebooted the server and those are still different? So they're different LUNs, not multiple paths to the same LUN?
 
2:45 AM
[root@orion-ieh-prodmessage1v ~]# pvscan -v
    Wiping cache of LVM-capable devices
    Wiping internal VG cache
    Walking through all physical volumes
  PV /dev/sdc1   VG vg2   lvm2 [200.00 GiB / 0    free]
  PV /dev/sda2   VG vg1   lvm2 [49.88 GiB / 0    free]
  Total: 2 [249.87 GiB] / in use: 2 [249.87 GiB] / in no VG: 0 [0   ]
well, one was resized... and that's what was rescanned to the 400GB size.
 
@ewwhite But that's not the one that's actually in use?
 
/dev/sdb is supposed to be there
 
What does fdisk -l /dev/sdb say?
 
Disk /dev/sdb: 429.5 GB, 429496729600 bytes
255 heads, 63 sectors/track, 52216 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00085d99

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       26108   209712478+  8e  Linux LVM
[root@prodmessage1v ~]# lvscan
  inactive          '/dev/vg2/data' [200.00 GiB] inherit
  ACTIVE            '/dev/vg1/swap' [4.00 GiB] inherit
  ACTIVE            '/dev/vg1/root' [45.88 GiB] inherit
I guess I can make it active again
I think the wrong SAN LUN was expanded
I quit.
2
 
0
Q: Windows XP And the Locked in Busniess Owner

Lost Business OwnerWhen Microsoft Windows XP ends its Offical Update Support on April 8, 2014, is there any way of updating re-installed clients, offline? Like keeping a copy of Service Pack 3 for Windows XP and all of the additional updates for future use? I could keep a copy of the hard drive in a GHOST IMAGE, b...

Why, oh god why?
I shall be employed forever.
writing documents that get IT people who think like that guy fired.
 
2:58 AM
okay, so no multipath...
just two LUNs that were 200GB...
# iscsiadm -m session -o show
tcp: [1] 10.243.126.10:3260,0 iqn.2001-07.com.isilon:lw:prodmessage-shared
tcp: [2] 10.243.126.10:3260,0 iqn.2001-07.com.isilon:lw:prodmessage1v
and well, I think someone expanded the wrong one.
although...
# pvscan
  WARNING: Duplicate VG name vg2: SUIThS-A3yl-aVc8-0dCS-1Ozm-rXET-557MCd (created here) takes precedence over hiFCqQ-LZc3-wBy1-jEAn-Ke91-gF4R-OcDJaX
 
30 mins ago, by ewwhite
and the iSCSI LUN is shared between two servers.
 
@freiheit Yeah... they manually mount the ext3 volume on one server or the other
 
@ewwhite Have they actually tested that recently?
I mean, and found the same data?
It seems quite possible that the other server is the one that is seeing that 400G LUN in its LVM pool...
 
who knows... I mean, it's soooo wrong that I can't even speak clearly when talking about it
 
Specifically, maybe they created two 200G LUNs assigned to both servers (thinking that's how to make multipath work?), did a pvcreate on LUN from one server, pvcreate on the other LUN from the other server...
 
3:08 AM
I look at both servers and only see the 200GB lun from iSCSI
wrong volume expanded
 
Basically, i'm confused how they even managed to make two volume groups with the same name... Maybe it involved some reinstalling of something along the way?
 
who knows... it's just a mess.
 
In summary: holy shit that setup is sooo fucking wrong that I'm having trouble comprehending how you could even accidentally get into that specific mess.
 
the LUN expansion on the san takes a couple of hours, so all of this is just for doubling drive space
but wouldn't you say LVM is a disadvantage here?
 
@ewwhite I do use LVM, but I don't use it on SAN LUNs.
One of the big advantages of LVM is that you can grow individual filesystems.
But you can already do that with SAN LUNs already.
Throw LUNs underneath LVM, and suddenly you have like 2 or 3 levels of scanning that need to happen to make the expanded LUN storage actually available to filesystems.
The kind of thing where you say "fuck it, maybe if we reboot this will work", and then somehow you end up needing to reboot twice...
 
3:19 AM
WTF. "vg2" as a volume group name? Somebody didn't think that would be a problem later on?
2
 
So I also have two types of SANs in this setup
when using Isilon, there's only one iSCSI path
with VNX, there's multipath, dm, LVM and the actual SAN LUNs to care about.
and this client has both
so that's why I was searching for multipath... here, there was just a random vg2 laying around
@MichaelHampton and @freiheit You know these things... a SAN has this capability, so LVM doesn't add much.
 
3:43 AM
VGs really need unique names, at least within the organization. I once had to deal with a situation where I had two VGs with the same name. That took me... probably a couple of hours to sort out. One of them has to be renamed.
 
@MichaelHampton you are very right.
but you all know how anti-LVM I am.
but I know... LVM isn't the problem here.
 
@ewwhite You have hardware that does LVM's job. So I'm not terribly surprised at your stance.
 
What happened tonight was that a NOC technician responded to a client request to go from 200GB to 400GB on an iSCSI LUN. The SAN took 3 hours to resize... The NOC person ended his shift. The next person didn't know how to handle the LVM and OS-side expansion...
A senior engineer was called... and he didn't know either... so I got called...
and I looked, saw an expanded LUN... made fdisk changes... couldn't see data. Realized it was the wrong LUN expanded.
My wife is cursing my team and clients. Ruined the evening.
 
3:59 AM
So, your coworkers are idiots.
 
Just untrained.
but I sound like a n00b trying to reverse engineer this mess.
 
You're just putting out fires. When are you going to get to do something interesting?
 
4:21 AM
@MichaelHampton I have a lot of big projects lined up... but the team is pretty set in their ways.
lotta fires
But I'm spec'ing a new large vSphere cluster..
small pleasures.
 
4:39 AM
I've run out of enthusiasm... I was writing this PHP webapp at a fast pace, and finished 70% of it in a few days... However, now I'm just sick of it, and out of energy. This is how I am at all my jobs, and now even with a personal project... Life sucks.
 
 
2 hours later…
6:47 AM
0
Q: Error while installing Windows Server 2012

cheese5505I am currently trying to install Windows Server 2012 on my server. Whenever I boot to the USB stick which has the ISO it gives me the "0xc000000e" and "The Boot Configuration Data" for your PC is missing or contains errors." I have used the Windows ISO to USB program twice to get thi...

This is not going to go well. I can tell.
 
7:03 AM
... oh dear cthulu
"This is a old PC that I got donated from a Kindergarten, I know it has 2GB of RAM, and a Pentium. I'll try get the exact model. "
and the next thing I would wonder is, why would someone accept a hardware donation from kindergarden, and yet have server licences.
 
7:18 AM
Hi all, I'm shocked in here
i've just figured out: when connecting to remote mysql server from another host via mysql/mysqldump etc
my.cnf is check on both sides O_O
*checked
despite wrong password O_O
wtf
looks like it's passing variables through the connection
O_O
 
7:43 AM
You can't explain that!
 
 
1 hour later…
9:10 AM
@MichaelHampton: the fact that its working properly is obviously an error!
 
 
2 hours later…
11:01 AM
guys what should I take, Ubuntu or Debian
I can't decide :(
Debian it is
 
Debian of course
 
now..
testing or stable? ;p
 
@LucasKauffman RHEL
 
@Chopper3 personal usage, I can't afford RHEL :(
 
@LucasKauffman Centos then
 
11:11 AM
@Chopper3 What are the advantages compared to Debian?
 
@LucasKauffman you can RHEL for 99 dollars, developers license
 
@LucasKauffman: ubuntu isn't half bad for a server, IMO
 
@LucasKauffman I like SL or Mint for desktop but I don't like dealing with different flavours - one distro to rule them all!
 
(but I run its KDE varient on a desktop sometimes so, I'm used to its quirks)
 
RHEL has strong support for SELinux, for example (which Debian lacks)
 
11:12 AM
It's just to run my Splunk on and some tunneling options
 
@LucasKauffman the only real advantage is that we use OEL (Oracle's RHEL) at work exclusively - so when I want to do something at home that's neither OSX or Windows I pick an RHEL variant so it's the same - that's the only thing I care about
 
(actually, I suspect I stick to ubuntu for the same reason most folks prefer centos/rhel - Its what they know)
 
@LucasKauffman You know we're one of the biggest Splunk users right?
 
@Chopper3 yea that's why I asked :p
@Chopper3 Share me your license!
 
11:13 AM
@LucasKauffman We run it on OEL but you can run it on anything
 
I thought centos was PRETTY darned close to RHEL?
 
95% my machines at $JOB are RHEL, some CentOS, I use a Fedora at $JOB and Debian at home.
 
@JourneymanGeek Yeah - RHEL/OEL/Centos/SL - all the same
 
@Chopper3 4 GB memory enough to run it comfortably?
 
@LucasKauffman which job? forwarder? indexer? search head? both?
 
11:15 AM
@Chopper3 both actually
I just have 5 machines running and I want to centralize the logging and the OSSEC parsing
 
@LucasKauffman for low volume stuff then yeah - we currently use 4 x DL980's with dual FusionIO 10.2TB drives in them but we're moving to 96 x BL460's with a 10 x 900GB R10 and 2 x 800GB SSD's in SmartCache in front setup
 
@Chopper3 O.O
 
@LucasKauffman what are you logging? ESXi logs a LOT - we pretty much stoppped splunking them, it was pointless
@LucasKauffman bear in mind we employ about a dozen people JUST to look after our Splunk stuff
as I said, we're big users of it - not as big as Apple or the 'security services' :)
 
@Chopper3 PRISM is actually Splunk
:P
wouldn't even surprise me if it was
 
@LucasKauffman I wouldn't be surprised
 
11:18 AM
lol
@LucasKauffman: just LOTS of instances of it?
 
@Chopper3 also the free license only allows one user account right?
 
@LucasKauffman I've no idea sorry, never used it
 
@Chopper3 you get to play with all the fun toys :(
 
and splunk 6 :)
 
@Chopper3 Splunk 6 ?
damnit
you rich BT people get all the nice previews :(
 
11:25 AM
yep, not long now, 3 weeks maybe? not as big a change as 4 to 5
 
@Chopper3 is it easy to upgrade?
like pushing a button
or painful
 
11:58 AM
http://serverfault.com/a/539039/7783

My comment at the top of the answer -- too snarky?
 
@RobM not at all. the guy is an idiot, and was asking for step by step for a trivial task.
and also he doesn't appear to understand IP subnetting
 
12:25 PM
Glad to know its not just me @FalconMomot
 
 
3 hours later…
3:13 PM
 
Z̪͎͓̺̩͖͡A̷L͇̦̰G̨̙̤O̤̳̻̮̺͓̝ ̣̣̯͚͇̖l̀i̥̬̱̣̣̦͍̕k̙̺̰͓e̴̦̫s̼̞͎͚̤͍̳ ̬y̝͖̙͉̺̺o̬̥̜̗͝u̱͔̤͙̼̘̝͝r ̧̭͓̥̯͇̪̘d̹e̷̞͖͖͎m͚̣̯̼̖i̘col͈̻͔͓͎͍ͅo̡͇̝̫͚̝̹͎ṉ̢͓̩͖̹̜,͕͈̟͡ ̞͈̝̦̳͎͉m̦͘o̞͘r̻̪̻t͙a̺̹̬̕l̴͔̪̭̹̥ͅ.
 
4:13 PM
Thank you for your answer. Of course I can reach the gateway on the same subnet. The setup is, that servers on the same subnet can only communicate with each other via the subnets gateway. — Bannane 2 hours ago
Why does nobody understand how IP subnetting works?!
or how IP works at all?!
 
IP? [what is IP?](http://www.acronymfinder.com/IP.html)
MARKDOWN, Y U NO RENDER?!
 
My personal favorite is Incontinentia Pigmenti
Actually, no. Not funny.
 
^ Rainbow does top gear. One for the brits
 
4:37 PM
Ahhh, web hosting. So incestuous and so crappy.
Endurance International Group, Inc. (EIG), formerly BizLand, is one of the world's largest website hosting companies. The company was founded in 1996 and is headquartered in Burlington, Massachusetts, It achieved its size by acquiring a large number of smaller hosts, which it continues to operate under the original names. In 2011, Endurance was bought from Accel-KKR by Warburg Pincus and GS Capital Partners, for around $975 million,. List of hosting brands * 2slick.com * AccountSupport * A Small Orange * ApolloHosting * Berry Information Systems L.L.C. * BizLand * BlueDomino * B...
 
4:52 PM
@MichaelHampton Is that the same Pincus family as the Zynga one?
 
5:25 PM
Um, I dunno?
 
5:37 PM
And does anybody have the slightest idea WHO UPVOTED THIS?
1
Q: Linux DNS: expand name with a choosen domain

ConlI need to expand a random name with a choosen domain. Example: nslookup abc 127.0.0.1 -> should look for abc@domainchoosen.dom. Suggestions?

 
@MichaelHampton we should close it unclear
it isn't even coherent
what is with all the people asking totally incoherent questions?
to the point where I am thinking... how exactly does this guy even use the internet
 
Unfortunately, it's quite possible to "use the internet" without understanding even the basics of it.
 
5:57 PM
@Cole Don't let your manager see this, but the question of whether indexing pre-2008 could be introduced alone, inspired me to launch a blog and put together a disorganized mess of powershell stuff: blog.iisreset.me/2013/09/…
 
6:47 PM
1
A: Two routes for localnet, how to delete the one without GW?

Falcon MomotIf you have an IP address on a subnet, which is what I assume you mean by local network, you will also have a route for it, intrinsically, because you have an interface on it. You can't direct all traffic to that network through a gateway because the gateway would be on that subnet too, and you ...

am I missing something here?
 
@FalconMomot Yeah, that ISP's network is... janky.
 
@MichaelHampton this is a common thing to do, if you are operating a CMTS or something
certainly not for ethernet networks
and most definitely not if you are under artificial pressure to "conserve" addresses because you are using RFC1918 space
 
 
2 hours later…
8:36 PM
Got my new Server 2012 R2 domain with two DCs set up perfectly... I think System Center is going to be the first thing I deploy in it.
 
 
1 hour later…
9:38 PM
When I refer to an RFC, you didn't do your homework. When I quote RFCs at you...
 
There's nothing like a weekend away from the internet.
 
@MichaelHampton Can I put a filesystem on a disk... say /dev/sda...
and then at a later time, add it to an mdadm software RAID 1 mirror?
preserving the data and all.
 
@ewwhite No, because md raid, like its hardware counterparts, writes metadata to the disks or disk partitions that it uses.
 
I see mr "I live in San Francisco Bay Area and I'm a shit sysadmin" is still trolling the ruby ferret thread.
 
@MichaelHampton When I use the SIOS software, it creates an md0 netraid over my existing data disks...
 
9:45 PM
@ewwhite Maybe it's doing something like resizing the FS and writing the necessary metadata. I'm not really that familiar with it.
 
it's all magical.
But then I can disconnect the network mdadm, and use the regular filesystems/partitions.
 
We totes need to canonise the linux ate my ram questions.
not this one, but something.
0
A: What is using up my RAM in VPS?

Tom O'ConnorNothing. It's Linux, and that's how it works. It'll use "free" memory for disk caching, because what's the point in having an under-utilised resource? When a process wants more memory, it'll ask for it from the kernel, and (generally), get it.

 
anywho, I'm in a bind. I overpromised a solution, and ZFS is letting me down
 
@TomO'Connor I'm leaning toward unclear what you're asking. He seems to be asking why MySQL is using memory, but I'm not really sure since he didn't provide any data about it.
 
@MichaelHampton I don't think MySQL is using memory. I think he can't read top.
doesn't appear there's anything wrong with his server.
 
9:51 PM
@TomO'Connor Yeah, nothing's wrong with his server that I can see.
 
10:12 PM
@TomO'Connor I can't even read top. Too many numbers. It's confusing.
 
@ewwhite If the load average is greater than the total number of cores, something's wrong.
If all the run-statuses are D or Z, something's wrong.
Other than that, it's fine.
 
perfect example...
 
Anyone know/know where I can find the light/button setup and beep codes on a Dell Powerdge 1800?
 
Client just wrote: Please Extend the Memory on 10.64.88.121 to 16 GB .
top - 18:17:55 up 24 days,  5:06,  3 users,  load average: 6.63, 12.76, 11.42
Tasks: 426 total,   2 running, 424 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.4%us,  1.3%sy,  0.0%ni,  8.8%id, 85.5%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8186056k total,  7954716k used,   231340k free,    51076k buffers
Swap:        0k total,        0k used,        0k free,  5132492k cached
Does client really need to go from 8GB RAM to 16GB @TomO'Connor ?
 
@ewwhite They're only using 3 GB!
 
10:21 PM
@ZachSmith See the answer I just posted to your question.
 
@MichaelHampton Exactly...
 
@ewwhite "Please read linuxatemyram.com and after that if you still want the extra RAM we'll be happy to add it."
 
@ewwhite There's a shitload free, a shitload in buffers and a massive shitload cached.
 
But it's what's soul-sucking about the job is that this is an urgent request... and I can't really say, "you're being stupid."
 
top - 23:23:02 up 78 days,  4:32,  1 user,  load average: 0.10, 0.14, 0.19
Tasks: 183 total,   1 running, 182 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us,  0.2%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4129972k total,  3371672k used,   758300k free,   449624k buffers
Swap: 11719376k total,   114692k used, 11604684k free,  2296692k cached
 
10:23 PM
@ewwhite How the fuck can that be urgent?
 
This is from my aging colo'd box. Proof you can be happy on 4GB.
 
I can make this change now. However, are you sure you need the additional RAM? 8GB is used, but 5GB of it is in cache. Is the application experiencing a problem at the moment?
 
@ewwhite how many cores?
@ewwhite 85.5%wa. Dafuq. There's something thrashing that poor server's IO.
 
Bad storage.
 
@ewwhite :(
 
10:26 PM
Hell, I'd be yelling about the storage problem long before that point.
 
@MichaelHampton so while I was typing the email, someone else added the RAM.
and it didn't take.
 
top - 18:29:49 up 24 days,  5:18,  2 users,  load average: 113.52, 63.99, 35.90
Tasks: 542 total,   2 running, 540 sleeping,   0 stopped,   0 zombie
Cpu0  :  5.8%us,  0.8%sy,  0.0%ni,  0.0%id, 93.4%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  0.8%us,  0.8%sy,  0.0%ni,  0.0%id, 98.3%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.8%us,  1.6%sy,  0.0%ni,  0.0%id, 97.6%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  1.6%us,  2.4%sy,  0.0%ni,  0.0%id, 95.9%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   8186056k total,  7809032k used,   377024k free,    29704k buffers
 
@ewwhite Don't you have a staff that does stupid shit like this without bothering you about it?
 
@MichaelHampton Yes, but I stepped in because this is the same client with all the overprovisioning issues
and the RAM increase didn't make sense.
 
10:37 PM
Yeah, they don't really need RAM, they need disk...
 
@MichaelHampton well, it's on a SAN.
but the hot-memory add failed because of kernel version
 
@ewwhite A SAN with performance like that ought to be taken out back and shot.
 
The relevant pastebin - pastebin.com/ExS1xgcB
@MichaelHampton You're right. It's all firefighting.
I'm afraid to touch anything.
Oh, more info...
[root@dev5v ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg1-root   26G  2.4G   22G  10% /
shmfs                 4.0G  2.8G  1.2G  72% /dev/shm
/dev/sda1             185M   50M  126M  29% /boot
/dev/shm contains Oracle goodness...
and that's why they needed moar RAM.
 
Oh, Oracle. Then they really do need more RAM.
Since it's Oracle, they probably also need more lube.
2
 
10:53 PM
But getting /dev/shm to increase probably requires a reboot
 
You have to reboot anyway.
 
normally, hot-ass works great!
hot-add, that is.
 
Not when they're running an old unpatched kernel!
 
so the other thing that happens is that we often lose LVM metadata during SAN operations like last night's.
 
11:08 PM
How the hell do you do that?
 
not sure... but the LVM, multipath, bad partition alignment...
 
Is there any DATA to support XEON being more reliable, or is XEON just a brand name, and i7 another brand name, for members of an identical CPU family from Intel, offered at different price points? For example, is the multi-core feature just a matter of unlocking a chipset feature, or is there actual silicon gates involved in enabling multi-core? Are the multi-core and ECC DRAM features on the die of the i7 and just not connected to the relevant pins in the LGA 2011 socket? — Warren P 2 hours ago
oh, Super User. You retarded subset of the technology universe.
 
151
A: LVM dangers and caveats

RichVelSummary Risks of using LVM: Vulnerable to write caching in kernel and hard drive/SSD on kernels before 2.6.33, or in VM guests Harder to recover data due to more complex on-disk structures Harder to resize filesystems correctly Snapshots are hard to use, slow and buggy Requires some skill to c...

 
@ewwhite Oh, you can fuck up just about anything if you're determined enough, or stupid enough. This is not limited to LVM.
@MDMarra He's a developer, what were you expecting?
 
@MichaelHampton Client calls and wants 200GB added to a VM... by having that on a SAN with iSCSI multipath/multiple initiators, with LVM on top...
a huge mess
the on-duty NOC guy left... and the next shift didn't know how to do it... he called the first shift guy, and was told to "google it"
 
11:24 PM
@ewwhite Really? Google it? Halfway through a process and he didn't even bother to document anything about what he was doing? WTF is wrong with that company?
 
Dec 17 '12 at 5:07, by Mark Henderson
Geezus I'm about to get yelled at soon; my boss asked me "How do you screenshot on a Mac?" I replied "I dunno - try Control+Shift+Google" and I heard him repeat that to a customer verbatim
2
 
So, @MichaelHampton ... since there will be a postmortem... what IS the right process for LUN expansion and LVM...
 
@ewwhite Ask @Basil?
 
use fdisk to add a new partition and add that to the PV group, or use fdisk to remove the partition and recreate using new boundaries?
 
Let the SAN handle it?
 
11:26 PM
well, the san does its part, but when LVM is in the mix, we have to recognize the new space
 
So... SAN... multipath... iSCSI... LVM on top of that? There's no point!
 
haha
from within a virtual machine...
 
I just want another 30 points before the day ends in half an hour.
0
Q: three-way handshake & sequence number

HexLooking at the image below, taken from the book of Ilya Grigorik - High Performance Browser Networking, I can understand that the SYN packet picks a random sequence number which is incremented in SYN ACK packet (N+1). Finally, in the ACK packet both sequence (N+1) and acknowledge (N+1) numbers a...

But all I'm seeing are crap and uninteresting questions.
 
@MichaelHampton and I want a steak dinner, the Bears to win and someone to buy my house...
 
11:44 PM
I'm still waiting for the Cubs to win the World Series.
 
@MichaelHampton +1 to that
I had a good run, picking up a little stuff
but, I spent the last few hours forking unrealircd so it's possible to trim the fat and make it something I'd want to use again
 
Look what I've been reduced to!
0
A: Spam Filters - Why does the same message fail via two different smtp servers

Michael HamptonIt appears that Google trusts mail arriving from Hotmail more than it trusts mail arriving from some random server on the Internet. This is completely out of your control. One thing that is in your control is: X-Mailer: PHPMailer 5.2.7 (https://github.com/PHPMailer/PHPMailer/) Considering how...

 
removed about 19500 lines of code :)
including a malloc() reimplementation for win32 from 2002 or so with contribs dating to 1995, unchanged since 2002.
 
@FalconMomot Why is RDRAND suddenly a huge topic of discussion? Did I miss a news story?
 
@MichaelHampton Remember that paper you linked me I'd already read? that's why.
oh and
it advertises that it generates the numbers according to a NIST standard
 
11:47 PM
Yeah, NIST has a huge PR problem now.
 
but it's come to light that it is totally unauditable (which is not true; I am currently working on an extremely time-consuming way to audit it similar in working to memtest86+)
 
Poor me, I'm just trying to figure out what the fuck is secure anymore.
 
@MichaelHampton fuck all
 
@FalconMomot Great, going back to one-time pads and dead drops is not my idea of a good time.
 
@MichaelHampton Mine neither, but that's what you do
I am not kidding about this: I have some friends who, when we need to talk about certain things, take all our electronics, remove batteries, and put them in another room
 
11:52 PM
I have an annoying habit of accidentally leaving all my electronics at home.
 
hahaha
not such a bad habit probably, in hindsight :P
 
@FalconMomot you need new friends
 
@Andrew disagree! some of the things I have come to know this way are very, very informative
and I must say I agree with the level of secrecy around them
 
Since I have a phone from which the battery can't be removed...
 
if I was a government they would be top secret, no foreign nationals, limited distribution
 
11:54 PM
@FalconMomot: you guys need to get an old fridge ;p
 
@MichaelHampton You need to embrace the use of burners...
 
@ewwhite I already use these extensively.
 
@ewwhite: is it even possible to get those in the US?
 
@ewwhite bears suck. Patriots rule
 
@MichaelHampton why, constant accidental loss of your phone? :P
 
11:55 PM
@JourneymanGeek Oh yes, they're in practically every shop.
 
I mean, when I was in the UK, I could walk into a store, plonk down 30 quid, and buy a phone
 
@MDMarra Chicagoans KNOW that our teams suck...
 
@MichaelHampton: oh, not cheap phones, anonymous prepaid cards - you end up needing to register for them here, cause TERRORISM
 
@FalconMomot I'm mean to my electronics. I keep dropping them in large bodies of water, or underneath bulldozers, or...
 
Except for the teams that win Stanley Cups and World Series... but we don't care about them.
 
11:56 PM
@ewwhite I have Brandon Marshall on my fantasy team. So I'm pulling for the bears offense.
 
@JourneymanGeek Yes, you can get anonymous phone cards here.
@JourneymanGeek HOWEVER... they generally have to be activated at the point of sale, where there's almost always a surveillance camera.
 
@MDMarra Superfan.
 
You know who's hilariously great. The eagles offense. you know who's hilariously bad? The eagles defense.
 

« first day (1100 days earlier)      last day (3881 days later) »