@MikeyB Are you sure that won't just drop what's in swap out of existence?

@KevinSoviero swap usage is normal and desirable. Do you want your idle programs sitting in memory taking up space which could be used for cache, or do you want them swapped out?

@MDMarra Good man.

@KevinSoviero Yes.

@MikeyB Swap if completely full, that's bad.

Man, this is taking forever, I guess that's a good sign though, it means things still exist somewhere.

@HopelessN00bGeniusofnetwork :(

@ewwhite what was up with that script yesterday?

@HopelessN00bGeniusofnetwork Is the chemical inflammable?

FreeBSD people... what's the equivalent to the Linux fdisk -l?

@ewwhite gpart list or gpart show (robots in disguise)

@ewwhite Probably fdisk -aersdfrris

Knowing them

@MikeyB I have no idea. Just arrived to the below email in my inbox:

Urk. I should've set this up properly from the beginning.

> This switch has been powered off for the fumigation that’s going on this weekend I’m going to un-manage this switch in Orion until Monday.

The regret now...

At least they're telling us shit that effects us now, so it's progress, even if it does make me die a little inside.

@MikeyB I'm trying to help this d00d, but FreeBSD is all weird and missing some ZFS features.

@ewwhite It shouldn't be. Such as?

Snapshots are not backups. I just trawled up some critical data that got corrupted by an incompetent windows admin during a migration by recovering a subfolder from a backup of a deleted volume. 250MB that would have cost us at least 250,000$ to recreate.

@Basil I hope you get a percentage. (Hah!)

Deleted volumes have no snapshots

@MikeyB I would have had a percentage had we taken the loss. As it is, I'm getting a big fat percentage of 0, our actual loss.

@MikeyB He got the ataboy like the rest of us

unless you count political currency.

@Basil Could you add "mirrors are also not backups" to that and ship it to my boss?

@jscott heh, technically, I only got this stuff because they discovered the corruption a month before the backups expired...

That's the kind of stuff that gives me anxiety about our horrible "backup" config.

I've since asked the netbackup admin to increase the expiry on this data just in case

@jscott Tape is cheap and getting cheaper. That said, this was backed up to a replicated data domain... but still

Very little of everything we have ever makes it off the SANs and onto actual, physical, tape.

@jscott fingers crossed for ya

@MikeyB snapdev

^^Question that comment's attached to could use some attention. And crushing.^^

@MikeyB He was told CEO: "OK, thanks...while you're here I have a question about my iPhone and the copier needs toner."

@Iain When we do have The Big One and there's nothing on tape to restore, I'll sign my resignation "Told you so".

@HopelessN00bGeniusofnetwork ??

@jscott I'll look forward to seeing that in a sad kinda way ;)

@Basil I was in a bunch of storage talks at TechEd. What are your thoughts on Microsoft pushing commodity storage for Hyper-V with scale out file servers and DAS over SMB3?

@jscott Write up a good business case and present it to whoever would be left holding the potato. Explain what your current situation protects from, and what it doesn't protect from. Make sure you're honest about it too- mirroring and snapshots are absolutely going to be able to keep you alive for a lot of different scenarios.

@ewwhite "Foreman" is destroying that guy's webservers. He clearly shouldn't be installing an electric grill on a server...

@Basil I should probably print those emails out too. They're not going to tape either. :)

@HopelessN00bGeniusofnetwork oh, thought you were talking about ZFS+FreeBSD

@MDMarra More power to them.They're worried about vsan, but that's not going to have any effect on the enterprise space. We store solaris, AIX, and mainframe data. Windows/VMWare is a whopping 8% of our data.

and shrinking

@Iain Thinking of it makes me feel sick, but were it to happen, it might be a huge relief. :)

@Basil VSAN has too many restrictions and unattractive pricing

@ewwhite Nono, that question's actually interesting, even if it's a few miles over my head.

Our backups go to a NAS then to an external drive that me or my boss take home every night...it works for us

Anyone use simple.com ?

@ewwhite Give it time. I predict it'll become the best option for a lot of medium sized IT shops.

We used to use Sugar Stink Sync but yeah

@Basil I was on a shuttle with a guy on the storage team from Microsoft and he was saying that they have a lot of hyper-v based hosting providers shifting to it, because the cost/GB is so low and performance is so good when everything in the stack speaks RDMA

@ewwhite OK, looks like on my version it doesn't but the snap devices show up.

@NathanC That's solid- you have local "oops" coverage, as well as remote "not on fire" coverage.

@MikeyB Ok, I think I understand now :P

@MikeyB in Linux, we want to turn that off, since frequent snapshots would create TOO many devices.

What I did wasn't too bad.

Now I just need to fix it a bit without killing my connection.

@Basil Aye. The only "issue" is manually having to rotate old backups off the backup drives since they fill every couple weeks

Funfunfun.

@MDMarra Hosting and big data (facebook/amazon/google etc) are driving a huge market, but it's not really like what we do.

The NAS helpfully alerts us when that happens though

@Basil @MDMarra We do stuff that runs real businesses :)

@ewwhite mind you this particular box is 9.1-RELEASE-p4 so kinda old

@MDMarra These people run self-built applications at massive scale, but are better off buying storage direct from the factory

@Basil Right. I just meant what are your thoughts in general since you're the resident storage guru and it's cool stuff that I don't get to play with much

Okay, Basil does stuff that runs real businesses.

@jscott I had emails to a former boss about the horrors we were experiencing with our backups. I had explained in detail that they were failing half the time and there was zero guarantee we'd recover from data loss. His response EVERY single time was "see what you can do to fix it...I don't think upper management will pay for a new system." -- which basically means "if it fails, TheCleaner will be blamed for incompetence and all the employees without jobs will hate him".

@ewwhite it would be perfect for produce. Get two little pizza boxes as scale-out file servers. Connect/cluster them with a shared SAS DAS. Share it out over regular 10GbE (or even teamed 1GbE) and poof, ghetto-ass HA storage for Hyper-V VMs

@MDMarra I think that local storage is absolutely capable of doing the work of centralized storage. The reason so many places ran to storage networks with central storage back in the 90s was because they needed to share and the networks couldn't keep up. Also, because it was a better built basket for their eggs. Now that the network and sharing protocols are as good or better as a SAN or NAS would use, it makes sense to start putting drives in servers again, imo

@MDMarra I wouldn't use Hyper-V

VMware is cheaper for my needs.

@mossy wanted to but no joint accounts

@ewwhite cheaper or cheap enough?

If you don't need clustering Hyper-V is free

@Basil Both our ESXi hosts are on local storage and it works well for us. Moving VMs does take longer, but the critical stuff is HA'd anyway (DC on both hosts, etc)

@MDMarra $500

@NathanC VSAN and what Microsoft are talking about would let you VMotion with local disks :)

HA from a storage perspective is a hard sell.

@Basil You can do that with ESXi too...it copies the VMs over live

@TheCleaner That's another of my concerns, besides the data loss. Every time I bring up "why aren't we using tape anymore?", my boss schedules another vendor to show us their latest "backup"-to-SAN product.... and we don't look at new tape libs.

just takes longer than telling the host "oh, you're hosting this VM now, here's where it is on the SAN"

@NathanC Storage VMotion?

or something else?

@Basil Just regular vmotion.

@jscott I prefer tape. My clients hate it

@ewwhite Damn. I have something in common with Ed's clients.

I need to brush up on VMWare

@ewwhite We had tape a while ago, but the unit we have doesn't support big enough tapes for our current usage...and tape drives be expensive :P

@Basil don't go too deep. VMware is getting funky these days

I hate tape but that's all we have! But I heard that the FDA/ISO is going to start requiring disk backup.

@cole We use two seagate passports with the rugged cases for off-site...it works well

@NathanC we.....can't do that

We have TBs of data.

And the cases do too ...i've dropped them more than once lol

@ewwhite I wouldn't even tell them that I was doing tape! In fact, let the mirroring/snapshots continue apace....

@cole Ah, fair enough.

@cole What don't you like tape for?

@MikeyB takes forever to do backups.

Our fileserver takes from Friday evening till Sunday at 5AM

@cole really? My tapes are pretty damned fast...

@cole How long would that take over the network to another disk?

@Basil dunno

@cole What are you running? LTO-6 is 160MBps uncompressed. Besides, all the cool kids backup to disk then destage to tape.

@MikeyB LTO4 or 5

Using EMC NetWorker - piece of shit.

Also, our LAN is only 1 gig

@cole Ah, that's a different problem than "tape is slow".

@MikeyB All the really cool kids backup to on and off site network storage.

I'd keep the tape library stashed in an old cabinet, running without anyone else knowing.

But we're a multi-billion dollar company - we shouldn't be just relying on tapes.

@cole You're backing up over 1Gb to tape? There's your problem. You want to direct attach tapes to that NAS

@Basil - yeah, not going to happen.

@KevinSoviero Yeah 'cause backing up at 1MBps is SO MUCH BETTER.

@cole So tapes can be made as reliable or more than any competing media. Your problem isn't the tape, it's the network and possibly the software.

We have a large datacenter in the UK as well.

I have a 600TB NAS I do weekly fulls on. Backups window is 48 hours, we usually complete in about 36

@MikeyB 1Mbps? 1Gbps on site, and 100Mbps off site.

All straight to tape

@Basil Damn - I know a lot of it has to be Networker and the network here.

@KevinSoviero I said 1MBps. We only have a 10MBps internet link :(

Our storage network is all 10 gig.

@cole that's faster than my tape connection- I think we're hooked up at 4 or 8Gb/s FC

We only be 24 port switches too -_-

@Basil yeah our FCs are 4Gb/s

for the SAP environment.

@cole You use it to write on tapes?

@Basil not really sure. I only know a sliver of our backup environment - I just know how to fix failed backups.

I know that two of the storage nodes are Solaris 10

We don't have any of the extra modules that supposed help make backups better/faster. On our MSSQL servers we have to do a backup to flat file within SQL then that gets backed up to tape.

@cole Best practice for large backup initiators is to get them using a high speed network that's separated from production for backup data. In our case, that's a FC SAN with tape drives. Traffic is coordinated by the backup media servers. We could have opted to use high speed ethernet networking to get the data onto the media servers and have them put it on tape (or disk or whatever), but this setup allows our backup server to do other work while the NAS is backing up.

It helps that we have a big tape library.

@Basil our issue is money.

@Basil like TS3500-class?

Corporate IT doesn't get any lol

@MikeyB It's an old storagetek- biggest one they have.

We have a StorageTEK

I took a stroll through it once

just kidding ;)

@Basil damn lol

@cole If what you're doing will cover you, then maybe it's ok. If you are no longer hitting your window, you can spend money to fix that, or a lot of money to fix it for a longer time

If they haven't defined a window, get them to.

Here, we're all about DR and tapes play a pretty important role in that. We have invested quite a bit of money in our backups, and rely on them heavily.

@Basil Or as our customers ask "Couldn't you just change the backups so they only back up important files?"

@Basil we seem to be moving in the right direction lately.

So I'm curious what will happen as a lot of upper management is retiring in the next 3-4 years.

@MikeyB We're working on that too!

I hate that I work Tue - Sat, which means that while everyone else has Monday off, and therefore a three day weekend, I still have my rubbish two day weekend, since I always have Mondays off...

Nobody's willing to give us money for archiving, though

@Basil I'll also point out that's all the direction they give us. What do I look like? A freaking djinn know-it-all?

Ahh! Forgot to take a picture of my ankle this morning.

Can I hand a question over to @MikeyB ?

Fix it!!

@ewwhite Well shit he's trying to mount a disk with a partition table as an ext2 volume.

@MikeyB OH... get rid of the partition #

I wasn't even looking at that.

or add the partition :)

one sec...

I recall freenas doing something odd with swap partitions at the beginning of each device

@ewwhite the disk with the partition table in this case is the zvol.

[root@freenas] /dev/zvol/vol01# file - < zvol01
/dev/stdin: x86 boot sector; partition 1: ID=0x83, starthead 4, startsector 256, 6291200 sectors, code offset 0xb8

which still shouldn't be an issue... but was he trying to mount the raw device or the partition. or will this just not work?

I'm not thinking straight today

You know, due to ransomware, a brief @MDMarra saving-the-day and some bad software downloaded from Tucows.

Oh, speaking of ransomware, we got hit with cryptolocker for the first time, earlier this week. Then got hit by it again. By the same user.

@HopelessN00bGeniusofnetwork yeah, it happened to produce company yesterday. I think they're going to pay.

$1000

@MikeyB kpartx?

I dunno what I was expecting, but it was a pretty disappointing experience. Restore from disk backups, go back to sleep.

@voretaq7 heeloooooooooo?

@Bob yeah need to do the freebsd equivalent by hand. Create a geom device that's consuming the other device except from 256 sectors in.

@ewwhite No backups, or is the effort just more than $1000 is worth?

wait, it's freebsd? damn.

@HopelessN00bGeniusofnetwork they have goodish backups

but yeah

@HopelessN00bGeniusofnetwork Do you know how they got it?

Probably email...

Yahoo email for this client

funny, I guess it's hard to protect people who use webmail

It's been two three decades. Why is it so hard to train people to not open unknown attachments?

Put a computer in front of them and they suddenly lose all common sense -_-

@Bob No, I punted that task (and dealing with the user in general) to the desktop team. Fortunately, the guy didn't have a whole lot of access to fileservers, so it wasn't worth the effort to investigate.

Just restore from backups, and reimaged his PC. Twice. Effing moron.

@Bob Based on the fallacious assumption that they have common sense to begin with. I'm about 99% sure it would be simple matter to get most of our users to go play in traffic.

@HopelessN00bGeniusofnetwork Do you not have perimeter/gateway antivirus scanning? Unless it's not easily captured by antivirus software...

When it rains, it pours. HD died, ordered replacement. Ordered a 146GB 10K SFF SAS drive; received a 72GB 15K SFF SAS drive.... Actually said "What the f--- am I supposed to do with this sh*t" out loud.

ow

Cryptolocker is scary here because a lot of people have access to the shared drive that holds a majority of the data people use on a day-to-day, but that's why we have backups that can't be accessed over SMB without being a domain admin.

@NathanC Cryptolocker hit us pretty good.. I was not amused.

ow

Ouch...

I won't say a word on fear of jinxing myself

Ugh, I hate it when software tries to compare the titles of open windows to determine if a copy of itself is already running

STOP MAKING ME CLOSE EVERYTHING THAT MENTIONS YOUR NAME!! GAH

@HopelessN00bGeniusofnetwork Yeah, luckily we have <1 TB of data total, so recovery wouldn't be too much of a loss.

<_<

@NathanC Everyone has access to all shared folders here. But that's what the daily backups and 1.5-hourly shadow copies are for.

you people are making me feel paranoid ;p

Which reminds me, really need to test a full restore from backup sometime.

@Bob Still, would you want to restore all that shit if you get hit?

@JourneymanGeek Inorite?

;p

@HopelessN00bGeniusofnetwork It's only ~250 GB... but it wouldn't be my decision.

@NathanC: I do weekly full system backups. Have most files backed up externally.

@HopelessN00bGeniusofnetwork Depending on the amount of data, I'd rather restore than pay some goons cash for our own stuff back

The weak link is my dad/bosses system which is a piece of legacy crap

@JourneymanGeek Come on, more often please :P

@HopelessN00bGeniusofnetwork I wonder how long a mass shadow copy rollback would take, actually. Never tested that.

@Bob: meh, personal systems. MOST serious stuff isn't stored on the systems themselves

@JourneymanGeek On my personal machine I have windows backup doing a daily image to an external...probably overkill, but it's set it and forget it, so yeah

either stored on the cloud, or external drives

@NathanC But if you have access to the external, it isn't gonna do much against cryptolocker

Hm, I suppose if you had it on a different user account (or elevated)...

I'm actually in a situation where I could wipe one of my systems, do a reinstall and get back to working condition in half a day or so

@Bob: write only memory!

@Bob I also use common sense and not get infected

@Bob Are you sure the shadow copies wouldn't get encrypted too?

I do NOT open attachments ever

@NathanC That helps, yes :P

@HopelessN00bGeniusofnetwork The shadow copies are on the server. They are obviously not shared.

anything suspicious gets forwarded to gmail then opened on a disposable linux install ;p

If CL got access to the domain admin accounts, I'd have bigger problems to worry about.

Most people at my work have some measure of it too...plus we have email scanning which kind of helps

Like, the DC going down.

@MikeyB Whaddya want?

Tree Fiddy

No one else has (local or otherwise) admin privileges on that server.

@voretaq7 Was going to ask about serverfault.com/a/598120/2101 but apparently that automagically happens for snapshots.

Even then, worst case no more than a day will be lost - the previous day's backup media is always offline.

Hm, WSUS says one computer needs XP SP2

I should hunt down that PC and burn it

@voretaq7 They're a bit more famous for this, I think :P

@Bob Hmm. This makes me realize I really know nothing about how shadow copies work at a low level. Making me want to find a copy of cryptolocker and test it out on shadow copies.

@MarkHenderson undoubtedly

@Bob Yes, but that doesn't have a snowman in it.

oh, I see what you were saying now. haha. very clever.

@HopelessN00bGeniusofnetwork Doesn't it delete shadow copies?

@HopelessN00bGeniusofnetwork IIRC they're block-level copy-on-write to a system/protected directory (\System Volume Information)

@MikeyB I was under the impression that shadow copies were shared in the same fashion that the original files are... after all, they are generally only accessible over SMB. But Bob's saying different, so I realized I really don't know and have just been taking it on faith/assuming all these years.

@MikeyB Maybe on the local machine. But there should be no way they can touch the shadow copies on the file server.

Cryptolocker probably erases shadow copies since it's really easy to do...

@HopelessN00bGeniusofnetwork Nice low-level explanation/analysis here: encase-forensic-blog.guidancesoftware.com/2012/06/…

Seems to match what I've seen in MS docs I can't find right now.

@NathanC Again, on the local machine. We're talking file servers and SMB shared folders here.

@Bob Ah we're talking about a file server.

@Bob Well, if you have access to the admin shares (c$) then it could get ahold of those as well.

@NathanC Only the domain admins do. And as I mentioned above, if CL got one of those accounts, we have bigger problems than just the user data.

@NathanC "if you're doing something wrong you could lose your data"

Heck, I think the admin shares are disabled by default now anyway.

@Bob Not in 7 or 8. Maybe you do it via GP.

Yup, as of Win8/Server2012: computerperformance.co.uk/win8/…

@jscott Apparently it is disabled on 8.

I dunno. Perhaps I should test and make sure.

My 2012/2012R2 beg to differ. :) Waiting for this 8 box to boot to test.

But that would mean doing something productive at 2am

@jscott Well, shit.

@Bob Useful, although I dislike sites that shove ads for software in your face

Woo trying a new BBQ place today.

@jscott Yea, our 2012r2 has it enabled :(

Still, only for domain admins, so not too bad.

@Bob For brand new scripted install of 8.1u1 Enterprise, non-domain, i see admin$, C$ and IPC$

Eww.

@MikeyB Ok, I didn't read that far.

Fuck that site.

Does anyone else find stretching to be painful? Especially my legs or back...

@BigHomie That fuckwit's latest comment just earned him a minimal understanding VTC and another downvote.

How can you not understand that you have to modify something in order to modify something?

@KevinSoviero Yeah, stretching is painful. That's why my only stretching-related activity is pointing my telescope at the yoga studio and watching hot women stretch.

@HopelessN00bGeniusofnetwork TIY you enjoy 'hot yoga'

@HopelessN00bGeniusofnetwork I mean like the stretching you do when you first wake up, or when you're tired. The muscles in my thighs and lower back will often spasm and burn, and the pain increases gradually, sometimes I'll be so bad, that I jump out of bed in a panic, unsure how to make the pain stop. Just wondering how common this is?

@KevinSoviero Sounds like tinea cruris

@BigHomie I'm at work, thanks for that... NSFW