« first day (222 days earlier)      last day (4759 days later) » 

10:00 PM
Hardware or software (RM is one of the two authorised hardware vendors)
 
Although I do have one curious thing on a server. I keep getting an event from source "bowser".
I still don't know how that typo got in there.
 
The dummies just do clerical things
 
What's really funny, is when you go into a new environment and fix some basic AD design problems, name resolution, roles, replication, and all of the sudden shit just starts working all over the place.
 
RM do software too? When I was at school (which wasn't so long ago) they just did the (crappy) whitebox PCs
 
Yup
 
10:01 PM
@SpacemanSpiff funny how that works out sometimes
 
Hate to break it to you kiddo but if your schools are similar to ours, your admins are underpaid, overworked, understaffed and don't have a test environment so every little "whoops" goes live.
 
What do you mean your login times just went down by 2 minutes?
:D
 
What @BartSilverstrim said
 
so that's a good thing right? You'd normally want nontechinical peple able to do non technical tasks?
 
"Fuck it! We'll do it live!"
 
10:01 PM
Pretty much.
 
@BartSilverstrim 6 servers, 1000 client computers, 2000 students, 3 dummies
@JimB Like adding users?
 
Extend schema...cross fingers...pentagram on the floor for the good luck spell.
 
@fahadsadah not too different then what I'm working with
 
More or less...
 
They really botched up the scripting of adding of users this year
 
10:02 PM
2700 machines 5200 users, 6 bodies who know very little.
 
And school networks are nothing like how "it's supposed to work."
 
yeah adding users should bed a relativly simple task that shouldn't require an admin- SCRIPTING usersadds should be an admin function
 
I don't complain its giving me a lot of work right now :D
 
Low pay, compared to private sector, and severely undermanned.
 
hooah
 
10:03 PM
Scripting is another joy in AD. We've had some funny glitches there too with some attributes not going in correctly. Fun times.
 
Assuming I get a CS degree, my dream job would be developer
Backup job: sysadmin
 
You should have seen the eyebrow raises I got when I asked why all their subnets were /24s
 
Backup backup: high school math teacher
 
Um...okay...
(hey guys...look at the naive one...all full of hope!)
(Careful or you'll crush him!)
 
heh Bart beat me to it.
 
10:05 PM
Ever get a student trying to be smart with your network, @BartSilverstrim?
 
We sort of crushed Jacob, it's only fair we do the same to fahadsadah :-)
 
Or, worse, succeeding in being smart?
 
If you want to be a developer, start coding. CS is more algorithms and such...why you do it, as opposed to just doing it. Which is a good thing, don't get me wrong, but nothing beats doing it on your own. No sex jokes on that, guys.
 
@SpacemanSpiff eh? How did they have it setup?
 
We have plenty of kids who think they're accomplishing something by being clever.
Usually they just resort to vandalism.
 
10:05 PM
Eww
 
Most are just morons.
 
@Holocryptic - Every switch stack in every closet was a new vlan with its own /24, in every school
 
and I started coding literally years ago
Sysadminning is a thing I recently got into
 
students? In the rest of the world it's wannabe sysadmins and developers trying to think they can prove they are better than you
 
I mean, it works... but why.
 
10:06 PM
Sysadmins and developers are...not usually comfortable together. Let's just say we're seated at different tables at the wedding, but we're required to be there.
 
@SpacemanSpiff was it broken up by floor or something at least?
 
@BartSilverstrim Anyone actually, for want of a better term, hacked your network?
 
...like what, altered files? infect us with malware?
 
Yeah
Gotten access to data they shouldn't
 
@Holocryptic oh it was nicely organized, but so many dhcp scopes, hilarious.
 
10:07 PM
They bypass my webfilter on a routine basis
 
Do some actual damage
(possible temporary)
 
meh, not really at a massive scale. Usually a teacher left their login on a machine unlocked, so they could get into grades.
Just for that teacher, though.
 
@SpacemanSpiff ours is a little like that, but it's based on physical location
 
@Holocryptic it was a lot for them to manage, more than anything, but they clearly never knew how to even use a different mask
 
10:09 PM
Usually it's attempting to get into facebook or youtube or something. Had a teacher leave his system logged in and he started triggering alerts on the filter for hitting gay porn sites. Turned out a kid was going into his office and trying to browse it.
Stuff like that.
 
@SpacemanSpiff admin buildings, floors of a dorm, etc
 
I can see any files anywhere on the school's server from home, but I generally avoid violating teacher's stuff, and school's records
That sort of thing can stray into data/child protection
 
Most kids are too self-centered to think of "sticking it to the man". They're too busy trying to access websites they're not supposed to get into...had a few boot from CD or USB stick to a Linux machine to show their l337 skillz
 
I did once test my school's sysadmins by creating a switching loop
 
@fahadsadah no spanning tree?
 
10:10 PM
They think home networks are like any other network, so they can't get the concept of a border proxy through their heads.
 
Wall port -> switch -> switch -> another wall port
No spanning tree
 
I hope that the problems with hack attempts get better as multipoint gets rolled out
 
@fahadsadah I hate you now. I had some clown do that and it took down half a floor.
 
haha, they deserve that.
 
Though they did later fix the issue
 
10:10 PM
That's not really a hack...that's just being a dick.
 
by ducttaping over unused wallports
 
facepalm
 
@Holocryptic that's not bad, at the last job we had someone do that and took out the entire building - including our primary datacenter
 
@Holocryptic I got more than I bargained for
I took out the district's school networks
all of em
 
@fahadsada you're a bastard :P (BTW I'd totally be gunning to have you sacked if you did that where I work....)
 
10:11 PM
That's like showing your mad electrical engineering knowledge by sticking a double ended plug from one outlet into another.
 
It's one large network, it seems
 
My first real job... yeah... they used 1.0.0.0/8 as the mask, left every switch default configuration, and daisy chained them together, bridging sites with fiber
 
The good thing was we have things segmented, and the switch detected a loop and shut down
 
@BartSilverstrim or convincing a retarded physics teacher that actually, she was standing on too many plastic trays?
 
900 hosts on one broadcast domain, including all servers.
 
10:12 PM
And the Van De Graff would work better if she stood directly on the floor?
 
@SpacemanSpiff EFFING SWEET!!!
 
2.7 million packet collisions per day
 
ouch
 
and the funny thing? those first 3 years it was like that, not one, freaking, outage
 
@SpacemanSpiff You're still using hubs?
 
10:12 PM
See this is what I meant about the young ones having all the answers, none of the experience...school networks aren't based on things resembling reality.
Neat network!
 
a new infrastructure manager came in, threw in Cisco 3750s to hub and spoke the design, and it failed 3 times in 2 days
screeching halt
 
Interesting...
 
@BartSilverstrim Their fault for not STPing
 
@fahadsadah nah they were all mostly HP procurve switches, but a loop anywhere would destroy the entire network
 
and later, not using the switch information to locate the fault
but spending hours walking around room to room
 
10:14 PM
yours truly produced the first loop
 
Several times (they didn't think someone would put a switch inbetween)
 
In many schools they just do what works. Then you don't touch it for fear of everything going kaplooey, and usually you're off fighting the next fire because you're supporting a few hundred people with a support team of two.
Or three, in our case.
 
@SpacemanSpiff anything really bad happen?
 
Yeah, nobody could sell anything for an hour
 
@BartSilverstrim I understand that if you actually do the job, it's a really hard job
 
10:15 PM
@BartSilverstrim bingo. Though we're making headway here in my neck of the woods.
 
but these guys are just asking for me to screw with them
 
@SpacemanSpiff and they were all broken up about that, weren't they?
 
Same with the physics teacher who hasn't actually succeeded in teaching me any physics
I've electrocuted her (5MV) twice now :D
 
@Holocryptic yes, some people were upset :)
 
Because of A) manpower B) budget C) user demands we end up chasing tails a lot, and we have plenty of wanna-be admin students who smugly offer comments not having any freakin' clue what it's like to deal with the tech+users+politics
 
10:16 PM
Yup, MV, not mV. MASSIVE Van De Graff.
 
Just the tech and crap they get from magazines (or pseudo-magazines in the form of l337 blogs)
 
@BartSilverstrim invite them for a day in the office or something?
Could be educational for them
 
@BartSilverstrim I could do with some more tail chasing wink wink
 
We've had recent graduates as summer workers.
You work in a college, right, @Holo?
 
Tail chasing is on the agenda this evening, methinks
 
10:17 PM
@BartSilverstrim nah, public boarding schools, 6-12
SEED
 
But you can't be overly educational with students when you're dealing with student issues (i.e., health/privacy laws, HR) and passwords. You can't get decent experience without sysadmin rights to a lot of stuff.
They end up with a lot of the grunt work. Install windows, configure this, replace the hard disk.
 
clean the @#% systems
 
Student records: big nono?
 
@BartSilverstrim Fuckin A'!
 
10:19 PM
But it's always hilarious to find the ones that "have their own networks" at home, run Linux, think they know everything there is to know and can school us on how to "do it right"...then they find they have no freakin' clue how things work.
 
In Britain, the law regarding student records is even worse
@BartSilverstrim If they're saying something correct, I'd listen
 
"hey, you. Clean off all that gunk that's on the keyboard. Yeah, whatever all that sticky stuff is!"
 
but otherwise, yeah, my answer would have two words
or, if I was bored, I'd tell them why they're wrong
 
Hint: managing your own little home network is not like managing multiple buildings with hundreds of users and services and your whims aren't what you're supporting, you're supporting user whim...and some of those users sign your paycheck.
 
back in a bit
 
10:21 PM
And in many cases, the "right thing" doesn't jive with what the user wants, and the user outranks you on the org chart.
Another hint: sysadmins are seen by their users as a necessary evil. You rank somewhere on the totem pole near the guy that mops puke up from the kindergarten room.
 
Please can I have an example of something a boss might want that doesn't jive with the "right thing"?
 
So for "this is how you're supposed to do it," you get the virtual finger a lot.
Um..."I need admin privileges on my computer."
Show of hands fellas! Who's heard that one?
 
"I need to access everyone's files"?
@BartSilverstrim raises hand
 
Developers love that one...
 
Why not? Developers often need to install software and stuff
and are technical people, not likely to screw up
 
10:24 PM
My response is now "No. When you encounter something you need admin privs for, call me and we'll see"
@fahadsadah NO devs ARE NOT technical people
We have some that can barely fucking turn their computers on
 
I may be biased here
bearing in mind my interest in development and in systems/networking is pretty much categorised in my head as 'IT'
and I administer servers for webapps I write and I develop programs for networks
 
Some little jerk thinks he's proving something by intentionally breaking a system or finding a neat trick on the network. Bleh. You're proving you have time on your hands. The sysadmins are usually out trying to solve every "FIX IT NOW" problem that springs up and then take some extra abuse for it. Schools aren't run with testbed networks, they don't have users that are tech-centered or savvy, and they're never given the resources to "do it right". We macguyver it into working.
 
but shouldn't devs at least know some systems administration?
 
Should != does.
 
@fahadsadah I insist they run as standard user, because then they can't write some shitty program that relies on admin privs they have but "normal" users don't
 
10:26 PM
Would you expect them to, though?
 
And most devs don't seem to give a damn. Try to find some sysadmin questions from devs on SF.
 
It sounds like you have shitty devs, Ben
 
Sounds like Ben has average devs.
They need higher privs (sometimes) for kernel debuggers and the like, but they often fail to test as restricted users. Then wonder why it generates support calls.
 
No, they're brilliant at what they do, which is programming. System administration is not their forté
 
@Ben has a good point.
You also seem to focus on (@fahadsadah) sysadminning being just plain sysadminning.
You'll find that we specialize too.
 
10:28 PM
My experience is helping sysadmins at my old school
 
You can find an awesome Linux admin who can't use AD worth a damn.
 
or administering a few servers for an online free shell community
 
They used to to the system administration before I was hired, and it shows. I've lost count of the amount of crap I have put right because they simply didn't have enough time to understand what they were doing and the impact of it
 
All really small places
 
Database admin...email...router work...you can have people who are great at A and B but suck at C.
Tech support is a special skill too, because you deal directly with...shit, what are they? People?
 
10:29 PM
Yup
but as I was saying, I have some experience with small organisations
and there are role isn't "DBA", "email admin", "router admin"
 
So it's not like you go to school and come out ready to take on the world in sysadmin work. I have a CS degree and really it did nothing to prep you for working in a real world environment.
 
In some cases there isn't a distinction between "dev", "support", and "sysadmin", it's all "IT"
 
It was helpful in other ways, but not for using terminal services, domain administration, dealing with creative users' problems...
You're right, it's an IT department.
And they're expected to just make things work.
 
Are you expected to create software in your job?
 
But if you can find someone working for the school that has in-depth knowledge of virtualization, large scale server deployment, desktop support, web server configuration, mail server deployment, proxy configuration, etc. etc. in one person...I'd love to see what you're paying.
No, we're not. There's no time for writing software.
 
10:32 PM
I'm sorry, I have to go now
 
I am so pissed I'm out of stars for that comment Bart
 
but it's been great talking to all of you
Really enlightening
 
Keeping hundreds of users on top of many times that many students and their workstations working is not possible in addition to writing software!
Plus we'd have to support it!
:-o
We're a jack of all trades, in-depth master of none.
With some exceptions :-)
I hate hardware...databases...and cisco routers.
Operating systems...I like that. Virtualization. Monitoring network health...
 
Ewww, hardware. Yeah, I'm very much a software guy, but I also get lumbered with hardware too. bleh
 
Don't worry, @fah. You'll be broken as reality sets in. It happens to all of us.
You'll learn that a sysadmin's secret power comes from a bottle with a flame arrestor on top.
There are people utterly fascinated with the specs on new processors and cable placement. I think they're the same ones with glowing lights in the cases. In the job, we just need to keep stuff running, and that's really all the users count on.
There are people who memorize what type of memory goes with what motherboard for optimal performance. It didn't take long for me to just say, "try this. See if it beeps when you power it up."
 
10:38 PM
I bet Chopper is one of those :P
 
He's a prime example of a specialized sysadmin.
 
When it comes to hardware specs, I'm more of a "look it up when you need it" kind of guy
 
He knows his shit, if you're dealing with his bathroom. He readily admits to that. Metaphorically.
Same here.
 
@BartSilverstrim that's too true, I used to worry about that until I realized that either that system is going to be replaced before I figure it out or it will be around so long that it won't matter
 
Yuppers! That's the thing kids don't understand. There's how it should be done, and how it practically gets done.
Not that experience saves people from stupid thinking. My horse isn't quite that high.
 
10:41 PM
and now that the cloud it taking over it still won't matter
 
There are plenty of mistakes and things that I would consider mistakes made in how things are done. But you deal with it. There are more important things to worry about.
Oh, yeah. The cloud. You know what else comes from clouds? Tornadoes.
 
I wanted to buck the whole clould thing, until about 90
 
Amazon back up yet? Can't get my @#% reddit fix. How can I tell people how stupid they are when it's in read-only mode!?
 
% of my customers started wondering how to get there
damn new keyboard
 
@BartSilverstrim I am so using that next time someone suggests a cloud service to me at work
 
10:43 PM
My word. I really wonder about people when they don't know how to create a group policy that is based on groups... serverfault.com/questions/262345/…
 
You have issues with cloud services Ben?
 
@BartSilverstrim, there are lots of people to talk down to on stack exchange if you can't get to reddit.
 
I'm always cautious of something I don't have total control over. I won't rule it out, but my first choice is almost never a cloud solution
 
dammit still in read-only mode.
@Ben: isn't that how StackOverflow's overlords designed the site?
I remember Jeff's podcasts basically saying he learned about sysadmin work on the fly by scaling up SO and adding to the sites because he didn't want to not know how it worked and/or lose control of how it worked. Something like that.
Guess it's good it's not on the Amazon @#$^%! cloud.
grr...
 
Ben, what makes you think you wouldn't have control over it? ( I used to think that too) most of the cloud solutions (I am most familiar with the azure stuff) gives control to the admins
 
10:47 PM
You don't have physical access to it or "know" where it resides. The physical part of it.
Hence...cloud.
You can't just replace a hard disk or know how much is backed up at their site...you get blamed for things you can't reach out and strangle. like WHERE THE HELL IS THE AMAZON CLOUD SERVICES NOW!@?
 
you can know where it resides (again I know that azure can), what would you need physical access to? Most of the solutions (amazon, azure in particular) use VMs
 
Maybe I've just not had enough exposure to it, but the services I've been asked to look at were far too rigid and we'd have to change business processes to fit with that particular cloud service's model
 
@ben
@BenPilbrow that is a real concern
however the savings usually pays for the change to business process, the downside to most cloud solutios is that the cloud is usually 1 size fits all
 
Some people don't like the idea of relying on someone else to have access physically to hard disks and such. I know who can get our email server, I know if someone is trying to access the data, while you don't know who could be doing stuff with your data if it's not in your company.
 
Don't get me wrong, we've got a few cloud services and they're usually fine. It is annoying however when they have a blip that I am effectively powerless to fix it but still get shit for it not working.
 
10:52 PM
Every time I hear about a solution that solves our problems, I cringe.
@BenPilbrow exactly
AD was supposed to clear up management issues. It cleared some, created others. Virtualization was supposed to be a huge savings. Worked for some things, created problems on other fronts. Now the cloud. /hums "stormy weather"
 
Prime example: We have cloud based email archiving. For about 23 days in March you could not see any email after ~2nd March. That is totally uncool that I got so much grief for that.
 
I would submit (now playing the role of cloud advocate :) ) that you can know who can access your data (private clouds) securitywise the cloud solutions (depending on provider) usually have literally millions invested in IPS/IDS firewalls etc
 
damn, I need more stars. @Zypher can we treat this like our national debt ceiling and just raise the limit when we get close to reaching it? I can't stand not having stars to hand out....
 
unfortuanely ... NO
 
Also, I'd have got my ass sacked if one of our services was down for 23 days. But no, since we pay £££££ for this, they get a dressing down, a few freebies for us and it's business as usual. Until they bugger it up again. That winds me up even more
 
10:56 PM
So Ben that sounds like an SLA problem that you got blamed for and couldn't pass on the pain. to me I'd be saying either bring that back in house (depeding on provider can be a pain) or change providers (which I know is easier said than done)
 
@Zypher :: pout ::
 
In the end, @JimB, you're relying on outsourcing services to someone else who isn't you or your company. For some things that's great. For others, it means you're taking a risk that they're going to provide equal or better work than you and/or your team can provide and take responsibility for.
 
a good provider lets you pass on that pain, and does more about it than simply pass out freebies.
 
There's still going to be potential friction because when certain things blow up, you take the heat, but you can't walk down to the NOC and get working on a solution.
You're basically getting some pay to take responsibility to sit on the phone and bitch to the people who are supposed to be getting access working again.
 

« first day (222 days earlier)      last day (4759 days later) »