The witch is dead

It's only taking prodding, and death threats to various factions of my company but last week I finally turned off our legacy Windows NT domain. This may sound like a small feat, but our payroll system was still running on an NT machine, and I was not under any circumstance going to bring that into the current domain. So there floated a PDC, BDC and application server running that I couldn't get rid of. Anyone with experience with payroll will know the cost and complexity of payroll systems, particularly when multiple unions are involved. We have accruals and seniority rules that we have keep careful track of for scheduling and some individuals can work multiple job codes sometimes all in the same shift. It was a mess, and migrating our legacy data to a new platform was going to be a costly endeavor.

In the end the threats of bodily harm did the trick, as they did bite the bullet and migrate their data (otherwise I might have had to operate that server in perpetuity). I was a little nervous about shutting down WINS, as a few other admins that had migrated off NT said that they ended up standing up WINS servers later on. My understanding of Active directory was that it could run on just DNS.  I guess I'm out of excuses, and will have to start working on migrating our Exchange 2003 server to Exchange 2010.

This years to do projects are Exchange 2010, Active directory 2008 or 2012 (need to see if all of our clients will work on 2012), the second half of our desktop deployment and the final death of XP in our organization, a completely overhauled Backup System (new software, tape hardware and a disk2disk with deduplication), expanding the coverage of our wireless network, and working on to get our preventative maintenance/asset tracking system integrated into a different software package so I can retire that creaky piece of shit.

How's that for some IT nerd?

Holy hot mess

So as I mentioned previously I've got no time, that would include time to go to the gym.  I've hoping to reverse that so I lined up some sessions with a personal trainer.  I've worked with him before and he's pretty good at breaking me so I knew generally speaking what I was getting into.  The gym that he works out of is about 30 minutes walking from my work, and I had an appointment for Monday at 3pm.  I figured if I brought a bike in I could cut the time to get to there down considerably (like 7 minutes instead of 30).  That would give me time to change, get there and work out them get back change and go back to work in theory. It was going to blow about an hour and a half, but I figured that wasn't too shabby.  Cue the Monday morning and like a jack-ass I didn't do any prep for my little workout, and had to scramble to get my bike out of the garage (no small task) and get all the gear for working out and locking up my bike downtown.  It was a cluster, but I managed to get on the road to work only adding about 20 minutes to the day.

I had an early(ish) lunch, but didn't have anything to eat just prior to leaving for my workout.  In the process of getting my bike out, I noticed the rear tire was low so I had dug out the compressor and filled it up (long ridiculous story but my bike pump was destroyed on accident the last ride I went on), the front tire was firm, but I din't really check it (mistake and a half).  I get changed, and get on the bike to ride out but the only water bottle I have on hand is the worst bottle ever (terrible prep for this I know) but it's all that I have on hand so I load that up and get to riding to my workout.  Right away I realize the error of my ways for not checking the front tire, when I brake and my weight transfers forward the tire flattens out enough that it's kind of scary to maneuver (ie brake and turn at the same time).  Scarier still is down one hill I'm matching traffic so 25-ish mph with a tire that is not safe.  Either way, whatever I get to the gym in about 7 minutes and kick myself for not having a protein bar or some fruit to eat before I work out.  But that's the sate of things so I roll with it and get into my workout.  40 minutes in the enormity of this mistake becomes abundantly clear, and I have to stop because I'm getting light headed and shaky.  Not the good work out shaky, rather the I'm going to throw up and then pass out shaky.  So I head to the gyms juice bar and get a smoothie to get some sugar into my body before I bonk, but it's nearly 20 minutes before I'm not shaking any more.

Now the fun part, the trip to the gym was largely down hill (gentle down hills, but still), crummy front tire and a body that is asking me for mercy; up the hill you bastard.  It wasn't pretty.  It took about 25 minutes to make it back to work, and another 10 to get changed.

When I got back to my desk I was a hot mess.  Maybe not hot, but sweaty and gross; so mostly just a mess.

Anyhow if you made it this far and didn't TL;DR me, I'll reward you with adorable Otters

Heavy is the head that goes to the conference

Ugh, and I've been in a user conference for the Enterprise management software we use at work all week. This might seem like it would be a good way to spend your time, but that would assume that the software was in any way interesting. It is not.
It is giant and all inclusive with modules for accounting, space management, work order tracking conflict management for resources and a hundreds of other things you don't care about.. Probably the best feature of this event is they are feeding me breakfast and lunch as well as snacks. Otherwise I've been using my phone turned tablet (it's a long story but I ended up with a phone that was run over and broken that I nursed back to life and use as a wifi tablet) reading blogs and other general joy.
How about you? What have you been up to this week?

Sudoku Puzzle training

I'm a sudoku player and was finding that my ability was stretched too thin on the Thursday and Friday puzzles so I sought out something to help me learn the strategies a little bit online.  Enter Sudoku wiki.

You have to pay close attention to what it changes, but the strategies of Naked pairs/Triplets and pointing pairs seems to have the most mileage for solving puzzles that you fell the need to guess.  Prior to finding this tool I would review my work against the answers in the next days paper and found that the correct number for a space was often one I wasn't considering for that space, (ie it was a number that could have been in 4 or 5 spaces so I didn't write it down).  In a sense I was finding that I was building myself a box that prevented me from ever getting the correct answer.  If you have any interest, but struggle with the harder puzzles this is a pretty good (though spartan) tool for learning some different strategies.

Networks are hard

So yeah, I got clobbered today.  A change an ex-coworker made to a switch config nearly a year ago kicked my ass today.  Ain't that a bitch?

Long story short see above, much longer version is around 10am I was having a conversation near one of my Wireless Access points.  It's a strange habit I'll admit, but I tend to look at it whenever I'm around it.  Why you ask?  Well these ones have handy dandy status lights, you can see how they are behaving by the color of the light, green is good with no clients blue is good with clients and anything in a red-ish hue is bad.

Well I noticed mine was magenta.

Typically not a huge deal, sometimes they dissociate from my wireless controller and eventually re-connect at their leisure.  Around 11am someone mentioned the wireless was not working so I looked into it.  I had traps in the log that told me around 10:18am every single AP disjoined the controller, not reason given.

Ruh Roh!  Out comes the investigator hat.

I reboot the controller and the configuration server, power cycle the AP closest to me and it joins the controller no problem, so I'm thinking hey that's not so bad.  Trouble is only a handful of my 51 AP's rejoin, and an hour later I only have 5 AP's and 10% isn't going to cut it.  Plus some people are starting to notice.

I have about 1 million square feet (I shit you not) of campus to try and cover so if one AP is down it's not a huge deal, but 46 kind of is.  So I pull out my magnifying glass and thinking pipe (whatever props help me think) and work on probing the recesses of my damn network.  I've got Cisco's network adviser running, a spreadsheet of IP addresses, 5 terminal sessions and my fluke network analyzer looking for issues.  I tracing links and verifying that trunks are working, making sure no ports are flapping, double checking vtp server settings, verifying STP is not doing something stupid, still only 12 AP's are joined and it's like 2pm.  I'm starving and not able to think any longer so I do the sensible thing and go get a Jimmy John's sandwich and play some sodoku.  It was a bad decision, it's Thursday and Sodoku is really hard when you're

1. distracted
2. hungry
3. stupid
4. tired

None of this stops me of course, and I fail at sodoku as I knew I would and return to my desk. At this point I still only have 12 AP's and I don't know why, so I do the sensible thing and start looking at errors in my Fluke Network Analyzer, low and behold I notice and IP conflict on the secondary IP of my master switch and the secondary IP of my wireless lan controller. I fixed that and boom all of the AP's joined. Son-of-a-bitch. I'm going to stab that guy, he's been gone almost 8 months and he is still fucking with me.

Alright the internet was fixed ...

Frack, yesterday was bad enough with everything being full of suck playing with the router but today my upstream provider took a dump in the middle of the day.

Facepalm.jpg

For real, it's fixed then it isn't and so people call me.  Like I'm coated in magic and can make it work.

So my trouble yesterday was a specific firewall rule that was allowing some external partners work for the last 6 months just stopped working in the middle of the day.

So yeah into the router config we go.  It's a piece of crap Juniper so yeah it's not the cisco routers I've come to know and love (seriously if you are a monkey you can follow most of the logic of the cisco, except sub-interfaces that shit is dumb).  Anyhow I have logging on so so many policies and none of them including my block any policy has any traffic in it.

Why?  Because it's a Juniper and the logging makes not one lick of sense.  It doesn't log all traffic, only the things you specifically call out, and even then only when the session is closed.  All in and out traffic is logged in the in policies for instance.

Anyhow my net take away was I added TCP Any and UDP Any to see what damn port that stupid program wanted (TCP 49292 for the curious), and find myself asking how on earth did it work for the last 6 months?????!!?

Yeah, so apparently my router is leaky and sucks at logging which is the worst combination of failure I can imagine.  I'm all Hey Boss guy how about we get a Cisco, and he's all "Hmm, those are only programmable from the command line right?"

For real.

I'm currently neck deep in router bullshit

I'm thinking the best way to fix it is to pitch a hand grenade into the damn chasis, and go for a beer.


Side note, people are crazy pushy when the internet is borked.

J is for Just Finish It

By way of Lifehacker comes a post that is fairly pertinent to my life at large Just Finish It.

Much like the author of the post I don't struggle with ambition to start things, or lack vision to design a more grand universe.  What I do lack is the wherewithal to finish or carry out these dreams that live in my head.  My project list is ever growing in both my personal and work life; the sensation is overwhelming in someways.  A real commitment in the form of my son is robbing me of my late night productivity and lack of interest is having further negative impact on my ever increasing project list at work.

The short list includes migrate from exchange 2003 to 2010, continue Windows7 rollout, update backup exec from disk to tape to disk to disk to tape (de-duplication for the win!) and hopefully making my backup windows shorter than 22 hours, upgrade the last NT server and the software that it is running (sometimes legacy just won't die, it has in the neighborhood of 80+k reasons why it's been kicking around on the really old version of the software) Mobile device strategy (people really think tablets are going to be some form of productivity boost, I think they are smoking that funny tobacco) completing the office 2010 deployment, some server automation scripts to make the co-workers lives easier.  Then I have a backlog of documentation, and future network planning to complete (really should be certified Cisco to run the network I have, let alone the network I need).

So yeah.  Sometimes my work sits on my mind a lot, but I have almost no interest in doing about half of that shit I just wrote down.

IT related post

Okay so it's about to get technical and ranty up in here so if you don't care for either thing piss off.

Last night my coworker and I stayed late to update an IBM Bladecenter S raid controller and upgrade the drives from 1tb drives to 2tb drives.  These drives are effectively the SAN for the blades so doubling the storage is kind of a big deal as we get ready to start adding more blades into this thing.  The trouble I had with this is the update process is really just bunch of bat files and python scripts that rely on default users that you cannot change and in typical IBM fashion it's not all built in house it just strung together oem parts glued together with code.  It was supposed to be a maintenance window of 6-8PM, but we didn't finish until 11:30pm and I have contractors in this morning (supposedly 30 minutes ago so some fuckers are about to get a phone call) all because of the kludgy as fuck nature of the update process I'm tired and cranky.

Also the SAS raid controllers on IBM bladecenters run MontaVista Linux professional version 3.1 on the Power PC platform and have a default userid and password and telnet is on by default.  For the technically inclined try to imagine what is wrong with this horrifying picture.  I can change the password, but telnet needs to be on (that's how you communicate the firmware updates ..... for serious) and I can hide the interfaces in VLAN's with ACL's which mitigates it to some degree, but I still bet this damn thing has busybox on it (it serves web pages and I think has an ssh interface) .  Anyhow the updates are done, and I'm tired but I'm rocking a 7.3tb array in one storage module bay(would be 9.1 but I assigned one drive as global hot spare) and a little shy of 2.8tb array on 15k rpm sas drives for running sql on

SAN Managment

F*CKSTICKS I was working with my ISCSI SAN yesterday because one of the volumes was getting full (seriously why does everyone save so much data) and I needed to delete an unused volume and use that space to expand the nearly full one.  Well genius boy over here deleted the live one.  ISCSI is really sticky so even though the volume is deleted it is still serving data and let me restore from the Friday backup to the new larger volume and overnight I remapped the shares to the restored files and started syncing the two.

Net result no one but me will know, but there is nothing like the rush of pure adrenaline when you realize you just deleted 300gigs of users data at 3pm!  Sweet Jesus thank you so much for LTO4 Tape drives, I restored at 1.5gb/m so restoring 300 gigs was only about 3 hours but either way I'm exhausted and not going to be in charge of making any decisions today.

Copy run start

On command on a cisco switch would have saved me a decent amount of stress on Tuesday.  Vlans that you didn't create, using a VTP server that you only half understand coupled with power outages = fuck yeah wasted time and day.

The guy that was Cisco certified decided he didn't want to work here any more, I'm not certed, but have somewhere between one half and three quarters of a clue, but sometimes I forget important things like copy run start.

This means nothing to anybody I realize, but I'm kicking myself for it.
conf t
interface gigabitethernet 0/44
switchport mode access
switchport access vlan 112
exit
exit
copy run start  <--- Important last step you idiot.

Those that work with their hands

If you haven't noticed I've got some strange and diverse interests, from cooking and food, to electronics and computers I'm working on being a broad and deep person. I used to be hyper focused on computers, particularly in college and I can say that I still like computers and enjoy pushing technology. The difference is when I go home now, I don't get on the computer often. Video games yeah, but more often I'm cooking or reading about my this or that hobby I have an interest in. I have been following some topic specific blogs for a while now, and one that consistently posts some great stuff is Toolmonger.

He posted this on Friday, and I really thing there is something to watching this man talk about the business he took over, built and run. The profound is open to those that work with their hands.  There is something about working hard, with your hands and having something to show at the end of the day. I don't sign up for it everyday by any means, but I do love the serenity I find at night after a day of working hard and getting dirty. Like I said it doesn't happen often, but I like it.

Crisis averted

Alright it was less terrible than I originally thought and it goes to show it's not what happens to you but how you react.  I didn't get much sleep last night so encountering a problem right away in the day I wasn't ready to deal with something that was broken.

Turns out the transaction logs on the SQL server had grown quite large and filled the volume.  Treesize and a few 20gigs deleted later, life is better and no one needs to know that the reboot wasn't exactly what fixed the server.  The server most definitely needed to be rebooted, I had a new version of anti-virus that was waiting to finish installing.

I'm just paranoid about this server it failed on me before, and I was stressed when I saw that the monthly full backup had failed to complete (the password reset from the co-worker leaving pissed off the backup service).  In the end it all worked out fine.  The monthly full this month is going to be a little spotty in terms of restoring, it is currently running and only about 20% after 5 hours.  1 terabyte of data to backup FTW?  We normally let it run from on Friday night and it goes all day Saturday.  The backup averages 22 hours and only a few things have been locked at the exact moment someone was trying to use them today so for the most part most people won't even notice.

I just got *ucked by Tuesday

It's 8:30 am and already this is starting to get ugly, I got a notice from a VP that we have been put on notice for e-discovery in relation to a Tort Claim that is being reviewed.  This is sort of my worst nightmare, e-discovery for email will literally be the worst thing to ever happen here, I have some users that literally delete every message that they get after they have read them.  I have no way to produce those records.   I've known about this issue for a few months, and have exchange 2010 purchased I just need to get it installed and setup the legal-hold mail store.

Next up the worthless accounting software won't launch.  The server needs to be rebooted I'm certain, but I haven't rebooted any servers since that fateful day about a year ago when we discovered a bad array controller and the accounting server couldn't see any of its disks.  Talk about panic, I found the same server model on craigslist drove out after work, came back on a Saturday to get this beast working.  Now I staring at the screen afraid of lighting striking twice as I hit reboot.

Same time frame my boss doesn't know the new WIFI password and has a contractor my coworker setup on the wireless trying to lead a class for our new ERP software.  Also I have people that need signage, and I'm quite literally the only one that knows how to run the software so, guess who is on the hook to do signage right now.  I'm in need of emergency Bourbon at this point.

Coworker

So my coworker is leaving in a short while and the stress of the departure is kind of starting to mount for me.  Next Thursday will be his last day and I plan on changing a shitload of passwords while he is in his exit interview.  The thought of what I'm talking about undertaking is literally keeping me up at night and frankly leaving me the tiniest bit pissed off at him.

In reality it's really a good thing that he is leaving, as he just never got on the same page as me and the boss man.  To make it worse not two weeks before his departure he managed to completely piss off our current President.  Enough that our boss was called down and yelled at, so had it not been his last two weeks it might have turned into being his last two weeks if you catch my meaning.  I'm sure that I'm going to get spotty here as the workload of making sure that a system admin is really gone once he leaves these doors comes into a clear picture.  Day to day we split the workload, but he had been given some tasks that I will have to familiarize myself with, and hope that there are no services that are running as him.  

Fare thee well little office mascot

So for the last few days my office has been all worked up because of a small 4 legged creature of the scurrying kind.  That's right a mouse is dancing around in our office and given that I sit in an area with 5 ladies and 1 other guy that is never at his desk I've been the designated man for looking under stuff for these ladies.  Yesterday out of annoyance for this disruption we cleaned out under a table that the little guy has been known to frequent.  The ladies bravely stood on chairs (I shit you not) and behind cube walls while I calmly looked through boxes to see if there was evidence the little guy had visited them.

After all the clutter was removed we laid out glue traps and eliminated hiding places.  I didn't bother checking glue traps this morning, it just didn't occur to me.  I was sitting at my desk still listening to music eating my breakfast when one of the ladies cursed loudly and yelled for me to come deal with the little fellow.  He met a glue trap and that was the end.  Custodians came and fetched the trap and hopefully so ends our little multi-day adventure.  I know that one of the ladies has been looking a little harried since the whole thing started last week, she looked down and it was just chilling under her desk watching her.

Work Conflict

It must have been something in the water, but yesterday was filled with conflict. Me and an accountant are going to find our selves in fisticuffs if she doesn't cool her heels, especially since it looks like the whole source of our conflict is really someone else's fault. I'm not shy about conflict, and I would dismiss it if that was the only conflict to be had yesterday. After she got me all riled up I went to a meeting (that I didn't want to attend) and my boss got a little lathered up talking to the CFO about a silly bit of training they wanted to get. While I agree that our users should learn how to use office on their own (literally they wanted training for how to use the ribbon interface) it is really unlike my boss to get so animated over something. Must have been a full moon or something, but man there was a lot of conflict (by my standards) yesterday.

College and its value

Lately several FB friends have posted articles that are critical of the value of college, at the same time several prominent blogs have been trashing the economic sense of attending college.  I think that even more than questioning the value of the undergrad experience vs the value of not starting your "adult" life in debt, some of these attacks are focused on if we are actually learning in college.  In that instance I am actually in agreeiance, as I was severely disappointed with a number of my undergraduate courses.  The material stale, the teachers on autopilot, and "knowledge" out of date.  Really the two best classes I had where from outsiders so to speak, professors that had just arrived fresh from industry.  Their current experiences really spoke to me more than one aging prof's long winded stories about the Kibbutz back in the 70's.  One of the newly minted professors had a side business that was leveraging the exploding commercialization of China by selling typewriters.  Typewriters, because then as now, the rural districts that where trying to modernize didn't have reliable electricity, so filling out forms on computer was not reliable enough for them.  He claimed to clear about $1 million dollars and had orders booked for the next several years, selling an antiquated technology to an exploding market.  The biggest lesson was opportunity is not where you might expect; sometimes large opportunities exist in where no one is looking for them.

I am conflicted about the college experience, as my position might not have been available to me without a degree and my wife's job as well might not have been available to her so on the one hand I absolutely believe I needed a degree on the other hand my coworker has an associates degree and has virtually the same pay and responsibilities.  Very little of my day to day job function can be tied back to my education and particularly my degree program.  Most of the things I took from college are social in nature, the network I gained from being there.  The friends, my wife, my time on the swim team are all tied to the social nature of the "full undergrad" experience.  I really enjoyed the social aspects of my time at college, and had it been fully covered by grants and scholarships I probably would be less bitter about my perception of the quality of the education.  That I paid for it, and continue to pay for the "education" I received makes me take the position that in dollars and cents my college didn't make sense for me.  My job and the way I started out in the industry would have been virtually identical, the difference is I got to make some mistakes in the relative safety of college, rather than loosing a real job.  I can say that I wouldn't do college over again, or at the very least I would have gone to a different college.

So I'll leave you with this.

Stupid Migraine

All right I was out of commission today with a migraine, and I actually was sad to miss work today.  This week is going to be pretty sweet as we start rolling out our new core-i5 desktops with windows 7 and office 2010.  My co-worker and I have been on a crash course in windows imaging, and desktop deployment this month, so we are excited to get the new hardware out of the server room and into the hands of the damn whinny users.  I do expect to hear about this is slow, or that is slow even though these machines are seriously about 4 times as fast as the desktops we are replacing.  You give people something new and all they do is bitch I tell you, they probably don't even want to know how close they came to getting thin clients and running on VDI.

Holy Batshit

Anyone have any idea for electrically charging fiber lines so contractors won't "accidentally" unplug your main trunk line overnight?

If you are smart and have a map of your network printed out it makes for quick troubleshooting, but won't fix that some of your sites couldn't process credit cards overnight.

FML