Networks are hard

So yeah, I got clobbered today.  A change an ex-coworker made to a switch config nearly a year ago kicked my ass today.  Ain't that a bitch?

Long story short see above, much longer version is around 10am I was having a conversation near one of my Wireless Access points.  It's a strange habit I'll admit, but I tend to look at it whenever I'm around it.  Why you ask?  Well these ones have handy dandy status lights, you can see how they are behaving by the color of the light, green is good with no clients blue is good with clients and anything in a red-ish hue is bad.

Well I noticed mine was magenta.

Typically not a huge deal, sometimes they dissociate from my wireless controller and eventually re-connect at their leisure.  Around 11am someone mentioned the wireless was not working so I looked into it.  I had traps in the log that told me around 10:18am every single AP disjoined the controller, not reason given.

Ruh Roh!  Out comes the investigator hat.

I reboot the controller and the configuration server, power cycle the AP closest to me and it joins the controller no problem, so I'm thinking hey that's not so bad.  Trouble is only a handful of my 51 AP's rejoin, and an hour later I only have 5 AP's and 10% isn't going to cut it.  Plus some people are starting to notice.

I have about 1 million square feet (I shit you not) of campus to try and cover so if one AP is down it's not a huge deal, but 46 kind of is.  So I pull out my magnifying glass and thinking pipe (whatever props help me think) and work on probing the recesses of my damn network.  I've got Cisco's network adviser running, a spreadsheet of IP addresses, 5 terminal sessions and my fluke network analyzer looking for issues.  I tracing links and verifying that trunks are working, making sure no ports are flapping, double checking vtp server settings, verifying STP is not doing something stupid, still only 12 AP's are joined and it's like 2pm.  I'm starving and not able to think any longer so I do the sensible thing and go get a Jimmy John's sandwich and play some sodoku.  It was a bad decision, it's Thursday and Sodoku is really hard when you're

1. distracted
2. hungry
3. stupid
4. tired

None of this stops me of course, and I fail at sodoku as I knew I would and return to my desk. At this point I still only have 12 AP's and I don't know why, so I do the sensible thing and start looking at errors in my Fluke Network Analyzer, low and behold I notice and IP conflict on the secondary IP of my master switch and the secondary IP of my wireless lan controller. I fixed that and boom all of the AP's joined. Son-of-a-bitch. I'm going to stab that guy, he's been gone almost 8 months and he is still fucking with me.

Alright the internet was fixed ...

Frack, yesterday was bad enough with everything being full of suck playing with the router but today my upstream provider took a dump in the middle of the day.

Facepalm.jpg

For real, it's fixed then it isn't and so people call me.  Like I'm coated in magic and can make it work.

So my trouble yesterday was a specific firewall rule that was allowing some external partners work for the last 6 months just stopped working in the middle of the day.

So yeah into the router config we go.  It's a piece of crap Juniper so yeah it's not the cisco routers I've come to know and love (seriously if you are a monkey you can follow most of the logic of the cisco, except sub-interfaces that shit is dumb).  Anyhow I have logging on so so many policies and none of them including my block any policy has any traffic in it.

Why?  Because it's a Juniper and the logging makes not one lick of sense.  It doesn't log all traffic, only the things you specifically call out, and even then only when the session is closed.  All in and out traffic is logged in the in policies for instance.

Anyhow my net take away was I added TCP Any and UDP Any to see what damn port that stupid program wanted (TCP 49292 for the curious), and find myself asking how on earth did it work for the last 6 months?????!!?

Yeah, so apparently my router is leaky and sucks at logging which is the worst combination of failure I can imagine.  I'm all Hey Boss guy how about we get a Cisco, and he's all "Hmm, those are only programmable from the command line right?"

For real.

I'm currently neck deep in router bullshit

I'm thinking the best way to fix it is to pitch a hand grenade into the damn chasis, and go for a beer.


Side note, people are crazy pushy when the internet is borked.

I don't understand Pepsi's ad

So Pepsi had an ad on that crazy Beserc Herks site and I thought to myself I like pepsi, what are you guys upto now.....

Go ahead, this is where it lead me.

This clusterfuck of a webpage is supposed to have information on it I think, but I tried to filter by city and it prompted me to subscribe to facebook, twitter or some archaic thing called E-Mail.  I couldn't figure it out, so way to go Pepsi, money well spent on that shitty ad campaign.

U is for Update

I thought long and hard about U's and came up with nothing, I couldn't come up with anything clever or new or original.  This is worse than K.  U, I bow before your mighty vowel self and just want to note that I did update the the S post with a picture of the painting I did.

In case it's too hard to click the link here it is:

J is for Just Finish It

By way of Lifehacker comes a post that is fairly pertinent to my life at large Just Finish It.

Much like the author of the post I don't struggle with ambition to start things, or lack vision to design a more grand universe.  What I do lack is the wherewithal to finish or carry out these dreams that live in my head.  My project list is ever growing in both my personal and work life; the sensation is overwhelming in someways.  A real commitment in the form of my son is robbing me of my late night productivity and lack of interest is having further negative impact on my ever increasing project list at work.

The short list includes migrate from exchange 2003 to 2010, continue Windows7 rollout, update backup exec from disk to tape to disk to disk to tape (de-duplication for the win!) and hopefully making my backup windows shorter than 22 hours, upgrade the last NT server and the software that it is running (sometimes legacy just won't die, it has in the neighborhood of 80+k reasons why it's been kicking around on the really old version of the software) Mobile device strategy (people really think tablets are going to be some form of productivity boost, I think they are smoking that funny tobacco) completing the office 2010 deployment, some server automation scripts to make the co-workers lives easier.  Then I have a backlog of documentation, and future network planning to complete (really should be certified Cisco to run the network I have, let alone the network I need).

So yeah.  Sometimes my work sits on my mind a lot, but I have almost no interest in doing about half of that shit I just wrote down.

DST Sucks

So I've been tired all week, and haven't really been able to figure it out.  I mean sure I worked late on Monday and have been playing video games at night, but that doesn't contribute right? (jokes people) Anyhow just like I'm blaming Canada for this piss cold weather I'm suffering though I'm going to blame my tiredness on Daylight savings time.

Gizmodo drops some knowledge on why it's pointless.  At this point I'll take anything (especially if it comes with a nap).

IT related post

Okay so it's about to get technical and ranty up in here so if you don't care for either thing piss off.

Last night my coworker and I stayed late to update an IBM Bladecenter S raid controller and upgrade the drives from 1tb drives to 2tb drives.  These drives are effectively the SAN for the blades so doubling the storage is kind of a big deal as we get ready to start adding more blades into this thing.  The trouble I had with this is the update process is really just bunch of bat files and python scripts that rely on default users that you cannot change and in typical IBM fashion it's not all built in house it just strung together oem parts glued together with code.  It was supposed to be a maintenance window of 6-8PM, but we didn't finish until 11:30pm and I have contractors in this morning (supposedly 30 minutes ago so some fuckers are about to get a phone call) all because of the kludgy as fuck nature of the update process I'm tired and cranky.

Also the SAS raid controllers on IBM bladecenters run MontaVista Linux professional version 3.1 on the Power PC platform and have a default userid and password and telnet is on by default.  For the technically inclined try to imagine what is wrong with this horrifying picture.  I can change the password, but telnet needs to be on (that's how you communicate the firmware updates ..... for serious) and I can hide the interfaces in VLAN's with ACL's which mitigates it to some degree, but I still bet this damn thing has busybox on it (it serves web pages and I think has an ssh interface) .  Anyhow the updates are done, and I'm tired but I'm rocking a 7.3tb array in one storage module bay(would be 9.1 but I assigned one drive as global hot spare) and a little shy of 2.8tb array on 15k rpm sas drives for running sql on

Gentlemen rant about the iPad

Yeah that about sums it up

Copy run start

On command on a cisco switch would have saved me a decent amount of stress on Tuesday.  Vlans that you didn't create, using a VTP server that you only half understand coupled with power outages = fuck yeah wasted time and day.

The guy that was Cisco certified decided he didn't want to work here any more, I'm not certed, but have somewhere between one half and three quarters of a clue, but sometimes I forget important things like copy run start.

This means nothing to anybody I realize, but I'm kicking myself for it.
conf t
interface gigabitethernet 0/44
switchport mode access
switchport access vlan 112
exit
exit
copy run start  <--- Important last step you idiot.

It was bound to happen

Ugh, what a awful week already.  So my truck wouldn't start on Tuesday so yeah for not work, boo for repair shop bills.  That is pretty sucky, but in more serious and sad news, I have finally actually failed in my attempt to produce booze.  This isn't just I missed the style or some such, I messed up so much that the cider is so acidic no yeast can survive the environment.  I was trying to make a more interesteing flavor in my hard cider and mixed in about 3/4lb of Acid blend.  I've since learned that it should have been about 3 tablespoons for the size of batch that I am working on.  So I over did it by about 50-90 time too much acid.  This stuff is tart, like rot your teeth sour patch tart.  I started with a 3 gallon batch and then upped it to 5 with more cider, then took it to over 8 with brown sugar and water.  The water is so base it doesn't register on my ph strips and the cider is so acid it just reads the lowest reading after diluting it as much as I have.  I think 1/2 a gallon of what I have in 5 gallons might still be too acidic.

It's a sad day for me but, the every dollar at this point is good money after bad.  For grins I might have gone to buy another huge thing of brown sugar and make a sugar wash to add 1/2 a gallon of this stuff to just to see if I can get any mileage out of it, but with a car in the shop I think my hobbies are on hiatus for some time.

Does Glee have a color?

Another Day, another Mac trojan.

Tick Tock Mac's Tick Tock

Another Mac Trojan, this time ported from Ye Olde Linux.  I promise I'm not smiling as I type this.

I just got *ucked by Tuesday

It's 8:30 am and already this is starting to get ugly, I got a notice from a VP that we have been put on notice for e-discovery in relation to a Tort Claim that is being reviewed.  This is sort of my worst nightmare, e-discovery for email will literally be the worst thing to ever happen here, I have some users that literally delete every message that they get after they have read them.  I have no way to produce those records.   I've known about this issue for a few months, and have exchange 2010 purchased I just need to get it installed and setup the legal-hold mail store.

Next up the worthless accounting software won't launch.  The server needs to be rebooted I'm certain, but I haven't rebooted any servers since that fateful day about a year ago when we discovered a bad array controller and the accounting server couldn't see any of its disks.  Talk about panic, I found the same server model on craigslist drove out after work, came back on a Saturday to get this beast working.  Now I staring at the screen afraid of lighting striking twice as I hit reboot.

Same time frame my boss doesn't know the new WIFI password and has a contractor my coworker setup on the wireless trying to lead a class for our new ERP software.  Also I have people that need signage, and I'm quite literally the only one that knows how to run the software so, guess who is on the hook to do signage right now.  I'm in need of emergency Bourbon at this point.

If it isn't one thing

So the pregnant wife is not comfortable in the heat of late so I pulled out our portable air conditioner to cool the bedroom off for her.  We have used this thing a grand total of about 20 days in the 3 years we have had it.  It wasn't exactly cheap and we read several reviews on several competing models and choose this particular one because it was generally believed to be a high build quality.  We have only used it for cooling or the fan and have never used the heater function but it apparently has been leaking like a sieve for the last 2 days on my carpet.  I left it running yesterday is the "dry" function that condenses moisture from the air so you can dehumidify a room, well all of that moisture is now in my carpet.

Awesome.

I put down towels to sop up the worst of it and then got out our carpet cleaner to try to soak up as much water as I could.  In two ours I had filled the thing 4 times and there was still a lot of water left in the carpet.  I worked on it until my arms where shot and will probably have to deal with it some more tonight.  I assume something is obstructing the drainage line inside the unit, but won't know more until after the carpet is dry and I start to take the AC unit apart.

Hurray for lowest cost manufacturing.

Get off Jobs dick

So this festering pile of shit article lit my fuse this morning.  An ode to how nothing was ever made nice before Steve "Aka God" Jobs descended to earth and made them nice.  My first problem is that Jonathan Ive designs Apple products, seriously look at the mans resume.  If you want to jump on someone's dick there is the man that makes these things look the way they do.  Period.

Next point Compaq before it was subsumed and defiled by HP made this product, now for the fan boys at Gizmodo that literally picked every ugly tablet that existed and ignored the best of class product for the time frame before iPad's existed I say I loathe your failure of an editorial process.  Now the downside it had a Transmeta processor, and as much as I would love for that company that paid Linus Torvold a ton of money to basically not work there to have put out a great product, it wasn't a good processor.  The revision that boasted an intel mobile processor was much better and share basically the same design.  Omitting this from your little glamour shot was shitty.

Long ass day....

Jeebus, I'll probably take forever to digest all the shit that went wrong today but I think once and for all I've proven that I am not smarter than the people that have taken the time to write brewing software.  I haven't calculated it yet, but my results are going to be terrible, like maybe 50% efficiency.  I have a few ideas of what went wrong, I know there where several mistakes on my part that made the day more stressful for sure, but I'm not positive what I did wrong.  I started the say with nearly 27lbs of grain and ended the day with just at 5 gallons of beer that was 1.0985 OG.  How I boiled 4 gallons off in a 90 minute boil is a little beyond me why I only had 9 gallons in the pot is a little mystery of shit math and poor planning.  I should not have brewed today.  I intended to brew tomorrow, but my wife wanted to go day tripping tomorrow so I compressed my day of planning and supply gathering and ended up screwing up.

Mistakes I made.  I had 25lbs of grist to work with my 5 gallon painters bags are not big enough to handle that.  They really cannot handle 13lbs very well but it is doable.  I only had 1 of 5 gallon painters bags.  I should not have pressed on at this point.  I should have stopped and rationally dealt with it.  I did not.  I ended up going and buying more bags anyway, but I not before over filling one and dunking it into the water.  This was an absolute fuck up.  I ended up with grain floating around in the pot that I took to boil and consequently in the finished wort that I tried to transfer to my fermenter.  The free floating grain was a major headache and I violated so many cold side rules that have kept my beer from getting infected with outside contaminates that I am not sure if there is any hope of it coming out ok.  I am at this point praying that brewers yeast can out compete and kill any potential infections but at this point I have mentally readied myself for throwing it out.

Reading that sentence turns my stomach, but I think it needs to be open as an option if I sample this and its infected.  I need to be ready for it.  The only thing that gives me a good deal of hope is the steady bubble of the airlock next to me, and the taste of the "failed" Mac & Jack's I'm drinking.  This beer needs to be my mascot for the night.  I missed on this beer in terms of the style and flavor I envisioned when I brewed it as my first "sour" beer (pro-tip it isn't a sour beer and that's why I failed) but it is a nice beer in its own way.  Little on the low gravity side with only 4.6% but still a nice beer when you want a light body amber.  When I bottled it I swore up and down that I was going to have to pitch it but the beer is fine.  Not the best I've ever done, but fine.  I'll tear apart the day tomorrow, but I started the brewing at noon and finished washing everything and putting my garage back together at 11pm.  I'm tired and feeling a little beat down.

As a sick the universe warned me, my horoscope told me not to spend my financial windfall, and as it would not turn out as I hoped.  Well fuck you horoscope, and superstition in general.

Why I won't buy another iPod and neither should you

Apple disrespects their customers and their accessory partners with pretty much every release of the iPod and iPhone. By breaking pin compatibility with previously released accessories they are forcing customers that have invested in their ecosystem to purchase new docks, charging stations, and other accessories. Some in-dash decks that boast iPod compatibility have to have the little * and tell you what generations they support, my wife's speaker dock and charger works for the iPod classic (aka 6th gen) but not for the iPod touch 2nd gen. This isn't rocket science people they did it to prevent unapproved iPod accessories, not because these devices might damage your device but because they where not paying Apple to make these devices. Who suffers in these cases though? Not the knock off manufactures, they just design around it, while people like me that bought my wife a Iluv 1155 dvd player with iPod dock to go with her fancy iPod only to find out that the manufacturer was unaware that it wouldn't work with the new generation of iPod that launched around the time it did. It turns out after reading recently they issued working replacements, but did nothing to remedy the affected users. I choose to not return the unit only because it was a decent dvd player, but it deffinately is a sore spot for me and Apple.