Networks are hard

So yeah, I got clobbered today.  A change an ex-coworker made to a switch config nearly a year ago kicked my ass today.  Ain't that a bitch?

Long story short see above, much longer version is around 10am I was having a conversation near one of my Wireless Access points.  It's a strange habit I'll admit, but I tend to look at it whenever I'm around it.  Why you ask?  Well these ones have handy dandy status lights, you can see how they are behaving by the color of the light, green is good with no clients blue is good with clients and anything in a red-ish hue is bad.

Well I noticed mine was magenta.

Typically not a huge deal, sometimes they dissociate from my wireless controller and eventually re-connect at their leisure.  Around 11am someone mentioned the wireless was not working so I looked into it.  I had traps in the log that told me around 10:18am every single AP disjoined the controller, not reason given.

Ruh Roh!  Out comes the investigator hat.

I reboot the controller and the configuration server, power cycle the AP closest to me and it joins the controller no problem, so I'm thinking hey that's not so bad.  Trouble is only a handful of my 51 AP's rejoin, and an hour later I only have 5 AP's and 10% isn't going to cut it.  Plus some people are starting to notice.

I have about 1 million square feet (I shit you not) of campus to try and cover so if one AP is down it's not a huge deal, but 46 kind of is.  So I pull out my magnifying glass and thinking pipe (whatever props help me think) and work on probing the recesses of my damn network.  I've got Cisco's network adviser running, a spreadsheet of IP addresses, 5 terminal sessions and my fluke network analyzer looking for issues.  I tracing links and verifying that trunks are working, making sure no ports are flapping, double checking vtp server settings, verifying STP is not doing something stupid, still only 12 AP's are joined and it's like 2pm.  I'm starving and not able to think any longer so I do the sensible thing and go get a Jimmy John's sandwich and play some sodoku.  It was a bad decision, it's Thursday and Sodoku is really hard when you're

1. distracted
2. hungry
3. stupid
4. tired

None of this stops me of course, and I fail at sodoku as I knew I would and return to my desk. At this point I still only have 12 AP's and I don't know why, so I do the sensible thing and start looking at errors in my Fluke Network Analyzer, low and behold I notice and IP conflict on the secondary IP of my master switch and the secondary IP of my wireless lan controller. I fixed that and boom all of the AP's joined. Son-of-a-bitch. I'm going to stab that guy, he's been gone almost 8 months and he is still fucking with me.

Alright the internet was fixed ...

Frack, yesterday was bad enough with everything being full of suck playing with the router but today my upstream provider took a dump in the middle of the day.

Facepalm.jpg

For real, it's fixed then it isn't and so people call me.  Like I'm coated in magic and can make it work.

So my trouble yesterday was a specific firewall rule that was allowing some external partners work for the last 6 months just stopped working in the middle of the day.

So yeah into the router config we go.  It's a piece of crap Juniper so yeah it's not the cisco routers I've come to know and love (seriously if you are a monkey you can follow most of the logic of the cisco, except sub-interfaces that shit is dumb).  Anyhow I have logging on so so many policies and none of them including my block any policy has any traffic in it.

Why?  Because it's a Juniper and the logging makes not one lick of sense.  It doesn't log all traffic, only the things you specifically call out, and even then only when the session is closed.  All in and out traffic is logged in the in policies for instance.

Anyhow my net take away was I added TCP Any and UDP Any to see what damn port that stupid program wanted (TCP 49292 for the curious), and find myself asking how on earth did it work for the last 6 months?????!!?

Yeah, so apparently my router is leaky and sucks at logging which is the worst combination of failure I can imagine.  I'm all Hey Boss guy how about we get a Cisco, and he's all "Hmm, those are only programmable from the command line right?"

For real.

I'm currently neck deep in router bullshit

I'm thinking the best way to fix it is to pitch a hand grenade into the damn chasis, and go for a beer.


Side note, people are crazy pushy when the internet is borked.

J is for Just Finish It

By way of Lifehacker comes a post that is fairly pertinent to my life at large Just Finish It.

Much like the author of the post I don't struggle with ambition to start things, or lack vision to design a more grand universe.  What I do lack is the wherewithal to finish or carry out these dreams that live in my head.  My project list is ever growing in both my personal and work life; the sensation is overwhelming in someways.  A real commitment in the form of my son is robbing me of my late night productivity and lack of interest is having further negative impact on my ever increasing project list at work.

The short list includes migrate from exchange 2003 to 2010, continue Windows7 rollout, update backup exec from disk to tape to disk to disk to tape (de-duplication for the win!) and hopefully making my backup windows shorter than 22 hours, upgrade the last NT server and the software that it is running (sometimes legacy just won't die, it has in the neighborhood of 80+k reasons why it's been kicking around on the really old version of the software) Mobile device strategy (people really think tablets are going to be some form of productivity boost, I think they are smoking that funny tobacco) completing the office 2010 deployment, some server automation scripts to make the co-workers lives easier.  Then I have a backlog of documentation, and future network planning to complete (really should be certified Cisco to run the network I have, let alone the network I need).

So yeah.  Sometimes my work sits on my mind a lot, but I have almost no interest in doing about half of that shit I just wrote down.

H is for Holy WTFBBQ?

Google catches so much shit lately, I just don't understand how a company that dares to dream about shit like the following can catch so much hate.




Augmented cyborg reality (also that guy's a tiny bit of a douche), yes please!  Also self driving cars?

E is for Energy Crisis

Gizmodo pointed me to a happy thought piece on Wired called no solution to the energy crisis until we run out. The article is really worth a read if you have any vague interest in energy policy and how it will shape our lives, but the real meat of this article is an example of how government mandates can make companies do the impossible.

Short version of his allegory, is that there used to be a real problem with old refrigerators trapping and killing children.  Thing is, for some reason it is really fun to crawl into refrigerators and close the door, which would trap them and cause them to suffocate.  When they where told that this was a problem they blamed the victims, and said it wasn't a problem, or if it was a problem it wasn't theirs to fix, ect, ect.  Any of this sound familiar (we're an oil company why do we have to come up with the next fuel source.  We should focus on producing what we started as...)  Any way the refrigerator lobby tried to convince Congress it was a problem that couldn't be solved, well when the Refrigerator safety act passed they started making fridges that weren't little death traps.  Go figure, and they say Government cannot do anything right.  I'm starting to think, businesses cannot do anything right without a little sharp pointing stick pocking their bollocks to get them moving.

As an aside I only 31 but my parents totally had an old refrigerator and I most certainly did crawl into it and close myself in it so I can relate.  Not to worry it was antique and the motor had been removed so I had an air hole, scary but true story.

Rotary Tumbler

I remember from my childhood having a Rock Tumbler[?] when I was a kid and really enjoying the idea of polished stones.  It must have been a fad in the late 80's or early 90's but for some reason I was really into it for about a month, and then the first batch came out of the tumbler and I was all, meh.  What does this all have to do with?  Well make posted a project about making a rotary tumbler from printer parts and I for some reason got all nostalgic and super excited about making one.

During the course of finding a tumbler on Amazon, and reading reviews about tumblers I remembered my own experience and am now sad that everything from my childhood must have actually been really lame.  I'm almost afraid to watch an episode of He Man or god help me The Snorks (oh shit I did just watch an episode there is not god) or any other show that I was obsessed with as a child.

IT related post

Okay so it's about to get technical and ranty up in here so if you don't care for either thing piss off.

Last night my coworker and I stayed late to update an IBM Bladecenter S raid controller and upgrade the drives from 1tb drives to 2tb drives.  These drives are effectively the SAN for the blades so doubling the storage is kind of a big deal as we get ready to start adding more blades into this thing.  The trouble I had with this is the update process is really just bunch of bat files and python scripts that rely on default users that you cannot change and in typical IBM fashion it's not all built in house it just strung together oem parts glued together with code.  It was supposed to be a maintenance window of 6-8PM, but we didn't finish until 11:30pm and I have contractors in this morning (supposedly 30 minutes ago so some fuckers are about to get a phone call) all because of the kludgy as fuck nature of the update process I'm tired and cranky.

Also the SAS raid controllers on IBM bladecenters run MontaVista Linux professional version 3.1 on the Power PC platform and have a default userid and password and telnet is on by default.  For the technically inclined try to imagine what is wrong with this horrifying picture.  I can change the password, but telnet needs to be on (that's how you communicate the firmware updates ..... for serious) and I can hide the interfaces in VLAN's with ACL's which mitigates it to some degree, but I still bet this damn thing has busybox on it (it serves web pages and I think has an ssh interface) .  Anyhow the updates are done, and I'm tired but I'm rocking a 7.3tb array in one storage module bay(would be 9.1 but I assigned one drive as global hot spare) and a little shy of 2.8tb array on 15k rpm sas drives for running sql on

Neil deGrasse Tyson

I swear this is the next incarnation of Carl Sagan.  These two men fill my head with dreams for the future, and our place in the sky.  With just the money we spent on the largely failed next generation fighter jet $66.7 Billion could fund NASA into the next decade.  Thanks GAS for posting this, I just don't get tired of listening to this man remind us of the importance of space and what it means for our society.

SAN Managment

F*CKSTICKS I was working with my ISCSI SAN yesterday because one of the volumes was getting full (seriously why does everyone save so much data) and I needed to delete an unused volume and use that space to expand the nearly full one.  Well genius boy over here deleted the live one.  ISCSI is really sticky so even though the volume is deleted it is still serving data and let me restore from the Friday backup to the new larger volume and overnight I remapped the shares to the restored files and started syncing the two.

Net result no one but me will know, but there is nothing like the rush of pure adrenaline when you realize you just deleted 300gigs of users data at 3pm!  Sweet Jesus thank you so much for LTO4 Tape drives, I restored at 1.5gb/m so restoring 300 gigs was only about 3 hours but either way I'm exhausted and not going to be in charge of making any decisions today.

Fracking awards season

Alright let's make this quick, some douchebags tagged me InfinitePlans and R probably some other jerk too, but I'll get to you soon enough.

So whoever thought up this blog version of a chain letter decided that they could arbitrarily add rules to this award, and in an act of extreme arrogance assumed that people would follow them.  Here's the deal, most do but I expect it's because they are sheep.  So the three rules to accept this award are that I must:

• Link back to the person who gave you the award. 
• Pick 5 people deserving of the award and notify them on their blogs. 
• Post the award on your blog and spread the love.

The point of this is to hightlight blogs with sub-200 followers that you think deserve a second look.  Well I give to you my two.  Why two because rules are dumb and I'm gonna do two.

Lord Phorzen over at Cirnopoly, I can barely understand the blog but hey what's life without whimsy?  also Dreadnaught tagged him already, but I barely give a fuck.
KingDota at the latest tech news, title says it all.  Click here to read about the suff you already read about on slashdot, engadget or gizmodo already.
ToTipota wanted to get an award so here he get's one

And I apparently need a nap, 'cause I'm feeling surly.  so whoever liebster is I hope you have really bad heartburn tonight.

Tazer Ball

Someone somewhere just found the shortcut to getting a Darwin Award.

Dried Apples

So one of the blogs I'm following is trying to kill me, I will not be posting about that other blog or the beignet recipe that I am studiously pretending I didn't read about today.

Instead I will post about dried apple slices and the how healthy and delicious these look see here is a picture

Oh sweet jesus help me, I'm weak and those fried dough balls call out to me with the force of 100 fat kids.

Seriously checkout the apple thing though, that looks pretty do able without needing a dehydrator and apple chips are the most delicious things.  That dough is making me drool on my keyboard just a bit, but you should totally make the apples instead.

What in the holy f*ck indie?

ElectricMustache is messing with me today.  I cannot say how I feel about this song because the video was so filled with wtfbbq that I cannot imagine the song without the video.  I whole heatedly apologize for what is about to happen to your brain as it leaks out your ear holes and puddles on your desk.  Grab a towel and don't panic.

Cloud Nothings - "No Future / No Past" Official Video from Urban Outfitters on Vimeo.

So it begins

I knew it was coming, but it was always in that sometime in the future sense.  In two days I'm going to be staying at home with my son to take on my turn of full time child care.  Yikes.

I just hope we get along well enough that he doesn't scream at me all day.

exactly this.

I don't understand ostensibly sane people and the end of the world garbage.  Freak out, but I predict a precipitous drop off in the talk about the Mayans sometime around December 22, also when is there a time that isn't appropriate to drop an XKCD link? Just saying.

Copy run start

On command on a cisco switch would have saved me a decent amount of stress on Tuesday.  Vlans that you didn't create, using a VTP server that you only half understand coupled with power outages = fuck yeah wasted time and day.

The guy that was Cisco certified decided he didn't want to work here any more, I'm not certed, but have somewhere between one half and three quarters of a clue, but sometimes I forget important things like copy run start.

This means nothing to anybody I realize, but I'm kicking myself for it.
conf t
interface gigabitethernet 0/44
switchport mode access
switchport access vlan 112
exit
exit
copy run start  <--- Important last step you idiot.

Saveur is trying to kill me

I've suspected it for some time now, but this post proves it.  Saveur is trying to kill me. With food.  Dead.

Hot Crab Dip with Pita Chips

 Fuck.

 INGREDIENTS
3 tbsp. unsalted butter
1 tbsp. extra virgin olive oil
1 red bell pepper, cored and finely diced
2 cloves garlic, minced
1 shallot, finely diced
2 tbsp. flour
1 cup heavy cream
1 cup grated parmesan, separated
Juice and zest of one lemon
½ tsp. salt
½ tsp. freshly ground black pepper
1 tbsp. Worcestershire sauce
½ tsp. Old Bay seasoning
¼ tsp. celery salt
1 lb. lump crab meat, cleaned
1 scallion, finely sliced


INSTRUCTIONS

1. Preheat oven to 350. Butter the inside of a 8" x 8" baking dish. 
2. In a 12" skillet, melt the butter and olive oil over medium-high heat. Cook the bell pepper and shallot until soft, about 5 minutes. Add the garlic and cook for 1 minute. Add the flour and cook, stirring, to make a roux, about 3 minutes. Whisk in the heavy cream until the mixture comes to a boil, about 1 minute. Reduce to a simmer, cook until thickened, about 1 minute. Remove from the heat and add 1/3 cup parmesan, lemon juice and zest, salt, pepper, Old Bay, and celery salt. Gently fold in the crab. 
3. Transfer the mixture to the buttered baking dish. Sprinkle the top with the remaining parmesan and bake for 20-25 minutes, or until the top is browned and the dip is bubbling. Garnish with the scallions and serve with pita chips. 

 NOTES: To make your own pita chips, cut fresh pita pockets in wedges and toss with 4 tbsp. of extra virgin olive oil, salt, and pepper. Cook at 400° for 12-15 minutes.

SOPA

I think I've mentioned it before, but given how unbelievably poorly this law is shaping up I thought it was time to post a little round up and information on what you can do should you find this proposed bill as abhorrent as I find it.

First Gizmodo has a pretty solid run down on why this bill is murderous for the internet as we know it.  The gist of it is that because sites that host infringing content typically are not US sites, court orders and take down notices do not have force of law.  To make it so rights holders have a way to see their interests looked after (keep in mind who the government works for here kids) they can have that sites DNS records blocked or erased without due process (remember foreign entities) with no appeal process.  This fundamentally undermines the way that DNS and the internet works and is from an infrastructure point of view a very bad way to implement this.  We would be following in the footsteps pioneered by The Great Chinese firewall.  Not the best role model, I would say.  Next up there is third party transfer of liability, that is a legal way of saying that if the search engines don't expunge records that rights holders get taken down fast enough they can be sued for the infringement that they are "facilitating".  Finally the whole framework amounts to a tool that can and in the hands of companies like UMG will be used to stifle free speech.  It literally blows my mind that a congress that claims to be in the business of lessening the power of government would like to handover such a huge power grab to private companies.  Or it would blow my mind if it wasn't business as usual over in congress.

Gizmodo has a link to a petition you can sign to ask Obama to veto the bill.

Gizmodo also has a story up of all the companies that are supporting this bill so you can know if you still want to give them your patronage.  The most notable of them is the Registrar GoDaddy, they have caught a decent amount of flack for this so I don't feel the need to pile on any more.  I will note that I have chosen to take my business elsewhere.  Yeah broken websites for a little while, but I think it is important enough to take the time to do.

Now I would like to post a link to a video that has a pretty decent attempt from one of the bills co-sponsors on what they want the bill to do.  They are trying to extend a physical metaphor to police the internet is the short story.  Once again Congress just doesn't understand the technology they are trying to regulate and should not be allowed to work on it.

Not covered in this post is the related and already passed 2008 Pro-IP law used to take websites down currently.  This is a really strange use of ICE and Department of Homeland Security, (read excessively broad interpretation of the Patriot Act powers).  Some people have gone as far as suggest that the current USAG should face criminal charges for his mis-use of laws.

You kind of knew it had to be true

So I'm about to spoil a great movie for you and I have to apologize for that.  It's not that I want to, but the video shows Raiders of the Lost Ark is a shot for shot recreation of another movie.  I had to stop in disgust after about 5 minutes.

Damn you Indy!!!!



Source: Binary Bonsai.  This guys is a huge star wars fan, his archives have fan made documentaries about the Star Wars movies that are better than anything Lucas would produce himself.  If that's your gig, you should really give this guy a read.