Networks are hard

So yeah, I got clobbered today.  A change an ex-coworker made to a switch config nearly a year ago kicked my ass today.  Ain't that a bitch?

Long story short see above, much longer version is around 10am I was having a conversation near one of my Wireless Access points.  It's a strange habit I'll admit, but I tend to look at it whenever I'm around it.  Why you ask?  Well these ones have handy dandy status lights, you can see how they are behaving by the color of the light, green is good with no clients blue is good with clients and anything in a red-ish hue is bad.

Well I noticed mine was magenta.

Typically not a huge deal, sometimes they dissociate from my wireless controller and eventually re-connect at their leisure.  Around 11am someone mentioned the wireless was not working so I looked into it.  I had traps in the log that told me around 10:18am every single AP disjoined the controller, not reason given.

Ruh Roh!  Out comes the investigator hat.

I reboot the controller and the configuration server, power cycle the AP closest to me and it joins the controller no problem, so I'm thinking hey that's not so bad.  Trouble is only a handful of my 51 AP's rejoin, and an hour later I only have 5 AP's and 10% isn't going to cut it.  Plus some people are starting to notice.

I have about 1 million square feet (I shit you not) of campus to try and cover so if one AP is down it's not a huge deal, but 46 kind of is.  So I pull out my magnifying glass and thinking pipe (whatever props help me think) and work on probing the recesses of my damn network.  I've got Cisco's network adviser running, a spreadsheet of IP addresses, 5 terminal sessions and my fluke network analyzer looking for issues.  I tracing links and verifying that trunks are working, making sure no ports are flapping, double checking vtp server settings, verifying STP is not doing something stupid, still only 12 AP's are joined and it's like 2pm.  I'm starving and not able to think any longer so I do the sensible thing and go get a Jimmy John's sandwich and play some sodoku.  It was a bad decision, it's Thursday and Sodoku is really hard when you're

1. distracted
2. hungry
3. stupid
4. tired

None of this stops me of course, and I fail at sodoku as I knew I would and return to my desk. At this point I still only have 12 AP's and I don't know why, so I do the sensible thing and start looking at errors in my Fluke Network Analyzer, low and behold I notice and IP conflict on the secondary IP of my master switch and the secondary IP of my wireless lan controller. I fixed that and boom all of the AP's joined. Son-of-a-bitch. I'm going to stab that guy, he's been gone almost 8 months and he is still fucking with me.