Data Clean Up

So you have some old equipment laying around the home. It is temping to simply take your old PC to your recycle center and do the right thing. Just be sure you erase anything left on the old hard drive.

You can download software that scrables the hard drive and that takes hour. You can remove the partition and hope for the best, or you go old school. A hammer always does the trick. Remove the hard drive and smash it with the front and backside of the hammer. Be sure to make an good indention in the case. Shake it and you should hear stuff rattling around. Job well done.

Now its ready for the recycle center.

EMC Clariion CX4-240 Self Install

OK. So you are working on getting a new storage system. If you're an EMC shop, you can do a self install of the CX4, and possibly save yourself a bunch of $$. Why do a self install? Well aside from the $$ savings, its easy, and you can do it way faster then EMC can coordinate and schedule an installation SE/CE.

A CX4 consists of a SPS, SPE, and multiple DAE's. The SPS is standby power supply that sits at the bottom and powers the SPE and the first DAE. The SPE or "Storage Processor Enclosure" contain two processors or SP's called SPA and SPB. Basically two storage processors to run the array. They are independent and if one fails, the other one continues to run the Array. It is very rare to have a multi-SP failure. It can happen, and would be more likely a bug, but its not likely. That is what makes the Clariions so reliable. They run and run and run. The last thing is the DAE or "Disk Array Enclosure". This holds all the drives. One of the DAE's you will receive is special in that it contains "Vault Drives" and vault drives contain the OS the Clariion boots from. This needs to be the first DAE attached to the SPE.

The EMC install guide is very good and if you follow it, you'll have a basic CX4 running in a few hours. So working from the bottom up. You'll get the following installed

• SPS. Install this on the bottom. If its in a cabinet, leave 1U space on the bottom. But an airflow cover over the space on the front if you have one. If you ever need to replace an SPS, you will need the space, or its a ?itch to replace in a fully loaded cabinet.
  
• SPE. On top of the SPS, install the SPE.
• DAE. On top of the SPE, install the DAE containing the vault drives.
  
• DAE - The Rest of them. Install the rest of the DAE's.

SPA/B and Backend Cabling

The backend cabling is the first cable job. Depending on the CX4 model, you will have either 1, 2, or 4 backend ports. A CX4-120 has one, a CX4-240 has 2, A CX4-480 had 4. Get it?

1st DAE. Now follow the install guide and start with SPA. Take the appropriate cable (SFP to HSSCD2 Cable) and plug it into the slot 0, port 0 on SPA and the other end to the corresponding primary port (PRI) on the 1st DAE. Be sure this DAE has the specially marked "array software" label across the 1st five disks. Then do the same for SPB

2nd DAE. Connect slot 0, port 1 on each SP to the 2nd DAE. Stop at the 2nd DAE for now. You should have the SPS, SPE, and two DAE's cabled. That is good to start. Be sure to review step 1 in the install guide and ensure the cabling is right. This is the hardest part. If its not, the SP's can not locate the vault drives and they will not boot.

Power is Next - To Equipment that is.

The next thing is to get the power cables installed. Be sure the SPS's are off. Its best to use a separate power source to each SP. i.e. Do not run from the same PDU. Put them on separate PDU's, breakers, etc. You will see that their is really two SPS's and they fit in a single tray. One SPS powers each SP. So once you have power to the SPS right under SPA, you will run a power cable from the SPS to SPA. Then another from the SPS to the power on the 1st DAE. Once you do this, you do the same for the other SPS and SPB and DAE. Once you have the SPS powering the SPE and 1st DAE, go a head and run another set of power cables from your PDU to the 2nd DAE. Once you do that, the 2nd DAE will power up and that is ok. The SPS should still be turned off and both the 1st DAE and SPE will be off. That is what we want.

There is also a SPS serial cable. One end plugs into the SPS and the other end plugs into the SPE. And there are two. One for each SP. In the install quide its got a picture and it is self explanatory.

Apply power and boot it up. Flip on each SPS. Remove the front bezel's and there are two lights for the SP. They flash during boot then go off. The bad light is the blinking yellow light. That tells you there is a fault. If any of your backend is wrong, that is likely the cause of the fault. Reexamine your cabling. It's usually a calbe that causes it to fail and not boot.

Did it boot? It should have. Look on the back. Thers is a light above each model. There is a management model, backend module, and iscsi module. You really want the mgmt and backend modules to be green. If not, back to the cable. Review it. It's likely wrong. If it is. Go ahead and shutdown the Array by turning off each SPS and waiting for it to shutdown the Array. It can take 2-3 minutes. Then fix the cables and start it back up. Once it comes up and all the yellow lights are out and the rest are green, then your ready to initalize it.

Recommended Reading

EMC Clariion CX4-240 Setup Guide (Requires Powerlink Access - You have that? Right!). This has all info needed to setup. Diagrams, pictures, etc. Just take your time. It is not very hard. Concentrate on the cabling and you will have no problem getting the physical installation going.

Next Steps. I'll post info on initializing the Array and how to troubleshoot.

Accidently Delete Entire OU of Computer Objects

We are so used to doing multiple things at once, multitasking as they say, and every now and then it bites you. So today, while doing some cleanup today in AD, one of my coworkers may have gotten a little overzealous, aka not paying full attention. He meant to delete a single computer object, but accidentally deleted an entire OU containing a significant numbers of Windows server computer objects.…. Serious DOH!!!!!! So for the sake of this article, let’s call him “JOE”

So it’s not the mistake, but how you react to it, because we have all made them before.

The good news is that we have multiple AD servers, and one is placed in a separate data center for DR purposes, and that is how we backed out of this Dooh! We leveraged the AD server in the remote data center as it had not received an update to the change yet. Why? When we configured it, it’s on a WAN segment, and therefore changes are batched and updated every three hours.

So Joe is a really good IT dude and just goofed. He is also smart and got on it right away. Joe did the following. They are at a high level, but you will get the idea.

• Went to the remote AD server and turned off receiving replication. So it will not get the change.
• Changed the update sequence number for the OU on the remote AD server. This is done via authoritative restore via Directory Services Recovery Mode.
• Then he was able to replicate everything back.
• And everything appeared to be good. And it was!

One Key! You have to have access to Directory Services Recovery Mode. This requires a password and it is set when you promote the server to an AD server. So you have to have the psw. If you do not have it, it’s easy to set. You just need to be able to login. So if you do not have it, or not sure, I suggest you set it and have it handy. It can bail you out of all sorts of goofs.

Recommended reading

Performing an Authoritative Restore of Active Directory Objects @ http://technet.microsoft.com/en-us/library/cc779573(WS.10).aspx

How To Reset the Directory Services Restore Mode Administrator Account Password in Windows Server 2003 @ http://support.microsoft.com/kb/322672

NAI/Mcafee Bad DAT

NAI/Mcafee, one of the biggest Anti Virus Vendors, goofed and sent out a bad virus definition update. This happened on 4/21 in the early afternoon.

The problem is a false positive which identifies a regular Windows binary, “svchost.exe”, as “W32/Wecorl.a”, a virus. The affected systems will enter a reboot loop and [lose] all network access.

We caught it pretty early and manually pushed out a fix @ approximately 3:45.. If you happen to have received the bad definition file, and shutdown your computer before you got the fix, then your computer may not start properly and will need a bit of work to repair. Unfortunately the bad definition file told the virus scan software that one of the critical system files had a virus. Bummer!

We used our epolicy console to push out the new DAT and verified that each computer got the update and installed it. The console is great for reports. Out of 500 computers, we only had one user hosed. And we fixed it in 10 minutes. BAM!

See https://kc.mcafee.com/corporate/index?page=content&id=KB68780

Improving Customer Serice - The old PBX is not old

Improving Customer Service

This year @ ACC2010 I wanted to improve the way we handled telephone calls and tracked requests. You see, each year, we put together 8 networks, 125 locations, 500 computers, 1000’s of wireless users, and make this all happen at one of the largest convention centers in the US. This year we are in Atlanta @ the George World Convention Center and the folks at CCLD handle the technology. We knew we had the center, partners, and people to help us step up our game.

We pretty much spend the year before handling all the logistics and nailing down requirements for it all. So once we are on site, we make it happen, and had tracked calls the old fashion way. Pen and Paper. This year we rolled our own portable PBX and contact system. We used an Asterisk based PBX via a cool ISO called PIAF or “Pbx in a flash”, Aastra handsets, and VOIP providers and local analog lines. We also used an open source CRM called “SugarCRM”.

What we Got!

Portable PBX supporting IVR and Queues

• Instant telephone system for 10 users
• Time of day routing, Queues, and custom prompts
• Voicemail and email notifications
• CRM tracking of all requests
• CRM reporting of all requests
• Improved customer service – Less chance of someone falling through the crack

Did it work?

Yes. Worked great. Dave got on site a week before the meeting started and got working to get our mini Network Operations Center (NOC) office setup and running. He received 4 skids worth of equipment, worked with the center and prime contractor (GES) to set the room, complete with desks, conference table, etc. Basically a working office away from home. Then Dave rolled out our tried and true network consisting of a pair of Cisco ASA5510 firewalls, switches, and access points. We utilize the centers network and access points as it is a comprehensive network built into every part of the center, but we augment several VLAN’s with our own private network in order to create a secure and private network for our own use. We then use a corporate VPN strategy to get critical offices connected back to HQ.

I rolled in a few days later and got the VOIP network setup. As Dave already had the network setup, I simply plugged in the server, handsets and bingo, we were online. I had already setup and tested in our lab back @ HQ, so I was pretty sure it would work. I did need to update the code on an analog gateway on site, but that was pretty much it.

We then started to receive calls and Chad, our lead helpdesk guy, used CRM to track requests, changes, and support issues. Overall, it went great, and we expect to use it for ACC2011 in New Orleans. Maybe with another team so more folks providing service for the meeting can take advantage of it.