How to upgrade a VBlock and not screw it up!

I have found myself over the past year doing a lot of VBlock upgrades and wanted to share some of my experience on the subject. Even though a VBlock upgrade can be a daunting task, it will be much easier if you have a game plan and are well prepared. Here is a high level overview of the tasks needed and some things to think about along your journey.

Decide on what matrix level you want to go to

 

 

The first step in doing a upgrade is deciding what matrix level you want to go to. There are a lot of different factors that may drive the necessity to do an upgrade.  You want to keep up on the latest bug fixes and security patches. Maybe your company just bought some new blades that are not compatible with the UCSM 2.0 version you are on or you are interested in a new feature found in UCSM 2.1. Maybe you want to upgrade to vSphere 5.5 or your Matrix version is falling out of support. Unfortunately with the VBlock you cannot just upgrade one component to whatever you want. VCE requires that you stay compliant – all component firmware levels must match a matrix version. Although this can be very annoying sometimes, the benefits outweigh the negatives in that it takes the guess work out of making sure everything will be bug free, compatible and work as intended.

A healthy VBlock is a happy VBlock

Before attempting a VBlock upgrade make sure your environment is healthy.  You can reduce your risk during an upgrade if your environment is free of bad practices and errors. I have walked into a few upgrades where things were delayed hours because Hosts in a cluster did not have access to shared storage, Hosts did not have multiple paths to the SAN or something as simple someone naming a service profile wrong in UCS; which unintentionally caused the reboot of a Host with live VMs (ouch!). Check and take care of all this little things before hand because they can bring a upgrade to a grinding halt!.

Read the Documentation!

Did I say read the documentation? I can not stress enough how important it is to read the documentation pertaining to the your upgrade. This includes but not limited to upgrade guides, release notes and blogs on the components you are upgrading. VCE provides release notes for each matrix level which can be found on their support site. They also provide upgrade guides for major versions of vSphere and VBlock systems which can be helpful in determining the correct upgrade path.

Upgrade in the right order

In order to upgrade the Vblock in the least disruptive manner and ensure a trouble-free upgrade it is recommended to follow the upgrade path that VCE prescribes. This ensures that any compatibilities and dependencies are met before moving on to the next major component.

A VBlock 320 upgrade would typically go in this order:  This is just one example, please refer to recent documentation for matrix upgrade path for your specific code levels.

• Cisco UCS
• EMC VNX
• Nexus 5ooo
• Cisco MDS Switches
• Cisco 1000v
• Cisco UCS C-Series Rack Servers (firmware)
• Cisco Catalyst Switches
• VMware vCenter Server, Client and ESXi Servers

Sub components have their own upgrade path. Take for example UCS. It is best practice to upgrade the UCS Manager first, then the Fabric interconnects, then the blades. This goes back to why reading the documentation is so important.

Create your own documentation

Document the process. Armed with information, make a spreadsheet comparing the old code to the new code. Document the upgrade path, things like who is responsible for what with dates and when each component will be completed. Assess the risk of each component upgrade and the potential for downtime so you can schedule the appropriate resources to support the task. Gather usernames and passwords, Database credentials and service accounts and any other info you will need during the upgrade. Having this handy saves time. It sucks having to waste an hour in the middle of an upgrade while some one looks for the password for a service account that was created 2 years ago.

Prepare yourself for battle

 

 

Now that you spent hours reading upgrade guides, release notes and blogs or maybe you watched some you tube videos on how to upgrade the firmware on your UCS or how to upgrade from vSphere 5.0 to 5.1, you are now ready for the big day. Although I have done Vblock upgrades over a single weekend, I can tell you from experience it comes with a lot of stress.

It is OK to upgrade your VBlock over several maintenance windows. VCE will not stop supporting you if your VBlock is temporarily out of compliance during an upgrade. Depending on how big your environment is, the upgrade could take several maintenance windows. One of the most time consuming parts is rebooting hosts after upgrading your UCS firmware and after upgrading ESXi. Depending on how many blades you have and the amount of resources available to shuffle around VMs this could take a few hours up to a few days. VCE provides estimates on how long each part will take, from my experience they are a little conservative, add at least 50%.

Download the code ahead of time. If this is first upgrade you are in luck. VCE now has a download center on their support site where you can download every piece of software pertaining to your matrix level. Gone are the days of going to cisco, powerlink and vmware.com to look for those hard to find executables and ISOs.

Be Proactive!

You paid for support, so use it! Contact your customer advocate if you need to they are their to help. They can assist in finding documentation or providing you with a TE to answer questions about your upgrade. It is also a good idea to open a proactive SR for your upgrade. That way if you mess something up that bad you already have a case open with VCE and can get someone on a Webex that much sooner to help you out.

Celebrate with shiny new firmware

Now that you have upgraded your VBlock, bask in its warm glow and revel in a job well done.