I recently completed a couple of vSphere upgrades for two different clients. One interesting project was from version 4.0 to 5.1 for a large enterprise running an internal facing infrastructure across two sites (active active).
A very nice solution conceptually, with stretched VLANs and vMotion between two physical locations.
Part of my engagement was to advise on scalability – one of my concerns was the vCenter virtual machine running SQL express and as a 32 bit server OS. Not a great foundation for a modern vSphere platform, however as it contains the object hierarchy, a minimal impact migration was a required deliverable with the new vCenter configuration as it was a core toolset for the IT Ops team.
The existing platform was running several hundred VMs across a 8 node cluster, 4 nodes in each site. Both VMware HA, and fully automated DRS was running. The use of RVtools and vCheck aided the current- state analysis and allowed rapid understanding of the platform and configuration.
As part of the planning phase it was determined the downtime or risk periods available would be quite short (risk period being hours, and VM downtime a few minutes) . With the heavily consolidated platform workloads needing addressing within the upgrade window also.
A business requirement stated that the work needed a roll back plan for each phase Although this is assumed 🙂
There are a number of possible approaches to complete this type of upgrade, however the one I used is shown in the high level upgrade plan below,;
Pre- migration work
1. Create a new remote SQL 2008 virtual server
2. Create a new database for vCenter 5.1
3. Create a system DSN ODBC connection to the SQL Server.
4. Install a new vCenter on v 5.1
5. Build a new separate VMware update manager instance and link to new vCenter.
6. Clone the existing vCenter running 4.0 for a DR point.
7. Snapshot the vCenter configuration using the inventory snapshot fling
8. Backup a host configuration using vicfg-backup
9. Restore the inventory snapshot configuration using powershell scripts from step 7.
10. Create a static baseline on vSphere update manager
11. Attach the baseline to the cluster
12. Scan the cluster
13. Remediate hosts in the cluster.
14. Upgrade VMware tools using a similar static baseline for VMs.
At this point The system was tested and left for a 24 hour period for VM backups to complete etc. This was a business driven decision , based on a requirement to cover an easy mass roll back plan for VMware hardware compatibility.
VM Migration Work.
14. Upgrade VMware hardware compatibility
15. Upgrade VMFS v3 to v5 using a scratch datastore and storage vMotion.
Overall this approach was clean, and successful. The minor issues I came across were DRS groups and some reservation minimums in the inventory snapshot migration , the settings were not completed carried over. I picked this up in a validation step at implementation time.
Although successful for me, please note that in Inventory snapshot is not a supported tool. I mitigated against this to meet requirements with a clone of the vCenter virtual machine containing the database, and vCenter config and a backup of a host in the cluster using vicfg-backup.
The final steps 14 and 15 were rolled in at a maintenance window and resulted in no issues.
The term upgrade normally causes worry. Ideas of data loss, or corruption.
VMware upgrades when planned can have very little impact to the end user. One of the comments I had from on of the end clients was “That seemed easy”.