Some quick notes and a nice work around for achieving a smooth NSX for vSphere upgrade as gleaned from upgrading my cross-vCenter home lab from v6.4.5 to v6.4.6.
Overview
Pre-Requisites
Check the VMware Product Interoperability Matrices. Check this matrix for NSX/vCenter/ESXi interoperability.
Ensure that your vCenter and ESXi servers are running compatible versions prior to starting the NSX upgrade!
Check the upgrade release notes. See here for all VMware NSX Data Center for vSphere release notes
Download
Download the upgrade bundle from the VMware download site
Once downloaded, confirm the check sum using PowerShell:
get-filehash VMware-NSX-Manager-upgrade-bundle-6.4.6-14819921.tar.gz
Confirm SHA256SUM value matches that given on the VMware download site
Upgrade Steps
NSX Data Center for vSphere components must be upgraded in the following order:
- Primary NSX Manager appliance
- All secondary NSX Manager appliances
- NSX Controller cluster
- Host clusters
- NSX Edge
- Guest Introspection
- Post-Upgrade Tasks
The top level VMware upgrade guide is available here
Upgrade Primary NSX Appliance
When: Anytime
How: Follow this guide
Further Notes: None
Upgrade All Secondary NSX Manager Appliances
When: Anytime
How: Follow this guide
Further Notes: None
Upgrade NSX Controller Cluster
When: During a maintenance window
How: Follow this guide
Further Notes: The VMware guidance of performing the upgrade “during a maintenance window” seems a little conservative. As long as there are no NSX changes being made whilst the upgrade is taking place, then I suggest that this can be done anytime
Upgrade Host Clusters
When: During a maintenance window
How: Follow this guide
Further Notes: This is where my one host per cluster lab had issues.
The problem was that the upgrade process wanted to put the host into maintenance mode to upgrade the ESXi NSX VIBs but couldn’t - because it had no where to vMotion the VMs to! (remember - only one host per cluster!)
Work Around
What follows is the NSX VIB upgrade process if for single host “lab” style clusters. Certainly in production you will have more than one ESXi host per NSX cluster, so this work around would (should) not be needed.
The solution was to upgrade the NSX VIBs manually, and then reboot the host and all it’s VMs to enable the updated NSX VIBs. Luckily I found this post that details a method to obtain the updated NSX VIBs from NSX manager.
For my lab upgrade to 6.4.6, I Opened https://nsx-site-a.lab/bin/bin/vdn/nwfabric.properties in a web browser to find the updated NSX VIBs:
Downloaded the VIB from https://nsx-site-a.lab/bin/vdn/vibs-6.4.6/6.7-14762108/vxlan.zip using IE as Chrome would just open the file
Extracted the VIB from the folder \vib20\esx-nsxv inside the zip:
WinSCP’ed the VMware_bootbank_esx-nsxv_6.7.0-0.0.14762108.vib file to the ESXi host (after enabling SSH on the ESXi host)
Shutdown all VMs, placed host into maintenance mode, SSH’ed to the host and upgraded the NSX VIB using the following command:
# esxcli software vib update -v /tmp/VMware_bootbank_esx-nsxv_6.7.0-0.0.14762108.vib
Rebooted my ESXi host, removed host from maintenance mode, booted all the VMs and boom!
Repeat for the other ESXi host at the other site. Double boom!
Upgrade NSX Edges
When: During a maintenance window
How: Follow this guide
Further Notes: More of an edge replacement process; a new Edge virtual appliance is deployed alongside the existing one. When the new Edge is ready, the old Edge’s vNICs are disconnected and the new Edge’s vNICs are connected.
Upgrade Guest Introspection
When: During a maintenance window
How: Follow this guide
Further Notes: I don’t run guest introspection in my lab, so did not complete this step
Upgrade NSX Services That Do Not Support Direct Upgrade
When: Depends on service(s)
How: Follow this guide
Further Notes: Nothing to do here as I don’t run any other NSX related services
Post-Upgrade Tasks
When: Anytime
How: Follow this guide
Further Notes: We know that our VIBs are OK as we manually installed them. I did not bother with resynchronising the host message bus as I had rebooted everything anyway!
Conclusion
In this post we learnt how to upgrade VMware NSX and the process required to complete an upgrade. We also worked around the issue of upgrading single host clusters and completed the upgrade without issue.
Until next time
-Chris