Chris Hall bio photo

Chris Hall

Making technology fit my requirements...

Windows, Linux, Virtualisation, Unified Computing & Cloud Offerings Design Engineer.

PolarCloudsUK Chris LinkedIn Github

pfSense + OpenVPN Some quick notes and a nice work around for achieving a smooth NSX for vSphere upgrade as gleaned from upgrading my cross-vCenter home lab from v6.4.5 to v6.4.6.

Overview

Pre-Requisites

Check the VMware Product Interoperability Matrices. Check this matrix for NSX/vCenter/ESXi interoperability.

Ensure that your vCenter and ESXi servers are running compatible versions prior to starting the NSX upgrade!

Check the upgrade release notes. See here for all VMware NSX Data Center for vSphere release notes

Download

Download the upgrade bundle from the VMware download site

Once downloaded, confirm the check sum using PowerShell:

get-filehash VMware-NSX-Manager-upgrade-bundle-6.4.6-14819921.tar.gz

Confirm SHA256SUM value matches that given on the VMware download site

Upgrade Steps

NSX Data Center for vSphere components must be upgraded in the following order:

  1. Primary NSX Manager appliance
  2. All secondary NSX Manager appliances
  3. NSX Controller cluster
  4. Host clusters
  5. NSX Edge
  6. Guest Introspection
  7. Post-Upgrade Tasks

The top level VMware upgrade guide is available here

Upgrade Primary NSX Appliance

When: Anytime
How: Follow this guide
Further Notes: None

Upgrade All Secondary NSX Manager Appliances

When: Anytime
How: Follow this guide
Further Notes: None

Upgrade NSX Controller Cluster

When: During a maintenance window
How: Follow this guide
Further Notes: The VMware guidance of performing the upgrade “during a maintenance window” seems a little conservative. As long as there are no NSX changes being made whilst the upgrade is taking place, then I suggest that this can be done anytime

Upgrade Host Clusters

When: During a maintenance window
How: Follow this guide
Further Notes: This is where my one host per cluster lab had issues.
The problem was that the upgrade process wanted to put the host into maintenance mode to upgrade the ESXi NSX VIBs but couldn’t - because it had no where to vMotion the VMs to! (remember - only one host per cluster!) :flushed:

Work Around

What follows is the NSX VIB upgrade process if for single host “lab” style clusters. Certainly in production you will have more than one ESXi host per NSX cluster, so this work around would (should) not be needed.

The solution was to upgrade the NSX VIBs manually, and then reboot the host and all it’s VMs to enable the updated NSX VIBs. Luckily I found this post that details a method to obtain the updated NSX VIBs from NSX manager.

For my lab upgrade to 6.4.6, I Opened https://nsx-site-a.lab/bin/bin/vdn/nwfabric.properties in a web browser to find the updated NSX VIBs:

VIB Locations

Downloaded the VIB from https://nsx-site-a.lab/bin/vdn/vibs-6.4.6/6.7-14762108/vxlan.zip using IE as Chrome would just open the file

Download VIB

Extracted the VIB from the folder \vib20\esx-nsxv inside the zip:

Extract VIB

WinSCP’ed the VMware_bootbank_esx-nsxv_6.7.0-0.0.14762108.vib file to the ESXi host (after enabling SSH on the ESXi host)

WinSCP VIB

Shutdown all VMs, placed host into maintenance mode, SSH’ed to the host and upgraded the NSX VIB using the following command:

# esxcli software vib update -v /tmp/VMware_bootbank_esx-nsxv_6.7.0-0.0.14762108.vib

Upgrade VIB

Rebooted my ESXi host, removed host from maintenance mode, booted all the VMs and boom! :boom:

VIB Done

Repeat for the other ESXi host at the other site. Double boom! :boom: :boom:

Upgrade NSX Edges

When: During a maintenance window
How: Follow this guide
Further Notes: More of an edge replacement process; a new Edge virtual appliance is deployed alongside the existing one. When the new Edge is ready, the old Edge’s vNICs are disconnected and the new Edge’s vNICs are connected.

Upgrade Guest Introspection

When: During a maintenance window
How: Follow this guide
Further Notes: I don’t run guest introspection in my lab, so did not complete this step

Upgrade NSX Services That Do Not Support Direct Upgrade

When: Depends on service(s)
How: Follow this guide
Further Notes: Nothing to do here as I don’t run any other NSX related services

Post-Upgrade Tasks

When: Anytime
How: Follow this guide
Further Notes: We know that our VIBs are OK as we manually installed them. I did not bother with resynchronising the host message bus as I had rebooted everything anyway!

Conclusion

In this post we learnt how to upgrade VMware NSX and the process required to complete an upgrade. We also worked around the issue of upgrading single host clusters and completed the upgrade without issue.

Until next time :thumbsup:

-Chris