Skip to main content

vSphere 6.0 w/ NSX-v 6.2 upgrade to vSphere 6.5 w/NSX-v 6.3 - EAM Agency issue

Problems with EAM (VMware ESX Agent Manager) and NSX

After upgrading ESXi 6.0 there can be problems with the NSX-v installation VIBs.

Symptoms

In the Installation section, of Network and Security the installation status of the host(s) will show as not ready. When you click the gear icon next to the host and attempt to run the “resolve” operation, it fails repeatedly. 
In the administration page of the vCSA under vCenter Server Extensions, double click on vSphere ESX Agent Manager and select the agency for the appropriate cluster. If you see the issue that “Agent VIB module is not installed” 
To confirm, ssh to the vCenter server and check the EAM log:
[msherian@MARS-E550 ~]$ ssh root@vc-01
VMware vCenter Server Appliance 6.5.0.11000
Type: vCenter Server with an external Platform Services Controller
root@vc-01's password:
root@vc-01 [ ~ ]# less /storage/log/vmware/eam/eam.log
2017-11-15T18:52:39.932Z |  INFO | host-1886-2 | VibJob.java | 288 | Installing VIBs on host: pesx-01.lab.adms.local (AgentImpl(ID:'Agent:d7006ecf-ce63-490d-a3c4-4a130c
ba2e94:null'))
URL: https://172.16.5.120/bin/vdn/vibs-6.3.4/6.5-6777105/vxlan.zip
Vibs:
[Vib [VibHeader [id=VMware_bootbank_esx-nsxv_6.5.0-0.0.6777105, VibId [name=esx-nsxv, version=6.5.0-0.0.6777105], vendor=VMware, summary=NSX datapath and host tools, null, release date=Sat Sep 30 17:36:24 GMT 2017], payload=https://vc-01.lab.adms.local:443/eam/vib?id=e61fdbde-7661-4880-82b2-55e2e43567f9]]
2017-11-15T18:55:33.823Z |  INFO | vim-async-0 | VimLRO.java | 54 | 'Task:task-284129:9098ceea-5e54-4142-bde1-a25c91f7b7a0' completed with result vim.host.PatchManager.Result {
   xmlResult = '<esxupdate-response>
<version>1.50</version>
<error errorClass="LiveInstallationError">
  <errorCode>29</errorCode>
  <errorDesc>Live vib installation failed. An immediate reboot is required to clear live vib installation failure.</errorDesc>
  <vibs>[]</vibs>
  <msg>Error in running ['/etc/init.d/nsxv-vib', 'stop', 'upgrade']:
Return code: 1
Output: ERROR: ld.so: object '/lib/libMallocArenaFix.so' from LD_PRELOAD cannot be preloaded: ignored.
Unloading datapath
Clearing extraConfig properties on "Management - VIO"
Disabling VTEPs on "Management - VIO"
Clearing opaque properties on "Management - VIO"
Failed to clear vdr dvptracking property [rc=3]: ioctl failed: No such file or directory
Operation failed: status = 0xbad0003
 
Clearing extraConfig properties on "vSAN-DVS"
Disabling VTEPs on "vSAN-DVS"
Clearing opaque properties on "vSAN-DVS"
Failed to clear vdr dvptracking property [rc=3]: ioctl failed: No such file or directory
Operation failed: status = 0xbad0003
 
Clearing extraConfig properties on "nsx-dvs"
Disabling VTEPs on "nsx-dvs"
Clearing opaque properties on "nsx-dvs"
Cleared VDR dvptracking property on "nsx-dvs"
Cleared VXLAN UDP port property on "nsx-dvs"
Cleared VXLAN overlay property on "nsx-dvs"
Clearing extraConfig properties on "vmo-dvs"
Disabling VTEPs on "vmo-dvs"
Clearing opaque properties on "vmo-dvs"
Failed to clear vdr dvptracking property [rc=3]: ioctl failed: No such file or directory
Operation failed: status = 0xbad0003
 
Clearing extraConfig properties on "Management"
Disabling VTEPs on "Management"
Clearing opaque properties on "Management"
Failed to clear vdr dvptracking property [rc=3]: ioctl failed: No such file or directory
Operation failed: status = 0xbad0003
 
Unloading kernel modules
Failed to unload nsx-dvfilter-switch-security [rc=1]: vmkload_mod: Can not remove module nsx-dvfilter-switch-security: module symbols in use
 
Retry unloading nsx-dvfilter-switch-security...
Failed to unload nsx-dvfilter-switch-security [rc=1]: vmkload_mod: Can not remove module nsx-dvfilter-switch-security: module symbols in use
 
Retry unloading nsx-dvfilter-switch-security...
Failed to unload nsx-dvfilter-switch-security [rc=1]: vmkload_mod: Can not remove module nsx-dvfilter-switch-security: module symbols in use

Resolution

In order to resolve the issue we have to disable the kernel modules that are preventing the upgrade of the VIB, I chose then to remove the VIB fully and let EAM run the install. Another, known issue is a conflict between EAM and VUM, please see https://kb.vmware.com/s/article/2053782
Per the above KB, reconfigure the EAM to by-pass vSphere Update Manager
#VUM Disable
https://vc-01.lab.adms.local/eam/mob/?moid=e5042115-75a4-4ecb-b5e5-b804b0626dfa
# Select Update
# Delete all lines save:
<config>
   <bypassVumEnabled>true</bypassVumEnabled>
</config>
#Press Invoke
Put the host into maintenance mode, ensuring that the objects will be accessible. Then we disable the applications that are part of the NSX-v VIB and disable the kernel modules provided by the package.
#pesx-03
esxcli system maintenanceMode set -e 1 -m ensureObjectAccessibility
chkconfig nsx-vibs off
chkconfig vShield-Protocol-Introspection off
chkconfig vShield-Stateful-Firewall off
esxcfg-module -d nsx-dvfilter-switch-security
esxcfg-module -d nsx-traceflow
esxcfg-module -d nsx-bfd
esxcfg-module -d nsx-vsip
esxcfg-module -d nsx-core
esxcli system shutdown reboot -r 'NSX-v Disable for repair of botched upgrade'
After the host reboots, we remove the out-of-date NSX-v vib (the names may be different for earlier releases, see The NSX-v Troubleshooting Guide and reboot again.
#pesx-03
esxcli software vib remove -n esx-nsxv
esxcli system shutdown reboot -r 'NSX-v Disable for repair of botched upgrade'
Confirm that the VIBs and modules are gone.
#pesx-03 Confirm no NSX-v
esxcli software vib list | grep nsx
esxcfg-module -l | egrep -e 'nsx-dvfilter-switch-security|nsx-traceflow|nsx-bfd|nsx-vsip|nsx-core'
Now we remove the disablement of the NSX-v kernel modules from the saved ESXi configuration, and the Variables related to RabbitMQ NSX, persist the configs, and reboot.
#EAM Reinstall will "stick" at 66%
egrep  -v 'nsx-traceflow.+enabled|nsx-core.+enabled|nsx-bfd.+enabled'  /etc/vmware/esx.conf > e && mv e /etc/vmware/esx.conf
egrep -iv Rmq /etc/vmware/esx.conf > e  && mv e /etc/vmware/esx.conf
 
auto-backup.sh
auto-backup.sh
reboot -ff
The ssh to the vCenter server and restart the EAM and UpdateManager service to kick-off the EAM managed VIB install.
#vCentre
[msherian@MARS-E550 i]$ ssh root@vc-01
 
VMware vCenter Server Appliance 6.5.0.11000
 
Type: vCenter Server with an external Platform Services Controller
root@vc-01's password:
Last login: Thu Nov 16 10:55:43 2017 from 192.168.255.110
 
service-control --stop vmware-eam vmware-updatemgr ; service-control --start vmware-eam vmware-updatemgr; tail -f /var/log/vmware/eam/eam.log /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log /var/log/vmware/vmware-updatemgr/updatemgr-utility.log
Ensure that the VIB is installed and the applications are enabled on boot.
#pesx-03
 esxcli software vib list | grep nsx
 
esx-nsxv                       6.5.0-0.0.6777105                        VMware   VMwareCertified   2017-11-15
 
 chkconfig nsxv-vib on
 chkconfig vShield-Protocol-Introspection on
 chkconfig vShield-Stateful-Firewall on

Comments

Post a Comment

Popular posts from this blog

HyperCube — The fastest little lab in the west of Europe.

In the coming days and weeks, I'll be documenting the construction of a very powerful nested cluster of ESXi hosts. These are based on the Intel NUC platform, and I will be linking to much of the source material used in constructing this lab. The hardware used is as follows: 3×Intel NUC 5i7RYH  3× Samsung 1TB 850 EVO Series SATA  SSD 3×SSD 850 EVO M.2 500GB 6×16GB SO-DIMM Intelligent Memory  The three ESXi nodes are running  6.0.0 (Build 2809209) customised with the E1000E driver, and have the ESXi UI Fling Installed. The first node of the cluster has been running for several weeks, running a nested ESX environment with all flash virtual SAN So, if you are interested, stay tuned here, and I will be documenting the steps I've already taken, Giving much credit to those other bloggers out there that have done a lot of the heavy lifting.  I'll be extending this deployment with all of the following: AD Services design for vSphere Environmen
Post a Comment
Read more

HyperCube — The easy bit, hardware assembly

Below I'll provide the first node as an example of the ridiculously simple hardware install. The unboxing: Removing the case: Installation of the 500 GB M.2 Drive: Remove the mounting screw from the chassis: Place the drive into the slot: Fix the screw in place: 1 TB SSD: Slot it into the drive bay: The correct way! ;-) Repeat two more times, cable and go:
Post a Comment
Read more