Elm License Server Is Not Responsive Try Again Later
Recently we've had some weird issues on one of our customers vCenter Servers. For starters the vMotion and Storage vMotion features weren't working anymore because of fourth dimension-outs. Which is weird and something I've never seen earlier. So we started troubleshooting the VCSA server and noticed that it couldn't retrieve the installed licenses (VMware vSphere Enterprise Plus) from the product ESXi hosts anymore.
Going to the "Licensed Features" tab in the vSphere Client (VCSA version 6.0 GA) normally gives you a nice overview of what vSphere license is installed, only this time information technology was but empty. Going to the ESXi host directly you could however see that the license was present and activated. We also noticed that the License module in the vSphere client was also providing united states of america with a timeout.
Once nosotros dove into the log files from the license service in "/var/log/vmware/cis-license/license.log" nosotros noticed some Security Token Service STS service, SSO service and web-customer service issues in regards to certificates. Which got me thinking and looking at the certificates for this vCenter Server Appliance. Below you can find some snippets of logs which might be interesting for you to match your problem to the one I was having:
| license . log : 2019-05 -13T13 :49:10.674Z Timer -3 WARN core . direction . maint . service . AssetInventoryMaintainerTimerTaskImpl Maintanance of the nugget inventory failed . com . vmware . cis . license . server . common . provider . ClientStubProviderException : com . vmware . vim . vmomi . customer . exception . SslException : com . vmware . vim . vmomi . core . exception . CertificateValidationException : Server certificate chain non verified Caused by : javax . net . ssl . SSLPeerUnverifiedException : peer not authenticated at sunday . security . ssl . SSLSessionImpl . getPeerCertificates ( Unknown Source ) at com . vmware . vim . vmomi . client . http . impl . ThumbprintTrustManager $HostnameVerifier . verify ( ThumbprintTrustManager . java :296) . . . 44 more 2019-05 -13T13 :22:43.443Z pool -3 -thread -one WARN common . vmomi . authn . impl . SsoAuthenticatorImpl STS signing certificates are missing or empty 2019-05 -13T13 :22:43.601Z pool -three -thread -1 ERROR server . common . sso . impl . SsoAdminProviderImpl Refetch STS certificates failed |
You tin can utilise the following cli cmdlets to cheque your certificate stores and the certificates that are in them:
| / usr / lib / vmware -vmafd / bin / vecs -cli entry list -- store MACHINE_SSL_CERT -- text | less / usr / lib / vmware -vmafd / bin / vecs -cli entry list -- shop machine -- text | less / usr / lib / vmware -vmafd / bin / vecs -cli entry list -- store vpxd -- text | less / usr / lib / vmware -vmafd / bin / vecs -cli entry list -- shop vsphere -webclient -- text | less |
All certificates checked out but estimate what, the "MACHINE_SSL_CERT" didn't. Turns out it was expired. Funny affair though is that this item vCenter Apparatus should'nt even be working anymore considering one time the document is expired, well-nigh of the fourth dimension it won't fifty-fifty kickoff all of the vCenter services once you reboot it. In our case somehow information technology did.
And so we went alee and fired up the "certificate-manager" tool which can exist found in "/usr/lib/vmware-vmca/bin/document-manager", picked option 3 to supercede the the Machine SSL with a VMCA certificate (which is a self-signed certificate but that's fine for this environment), entered the data which was nowadays in the electric current certificate such as hostnames and IP-address information and accepted all changes.
Once y'all accepted the change it is proposing it will update the certificates in the locations information technology is needed and end and first all services. Slice of cake. Our certificate-manager however decided it was time to throw an error:
| Mistake certificate -director Mistake while starting services , please see log for more details document -director Mistake while replacing Machine SSL Cert , please see / var / log / vmware / vmcad / document -manager . log for more information . |
Once we checked that log we saw that the certificate-manager tooling couldn't start the "vmware-eam" service, see the below log snippet which can be establish in "/var/log/vmware/vmcad/certificate-manager.log":
| 1 2 iii 4 5 six 7 8 9 x xi 12 13 xiv fifteen 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 | Waiting for VMware ESX Amanuensis Manager . . . . . . . Alert : VMware ESX Amanuensis Managing director may accept failed to start . Last login : Monday May 13 13:22:44 UTC 2019 on console Stderr = 2019-05 -13T13 :47:40.139Z { "resolution" : null , "detail" : [ { "args" : [ "Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: " ] , "id" : "install.ciscommon.command.errinvoke" , "localized" : "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: '" , "translatable" : "An error occurred while invoking external command : '%(0)s'" } ] , "componentKey" : goose egg , "problemId" : nada } ERROR : root : Unable to get-go service vmware -eam , Exception : { "resolution" : zero , "particular" : [ { "args" : [ "vmware-eam" ] , "id" : "install.ciscommon.service.failstart" , "localized" : "An error occurred while starting service 'vmware-eam'" , "translatable" : "An error occurred while starting service '%(0)s'" } ] , "componentKey" : zippo , "problemId" : null } |
Sure enough we were hitting a bug in our vCenter Server Appliance. This bug prevented the EAM service from starting subsequently a vCenter reboot. This bug basically deletes the "eam.backdrop" file in the "/etc/vmware-eam/" directory. This file is crucial for the service to first and know what to exercise. Since this file was missing in our surround, the "vmware-eam" service was cleaved. This VMware KB explains how to set this. Which basically ways that you take to download the attachment chosen "Recreate_eam.properties.sh" and run it. This script recreates the eam.properties file so that your "vmware-eam" service can start again. Delight not that you can merely run this when yous run the EAM service on the vCenter Server yous are working on. The steps to run this script are described beneath:
| ane 2 3 four v six 7 viii 9 ten 11 12 13 14 15 16 17 | Step 1: Download the script and upload it to your vCenter Server Footstep 2: Create a backup from the current eam . properties file ( if present ) . Don't forget to create a VM snapshot either Step iii: Decide the host ID : cat / etc / vmware / install -defaults / sca . hostid Step 4: Determine the vCenter Server appliance hostname hostname -f Footstep 5: Ready permissions on the Recreate_eam . properties . sh file chmod 777 Recreate_eam . properties . sh Step 6: Run the Recreate_eam . properties . sh file : . / Recreate_eam . properties . sh and enter the required information Stride vii: Check the / eam . properties for the correct hostname and host ID which you collected earlier Step 8: Offset the "vmware-eam" service . service -control vmware -eam start Stride ix: re -run the certificate managing director with your previously entered information / usr / lib / vmware -vmca / bin / certificate -managing director and select Pick 3 |
In our situation this almost stock-still our issues. We were forced to break the certificate-director procedure in the middle where it starts starting the services once more after information technology updated the "MACHINE_SSL_CERT" in the places information technology has to. You tin can do this past just pressing CTRL+C on the right time in the procedure. To find this correct time you can open another putty session to the VMware vCenter server and using the following control:
| tail -f / var / log / vmware / vmcad / certificate -manager . log |
Simply press CTRL+C when the following log entries pass by:
| ane 2 3 4 five 6 7 8 ix 10 eleven 12 xiii xiv 15 sixteen 17 18 nineteen xx 21 22 23 24 25 26 27 28 29 thirty 31 32 33 34 35 36 | 2019-05 -13T14 :15:06.607Z INFO certificate -manager Running control : - service -command -- stop -- ignore -- all 2019-05 -13T14 :15:06.608Z INFO certificate -manager delight see service -control . log for service status INFO : root : Service : vmware -psc -client , Action : stop INFO : root : Service : vmware -syslog -wellness , Action : stop INFO : root : Service : vmware -vsan -health , Action : end INFO : root : Service : applmgmt , Action : terminate INFO : root : Service : vmware -eam , Activeness : stop INFO : root : Service : vmware -mbcs , Action : terminate INFO : root : Service : vmware -netdumper , Action : stop INFO : root : Service : vmware -perfcharts , Action : stop INFO : root : Service : vmware -rbd -watchdog , Action : stop INFO : root : Service : vmware -sps , Action : stop INFO : root : Service : vmware -vapi -endpoint , Action : stop INFO : root : Service : vmware -vdcs , Activeness : end INFO : root : Service : vmware -vpx -workflow , Activity : finish INFO : root : Service : vmware -vsm , Activity : stop INFO : root : Service : vsphere -client , Action : stop INFO : root : Service : vmware -vpxd , Activeness : stop INFO : root : Service : vmware -cis -license , Activeness : stop INFO : root : Service : vmware -invsvc , Action : stop INFO : root : Service : vmware -vpostgres , Action : stop INFO : root : Service : vmware -syslog , Activeness : stop INFO : root : Service : vmware -sca , Action : stop INFO : root : Service : vmware -vws , Action : terminate INFO : root : Service : vmware -cm , Action : cease INFO : root : Service : vmware -rhttpproxy , Activeness : stop INFO : root : Service : vmware -stsd , Activity : stop INFO : root : Service : vmware -sts -idmd , Activity : stop INFO : root : Service : vmcad , Action : stop INFO : root : Service : vmdird , Action : stop INFO : root : Service : vmafdd , Activeness : stop 2019-05 -13T14 :15:52.728Z INFO document -manager Command executed successfully 2019-05 -13T14 :xv:52.728Z INFO certificate -manager all services stopped successfully . 2019-05 -13T14 :15:52.728Z INFO certificate -manager None 2019-05 -13T14 :15:52.729Z INFO document -manager Running control : - service -control -- showtime -- all 2019-05 -13T14 :fifteen:52.729Z INFO certificate -director please see service -control . log for service condition |
Once you are at this point just start the services yourself with:
| service -control -- kickoff -- all |
This should commencement all the services nicely. Subsequently this betoken we had our VMware vCenter Server Appliance working once again with a new fresh "MACHINE_SSL_CERT" document. As a last check yous can execute the following command and verify the expiration date:
| / usr / lib / vmware -vmafd / bin / vecs -cli entry list -- store MACHINE_SSL_CERT -- text | less |
In that location you have it. I figured information technology would be easy plenty and fix this quickly, turned out we were facing a bug in the "vmware-eam" service. I hope this post helps when yous are finding the aforementioned issues we plant.
Source: https://vcloudvision.com/2019/05/13/how-to-fix-an-expired-vcsa-machine-ssl-certificate-with-a-bugged-vmware-eam-service/
0 Response to "Elm License Server Is Not Responsive Try Again Later"
Post a Comment