NSX: Missing service definition due to expired API key
6810
Created On 03/12/20 03:47 AM - Last Modified 04/04/20 00:37 AM
Symptom
- Panorama Service Manager is in “Registered” state, but service definitions are not pushed to NSX. Plugin logs shows REST API error 401
- Set logging level on plugin to high from Panorama CLI and trigger NSX config sync under Panorama > VMware NSX > Service Manager > NSX Config Sync
- Review Plugin logs to understand and verify the failure events.
Corp_TME_SM: Resp from NSX for not creating. <error><errorCode>1803</errorCode><details>Error during REST callback. POST to the registered ServiceManager at : https://10.8.59.222/api/?type=plugin&plugin=vmware_nsx&nsx- mgr=37&action=update&key=LUFRPT1UdGZhQ21oQW5rQ3pwUis3NkxJQklUTkVXcm89ZGtkRVlPL09hYmlZSERHczU1MU5iVjA3L29KN3BnRnVvUUR6NkhRTEU5R3U0cmltY1lhSmNNbC9Hdy8xVVRZZA==&file-name=f&client=wget&url=/vmware/2.0/si/serviceinstance/. Please use Service Manager in Non-Operational Mode to proceed. This issue is caused by : 401 LUFR...ZA== has expired..</details><moduleName>core-services</moduleName></error> 2019-05-15 07:08:43.912 -0700 ERROR: Corp_TME_SM: Failed to create object service-definition. Ret code is 400
- Above log snippet shows plugin was not able to create service definition and the API call returned error 401 which implies the request made was unauthorized.
Environment
- Platform: Panorama
- PAN-OS / Plugin Version: 9.0.1 / 2.0.4
- Deployment: Operations Centric
Cause
- "Expire All API keys" was manually triggered under Panorama > Setup > Management > Authentication Settings which also caused the API key for service account "_vmware_nsx" to expire in addition to all other API keys which caused the 401 Unauthorized error on the API call to NSX Manager
Resolution
- This issue has been addressed in plugin version 2.0.6 or later
- Workaround needs TAC engagement to access root shell and remove the expiration flag