Panorama Service Manager was in “Registered” state however no service definition was seen under NSX.
Symptom
- Panorama Service Manager is in “Registered” state, however no service definition is seen under NSX.
- Check connectivity from management interface to NSX Manager using ping. If this fails, troubleshoot network connectivity from Panorama to NSX Manager.
- If ping succeeds that implies, we have layer3 connectivity. Perform traceroute to identify any devices inpath. Only device in Layer 3 mode can be identified with this step.
- Proceed with testing layer 7 connectivity using curl from root shell using below:
# curl -s -k -H 'Content-Type: application/xml' -u 'admin:<password>' -X GET 'https://<ipaddress>/api/2.0/si/servicemanagers' --tlsv1.2
Response: {"errorCode":1803,"details":"Error during REST callback. POST to the registered ServiceManager at : https://10.2.99.100/api/?type=plugin&plugin=vmware_nsx&nsx-mgr=29&action=update&key=LUFRPT0rbENzR3haOHBKL0pmRTVLODhBV00xTllmMEE9OXVnUG0rdXV2Zmg4RVFvZm8rRmE2RSt0Ri9WOTJ3VDNXUzlNVjg3cDRvYVRvNTBsZ09GMDBQYVljU2RKOTRmZQ==&file-name=f&client=wget&url=/vmware/2.0/si/serviceinstance/. Please use Service Manager in Non-Operational Mode to proceed. This issue is caused by : REST Request Failed. Unable to establish a connection to the Service Manager.","rootCauseString":null,"moduleName":"core-services","errorData":null}
- curl response shows plugin is unable to establish layer 7 connection with NSX Manager
- Set logging level on plugin to high from Panorama CLI and trigger NSX config sync under Panorama > VMware NSX > Service Manager > NSX Config Sync
- Review System Logs and/or Plugin logs to understand and verify the failure events.
2019-11-15 17:00:24.962 -0600 INFO: [NSX-MON] PAN-NSX status changed from Unknown State to Registered
2019-11-15 17:03:45.396 -0600 ERROR: Error while sending license api key request to configd. 'NoneType' object has no attribute 'text'
2019-11-15 17:03:45.397 -0600 INFO: Config from panorama validated!
2019-11-15 17:04:00.979 -0600 INFO: PAN-NSX: Service Manager servicemanager-11 updated.
2019-11-15 17:05:05.535 -0600 ERROR: Curl call to NSX Manager failed
Curl Request: /usr/bin/curl -X POST -k -s -m 120 --config /root/.curlrc --connect-timeout 3 --tlsv1.2 -w
%{http_code} -H Content-Type: application/xml --data @/tmp/post_data_file_140245416109824.txt https://10.2.12.6/api/2.0/si/service
Reason: Bad Request
Return code: 400
- Plugin logs show curl call failed with http error 400, which implies something in the path may be filtering the curl request being sent
- Proceed with troubleshooting network issue with devices in path
Environment
- Platform: Panorama
- PAN-OS / Plugin Version: 9.0.4 / 2.0.5
- Deployment: Operations Centric
Cause
- 3rd party WAF (Web Application Filter) device was in path that was blocking the specific SSL stream between Panorama and NSX Manager
Resolution
- Whitelisting the traffic stream between Panorama and NSX Manager on WAF device resolved the issue.