NSX Service Manager status shows “No Connection”
6291
Created On 03/09/20 18:44 PM - Last Modified 04/03/20 22:26 PM
Symptom
Service Manager is configured on Panorama under VMware NSX -> Service Managers in an existing deployment and the connection status changed from “Registered” to “No Connection” or “Error! Trigger NSX-Config-Sync!”
- Review System Logs and/or Plugin logs to understand the failure event.
System Logs:
2019/12/20 16:18:50 2019/12/20 16:18:50 medium plugin general Plugin vmware_nsx: Curl call to NSX Manager failed Curl Request: /usr/bin/curl -k -s -m 120 --config /root/.curlrc --connect-timeout 3 --tlsv1.2 -w %{http_code} https://10.46.78.41/api/2.0/si/servicemanagers
plugin_vmware_nsx.log
2019-12-20 16:22:16.456 -0800 ERROR: Curl call to NSX Manager failed
Curl Request: /usr/bin/curl -k -s -m 120 --config /root/.curlrc --connect-timeout 3 --tlsv1.2 -w
%{http_code} https://10.46.78.41/api/2.0/si/servicemanager
Reason:
Return code: 0
Response:
2019/12/20 16:18:50 2019/12/20 16:18:50 medium plugin general Plugin vmware_nsx: Curl call to NSX Manager failed Curl Request: /usr/bin/curl -k -s -m 120 --config /root/.curlrc --connect-timeout 3 --tlsv1.2 -w %{http_code} https://10.46.78.41/api/2.0/si/servicemanagers
plugin_vmware_nsx.log
2019-12-20 16:22:16.456 -0800 ERROR: Curl call to NSX Manager failed
Curl Request: /usr/bin/curl -k -s -m 120 --config /root/.curlrc --connect-timeout 3 --tlsv1.2 -w
%{http_code} https://10.46.78.41/api/2.0/si/servicemanager
Reason:
Return code: 0
Response:
- Above log snippet shows the curl call to NSX manager has failed
- Check connectivity from management interface to NSX Manager using ping. If this fails, troubleshoot network connectivity from Panorama to NSX Manager.
- If ping succeeds that implies, we have layer 3 connectivity, perform traceroute to identify any devices in path.
- Perform tcpdump on management interface of the Panorama. Generally, the filter should be with destination as NSX Manager IP however, if Proxy server is configured on Management interface, the filter should be the proxy server’s IP as all outgoing packets would be destined to Proxy server IP. In this case, you would notice RESET being sent from the Panorama to Proxy Server
> tcpdump filter “host <nsx_manager_IP>” or > tcpdump filter “host <proxy_IP>”
- If you notice the connection being reset on the above packet captures, try bypassing the proxy for NSX Manager connection using below CLI:
> request plugins vmware_nsx proxy bypass yes
Environment
- Platform: Panorama
- PAN-OS / Plugin Version: 9.0.5 / 2.0.5
- Deployment: Operations Centric
Cause
- Since bypassing the proxy server resolved the issue, we can conclude TCP traffic was blocked by Proxy Server
- Below logs can also be monitored in case one has access to proxy server
Logs on Proxy [tail -f /var/log/squid/access.log]
1576889092.153 0 10.46.78.78 TCP_DENIED/407 3738 CONNECT 10.46.78.41:443 rohit1 HIER_NONE/- text/html
1576889098.342 0 10.46.78.78 TCP_DENIED/407 3738 CONNECT 10.46.78.41:443 rohit1 HIER_NONE/- text/html
Resolution
- Bypassing Proxy server configured on Panorama for connections to NSX Manager resolved the issue
- Plugin 2.0.5 [PLUG-1596] introduced a feature to bypass Proxy
> request plugins vmware_nsx global proxy bypass {yes | no}