NSX: Empty Dynamic Address Group on Firewalls

NSX: Empty Dynamic Address Group on Firewalls

8377
Created On 03/12/20 23:48 PM - Last Modified 04/04/20 00:49 AM


Symptom


  • Panorama shows IP addresses being populated on Address Groups however the managed firewalls do not have any IP’s registered on Dynamic Address Group.
User-added imageUser-added image
  • Run the below CLI command on PA-VM to verify if any IP addresses are being registered on the firewall
           > show object registered-ip all
  • Next run the below CLI command on PA-VM firewall to verify the last “register” event for the IP in question. In this case IP’s were pushed to firewall about 4 hours ago.
          > show log iptag datasource_type equal xml-api ip in x.x.x.x
  • Set logging level on useridd process on PA-VM to “debug” by using below CLI:
         > debug user-id on debug 
  • Manually trigger “Synchronize Dynamic Objects” on Panorama Service Manager and monitor useridd on PA-VM firewall to check below events:
useridd Logs:
2020-01-31 10:26:10.146 -0800 clear all registered ip adddresses upon XMLAPI request
2020-01-31 10:26:29.724 -0800 Processing dnld delta : 4, full : 39
2020-01-31 10:26:29.724 -0800 dnld 4 registered ip takes 0 second
  •  Above logs snippet shows a working scenario for registered IP’s being downloaded from Panorama however in this case no such log events were observed.
  • Verify resource utilization on Panorama. In this case, the management CPU was running high.
  • Adopt PAN-OS troubleshooting steps to identify the root cause for high CPU. In this case, it was the elasticSearch process consuming high CPU due to heavy log indexing

Environment


  • Platform: PA-VM    
  • Deployment: Operations Centric

Cause


  • Log Ingestion rate seen under mp-monitor log was 47K logs/s which is above the system supported limit for log ingestion in hybrid or mixed mode.
https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000Clc8CAC
 

Resolution


  • [Temp Fix] Kill the elasticSearch 'es' process from root to bring the load down to 26% temporarily and triggered “Synchronize Dynamic Objects” under Panorama > Service Manager to register the IP’s on Firewall
  • Logs Forwarded should be reduced / re-designed so that log ingestion rate on Panorama lies within the supported limits.


 
Other users also viewed:
Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000PP2ZCAW&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail