Firewall frequently disconnecting from Panorama | Commit-All stuck
8276
Created On 01/17/23 03:12 AM - Last Modified 11/16/23 03:04 AM
Symptom
- When doing a push to devices from Panorama to managed firewall, the Commit-All job is stuck at 0% for a long time.
admin@LAB(primary-active)> show jobs all
Enqueued Dequeued ID PositionInQ Type Status Result Completed
------------------------------------------------------------------------------------------------------------------------------------------
2023/01/15 19:32:35 19:32:35 5185871 CommitAll ACT OK 0 %
2023/01/15 19:18:58 19:18:58 5185724 CommitAll ACT OK 0 %
2023/01/15 19:11:30 19:11:30 5185634 CommitAll ACT OK 0 %
- Firewall is frequently seen disconnecting from Panorama (In Panorama-->Managed Devices-->Summary OR system logs)
- In packet capture between firewall and Panorama, frequent TCP Window Full or TCP ZeroWindow seen from Panorama IP.
Environment
- PanOS firewalls managed by Panorama
Cause
- Check the Recv-Q on Panorama (Or Send-Q on Firewall) by running netstat command for the firewall in question:
admin@LAB(primary-active)> show netstat numeric-hosts yes numeric-ports yes | match 10.129.70.112
tcp6 2479433 0 10.129.70.180:3978 10.129.70.112:49021 ESTABLISHED
- A high value on Recv-Q means Panorama is not able to ingest the logs as fast as the firewall is sending.
- Panorama does management & logging both over TCP 3978 on one single channel. This can overload the channel and cause Firewall disconnect issues.
Resolution
- Separate the management and logging channel by configuring "Log Collector Preference List " under Panorama-->Collector Groups-->Device Log Forwarding-->Log Forwarding Preferences.
Additional Information
- More on Log Collector Preference List