Repeating Data Plane restarts seen after abrupt power outage
91057
Created On 01/26/21 15:51 PM - Last Modified 04/02/24 20:08 PM
Symptom
- Repeating Data Plane restarts seen after an abrupt power outage.
- These are due to path monitoring failures and may be followed by device reboots.
- Multiple error messages related to "path monitoring failure" "DP down" is seen under GUI: Monitor > Logs > System:
high general general 0 9: dp2-path_monitor HB failures seen, triggering HA DP down
high general general 0 9: dp1-path_monitor HB failures seen, triggering HA DP down
high general general 0 9: cp-path_monitor HB failures seen, triggering HA DP down
critical general general 0 Chassis Master Alarm: HA-event
critical ha state-c 0 HA Group 1: Moved from state Active to state Non-Functional
critical ha datapla 0 HA Group 1: Dataplane is down: path monitor failure
high general general 0 9: dp0-path_monitor HB failures seen, triggering HA DP down
info hw slot-st 0 Slot 1 (PA-5260) is starting.
critical general general 0 internal_monitor: Exited 1 times, must be manually recovered.
critical general general 0 cp-path_monitor: Exited 1 times, must be manually recovered.
high general general 0 cp-path_monitor: exiting because service missed too many heartbeats
critical general general 0 Internal packet path monitoring failure, restarting dataplane
high general general 0 9: dp1-path_monitor HB failures seen, triggering HA DP down
high general general 0 9: cp-path_monitor HB failures seen, triggering HA DP down
critical general general 0 Chassis Master Alarm: HA-event
critical ha state-c 0 HA Group 1: Moved from state Active to state Non-Functional
critical ha datapla 0 HA Group 1: Dataplane is down: path monitor failure
high general general 0 9: dp0-path_monitor HB failures seen, triggering HA DP down
info hw slot-st 0 Slot 1 (PA-5260) is starting.
critical general general 0 internal_monitor: Exited 1 times, must be manually recovered.
critical general general 0 cp-path_monitor: Exited 1 times, must be manually recovered.
high general general 0 cp-path_monitor: exiting because service missed too many heartbeats
critical general general 0 Internal packet path monitoring failure, restarting dataplane
Environment
- Palo Alto 5200 Series Firewalls
- Palo Alto 3200 Series Firewalls
- PAN-OS Versions: 10.2.4, 10.1.10, 10.1.12,10.1.9, 9.1.6 and below.
Cause
- Communication between the Management Plane and Control Plane uses specific internal ports
- When the internal ports are down the communication between management and control plane fails
- This triggers Path monitoring failure
Resolution
- The issue is resolved under PAN-160633 in PAN-OS versions of 11.1.0, 11.0.3, 10.2.6, 10.2.5, 10.1.11, 9.1.17, 10.1.10-h2
- Upgrading to the fixed codes will resolve the issue
- Check the status of xe8 and xe9 ports using the command "debug dataplane internal pdt bcm show port status"
- Note:xe8 and xe9 ports (xe9 not part of PA-3200) are connected to the Management Plane and used to transmit/receive data to/from the Management Plane from/to other hardware components.
- if xe8 and xe9 ports are down then Power Cycle the firewall (Plug out the power cable and insert back without delay). This will bring up the firewall. (In some cases you may need to wait a couple minutes then plug back in)
> debug dataplane internal pdt bcm show port status
ena/ speed/ link auto STP lrn inter max loop
port link duplex scan neg? state pause discrd ops face frame back
xe1( 1) up 1G FD SW Yes Disable TX RX None D GMII 1856
xl2( 2) up 40G FD SW No Disable TX RX None D XLAUI 16360 PHY
ce3( 3) up 100G FD SW No Forward RX None FA CAUI 16360
xe4( 4) up 10G FD SW No Disable TX RX None D XFI 16360
xe5( 5) up 10G FD SW No Disable TX RX None D XFI 16360
xe6( 6) !ena 10G FD SW No Disable TX RX None D XFI 1856
xe7( 7) !ena 10G FD SW No Disable TX RX None D XFI 1856
xe8( 8) down - SW Yes Disable TX RX None D KR 16360
xe9( 9) down - SW Yes Disable TX RX None D KR 16360 (xe9 on PA-5200)
xe10( 10) !ena 10G FD SW No Disable TX RX None D XFI 1856
- Note: These ports are also used by path monitoring packets.