High Management Plane CPU Due To SSH Attempts

12203

Created On 11/25/20 21:25 PM - Last Modified 01/12/21 04:34 AM

Management Interface

Device Management

8.1

9.0

9.1

10.0

PAN-OS

Symptom

Firewall GUI is showing high CPU percentage loading on Dashboard page
"show system resources follow" shows high "load average" and/or high CPU

mp-monitor.log 2020-10-22 12:54:02* Cpu(s): 11.7%us, 4.1%sy, 0.1%ni, 83.8%id, 0.1%wa, 0.0%hi, 0.2%si, 0.0%st
mp-monitor.log 2020-10-22 12:57:00* Cpu(s): 11.8%us, 4.1%sy, 0.1%ni, 83.8%id, 0.1%wa, 0.0%hi, 0.2%si, 0.0%st
mp-monitor.log 2020-10-22 13:00:00* Cpu(s): 11.8%us, 4.1%sy, 0.1%ni, 83.8%id, 0.1%wa, 0.0%hi, 0.2%si, 0.0%st

mp-monitor.log 2020-10-22 12:54:02* top - 12:54:01 up 54 days, 18:00, 1 user, load average: 15.35, 11.06, 10.59
mp-monitor.log 2020-10-22 12:57:00* top - 12:57:00 up 54 days, 18:03, 1 user, load average: 20.39, 13.85, 11.66
mp-monitor.log 2020-10-22 13:00:00* top - 13:00:00 up 54 days, 18:06, 1 user, load average: 10.83, 13.85, 12.13

In this case, the CPU is not high, but the load averages show the 1 minute, 5 minute, and 15-minute loading is elevated - these numbers represent the average number of processes waiting for a CPU during the respective interval. (In this particular example there are 4 Management Plane cores on this firewall - so this indicates on average there are 10-20 processes waiting to be serviced by 4 cores). The higher numbers indicate the firewall could be receiving a lot of requests.

The "Monitor -> System" log show many failed SSH login attempts from public IP addresses (this is showing many connections per second)

2020/10/22 12:57:31 medium general general 0 Failed password for root from 218.92.0.191 port 40760 ssh2
2020/10/22 12:57:31 medium general general 0 Failed password for root from 168.90.89.35 port 50888 ssh2
2020/10/22 12:57:31 medium general general 0 Failed password for root from 218.92.0.191 port 40760 ssh2
2020/10/22 12:57:31 medium general general 0 Failed password for share from 191.43.12.85 port 35904 ssh2
2020/10/22 12:57:31 medium general general 0 Failed password for root from 107.170.20.247 port 42771 ssh2
2020/10/22 12:57:31 medium general general 0 Failed password for root from 188.131.166.98 port 45944 ssh2

"show netstat" shows many SSH connections in various connection states:

2020-10-22 04:18:35.238 -0700 --- netstat
tcp 40 0 67.109.26.210:22 218.92.0.191:19919 CLOSE_WAIT 15305/sshd
tcp 23 0 205.158.104.17:22 163.172.101.48:55226 ESTABLISHED 15630/sshd
tcp 40 0 205.158.104.26:22 222.186.180.130:64031 CLOSE_WAIT 14690/sshd
tcp 1 0 205.158.104.15:22 103.238.69.138:36270 CLOSE_WAIT 14860/sshd
tcp 1 0 205.158.104.12:22 211.193.58.225:59800 CLOSE_WAIT 14887/sshd
tcp 40 0 205.158.104.16:22 222.186.180.130:41087 CLOSE_WAIT 14698/sshd
tcp 1 0 205.158.104.28:22 113.53.238.195:37006 CLOSE_WAIT 15225/sshd
tcp 1 0 205.158.104.30:22 186.47.213.34:39554 CLOSE_WAIT 15112/sshd
tcp 1 0 205.158.104.28:22 211.108.168.106:32882 CLOSE_WAIT 15490/sshd

Environment

Palo Alto Firewall.
Any PAN-OS.
GlobalProtect (GP) Gateway or Portal configured.
An interface configured for GP assigned with management profile permitting SSH.

Cause

The firewall has an Interface Management profile which permits SSH access that is assigned to the dataplane interface that hosts the GlobalProtect Portal and/or Gateway
Many internet based users and programs are attempting to access the firewall using SSH via the exposed public IP of the Globalprotect Portal and Gateway, thereby increasing the Management Plane loading to service those requests.

Resolution

Remove the SSH access from the Management profile assigned to the interface hosting GlobalProtect Gateway or Portal. This can be done under GUI: Network > Network Profiles > Interface Mgmt > (select the profile and uncheck SSH)
Click OK and Commit the configuration changes.

Additional Information