Enable Aggressive Cleaning (PAN-OS 7.1.14+, 8.0.7+, 8.1.0+)
This will automatically truncate all old log files (entries under all *var/log/pan directories matching *.1, ... *.4, *.log.old) if the 95% occupancy alarm is tripped.
> debug software disk-usage aggressive-cleaning <enable|disable>
Note: Enabling aggressive clean up may clear up logs, rendering logs unavailable for troubleshooting purposes.
To verify the changes or if it is already enabled use the command below.
> show system state | match aggressive-cleaning
cfg.debug-sw-du.config: { 'aggressive-cleaning': True, }
Run disk-usage cleanup command (PAN-OS 8.1.0+)
disk-usage cleanup can be done manually using the command below:
> debug software disk-usage cleanup ?
+ deep cleanup with deleting backup logfile
* threshold percentage threshod of size of system partition to kick off cleanup
> debug software disk-usage cleanup deep threshold 90
Note that this command has to be run manually each time to bring the disk usage to below 90% and is not persistent across reboots.
Note:
1- Enabling aggressive clean up may clear up logs, rendering logs unavailable for analysis.
2- If the following error message shows up after the disk-usage cleanup deep threshold 90 command:
Server error : op command for client dagger timed out as client is not availableThen start with a higher threshold example 94 instead of 90 so the cleaning time will be less which will prevent the server timeout error.
> debug software disk-usage cleanup deep threshold 94
You can afterwards keep lowering the threshold value to reach 90.
Check and Delete Unnecessary Core Files
- Check the output of > show system files to see core files using up a large amount of disk space.
> show system files
/opt/dpfs/var/cores/:
total 4.0K
drwxrwxrwx 2 root root 4.0K Jun 10 20:05 crashinfo
/opt/dpfs/var/cores/crashinfo:
total 0
/var/cores/:
total 115M
drwxrwxrwx 2 root root 4.0K Jun 10 20:15 crashinfo
-rw-rw-rw- 1 root root 867M Jun 12 13:38 devsrvr_4.0.3-c37_1.gz
-rw-rw-rw- 1 root root 51M Jun 12 13:39 core.20053
/var/cores/crashinfo:
total 16K
-rw-rw-rw- 1 root root 15K Jun 10 20:15 devsrvr_4.0.3-c37_0.info
- Delete unnecessary core files. A core file can be deemed unnecessary if investigation around the core file is complete or they are very old files.
> delete core management-plane file devsrvr_4.0.3-c37_1.gz
- These files can be exported using the scp or tftp export command, then deleted to free up space on the firewall
> scp export core-file management-plane from mgmtsrvr_7.0.3_0.tgz to user@10.0.0.10:/home/
user@10.0.0.10 's password: ********
mgmtsrvr_7.0.3_0.tgz 100% 453MB 46.4MB/s 00:12
Note: The steps below are only applicable to firewall and not to Panorama.
Delete Rotated Files and Files with Extension .old
These files contain monitoring details and service related logs on the firewall. They can be deleted safely if you don't need them. If TAC investigates an ongoing issue, you may prefer to keep them until you upload the tech support file to the case manager.
> delete debug-log ?
cp-log Remove cp-log at /opt/var.cp/log/pan/
dp0-log Remove dp0-log at /opt/var.dp0/log/pan/
dp1-log Remove dp1-log at /opt/var.dp1/log/pan/
dp2-log Remove dp2-log at /opt/var.dp2/log/pan/
mp-global Remove mp-global at /opt/mp-global/
mp-log Remove mp-log at /var/log/pan/ <--- mp-log example used below
> delete debug-log mp-log file *.1
> delete debug-log mp-log file *.2
> delete debug-log mp-log file *.3
> delete debug-log mp-log file *.4
> delete debug-log mp-log file *.old
Clear Any packet-diag Logging If Enabled
Run > debug dataplane packet-diag show setting to check if packet-diag is enabled.
> debug dataplane packet-diag show setting
DP dp0:
--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
Enabled: no
Match pre-parsed packet: no
--------------------------------------------------------------------------------
Logging
Enabled: no <-- No is the default value and means not enabled.
> debug dataplane packet-diag show setting
DP dp0:
--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
Enabled: no
Match pre-parsed packet: no
--------------------------------------------------------------------------------
Logging
Enabled: yes <--- Means packet-diag is enabled.
To clear out packet-diag setting and the logs, run below commands:
> debug dataplane packet-diag clear all
Packet diagnosis setting set to default.
> debug dataplane packet-diag clear log log
dataplane debug logs cleared
Delete Any Debug pcaps or Debug-filter pcaps
> delete debug-filter file <file-name>
> delete pcap directory *
Set the logging-level back to its default for all processes running on the firewall:
> debug software logging-level set level default service all-services
If none of the above remediation steps resolve the issue, it is recommended to collect the following Troubleshooting Data below and open a Support Case.
- Collect Tech Support File (GUI: Device > Support Click Generate Tech Support File)
- Collect the output of the CLI show system disk-space
Important to note:If your FW is a
PA-220 running PAN-OS version 10.1.x but less than 10.1.9 then upgrade your device to
10.1.9-h3 or higher to fix issue mentioned in
PAN-193452 regarding firewall reaching the maximum disk usage capacity repeatedly in one day.
If your FW is a
PA-220 running PAN-OS version 10.2.x the root disk-space issue is still under investigation reach out to support if the steps listed above don't help you solve this problem.