High Disk Space Usage on / root partition and How To Clear

High Disk Space Usage on / root partition and How To Clear

925913
Created On 09/25/18 19:37 PM - Last Modified 04/05/24 02:34 AM


Symptom


Some common symptoms of the root partition getting filled up are:
  • Unable to log in or access Web UI
  • Certain daemons processes not starting
  • Incomplete tech support bundles
  • System log alerts with "Disk usage for / exceeds limit,  X percent in use, cleaning file system"
  • Root partition is high
> show system disk-space 
Filesystem            Size  Used Avail Use% Mounted on
/dev/md2              3.8G  3.1G  581M  85% /                   <----- root partition
/dev/md5              7.6G  4.2G  3.0G  59% /opt/pancfg .        
/dev/md6              3.8G  3.0G  666M  82% /opt/panrepo
tmpfs                 2.0G  210M  1.8G  11% /dev/shm
cgroup_root           2.0G     0  2.0G   0% /cgroup
/dev/md8               88G  2.4G   81G   3% /opt/panlogs
tmpfs                  12M     0   12M   0% /opt/pancfg/mgmt/lcaas/ssl/private


Environment


  • PAN-OS
  • Palo Alto Firewall
  • Panorama


Cause


Some of the common causes of a filled partition:

  • An admin troubleshooting certain processes and creates core files. These files are stored on the root and remain there until deleted by the administrator.
  • Enabling of diagnostic logs for the dataplane (packet diags) can also take up space on the root partition.


Resolution


Enable Aggressive Cleaning (PAN-OS 7.1.14+, 8.0.7+, 8.1.0+)

This will automatically truncate all old log files (entries under all *var/log/pan directories matching *.1, ... *.4, *.log.old) if the 95% occupancy alarm is tripped.

 > debug software disk-usage aggressive-cleaning <enable|disable>

Note: Enabling aggressive clean up may clear up logs, rendering logs unavailable for troubleshooting  purposes.

To verify the changes or if it is already enabled use the command below.

> show system state | match aggressive-cleaning
cfg.debug-sw-du.config: { 'aggressive-cleaning': True, }

 

Run disk-usage cleanup command (PAN-OS 8.1.0+)

disk-usage cleanup can be done manually using the command below:
> debug software disk-usage cleanup ?
+ deep        cleanup with deleting backup logfile
* threshold   percentage threshod of size of system partition to kick off cleanup

> debug software disk-usage cleanup deep threshold 90

Note that this command has to be run manually each time to bring the disk usage to below 90% and is not persistent across reboots.
Note:
1- Enabling aggressive clean up may clear up logs, rendering logs unavailable for analysis.
2- If the following error message shows up after the disk-usage cleanup deep threshold 90 command:
Server error : op command for client dagger timed out as client is not available
Then start with a higher threshold example 94 instead of 90 so the cleaning time will be less which will prevent the server timeout error.
> debug software disk-usage cleanup deep threshold 94
You can afterwards keep lowering the threshold value to reach 90.
3- The threshold value should be higher than the current disk usage. For example, threshold value 90 is not acceptable if disk usage is 94.
 

Check and Delete Unnecessary Core Files

  • Check the output of show system files to see core files using up a large amount of disk space.
> show system files
/opt/dpfs/var/cores/:
total 4.0K
drwxrwxrwx 2 root root 4.0K Jun 10 20:05 crashinfo

/opt/dpfs/var/cores/crashinfo:
total 0
 
/var/cores/:
total 115M
drwxrwxrwx 2 root root 4.0K Jun 10 20:15 crashinfo
-rw-rw-rw- 1 root root 867M Jun 12 13:38 devsrvr_4.0.3-c37_1.gz
-rw-rw-rw- 1 root root  51M Jun 12 13:39 core.20053
 
/var/cores/crashinfo:
total 16K
-rw-rw-rw- 1 root root 15K Jun 10 20:15 devsrvr_4.0.3-c37_0.info
  • Delete unnecessary core files. A core file can be deemed unnecessary if investigation around the core file is complete or they are very old files.
> delete core management-plane file devsrvr_4.0.3-c37_1.gz
  • These files can be exported using the scp or tftp export command, then deleted to free up space on the firewall
> scp export core-file management-plane from mgmtsrvr_7.0.3_0.tgz to user@10.0.0.10:/home/ 

user@10.0.0.10 's password: ********
mgmtsrvr_7.0.3_0.tgz                       100%  453MB  46.4MB/s   00:12  
  Note: The steps below are only applicable to firewall and not to Panorama.

Delete Rotated Files and Files with Extension .old 

These files contain monitoring details and service related logs on the firewall. They can be deleted safely if you don't need them. If TAC investigates an ongoing issue, you may prefer to keep them until you upload the tech support file to the case manager. 

> delete debug-log ?
cp-log      Remove cp-log at /opt/var.cp/log/pan/
dp0-log     Remove dp0-log at /opt/var.dp0/log/pan/
dp1-log     Remove dp1-log at /opt/var.dp1/log/pan/
dp2-log     Remove dp2-log at /opt/var.dp2/log/pan/
mp-global   Remove mp-global at /opt/mp-global/
mp-log      Remove mp-log at /var/log/pan/   <--- mp-log example used below

> delete debug-log mp-log file *.1
> delete debug-log mp-log file *.2
> delete debug-log mp-log file *.3
> delete debug-log mp-log file *.4
> delete debug-log mp-log file *.old

Clear Any packet-diag Logging If Enabled

Run > debug dataplane packet-diag show setting to check if packet-diag is enabled.

> debug dataplane packet-diag show setting

DP dp0:
--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
  Enabled:                   no
  Match pre-parsed packet:   no            
--------------------------------------------------------------------------------
Logging
  Enabled:                   no     <-- No is the default value and means not enabled.

> debug dataplane packet-diag show setting

DP dp0:

--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
  Enabled:                   no
  Match pre-parsed packet:   no            
--------------------------------------------------------------------------------
Logging
  Enabled:                   yes   <--- Means packet-diag is enabled.
To clear out packet-diag setting and the logs, run below commands:
> debug dataplane packet-diag clear all
Packet diagnosis setting set to default.

 > debug dataplane packet-diag clear log log 
dataplane debug logs cleared

Delete Any Debug pcaps or Debug-filter pcaps

> delete debug-filter file <file-name>
> delete pcap directory *        

Set the logging-level back to its default for all processes running on the firewall:

> debug software logging-level set level default service all-services

If none of the above remediation steps resolve the issue, it is recommended to collect the following Troubleshooting Data below and open a Support Case.

  1. Collect Tech Support File  (GUI: Device > Support  Click Generate Tech Support File)
  2. Collect the output of the CLI show system disk-space 
Important to note:
If your FW is a PA-220 running PAN-OS version 10.1.x or 10.2.x but less than 10.1.10-h1 or 10.2.5 then upgrade your device to 10.1.10-h1, 10.2.5 or higher to fix issue mentioned in PAN-219659.
Also we are still trying to create more free space on PA-220 in future release. Please reach out to support if the steps listed above don't help you solve this problem on PA-220 running PAN-OS version 10.1.10-h1 or 10.2.5 or higher.


Additional Information


Related Issues
Disk Usage Exceeds Limit 95 Percent After Upgrade To PAN-OS 10.2.0

For additional information, please review the following articles:
How to Delete Unnecessary Downloaded Software Versions
How to delete configurations through the CLI
How to Delete Saved Configuration Files


Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClaJCAS&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Choose Language