High Disk Space Usage on / root partition and How To Clear

High Disk Space Usage on / root partition and How To Clear

429803
Created On 09/25/18 19:37 PM - Last Modified 04/15/22 18:21 PM


Symptom
Some common symptoms of the root partition getting filled up are:
  • Unable to log in or access Web UI
  • Certain daemons processes not starting
  • Incomplete tech support bundles
  • System log alerts with "Disk usage for / exceeds limit,  X percent in use, cleaning file system"
  • Root partition is high
> show system disk-space 
Filesystem            Size  Used Avail Use% Mounted on
/dev/md2              3.8G  3.1G  581M  85% /                   <----- root partition
/dev/md5              7.6G  4.2G  3.0G  59% /opt/pancfg .        
/dev/md6              3.8G  3.0G  666M  82% /opt/panrepo
tmpfs                 2.0G  210M  1.8G  11% /dev/shm
cgroup_root           2.0G     0  2.0G   0% /cgroup
/dev/md8               88G  2.4G   81G   3% /opt/panlogs
tmpfs                  12M     0   12M   0% /opt/pancfg/mgmt/lcaas/ssl/private


Environment
  • PAN-OS
  • Palo Alto Firewall
  • Panorama


Cause

Some of the common causes of a filled partition:

  • An admin troubleshooting certain processes and creates core files. These files are stored on the root and remain there until deleted by the administrator.
  • Enabling of diagnostic logs for the dataplane (packet diags) can also take up space on the root partition.


Resolution

Enable Aggressive Cleaning (PAN-OS 7.1.14+, 8.0.7+, 8.1.0+)

This will automatically truncate all old log files (entries under all *var/log/pan directories matching *.1, ... *.4, *.log.old) if the 95% occupancy alarm is tripped.

 > debug software disk-usage aggressive-cleaning <enable|disable>

Note: Enabling aggressive clean up may clear up logs, rendering logs unavailable for troubleshooting  purposes.

To verify the changes or if it is already enabled use the command below.

> show system state | match aggressive-cleaning
cfg.debug-sw-du.config: { 'aggressive-cleaning': True, }

 

Run disk-usage cleanup command (PAN-OS 8.1.0+)

disk-usage cleanup can be done manually using the command below:
> debug software disk-usage cleanup ?
+ deep        cleanup with deleting backup logfile
* threshold   percentage threshod of size of system partition to kick off cleanup

> debug software disk-usage cleanup deep threshold 90

Note that this command has to be run manually each time to bring the disk usage to below 90% and is not persistent across reboots.
Note: Enabling aggressive clean up may clear up logs, rendering logs unavailable for analysis.
 

Check and Delete Unnecessary Core Files

  • Check the output of show system files to see core files using up a large amount of disk space.
> show system files
/opt/dpfs/var/cores/:
total 4.0K
drwxrwxrwx 2 root root 4.0K Jun 10 20:05 crashinfo

/opt/dpfs/var/cores/crashinfo:
total 0
 
/var/cores/:
total 115M
drwxrwxrwx 2 root root 4.0K Jun 10 20:15 crashinfo
-rw-rw-rw- 1 root root 867M Jun 12 13:38 devsrvr_4.0.3-c37_1.gz
-rw-rw-rw- 1 root root  51M Jun 12 13:39 core.20053
 
/var/cores/crashinfo:
total 16K
-rw-rw-rw- 1 root root 15K Jun 10 20:15 devsrvr_4.0.3-c37_0.info
  • Delete unnecessary core files. A core file can be deemed unnecessary if investigation around the core file is complete or they are very old files.
> delete core management-plane file devsrvr_4.0.3-c37_1.gz
  • These files can be exported using the scp or tftp export command, then deleted to free up space on the firewall
> scp export core-file management-plane from mgmtsrvr_7.0.3_0.tgz to user@10.0.0.10:/home/ 

user@10.0.0.10 's password: ********
mgmtsrvr_7.0.3_0.tgz                       100%  453MB  46.4MB/s   00:12  
 

Delete Rotated Files and Files with Extension .old 

These files contain monitoring details and service related logs on the firewall. They can be deleted safely if you don't need them. If TAC investigates an ongoing issue, you may prefer to keep them until you upload the tech support file to the case manager. 

> delete debug-log ?
cp-log      Remove cp-log at /opt/var.cp/log/pan/
dp0-log     Remove dp0-log at /opt/var.dp0/log/pan/
dp1-log     Remove dp1-log at /opt/var.dp1/log/pan/
dp2-log     Remove dp2-log at /opt/var.dp2/log/pan/
mp-global   Remove mp-global at /opt/mp-global/
mp-log      Remove mp-log at /var/log/pan/   <--- mp-log example used below

> delete debug-log mp-log file *.1
> delete debug-log mp-log file *.2
> delete debug-log mp-log file *.3
> delete debug-log mp-log file *.4
> delete debug-log mp-log file *.old

Clear Any packet-diag Logging If Enabled

Run > debug dataplane packet-diag show setting to check if packet-diag is enabled.

> debug dataplane packet-diag show setting

DP dp0:
--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
  Enabled:                   no
  Match pre-parsed packet:   no            
--------------------------------------------------------------------------------
Logging
  Enabled:                   no     <-- No is the default value and means not enabled.

> debug dataplane packet-diag show setting

DP dp0:

--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
  Enabled:                   no
  Match pre-parsed packet:   no            
--------------------------------------------------------------------------------
Logging
  Enabled:                   yes   <--- Means packet-diag is enabled.
To clear out packet-diag setting and the logs, run below commands:
> debug dataplane packet-diag clear all
Packet diagnosis setting set to default.

 > debug dataplane packet-diag clear log log 
dataplane debug logs cleared

Delete Any Debug pcaps or Debug-filter pcaps

> delete debug-filter file <file-name>
> delete pcap directory *        

 


If none of the above remediation steps resolve the issue, it is recommended to collect the following Troubleshooting Data below and open a Support Case.
  1. Collect Tech Support File  (GUI: Device > Support  Click Generate Tech Support File)
  2. Collect the output of the CLI show system disk-space 


Additional Information
Related Issues
Disk Usage Exceeds Limit 95 Percent After Upgrade To PAN-OS 10.2.0

For additional information, please review the following articles:
How to Delete Unnecessary Downloaded Software Versions
How to delete configurations through the CLI
How to Delete Saved Configuration Files


Attachments
Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClaJCAS&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Attachments
Choose Language