High Disk Space Usage on /opt/pancfg partition and How To Clear

204772

Created On 04/08/19 13:54 PM - Last Modified 04/09/24 15:11 PM

Dynamic Updates

Content Release

Investigation and Response

Hardware

PAN-OS

Panorama

Symptom

Unable to log into Web UI
Unable to download PAN-OS release images (Device > Software Updates)
Unable to download Dynamic Wildfire, Content images (Device > Dynamic Updates)

The /opt/pancfg partition is greater than 90 percent

> show system disk-space 

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             7.6G  2.8G  4.4G  40% /
/dev/sda5              23G   21G  1.4G  94% /opt/pancfg         <------ Over 90%
/dev/sda6              16G   13G  1.5G  91% /opt/panrepo
tmpfs                 7.9G  110M  7.8G   2% /dev/shm
cgroup_root           7.9G     0  7.9G   0% /cgroup
/dev/sda8              56G   17G   37G  32% /opt/panlogs
/dev/loop0             16G  173M   15G   2% /opt/logbuffer

Environment

PAN-OS
Palo Alto Firewall
Panorama

Cause

Some of the usual causes of /opt/pancfg filling up are:

Large number of saved configurations
Large number of downloaded PAN-OS software image files
In Panorama, large number of downloaded PAN-OS software image files for managed devices under Panorama> Device Deployment
Although there is still 1.4GB free space left in the /opt/pancfg partition, the downloads will still fail. The PAN-OS runs a strict check on the disk space on /opt/pancfg. If it is 90% or above, the downloads will fail regardless of the available free disk-space.

PA-50XX 8.1 - Returns following error in the ms.log

Example:
Error:  _pan_mgmtop_content_upgrade_install_file(pan_ops_content.c:5810): Command /usr/local/bin/masterd_batch -s -p 10 -o/tmp/.xml_string.139789456291584.1466907301 content_install /usr/local/bin/paninstaller.sh  -tcontent -f/opt/pancfg/mgmt/content-images/.latest-content.tar.gz -
d/opt/pancfg/mgmt/content-images -o/opt/pancfg/mgmt/updates/curcontent -n/opt/pancfg/mgmt/updates/newcontent return failure, sys_rc=-256, with the following message:cp: cannot stat `global/2.1.0/global_app.xml': No such file or directory

Important Note About Panorama:
Whenever a candidate configuration is saved on a firewall managed by Panorama, either via the web interface (Device > Setup > Operations > Save named configuration snapshot) or via the CLI (“save config to <filename>”), the same file is also saved on Panorama in the directory /opt/pancfg/mgmt/devices/<SN>/. In this case, <SN> is the serial number of the firewall.

Whenever a saved candidate configuration on the firewall is deleted (“delete config saved <filename>”), the configuration file is NOT automatically deleted from Panorama. By having numerous firewalls managed by the same Panorama, some of them with large configuration files will eventually lead to exhaustion of the available disk space on the /opt/pancfg/ partition.

Resolution

From the CLI
1. Check which pancfg subdirectory usage is high using CLI available starting PAN-OS 10.2:

 show system pancfg-directory-usage

2. To delete Software and Dynamic Update Images use CLI:

delete software version <version-number>
delete config saved <file-name>
delete config repo <Named snapshot>
delete content cache old-content
delete wildfire update <value>
delete anti-virus update <value>

3. To delete saved config files from specific devices (Panorama only) use CLI:

Panorama> delete config repo device <device SN> file <Named snapshot>

Note: No bulk delete is possible through the Web Interface or CLI.

4. To lower the number of maximum images to keep use CLI below. Default is set to 5.

> show max-num-images
Maximum images to keep is set to 5

To change the number stored to a minimum of 2.
> set max-num-images count 2

5. To limit the device monitoring allocation which is set to default at 6GB use CLI: (Panorama only)

> debug management-server device-monitoring disk-quota set force yes size 2048

6. To limit the size of the report storage for Panorama use CLI:

> request report-storage-size set size <0-4>

From the Web-GUI

Delete any unused Software images (Device>Software)
Delete any unused Content images (Device>Dynamic Updates)

If none of the above remediation steps resolve the issue, it is recommended to collect the following Troubleshooting Data below and open a Support Case.

Collect Tech Support File (GUI: Device > Support Click Generate Tech Support File)
Collect the output of the CLI show system disk-space

Additional Information

For Virtual Panorama Devices in Legacy mode see KB below on increasing Disk space

HIGH DISK UTILIZATION ON /OPT/PANCFG PARTITION ON VIRTUAL PANORAMA