/opt/pancfg disk partition full preventing download of dynamic updates or logging into device
Created On 04/08/19 13:54 PM - Last Modified 05/11/21 06:43 AM
You may observe that the scheduled dynamic updates do not work. When trying to download dynamic updates on PAN-OS software, you may encounter an error that indicates that there is not enough free disk space to complete the desired operation.
Error: There is not enough free disk space to complete the desired operation. Delete older software, dynamic update, or client versions to free additional disk space before trying the operation again. Use the 'set max-num-images count' CLI command to adjust the number of versions stored in order to avoid this problem in the future.
This can happen on any PAN-OS device.
This error happens when the /opt/pancfg partition is greater than 90 percent. To check disk space, run the following command:
> show system disk-space.
Here is an example of the output from a Panorama device:
user@panorama> show system disk-space Filesystem Size Used Avail Use% Mounted on /dev/sda3 7.6G 2.8G 4.4G 40% / /dev/sda5 23G 21G 1.4G 94% /opt/pancfg /dev/sda6 16G 13G 1.5G 91% /opt/panrepo tmpfs 7.9G 110M 7.8G 2% /dev/shm cgroup_root 7.9G 0 7.9G 0% /cgroup /dev/sda8 56G 17G 37G 32% /opt/panlogs /dev/loop0 16G 173M 15G 2% /opt/logbuffer /dev/md2 917G 92M 871G 1% /opt/panlogs/ld2 /dev/md3 917G 111M 871G 1% /opt/panlogs/ld3 /dev/md4 917G 91M 871G 1% /opt/panlogs/ld4 /dev/md1 917G 91M 871G 1% /opt/panlogs/ld1
Although there is still 1.4GB free space left in the /opt/pancfg partition, the downloads will still fail. The PAN-OS runs a strict check on the disk space on /opt/pancfg. If it is 90% or above, the downloads will fail regardless of the available free disk-space.
Some of the usual causes of /opt/pancfg filling up are as follows:
- Large number of saved configurations
- Large number of downloaded PAN-OS software image files
- In Panorama, large number of downloaded PAN-OS software image files for managed devices under device deployment
- Large size of the mongodb database. This database holds configuration and device monitoring data.
Important Note About Panorama:
Whenever a candidate configuration is saved on a firewall managed by Panorama, either via the web interface (Device > Setup > Operations > Save named configuration snapshot) or via the CLI (“save config to <filename>”), the same file is also saved on Panorama in the directory /opt/pancfg/mgmt/devices/<SN>/. In this case, <SN> is the serial number of the firewall.
Whenever a saved candidate configuration on the firewall is deleted (“delete config saved <filename>”), the configuration file is NOT automatically deleted from Panorama. By having numerous firewalls managed by the same Panorama, some of them with large configuration files will eventually lead to exhaustion of the available disk space on the /opt/pancfg/ partition.
The idea is to recover disk-space from the /opt/pancfg partition.
1. In order to recover the disk-space, saved configuration files should be manually deleted from Panorama either by using the web interface or via the CLI.
delete config repo device <SN> file <filename>
Caveat: No bulk delete is possible through the web interface or CLI, and the CLI command does not accept wildcards.
2. Apart from the above, other commands that can help clear up space are as follows:
delete software version <version-number> delete config saved <file-name> delete config repo <Named snapshot> delete content cache old-content delete wildfire update <value> delete anti-virus update <value>3. We can see the number of images we store with this command:
We can change the number stored to a minimum of 2. Default is 5.
set max-num-images count 2
4. To reduce the size of mongodb. You will lose some device monitoring data with this command.
- Cap total size of device monitoring database in mongodb via command "debug management-server device-monitoring disk-quota set size 2048"
- If disk usage is still high, open a TAC case. The database might need to be defragmented
Here are some other useful articles related to failing dynamic updates:
Dynamic Updates Display Error after Clicking on Check Now Button
Dynamic Update Fails with Image File Authentication Error Message
Unable to Perform Dynamic Updates with updates.paloaltonetworks.com FQDN Address Object
Dynamic Updates for AntiVirus Fail
Dynamic Updates for Applications and Threats will not Install
Cannot Schedule Dynamic Updates from Panorama for Firewalls
Dynamic Updates Error failed to get a response from the device server