/opt/pancfg disk partition full preventing download of dynamic updates or logging into device

/opt/pancfg disk partition full preventing download of dynamic updates or logging into device

44981
Created On 04/08/19 13:54 PM - Last Modified 05/11/21 06:43 AM


Symptom
You may observe that the scheduled dynamic updates do not work. When trying to download dynamic updates on PAN-OS software, you may encounter an error that indicates that there is not enough free disk space to complete the desired operation.
WebGUI_Error
Error: There is not enough free disk space to complete the desired operation. Delete older software, dynamic update, or client versions to free additional disk space before trying the operation again. Use the 'set max-num-images count' CLI command to adjust the number of versions stored in order to avoid this problem in the future.


 


Environment
This can happen on any PAN-OS device.

Cause
This error happens when the /opt/pancfg partition is greater than 90 percent. To check disk space, run the following command: 
> show system disk-space.

Here is an example of the output from a Panorama device:
user@panorama> show system disk-space 

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             7.6G  2.8G  4.4G  40% /
/dev/sda5              23G   21G  1.4G  94% /opt/pancfg
/dev/sda6              16G   13G  1.5G  91% /opt/panrepo
tmpfs                 7.9G  110M  7.8G   2% /dev/shm
cgroup_root           7.9G     0  7.9G   0% /cgroup
/dev/sda8              56G   17G   37G  32% /opt/panlogs
/dev/loop0             16G  173M   15G   2% /opt/logbuffer
/dev/md2              917G   92M  871G   1% /opt/panlogs/ld2
/dev/md3              917G  111M  871G   1% /opt/panlogs/ld3
/dev/md4              917G   91M  871G   1% /opt/panlogs/ld4
/dev/md1              917G   91M  871G   1% /opt/panlogs/ld1

Although there is still 1.4GB free space left in the /opt/pancfg partition, the downloads will still fail. The PAN-OS runs a strict check on the disk space on /opt/pancfg. If it is 90% or above, the downloads will fail regardless of the available free disk-space.

Some of the usual causes of /opt/pancfg filling up are as follows:
  • Large number of saved configurations
  • Large number of downloaded PAN-OS software image files
  • In Panorama, large number of downloaded PAN-OS software image files for managed devices under device deployment
  • Large size of the mongodb database. This database holds configuration and device monitoring data. 

Important Note About Panorama:
Whenever a candidate configuration is saved on a firewall managed by Panorama, either via the web interface (Device > Setup > Operations > Save named configuration snapshot) or via the CLI (“save config to <filename>”), the same file is also saved on Panorama in the directory /opt/pancfg/mgmt/devices/<SN>/. In this case, <SN> is the serial number of the firewall.

Whenever a saved candidate configuration on the firewall is deleted (“delete config saved <filename>”), the configuration file is NOT automatically deleted from Panorama. By having numerous firewalls managed by the same Panorama, some of them with large configuration files will eventually lead to exhaustion of the available disk space on the /opt/pancfg/ partition.

 


Resolution
The idea is to recover disk-space from the /opt/pancfg partition.
1. In order to recover the disk-space, saved configuration files should be manually deleted from Panorama either by using the web interface or via the CLI.
delete config repo device <SN> file <filename>

Caveat: No bulk delete is possible through the web interface or CLI, and the CLI command does not accept wildcards.

2. Apart from the above, other commands that can help clear up space are as follows:
delete software version <version-number>
delete config saved <file-name>
delete config repo <Named snapshot>
delete content cache old-content
delete wildfire update <value>
delete anti-virus update <value>
3. We can see the number of images we store with this command:
show max-num-images

We can change the number stored to a minimum of 2.  Default is 5.
set max-num-images count 2

4. To reduce the size of mongodb. You will lose some device monitoring data with this command. 
  • Cap total size of device monitoring database in mongodb via command "debug management-server device-monitoring disk-quota set size 2048"
  • If disk usage is still high, open a TAC case. The database might need to be defragmented 


Additional Information
Here are some other useful articles related to failing dynamic updates:
Dynamic Updates Display Error after Clicking on Check Now Button
Dynamic Update Fails with Image File Authentication Error Message
Unable to Perform Dynamic Updates with updates.paloaltonetworks.com FQDN Address Object
Dynamic Updates for AntiVirus Fail
Dynamic Updates for Applications and Threats will not Install
Cannot Schedule Dynamic Updates from Panorama for Firewalls
Dynamic Updates Error failed to get a response from the device server


Attachments
Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000PLSJCA4&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Attachments