High policy config memory usage and commit failures with error "failed to handle CONFIG_UPDATE_START"
7631
Created On 04/12/24 21:58 PM - Last Modified 04/24/24 15:31 PM
Symptom
- High percentage of policy cache usage in the output of the CLI:
> debug dataplane show cfg-memstat statistics VSYS POLICY CACHE Allocator Usage POLICY CACHE (ACTIVE) : 61461KB POLICY CACHE USAGE : 60032KB (97% of 61440KB) << : POLICY CACHE (PASSIVE) : 61461KB POLICY CACHE USAGE : 60032KB (97% of 61440KB) <<
- Commit failure
- Example of commit failure error messages seen in this case:
"Details:Error: Failed to get vsys config, already allocated (2621440 bytes) failed to handle CONFIG_UPDATE_START"
Environment
- Any platform that is running PAN-OS including VMs used by Prisma Access.
Cause
- Too many IP addresses/FQDNs/tags(registered IPs)/DAGs are configured as the source or destination addresses of the policy rules.
- Too many IP addresses/FQDNs are configured in EDLs and the EDLs are configured as source or destination addresses of the policy rules;
- Too many policy rules are configured (in most cases, they are security policy rules).
Resolution
- Reduce the number of IP addresses/FQDNs/tags(registered IPs)/DAGs configured as the source or destination address of policy rules.
- Reduce the number of IP addresses/FQDNs configured in EDLs that are configured as the source or destination address of policy rules.
- Reduce the number of policy rules configured.
- For a Multi Virtual System (multi-vsys) firewall, consider reducing the number of vsys by deleting any unused vsys.
- If even after following the recommendation listed above you are unable to reduce the policy config memory usage then:
- For a hardware FW consider upgrading your FW to a higher capacity platform, for a multi-slot firewall consider upgrading the card affected to a higher capacity card if possible.
- For a VM-Flex FW if its running a version lower than 10.2.0, consider upgrading to a version greater than 10.2.0 to take advantage of the increased configuration capacity offered by the Memory Scaling of the VM-Series Firewall Feature. Also consider increasing the FW memory/RAM to increase the capacity of your VM-Flex FW.
Additional Information
- If the problem is determined to be related to the predefined external dynamic list, it's important to understand that the firewall receives updates for these feeds through antivirus dynamic updates. To mitigate this issue:
- Reduce the EDL entries in the EDL files by excluding entries as needed until the commit succeeds and the firewall moves out of error condition. Refer to https://docs.paloaltonetworks.com/pan-os/10-1/pan-os-admin/policy/use-an-external-dynamic-list-in-policy/exclude-entries-from-an-external-dynamic-list.
- If the problem is determined to be related to a custom external dynamic list and there is a need to delete it then use CLI command:
> debug external-list delete-file type ip name <name of the custom edl list>
POLICY CACHE (ACTIVE): This represents the currently allocated cache for policy configuration. It is also referred to as "Current Policy Config Memory Usage".
POLICY CACHE (PASSIVE): This represents the allocated alternate cache for policy configuration. It is the one that will be used for policy configuration in the upcoming commit. It is also referred to as "Alternate Policy Config Memory Usage".