BuildXMLCache job failing due to 'CACHE BUILD ERROR: Could not find the DG node for <device-group>'
6311
Created On 08/01/23 21:23 PM - Last Modified 08/23/23 18:37 PM
Objective
After certain job operations (such as Load, Commit, etc.), a BuildXMLCache job may be initiated. Under some rare circumstances, this job may fail:
> show jobs all
Enqueued Dequeued ID Type Status Result Completed
------------------------------------------------------------------------------------------------------------------------------------------
2023/05/02 00:07:56 00:07:56 441881 BuildXMLCache FIN FAIL 00:08:00
2023/05/02 00:07:09 00:07:09 441880 Load FIN OK 00:07:57
One common cause of this issue is an unused Device Group (DG) is referenced in the configuration but is not configured as part of any Device Group Hierarchy which is actually used to push configurations to Firewalls. This configuration state may occur after a Device Group Hierarchy is changed, a specific series of object renaming and partial commits done by users within a Device Group Hierarchy, or other similar operations.In the example below, the Device Group named Device-Group-1 is present in the <readonly> section of the running-config.xml. However, the Device Group 'Device-Group-1' is not actually assigned to be pushed to any devices in the rest of the running-config.xml. Also, there were references left to 'Device-Group-1' throughout our running-configuration.xml, specifically in the '<readonly>' section of the running-config.xml and in child objects throughout the configuration.
While Device-Group-1 is configured and the Device Group does exist, that Device Group is not used in any Device Group Hierarchy which gets pushed down to any firewalls. Thus, if it is referenced in any child objects (address objects, etc.) in the firewall configuration but is not a part of any Device Group Hierarchy nor any Templates/Template Stacks, the BuildXMLCache job may fail. If removing this Device Group would have no effect on your configurations, it may be safely removed/deleted, and the BuildXMLCache job, Load job, and Commit and Push job will succeed after that.
Warning: Verify that this Device Group is not configured to push to any devices before deleting
If this state occurs, perform the steps below to resolve the issue of the BuildXMLCache job failing:
Environment
- Panorama
Procedure
- Identify the Device Group object and/or other object that is causing the issue by searching for this error in configd.log:
> less mp-log configd.log
2023-05-22 16:49:00 Error: _traverse_dg_hierarchy(pan_xml_cache_mgr.c:3690):
CACHE BUILD ERROR: Could not find the DG node for <Device-Group-1>
- Identify if the Device Group 'Device-Group-1' entry appears in the <readonly> section in your configuration. Ensure that the Device Group Device-Group-1 is not present anywhere else in the configuration and is not part of a Device Group Hierarchy. If this Device Group is only in the <readonly> section and not referenced or used anywhere else in your configuration and is safe to remove, delete the Device Group named 'Device-Group-1' from the configuration. If applicable, remove any instances of 'Device-Group-1' being referenced anywhere else in your configuration (as the DG those objects reference now does not exist since you deleted it in this step).
>show config running
<readonly>
<parent-dg>Remote Offices</parent-dg>
</entry>
<entry name="Device-Group-1">
<id>1234</id>
<address-group>
<entry name="Block List">
<id>1235</id>
</entry>
</readonly>
Once verified that Device-Group-1 and all references to it in the configuration have been resolved/removed, proceed to the steps below
- Perform the Load config operation again. Verify the Load job and BuildXMLCache job now both succeed:
Web UI: Panorama > Setup > Operations
CLI: #configure
#load config from running-config.xml
CLI: #configure
#load config from running-config.xml
- Run the CLI Command below to regenerate all the Device Group and Template configurations:
> debug md5sum_cache clear
Note: If the BuildXMLCache job is still failing, do not proceed below - return to step 1 above.
Note: If the BuildXMLCache job is still failing, do not proceed below - return to step 1 above.
- Perform a Commit Force:
>configure
#commit force
#commit force
Additional Information
To prevent this issue from occurring in the future, refer to the link below:
"After adding, editing, or deleting a device group, perform a Panorama commit and device group commit (see Panorama Commit Operations). Panorama then pushes the configuration changes to the firewalls that are assigned to the device group"
Reference: PAN-OS Web Interface Help - Panorama > Device Groups