Commit failing on panorama after upgrade to 10.2.3 due to "meta data not found for dg" validation error

• Panorama upgraded to PAN-OS 10.2.3
• After upgrade, commit fails with device group error seen below

• Configd logs (less mp-log configd.log) report "meta data not found" and "schema verfication failed" messages.

-0800 Error: pan_cfg_dgname_validate(pan_cfg_devicegroups.c:2245): meta data not found for dg mu-dg-ECS_Prisma_Tenant
-0800 Error: pan_schema_verify_attribute(pan_schema_types.c:1054): 'mu-dg-ECS_Prisma_Tenant' is invalid. meta data not found for dg mu-dg-ECS_Prisma_Tenant near line 14395
-0800 Error: pan_schema_verify_attr(pan_schema_obj.c:5762): attribute name breaks schema at line 14395
-0800 Error: pan_cfg_verify_ex(pan_cfg_commit_handler.c:2987): invalid configuration. Schema verification failed.
-0800 Clearing commit completion cache2023-01-25 00:36:37.148 -0800 Error: pan_jobmgr_process_job(pan_job_mgr.c:3938): error verifying commit candidate

• Panorama upgraded to 10.2.3
• Panorama with Prisma Access plugin 3.2.1
• Panorama with Prisma Access multi tenancy

• This is due to the old device-group (DG) configuration which were not removed when the sub tenant was deleted.
• Now Panorama treats it as invalid configuration as it couldn't find metadata related to the device group.

To resolve this issue, Use option1 below. If it does not resolve the same, proceed to option 2

1. Login to Panorama.
2. Ensure there are no pending changes to be committed (usually by other admins) using GUI: Commit > Commit to Panorama > Commit all changes > Preview changes. If there are changes it can be noted down to perform the changes later.
3. Note down the process ID (PID) of management process

>show system resources | match mgmt 

3. Restart the process: 

> debug software restart process management-server

4. After awaiting a few minutes, Re-login and check if the service successfully restarted:

 show system resources | match mgmtsrvr

5. Perform a commit force

> configure
# commit force
# exit

1. Confirm with the customer if the device group is needed or can be delete.
2. Once confirmed, enter configure mode in the cli  and delete the device group.

> configure
# delete device-group [Group Name]

3. Once deleted, Confirm the device-group is no longer in the configuration by using "show device-group"

# show device-group

4. Once the device group is deleted,  Perform a commit force and exit the configuration.

# commit force
# exit

5. Once successful, push it to Prisma Access.
6. If the issue is still unresolved, Open a support case.

• Confirm if the device-group is being used currently. In this case it is not being used as it was created during POC stage
• Here the commit fails because the panorama has the unwanted or unused device group from the old sub tenant
• POC sub tenant was already deleted before. however, the configs related to sub tenant such as device group, templates were not deleted. Hence it showed up after the upgrade