Auto commit failing after PanOS Upgrade to 10.2.4 with error "Management server failed to send phase 1 to client useridd"
Symptom
After upgrading PanOS to 10.2.4 or higher, auto commit is failing with the error below:
> show jobs id <JobID>
Enqueued Dequeued ID Type Status Result Completed
------------------------------------------------------------------------------------------------------------------------------
2023/11/16 10:06:33 10:06:33 73 AutoCom FIN FAIL 10:06:49
Details:config commit phase 1 aborted(Module: routed)
panike_daemon phase 1 aborted(Module: ikemgr)
Management server failed to send phase 1 to client useridd <<<<<<<<
Commit failed
Failed to commit policy to device
When checking the useridd.log, we see the below error messages:
2023-11-16 09:12:25.341 +0000 debug: pan_alloc_nofree_chunk(pan_alloc.c:1104): allocating 17328928 , bigger than chunk size 16777184
2023-11-16 09:12:25.347 +0000 reach 99 percent. start to trim down to 95 percent
2023-11-16 09:12:25.347 +0000 Disk quota (148838KB) is reached: 155112KB
2023-11-16 09:12:25.693 +0000 delete 678 entries
Environment
- PA-VM upgraded to PanOS 10.2.4
- Heavy utilization of GP HIP Reports
Cause
In PanOS 10.2.4, a change enforces the Quota for HIP Reports to be honoured (PAN-192681), and thus if there is not enough Quota allocated for HIP Reports, then useridd process will not be able to process Commits or Auto Commits from the Management Server.
Resolution
- Since AutoCommit is failing and an increase in Quota for HIP Reports cannot be Committed, a PanOS Downgrade is needed.
- Once the Downgrade is complete, if AutoCommit is still failing, then deleting the "HIP Report DB" from Root might be needed (Please contact TAC).
- Once AutoCommit is successful after the PanOs downgrade, then a higher Quota needs to be applied to "HIP Reports" as needed, and as described in this article.
- When the change is Committed, an upgrade to PanOS 10.2.4 or higher should not result in the same issue any longer.