CommitAll from Panorama may sometime result in failure
727
Created On 01/12/23 16:21 PM - Last Modified 11/04/25 19:10 PM
Symptom
- CommitAll job from Panorama to managed devices (Prisma Access or on-prem) may sometime result in failure with the error message "Client sslvpn requesting last config in the middle of a commit/validate. Aborting current commit/validate".
- This is generally observed in situations wherein the Panorama is performing multiple commits to the same firewall in quicker-than-normal succession. A couple of use cases would be -
- Committing from Panorama to Prisma Access (PA) instances wherein multiple changes (via multiple commits) are supposed to be pushed to the same firewall instance in a set window
- Scheduled CommitAll task from Panorama via XML-API that results in quick succession of commit jobs on firewalls.
- Also, if another commit is executed on the same erring firewall (either from the same panorama or otherwise) a couple of mins after the failed commit, it would be successful.
- The reason of commit failure seen in the corresponding job details is "Client sslvpn requesting last config in the middle of a commit/validate. Aborting current commit/validate". Also, system logs on the firewall during this timeframe show "SSLMGR daemon configuration load phase-1 aborted" message.
- The above logs generally mention sslvpn/SSLMGR process but may mention other processes with the remainder phrasing staying the same.
Environment
- Panorama managing firewalls (Prisma Access firewall instances or on-prem firewalls)
- CommitAll initiated from Panorama in such a manner that causes multiple commit jobs to be enqueued on the firewalls
- No specific PANOS requirements (as long as it adheres to the Panorama-firewall PANOS hierarchy)
Cause
During to the rapid succession of commits on the firewall, one commit may include changes to the GP settings (keepalive, timeout, etc). The processing of such a commit would cause devsrvr to trigger a restart of sslvpn process and is visible in masterd logs as such -
2020-10-08 07:29:18.520 +0100 INFO: sslvpn: User restart reason - triggered by devsrvr commit
The issue is caused when this sslvpn restart happens after the initiation of the subsequent commit. After sslvpn restart, it requests the lastcfg but since the subsequent commit was already underway, the request is invalidated.
The sequence would be as follows:
1. Commit succeeded around 07:28:55 -
2020-10-08 07:28:55.049 +0100 client device reported Phase 2 was SUCCESSFUL
2. Second commit triggered at around 07:29:23 -
2020-10-08 07:29:23.794 +0100 detail : Commit from Panorama. Merged with candidate config: Yes. Commit parameters: force=false, device_network=true, shared_object=true. Commit All Vsys.
3. Sslvpn restarted triggered by first commit and requested lastcfg -
2020-10-08 07:29:23.022 +0100 debug: pan_sys_init(pan_sys.c:506): appweb3-sslvpn 2020-10-08 07:29:23.023 +0100 waiting to be connected to sysd... 2020-10-08 07:29:23.029 +0100 sysd worker[0]: 7fffdf169700: starting up... ..... 2020-10-08 07:29:26.080 +0100 client sslvpn disabled 2020-10-08 07:29:28.143 +0100 Error: pan_mgmt_client_lastcfg_read_callback(pan_cfg_commit_jobs.c:652): get lastcfg is disabled due to outstanding commit/validate job. Cancelling the commit/validate and resetting the state machine to receover.
4. Second commit failed -
2020-10-08 07:29:28.143 +0100 Error: pan_cfg_commit_to_local_device(pan_cfg_commit_handler.c:3629): Commit failed
Resolution
No resolution. A couple of workarounds would be -
1. Manually trigger a commit a couple of mins after the failure.
2. If this issue is observed quite frequently for any kind of automated commit process, increase the duration between two consequent commits by 2 mins in the automation process.