How to Troubleshoot Commit/Push Job in Status 'Reverted' due to Panorama Automated Commit Recovery
23199
Created On 10/18/23 03:23 AM - Last Modified 10/18/23 08:08 AM
Objective
When a user Commits/Pushes a configuration from Panorama to the firewall which will break the connection between Panorama and the managed firewall after the pushed changes successfully take effect, the Automated Commit Recovery feature in Panorama (enabled by default) will check to ensure the Panorama and firewall can still reach each other with the newly successfully-pushed configuration in place. Additional info on this feature can be found in Automatic Panorama Connection Recovery.
Below are some common configuration changes made by users that may cause Panorama to lose connectivity to Firewalls after a commit/push:
- Management Interface or Service Route changes
- Security Policy or Interface Management Profile changes
- Changes to routes or routing protocols
- Introducing decryption, QoS, or other network features in the connection path between the Panorama and firewall which interfere with the normal flow of traffic/packets
- Change in MTU/MSS of interfaces along the path between Panorama and Firewall causing fragmentation in the connection
Tasks - Commit Job
>show jobs all
> show jobs all Enqueued Dequeued ID PositionInQ Type Status Result Completed ------------------------------------------------------------------------------------------------------------------------------------------ 2023/02/19 04:45:06 04:41:06 3 CommitAll FIN REVERT 04:51:11 <<<<<<<<<<<<<<<<<< 2023/02/19 04:42:17 04:43:17 2 AutoCom FIN OK 04:46:50 2023/02/19 04:41:02 04:40:02 1 AutoCom FIN OK 04:42:29
> show jobs id 3 Enqueued Dequeued ID Type Status Result Completed ------------------------------------------------------------------------------------------------------------------------------ 2020/07/13 22:28:23 21:45:29 3 CommitAll FIN REVERT 21:22:44 Details:Performing panorama connectivity check (attempt 1 of 1) Panorama connectivity check failed for 10.30.20.10. Reason: TCP channel setup failed, reverting configuration Configuration reverted successfullySystem Log
> show log system 2023/05/21 08:42:23 critical panoram panoram 0 JobId=6231: Panorama connectivity check failed for panorama.example.com. Reason: TCP channel setup failed, reverting configuration
Environment
- Panorama
- PAN-OS
- Commit & Push
Procedure
- Verify that the configuration being pushed to the firewall does not inherently break the connection between Panorama and the firewall. Once identified, change/delete that configuration change so that the connection (TCP Port 3978) between Panorama and the firewall is no longer broken, disrupted, or blocked/denied the next time it is pushed to the firewall. See Troubleshoot Automatically Reverted Firewall Configurations for help with identifying which configuration change was made that caused this connection failure to occur.
- Increase 'Automated Commit Recovery' retry attempts:
Navigate to Device > Setup > Management > Panorama Settings > Number of attempts to check for Panorama connectivity > Increase the number to a higher value (such as 5 or 10 retries, for example)
This setting may be useful in network environments where a temporary/brief connectivity loss is expected between Firewall and Panorama after the push is done. Examples of this scenario include: low bandwidth environments, routing protocols need time to re-converge, VPN tunnels need to re-form, path re-optimization needs to occur with the new configuration, interfaces need to come back up after interface configuration change, etc. after the Commit and Push is complete. In these cases, increasing the value of Number of attempts to check for Panorama connectivity and/or Interval between retries can allow extra time for those operations to complete before the firewall reverts the configuration back.
This setting may be useful in network environments where a temporary/brief connectivity loss is expected between Firewall and Panorama after the push is done. Examples of this scenario include: low bandwidth environments, routing protocols need time to re-converge, VPN tunnels need to re-form, path re-optimization needs to occur with the new configuration, interfaces need to come back up after interface configuration change, etc. after the Commit and Push is complete. In these cases, increasing the value of Number of attempts to check for Panorama connectivity and/or Interval between retries can allow extra time for those operations to complete before the firewall reverts the configuration back.
- Verify the connection between Panorama and the firewall is stable, has sufficient bandwidth, and is not experiencing congestion, slowness, fragmentation, or disruption of any kind.
Additional Information
Warning: If the Automated Commit Recovery setting is disabled and a configuration change is committed which breaks connectivity between Panorama and the firewall, access to the firewall may be lost and no option will be available but to physically access the firewall via Console Cable onsite. (i.e. the firewall is isolated and must be recovered to become managed/reachable by Panorama again). It is advised to not disable the Automated Commit Recovery setting whenever possible.