Palo Alto Networks Knowledgebase: Best Practices for PAN-OS Upgrade

Best Practices for PAN-OS Upgrade

91045
Created On 04/23/19 09:41 AM - Last Updated 04/23/19 09:42 AM
Upgrade-Downgrade Content Release High Availability Installation 8.1 8.0 7.1 9.0 PAN-OS Panorama
Resolution

Best Practices for PAN-OS Upgrade

The following procedure documents best practices for customers who are new to the PAN-OS upgrade process. It's intended as a foundation for customers who want to create their own more-specific upgrade procedures.
 

About the PAN-OS upgrade and customer responsibilities

You cannot skip installation of any feature release versions in the path to your target PAN-OS release. Additionally, it is best practice to always download and install the latest maintenance release for each feature release and then reboot before you install the base image for the next feature release, which applies to each feature release through which you pass in the upgrade path.
 
In this example, we are upgrading a hypothetical customer (ACME, Inc.) from PAN-OS 7.1.16 to 8.1.6 (with 8.0 as an interim step).

ACME firewall is configured in Active/Passive HA cluster managed by Panorama (this is the most common configuration in use today). We are not covering Active/Active, non-HA scenarios or scenarios where there is no Panorama installed.

This is a best practice document. It is not meant to be a step-by-step procedure. Please create your own step-by-step procedure, if necessary.

Customer is responsible for verifying all steps before the upgrade.

Terminology

Active firewall

The firewall in an HA cluster that's passing traffic

Passive firewall

The firewall in an HA cluster that's not passing traffic

Primary firewall

The firewall in an HA cluster that's usually the active firewall

Secondary firewall

The firewall in an HA cluster that's usually the passive firewall

Feature release

 

Contains new features and bug fixes, typically ends with .0 (i.e. 8.1.0)

Maintenance release

Bug fixes only, typically ends with .number (8.1.2)

 

Dependencies

Before you upgrade, make sure the firewall is running a version of app + threat (content version) that meets the minimum requirement of the new PAN-OS (see release notes). We recommend always running the latest version of content to ensure the most accurate and effective protections are being applied.


Panorama should be running the same or a later version of a feature release than the firewall (up to two feature versions is supported but not recommended as of June 2016).

If the firewall is running

Panorama versions supported are

PAN-OS 7.1.x

 7.1.x, 8.0.x, 8.1.x

PAN-OS 8.0.x

8.0.x, 8.1.x, 9.0.x

PAN-OS 8.1.x

8.1.x, 9.0.x

 

TABLE OF CONTENTS

  1. Determine the need to upgrade
  2. Pre-upgrade checklist
  3. Panorama upgrade procedure
  4. Firewall upgrade procedure (HA)
  5. Post-upgrade checklist
  6. Troubleshooting resources
  7. Downgrade procedure
 

 

1) Determine the Need to Upgrade


In most cases, upgrade should be considered for only the following reasons:

New features that are not available in current version

Patches for security vulnerabilities in PAN-OS (see Security Advisories page)

Bug fixes that are not available in current version

Current version is going to End of Life soon (see PAN-OS EOL policy)

We highly encourage customers to consult with Palo Alto Networks account team for upgrade decision. Your Palo Alto Networks account team can provide you with a recommended PAN-OS version.

For the purpose of this document, we will be upgrading from 7.1.16 to 8.1.6 to demonstrate the upgrading process across two major releases (7.1 > 8.0 > 8.1).

NOTE: For any PAN-OS version prior to PAN-OS 8.0 (PAN-OS 7.1 and lower), it is recommended to go to the latest maintenance release to prevent running into issues during the upgrade.

NOTE about PAN-OS 8.0 base installation: For PAN-OS 8.0, the additional step of installing  and rebooting the base image was added to accommodate the larger size of the base image. This is considered best practice. (This is optional when installing from the updates server, but it is mandatory when installing from a manual file.)

HA Upgrade NOTE: When upgrading across two major release versions at a time, you will likely experience a network outage. However, if the devices are upgraded one major version at a time, HA will remain active and continue to synchronize sessions and no network outage will be seen.

To maintain HA sync and activity, upgrade the HA pair in tandem one major release at a time. If you upgrade one device by two major upgrades, the newly upgraded device will stay in suspended mode with the error peer OS too old. When you start the first OS upgrade on the second HA device, you will lose network connectivity until the upgrade is completed and the first device is moved out of suspended mode and into passive mode and HA capabilities resume functioning.



2) Pre-Upgrade Checklist

Review release notes.

Do not schedule Panorama and firewall upgrades at the same time.

Upgrade Panorama first, wait at least 24 hours and then upgrade the firewall.

A pro-active support case may be desirable for some critical environments. Please review this article: When to request a Support Event

Upgrade should be carried out during non-business hours to minimize impact.

Allocate sufficient time in the change window for upgrade, troubleshooting and possible downgrade procedures. It may take up to 2-3 hours to upgrade a slower or older system, depending on config. Multiply if upgrading across multiple versions.

Identify business contacts who will help verify application and network functionalities after the upgrade.

Back up configuration and device state before upgrade.

View of PAN-OS Device setup screen in Configuration Management
 
  • Device > Setup > Operations > Save Named Configuration Snapshot
  • Device > Setup > Operations > Export Named Configuration Snapshot
  • Device > Setup > Operations > Export Device State
  • Device > Support > Generate Tech Support File


View of PAN-OS Device support screen in Tech Support File

Document any non-standard settings that should be applied post-upgrade

Disable TCP state checking

Non-syn TCP reject

Any debug setting if existing (not recommended)

Stage/Download necessary PAN-OS images ahead of time. You will need both the base image and the latest maintenance release. The base image should be installed but not rebooted. In this case, we need to download the following versions:

– 7.1.22 (it is recommended to bring your current Feature release to the latest recommended maintenance release before preceding)
– 8.0.0 for firewalls, 8.0.2 for Panorama (base image) NOTE: the 8.0 base image and maintenance versions will not become visible in the download section until you have a version of 7.1 installed.
– 8.0.16
– 8.1.0 (base image)
– 8.1.6


Following the PAN-OS upgrade, you may need to upgrade associated software (such as GlobalProtect agent or User-ID agent).

– See the Associated Software Versions chart in the release notes.

(Optional but recommended) Arrange for Out-of-Band access (Console access) to the firewall if possible. This is to help recover from any unexpected situations where we lose connectivity to the firewall after upgrade.



3) Panorama Upgrade Procedure

Make sure no policy or configuration changes are being made by acquiring a config lock.
View of PAN-OS GUI PadLock

Click the padlock on the upper-righthand corner of GUI

If there are any locks, remove the locks and talk with the admin who placed the lock there and remove or commit.

Clear or complete any pending commit job making a commit to panorama before starting the upgrade.

(Optional but recommended) Post-upgrade failover testing:

Suspend Secondary Panorama to fail connection back to Primary Panorama to make sure failover still works after upgrade.

– CLI: > request high-availability state suspend
– GUI: Device > High Availability > Operations > click Suspend Local Device
– Verify suspend status on Secondary Panorama
– Verify all firewalls are connected to Primary Panorama
– Re-enable Secondary Panorama and High Availability function
– – CLI: > request high-availability state functional
– – GUI: Device > High Availability > Operations > click Make Local Device Functional

Primary Panorama Upgrade procedure

– Disable Pre-emption if enabled.
(Disable preemption on High Availability settings to avoid unexpected fail overs. Disabling preempt configuration change must be committed on both peers. Once completed, re-enabling must be committed on both peers)

– Go to Device > High Availability > Election Settings and uncheck Preemptive. Then, commit the change.


View of PAN-OS Device High Availability screen in Election Settings

– Suspend Primary Panorama to make Secondary Panorama as active
– From Primary Panorama, suspend High Availability function:
– – CLI: > request high-availability state suspend
– – GUI: Device > High Availability > Operations > click Suspend Local Device

View of PAN-OS Device High Availability screen in Optional Commands

– Verify suspend status on Primary/passive Panorama.
– Verify all firewalls are connected to Secondary/active Panorama.
– On the Primary Panorama, download, install and reboot 7.1.22.
– Save/export tech support and device state and save named device config snapshots (this is in case downgrade is needed).
– Download 8.0.2 (base version) (Recommended) and install the 8.0 base image, and reboot before you install the target maintenance release.
– Download and install 8.0.16, and reboot to complete the upgrade.
– Save/export tech support and Device state and save named device config snapshots (this is in case downgrade is needed).
– Download 8.1.0 (base version) (Recommended) install the 8.1 base image and reboot before you install the target maintenance release.
– Download and install 8.1.6, and reboot to complete the upgrade.
– re-enable pre-empt if desired
– This concludes upgrade on Primary Panorama.

Secondary Panorama Upgrade procedure

– Download, install, and reboot 7.1.22.
– Download 8.0.2 (base version) (Recommended) install the 8.0 base image and reboot before you install the target maintenance release.
– Download and install 8.01.6 and reboot to complete the upgrade.
– Save/export tech support and device state and save named device config snapshots (this is in case downgrade is needed).
– Download 8.1.0 (base version) (Recommended) install the 8.1 base image and reboot before you install the target maintenance release.
– Download and install 8.1.6, and reboot to complete the upgrade.
– This concludes upgrade on Secondary Panorama.

Backup config and device state files just in case: (BACK UP CONFIGURATION AND DEVICE STATE FROM THE CLI)

(Optional but recommended) Post-upgrade verification:

– Verify connectivity between Panorama and Firewalls. If something is not working, skip to troubleshooting section
(For example, check if Panorama is receiving logs with correct time stamp from firewalls after upgrade is completed)

– Test commit-all operations to managed devices, and verify new changes are applied as expected locally on the devices.
 


4) Firewall Upgrade Procedure (HA)


It is recommended to upgrade the Primary firewall first and then upgrade the Secondary firewall. This is done for two reasons:

1) Ensure that HA failover is functioning properly
2) Ensure that the passive firewall is functioning properly and is able to pass traffic without issues

Disable Preemption if enabled.

Disable Preemption on High Availability settings to avoid unexpected fail overs.

Disabling preempt configuration change must be committed on both peers. Once completed, re-enabling must be committed on both peers.

To Disable, got to: Device > High Availability > General > Election Settings > Edit > Uncheck Preemptive.

Then, perform a commit.

View of PAN-OS Device High Availability screen in Election Settings

NOTE: This procedure relies on the administrator having foreseen access to their devices at all times, either by being local or having OOB connectivity to the management network, which is a good practice when upgrading the firewall. In the case where you do not have the option of achieving either, it is a good idea to change the procedure slightly to ensure you don't lose connectivity at the cost of having a less rigid upgrade path.

Having the preempt enabled will require you to keep this config change in mind during the whole process as it could unexpectedly switch over the active membership while upgrading.

Primary firewall upgrade Procedure

– On Primary firewall, Suspend Primary firewall to make Secondary firewall active
– – CLI: > request high-availability state suspend
– – GUI:  Device > High Availability > Operations > click Suspend Local Device

NOTE: This will cause an HA failover. We recommend doing this first to verify the HA functionality is working before initiating the upgrade. Production traffic is now going through the Secondary firewall which is now active.

– Ask your business owners to verify all applications are working on the network. If there is a problem, skip to troubleshooting section. If there is any problem, fix it before proceeding with upgrade.
– Upgrade Primary firewall. You can do this by either directly downloading and installing software onto the firewall itself or via Panorama Device Deployment > Software option.
– Download, install and reboot 7.1.22.
– Download 8.0.0 (base version)(Recommended) Install the 8.0 base image and reboot before you install the target maintenance release.
– Download and install 8.0.16, and reboot to complete the upgrade.
– Save/export tech support and Device state and save named device config snapshots (this is in case downgrade is needed).
– Download 8.1.0 (base version)(Recommended) Install the 8.1 base image and reboot before you install the target maintenance release.
– Download and install 8.1.6, and reboot to complete the upgrade.
​​​​– On the Primary firewall, verify auto commit completes successfully (FIN OK) by running the command before proceeding to the next step: > show jobs all
– Run the following command to make Primary firewall functional again: > request high-availability state functional
– This concludes upgrade on the Primary firewall.

Secondary firewall upgrade procedure
– Suspend Secondary firewall to make Primary firewall active.
– From Secondary firewall, suspend High Availability function
– – CLI: > request high-availability state suspend
– – GUI: Device > High Availability > Operations > click Suspend Local Device

NOTE: This will cause an HA failover. Production traffic is now going through Primary firewall with new software installed.

– Ask your business owners to verify all applications are working on the network. If there is a problem, skip to troubleshooting section. If there is any problem, fix it before proceeding with upgrade.
– Upgrade Secondary firewall. You can do this by either directly downloading and installing software onto the firewall itself or via Panorama Device Deployment > Software option.
– Download, install and reboot 7.1.22.
– Download 8.0.0 (base version)(Recommended) install the 8.0 base image and reboot before you install the target maintenance release.
– Download and install 8.0.16. reboot to complete the install.
– Save/export tech support and Device state and save named device config snapshots (this is in case downgrade is needed).
– Download 8.1.0 (base version)(Recommended) install the 8.0 base image and reboot before you install the target maintenance release.
– Download and install 8.0.6. reboot to complete the install.
– Verify auto commit completes successfully (FIN OK) by running the command before proceeding to the next step: > show jobs all
– Run the following command to make Secondary firewall functional again: > request high-availability state functional
– This concludes upgrade on the Secondary firewall

(Optional but Recommended) Arrange for Out-of-Band access (Console access) to the firewall if possible. This is to help you recover from any unexpected situations where we are unable to login to the firewall.

– Backup config and device state files just in case.
– Make sure no policy or configuration changes are being made by acquiring a config lock.
– Click on the padlock in the upper-righthand corner of the GUI
– Make sure no pending commit jobs on firewall.

(Optional but recommended) Post-upgrade verification:

– Now that both Primary and Secondary firewalls are both running the new software, it is a good idea to test failover functionality again.
– Run the following comma to suspend Primary firewall to fail traffic to the Secondary firewall
     > request high-availability state suspend

– Ask your business owners to verify all applications are working on the network through the Secondary firewall. If there is a problem, skip to troubleshooting section.
– Run the following CLI command to make Primary firewall functional again:
     > request high-availability state functional

– Repeat the process to verify traffic works fine through Primary firewall (suspend the Secondary firewall, test functionality on Primary firewall, then re-enable Secondary firewall).
– This concludes failover test.

(Optional but recommended) Enable preemption if it was disabled due to upgrade.

– Re-enabling preempt configuration change must be committed on both Likewise, once completed, re-enabling must be committed on both peers.

– Go to Device > High Availability > Election Settings and check Preemptive. Then, perform a commit.

– This completes upgrade on the HA pair.

View of PAN-OS Device High Availability screen in Election Settings for Preemptive



5) Post-Upgrade Checklist

The following Post-Implementation Activities should be performed prior to the change window end time. Performing these Post-Implementation Activities prior to the change window end time allows time to complete any potential corrective action that might be required after performing these activities.

Compare Post-Implementation results with Pre-Implementation results.

– Reapply the non-persistence settings from the pre-checklist.
– Check system logs for any unexpected errors.
– Check traffic logs for any traffic that is unexpectedly allowed or denied.
– If applicable, verify connectivity and network functionality between firewall and panorama, specifically make sure log forwarding from firewall to Panorama is working.
– Also, validate changes will commit between Panorama and the managed devices.
– Check system reports and incidents for any relevant outages.

– – Monitor help desk tickets post upgrade, this may take 24 to 48 hours.
– – Let administrators know about the completion of the change.

– Contact all stake holders to communicate any change IDs, describe change activity results, and verify that there are no relevant network alarms, incidents, or outages.
– Monitor the appliance for any anomalies.



6) Troubleshooting Resources

If business applications no longer works after upgrade, reference the links below:
– PAN-OS Upgrade or Content Update Failure
– CONTENT VERSION ERROR UPGRADING MAJOR PLATFORM OS WITH AN OLDER CONTENT DATABASE 

If the device fails to complete auto-commit, reference the links below:
– HOW TO DETERMINE WHEN AUTO-COMMIT IS COMPLETE 

If software fails to install, reference the links below:
– SOFTWARE INSTALL OR DOWNLOAD PUSH FROM PANORAMA TO DEVICE WILL NOT COMPLETE
– COMMIT FINISHES WITH AN ERROR RESPONSE: CFGPUSH.S1.DP1.COMM.CFG-DP: ERROR PRE-INSTALLING CONFIG 
– LICENSE ERROR: FAILED TO INSTALL LICENSES. UNEXPECTED ERROR OCCURRED

If software fails to download, reference the links below:
– Error downloading 7.0.4, with 7.0.0 previously downloaded (General User Discussion) 
– SOFTWARE DOWNLOAD ERROR: 'FAILED TO DOWNLOAD DUE TO SERVER ERROR. FAILED TO DOWNLOAD FILE'

Panorama checklist reference links below:
– QUICK REFERENCE GUIDE: HELPFUL COMMANDS
– TROUBLESHOOTING PANORAMA CONNECTIVITY

If issues cannot be resolved:
– Contact Palo Alto Networks TAC using proactive case number.
– Save configurations of affected network devices.
– Save configurations of the Palo Alto Network devices.
– Add to pcaps, configurations, tech support files, and logs from near by networking devices for post mortem and troubleshooting by support teams.
– Go to the downgrade/back out procedure section.
 


7) Downgrade Procedure

If the issue cannot be resolved within the allotted change window, you should revert all changes:
– Verify 7.1.0 (base image version) is still present on the system.
– Verify and install 7.1.14. reboot to complete the install. When prompted, use 7.1.14 snapshot file saved during the upgrade.
– Verify 7.0.1 (base version) is still on the system.
– Download and install 7.0.18, and reboot to complete the downgrade. When asked, use 7.0.18 snapshot file saved before the upgrade.

NOTE: After the Secondary firewall is rebooted, the CLI prompt should show non-functional.
– On the primary firewall, verify auto commit completes successfully (FIN OK) by running the command before proceeding to the next step:
     > show jobs all
– Run the following command to make it functional again:
     > request high-availability state functional
– This concludes downgrade on the Primary firewall.

Downgrade the Primary firewall
– Suspend the primary firewall to fail traffic to Secondary.
– On the Primary device, suspend the unit from the CLI. Run the command:
     > request high-availability state suspend

Save the following files for analysis:
– Save and export Tech support and device state from both active and passive firewalls.
– All core dump files if there is any.
– Packet capture of problem traffic, if observed.
– Assuming the firewall is currently running 8.0.6-h3 already and traffic is currently going through the Primary firewall, follow the same upgrade process but in reverse order.

Downgrade the Secondary firewall:
– Verify 7.1.0 (base image version) is still present on the system.
– Verify 7.1.14 is still present on the system and install, reboot to complete the install. When prompted, use 7.1.14 snapshot file saved during the upgrade.
– Verify 7.0.1 (base version) is still on the system.
– Download and install 7.0.18. reboot to complete the install. When prompted, use 7.0.18 snapshot file saved before the upgrade.

NOTE: After the Secondary firewall is rebooted, the CLI prompt should show non-functional.
– On the Secondary firewall, verify auto commit completes successfully (FIN OK) by running the CLI command before proceeding to the next step:
     > show jobs all
– Run the following command to make it functional again:
     > request high-availability state functional
– This concludes downgrade on the Secondary firewall.

(Optional but recommended) Enable preemption if it was disabled due to upgrade.
– Re-enabling preempt configuration change must be committed on both. Once completed, re-enabling must be committed on both peers.
– Go to Device > High Availability > Election Settings& and check Preemptive. Then, perform a commit.

View of PAN-OS Device High Availability screen in Election Settings for Preemptive Setting

(Optional but recommended) Ask your business owners to verify all applications are working on the network. If there is a problem, skip to troubleshooting section.
– Upload all files to the Palo Alto Networks proactive support case for troubleshooting later.
– This concludes the downgrade process.


Attachments
Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClRrCAK&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Attachments
Choose Language