“certificate has expired” log entry in a PA-VM (2024 Certificate Expiration).

There are multiple ways this issue manifests itself:

• A newly-bootstrapped PA-VM doesn't register itself to your Panorama, despite receiving licenses and despite evidence of said PA-VM attempting to connect to your Panorama.
• PA-VMs show up as being disconnected from your Panorama but connect once again if, for instance, useridd is restarted on your Firewall.
• “device-certificate-status: None” in the output of “show system info” when run on your PA-VM.

“Error:  pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:1614): cms sent untrusted cert!!”

“Error:  valid_cert(cs_client.c:17): commssl: Cert verify failed: error: 10 (certificate has expired)”

“Error:  pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:1344): cms agent: cs_load_certs_ex failed”

“Error:  cs_load_certs_ex(cs_common.c:544): keyfile not exists”

PA-VM:

• Runs a PAN-OS version not listed in the PAN-OS Certificate Expirations LIVEcommunity article (hereafter referred to as LIVEcommunity article)
• Has a dynamic content version that is not 8795-8489 (per the LIVEcommunity article) or higher (this can be found in the “app-version” field when running “show system info” on your PA-VM).
• Runs both a PAN-OS version listed in the LIVEcommunity article and also has a dynamic content version >= 8795-8489, but hasn’t been rebooted.
• Shows up as being disconnected on your Panorama.

• Similar to PA-VM.
• Refer to the LIVEcommunity article for more up-to-date information.

Since the PA-VM might not have the requisite version updates (as noted in the LIVEcommunity article), it may not have a valid device certificate. As a result, while it may still be able to connect with the Panorama (and even obtain licenses from it), it will show up as being “disconnected” on the Panorama.

 This article is part of our comprehensive certificate management plan to mitigate the November / December 2023 PAN-OS Root and Default Certificate Expiration (Khans, 2024) (Khans, 2024).

1. Use PAN-OS images that are available on your Cloud Service Provider’s Marketplace:

• The PAN-OS versions contained in these images don’t come across the PAN-OS Root and Default Certificate Expiration issue.
• Older images might come across this issue.
• For example, on Azure, the following images are available on the Marketplace in the East US region (10.2.901 is actually 10.2.9-h1; the “0” can be considered as “-h”):

2. If this issue occurs on an existing PA-VM:

• Update your PA-VM to one of the PAN-OS versions listed in the LIVEcommunity article.
• Update your dynamic content to a version that is >= 8795-8489 (again, in line with the LIVEcommunity article).
• Reboot the Firewall in order to allow the new dynamic content to take effect; an mgmtsrvr restart will not suffice.

3. If you’re using a custom image to bootstrap your PA-VM, you may have to create a new image:

• The idea is to create a new image by following the respective documentation for creating a custom image on your Cloud Service Provider but with a few extra steps.
• Said extra steps are just the steps listed in point 2 (“If this issue occurs on an existing PA-VM”; just above this point; hereafter referred to as “extra steps”) on the PA-VM that you will use to create your image.
• These extra steps should be performed before delicensing the PA-VM. 
• e.g. creating a custom image (AMI) on AWS (Palo Alto Networks, Inc., 2024): 
  • The documentation describing the steps is: https://docs.paloaltonetworks.com/vm-series/11-1/vm-series-deployment/set-up-the-vm-series-firewall-on-aws/deploy-the-vm-series-firewall-on-aws/create-custom-ami
  • In this case, you will perform the extra steps during step 3 of the document (i.e. prior to step 4, which involves delicensing the PA-VM).

References

Khans. (2024, 01 26). Emergency Update Required - PAN-OS Root and Default Certificate Expiration. Palo Alto Networks LIVEcommunity. https://live.paloaltonetworks.com/t5/customer-advisories/emergency-update-required-pan-os-root-and-default-certificate/ta-p/564672

Khans. (2024, 04 11). Additional PAN-OS Certificate Expirations and New, Comprehensive Certificate Management Process. Palo Alto Networks LIVEcommunity. https://live.paloaltonetworks.com/t5/customer-advisories/additional-pan-os-certificate-expirations-and-new-comprehensive/ta-p/572158

Palo Alto Networks, Inc. (2024, 05 31). Create a Custom Amazon Machine Image (AMI). Palo Alto Networks TechDocs. https://docs.paloaltonetworks.com/vm-series/11-1/vm-series-deployment/set-up-the-vm-series-firewall-on-aws/deploy-the-vm-series-firewall-on-aws/create-custom-ami