Critical Issues Addressed in PAN-OS Releases

Critical Issues Addressed in PAN-OS Releases

373775
Created On 09/26/18 21:07 PM - Last Modified 09/30/22 19:05 PM


Symptom
Historical Critical Issue List Addressed in PAN-OS Releases

Environment
All current PAN-OS

Resolution

Last Updated On : Sep  30th , 2022


This list is limited to critical severity issues as determined by Palo Alto Networks and is provided for informational purposes only.

  • Please doublecheck the information in release notes to see the latest info about fixed versions.
  • Please create a case with your support provider for a detailed investigation if you feel you have encountered one of these issues.
  • Maintenance releases are the primary mechanism to fix issues.
  • A maintenance release is signified by the third digit in the release version number (for example the .2 in PAN-OS 10.1.2 ).
  • asterisk(*) in Fixed release is used for internal check. please ignore it.

 

 
Bugs

Affected Platform(if any)

/Affected Version

Description (release note)Impact

Root cause

Workaround

Fixed release

PAN-198266PA-400, PA-3400, PA-5400
10.2.2
Fixed an issue where, when predicts for UDP packets were created, a configuration change occurred that triggered a new policy lookup, which caused the dataplane stopped responding when converting the predict. This resulted in the policy lookup returning a policy denial.DP crashThe logging code access a non-existent field when generating a deny log for a predict. This happens when an allow policy is removed or changed to deny and pre-exiting predicts created by ALG are no longer valid.clear all predicts before a config commit.
"clear session all filter type predict"
10.1.8,10.2.3
PAN-19121610.2.0-10.2.2Fixed an issue where, on Apple iOS devices, SAML authentication did not connect to the GlobalProtect portal.GP on iOS with SAML does not workSince 10.2.0, GP server is missing to SAML related result in HTTP headerN/A10.2.3
PAN-196005PA-3200 Series, PA-5200 Series, and PA-5400 Series firewalls only

10.1.0-10.1.6
10.2.0-10.2.2
(only 10.1.6 is reported)
 
Fixed an issue where GlobalProtect IPSec tunnels disconnected at half the inactivity logout timer value.GP tunnel goes down every 30minutesBecause of local time handling difference in MP and DP for a GP tunnel timeout feature, NGFW mistakenly disconnects GP tunnel.To sync time for this, power off the fw then power up. NOT reboot.10.1.7,10.2.3
PAN-19155810.0.10, 10.1.5-10.1.6, 10.2.1-10.2.2Fixed an issue where, after an upgrade to PAN-OS 10.1.5, Global Find did not display all results related to a searched item.Global does notA searchAttribute instance throwing a null pointer error on searching causes endless loadingN/A10.0.11, 10.2.3, 10.1.7, 10.1.6-h3
PAN-189395PA-400
10.2.0-10.2.1
PA-400 Series firewalls only: Fixed an issue where running a PAN-OS 10.2 release caused dataplane processes to restart unexpectedly.dataplane process restartmemory leak in memory bufferNo workaround10.2.2
PAN-1894689.1.13
10.0.10
10.2.0
Fixed an issue where sessions were dropped with the message `resource-unavailable` due to the content inspection queue filling up.session drops due to 'resource-unavailable'ctd memory space is held due to wrong memory freeingset system setting ctd nonblocking-pattern-match disable
(This will cause higher packet buffer CPU usage.)
9.1.14,10.0.10-h1,10.0.11,10.1.5,10.2.1
PAN-1838269.1.12-9.1.13
10.0.8
10.1.0-10.1.6
Fixed an issue where, after clicking **WildFire Analysis Report**, the web interface failed to display the report with the following error message: `refused to connect`.WildFire Analysis Report can't be seen in WebUI
The issue is because the x-frame-options is set to deny so the WF report is unable to display within the iframe
"View frame source" on right click menu on failed analysis report.
remove "viewsource" from the opened link. the link starts with "viewsourcehttps://x.x.x.x/wf_report/".
then open the page.
9.1.14,10.0.9,10.1.7,10.2.1
PAN-1752119.0.0-9.0.15
,9.1.0-9.1.12
,10.0.0-10.0.8
,10.1.0-10.1.3
Fixed a memory leak issue in the mgmtsrvr process.mgmtsvr process memory leakWhen there is constant reconnect from FW to Panorama, old SSL structure is not freed and newly allocated SSL structure overwrites a memory space leaks.No workaround9.0.16, 9.1.13, 10.0.9, 10.1.4
PAN-187183(PLUG-10024)All PA-VM in 10.1.4
VM Plugin 2.1.4
Fixed an issue with `vm_license_response.log` that consumed a large portion of the root partition.root partition full
License fetch log is consuming root space
From admin CLI,
admin@PA-VM> delete debug-log mp-log file vm_license_response.log_backup.gz
successfully removed vm_license_response.log_backup.gz
VM Plugin: 2.1.5, 3.0.0
PAN-18111610.1.0-10.1.4Fixed memory corruption issues in PAN-OS 10.1.3 and 10.1.4 that caused the *pan_comm* process to stop responding and the dataplane to restart. These issues also caused GlobalProtect tunnels to fall back to SSL instead of IPSec due to the inadvertent encapsulation of the ICMP keepalive response from the firewall.GP does not connect with IPSEC ESP and instead switches to SSLIn original design, mix mode was not supported.
If ssl tunnel and ipsec tunnel established together, their config are messed up.
It caused tunnel failed.
N/A10.1.5
PAN-18575010.1.4
Updated an issue to eliminate failed `pan_comm` software issues that caused the dataplane to restart unexpectedly
pan_comm process crashtimestamp variable was not cleared properly and it'No workaround10.1.5, 10.1.4-h4
PAN-1869379.1.0-9.1.11Fixed an issue where the firewall dropped packets decrypted using the SSL Decryption feature and Encapsulating Security Payload (ESP) IPSec packets that originated from the same firewall. This occurred when **Strict IP Address Check** was enabled in the zone protection profile (**Packet Based Attack > IP Drop**) and the packet's source IP address was the same as the egress interface address.packet drop on SSL decryption and ESP IPsec on the same FW

The bug was caused when strict IP was on and packet source IP == egress IP. This caused packets, like ESP and SSL decrypt for example, to be erroneously dropped"

Disable the Strict IP Address Check option in the Zone Protection profile. Alternatively, downgrade to 9.1.11 or earlier or upgrade to 10.0.0 or later if you want to enable the Strict IP Address Check.9.1.14
PAN-1792749.1.0-9.1.12,10.0.0-10.0.9,
10.1.0-10.1.4
Fixed an issue on high availability configurations where, after upgrading to PAN-OS 9.1.10, PAN-OS 10.0.6, or PAN-OS 10.1.0, the HA1 and HA1-Backup link stayed down. This issue occurred when the peer firewall IP address was in a different subnet.HA1/HA1 backup link not coming upInternal routing lookup mechanism didn't work as expectedNo workaround9.1.13,10.0.10,10.1.5,10.2.0
PAN-17776210.0.0-10.0.8,10.1.0-10.1.3Fixed an issue where `wificlient` in PAN-OS 10.0 and later releases caused processing delays, on-chip descriptor spikes, and buffer usage.Traffic is intermittently droppedfrom 10.0, new feature tends to hold cores. It can cause high packet descriptor on-chip or buffer usage.Disable EAL10.0.9,10.1.4
PAN-1722438.1.0-8.1.21,9.0.0-9.0.14,
9.1.0-9.1.12,10.0.0-10.0.8,
10.1.4-10.1.4
Fixed an issue where NetFlow traffic triggered a packet buffer leak.packet buffer full should cause general traffic processing in DPNetflow saved packet leaked on commit as netflow profile changes memory spaceDisable Netflow8.1.22,9.0.15,9.1.13,
,10.0.9,10.1.5*
PAN-1837678.1.21,9.1.12,10.0.8, 10.1.3Fixed an issue where downloading Dynamic Updates files failed when connected to the static update server at `us-static.updates.paloaltonetworks.com`.PAN-OS is not abl e to download software image from update serverA code change in affected version provided wrong option for a download command.use "updates.paloaltonetworks.com" instead.8.1.21-h1, 9.0.15,9.1.12-h3, 10.0.8-h8,10.0.9,10.1.4
PAN-177941PA-70x0 (100G-NPC)/
10.0.0-10.0.7
10.1.0-10.1.2
Fixed an issue where the `bcm.log` and `brdagent_stdout.log-<datestamp>` files filled up the root disk spaceRoot partition fullUnnecessary logs are generated on file systemUse ports 1-8 on LFC for log forwarding.10.0.8, 10.1.3
PAN-17258010.0.0-10.0.7
10.1.0-10.1.2
Fixed an intermittent issue where commits failed after a commit validation and were modified for custom URL category objects.Intermittent commit failuresCandidate internal ids are not cleaned up for validate job during phase1 abort. It affects the subsequent commit for such.Restore the url pattern changes made after the validate job and commit.
OR
Skip Validate and enforce commit
10.0.8, 10.1.3
PAN-1690649.1.0-9.1.10
10.0.0-10.0.6
10.1.0
Fixed an issue where the management CPU remained at 100% due to a large number of configured User-ID agents.memory leak on useridd1) hip report xml buffer was not released after message was sent out which caused memory leak
2) High CPU issue is caused by a busy loop ,because a big number of jobs are scheduled and FD is alway readable during the job waiting period.
 
Reducing the number of configured userid agents/clients can alleviate the issue.10.1.1, 10.0.7 and 9.1.11
PAN-1695519.1.8-9.1.9Fixed an issue where custom URL categories hit incorrect URL categories, which caused the firewall to miss or deny the security policies for the configured custom URLURL category lookup failsId-manager mis-manage the table on commit , caused URL pattern lost on DP

For customers using custom URL categories only (NO EDL-URL), before committing any URL pattern changes,

  1. perform a "Commit force" job first. Then
  2. commit the URL pattern changes (or push changes from panorama).

For customers using EDL-URL,

  • if customer host the EDL-URL themselves, before they update the URL patterns, perform a "Commit force" job first.
  • if customer uses 3rd party provided EDL, there's no good workaround.
    • They can lower the EDL refresh frequency to reduce the chance of hitting this issue, and
    • if customer has hit this issue, do commit-force twice to temporarily resolve this issue.
9.1.10
PAN-1638009.1.0-9.1.10,
10.0.0-10.0.6,
10.1.0
Fixed an intermittent issue where the presence of an Anti-Spyware profile in a Security policy rule that matched DNS traffic caused DNS responses to be malformed in transit.dns response is corruptedcode of license check and TTL modification had a bug to handle DNS responseRemove anti-spyware that contains dns security profile9.1.11,10.0.7,10.1.1
PAN-1462508.1.0-8.1.19,
9.0.0-9.0.13,
9.1.0-9.1.9,
10.0.0-10.0.6
Fixed an issue where, in two separate but simultaneous sessions, the same software packet buffer was owned and processed.DP crashFor inter-vsys scenario,  the same sw packet buffer could be processed in two different sessions at the same time, which in turn cause the issue.Use IPsec VPN instead of using SSL8.1.20,9.0.14,9.1.10,10.0.7
PAN-156017 9.1.0-9.1.6, 10.0.0-10.0.2Fixed an issue where a host information profile (HIP) report XML buffer caused a memory leakOut of Memory in MPHIP report buffer was not released after message was sent out which caused memory leakDisable hip redistribution9.1.7,10.0.3
PAN-156225

PA-3200series
/
8.1.0-8.1.19, 9.0.0-9.0.13, 9.1.0-9.1.8 ,10.0.0-10.0.4

Fixed an issue where HA1-B port on PA-3200 series remain down after upgrade from 9.1.4 to 9.1.5HA1-B link downfailed to fetch a related sysd nodeNone8.1.20,9.0.14, 9.1.9,10.0.5
PAN-1363478.1.0-8.1.18, 9.0.0-9.0.13, 9.1.0-9.1.8 , 10.0.0-10.0.4Fixed an issue wherer DNS proxy TCP connections were processed incorrectly, which caused a process (`dnsproxy`) to stop responding.dnspropyd crash / high CPUtcp_wait_timer on the daemon didn't cleared correctlyWorkaround is to disable TCP connection through DNSproxy daemon, to safely avoid any ability issues with proxied TCP requests.8.1.19, 9.0.14, 9.1.9,10.0.5
PAN-1508528.1.0-8.1.18 ,9.0.0-9.0.12 ,9.1.0-9.1.6 ,10.0.0-10.0.4Fixed an issue with SMTP that occurred when attachment file names were longer than the allocated buffer. If the file name was longer than the buffer and Layer 7 inspection was enabled, the file was dropped, which caused session errors and an email to not be sent.DP crash /SMTP packet dropbuffer handling issue when processing SMTP mult-part filenameNone8.1.19
9.0.13
9.1.7
10.0.5
PAN-1434858.1.0-8.1.18, 9.0.0-9.0.12 , 9.1.0-9.1.6, 10.0.0-10.0.4Fixed a memory leak issue related to a process (*devsrvr*).device server memory leakmultiple leaks (URL,confg,etc) are fixedRestarting devsrvr before device  memory gets depleted9.0.13,9.1.8,10.0.0
PAN-1568919.1.0-9.1.7
10.0.0-10.0.4
Fixed an issue where some zip files did not download and the following error message displayed: `resources-unavailable`.L7 feature does not work when hitting 'resource-unavailable' errorThe decoder buffer would go through a high number of loop in L7 processing. It hits the max limit.

"set deviceconfig setting session resource-limit-behavior bypass" helps to bypass sessions hitting the error.

Technically it generally happen. but "strip ALPN" in decryption profile may resolve the issue if it's caused by decoding http2.

"disabling hardware dfa" can be a workaround since it helps to reduce the number of loop.That can be done by the following commands

debug dataplane fpga set sw_aho yes
debug dataplane fpga set sw_dfa yes
9.1.8
10.0.5
PAN-145417

9.0.0-9.0.12
9.1.0-9.1.7
10.0.0-10.0.3

Debug commands were added to address an issue where the firewall connect to Cortex Data Lake due to the Online Certificate Status Protocol (OCSP) message missing the `nextUpdate` value in the OCSP response.sslmgr memory leak caused an issue on OCSPFailed OCSP queries are cached for long time. It affects normal behaviour of sslmgr and its memory usage goes upRestart sslmgr process9.0.13,9.1.8,10.0.4
PAN-1492979.1.0-9.1.6
10.0.0-10.0.1
Fixed a buffer overflow issue on the management server, which forced the administrator to log out on the web interface.management server crashMissing close calls for an internal dbsAvoid doing multiple validate commits, commitAlls9.1.7,10.0.2
PAN-153673

Technically all FW platform can be affected. but we only get reports from PA5200,PA7000series
8.1.15-8.1.17
9.0.9-9.0.10
9.1.1-9.1.5
10.0.0-10.0.1

Fixed an issue where traffic logs were not shown due to a thread timeout that was causing the reading of the logs from the dataplane to slow.Logging intermittently stopsthe main thread was busy doing cache age out, cause the reading of the logs from the link from the DP slows down greatly.None8.1.18, 9.0.11, 9.1.6, 10.0.2
PAN-1521068.1.14-8.1.16
9.0.8-9.0.10
9.1.0-9.1.5
10.0.0-10.0.1
Fixed an issue where a process (*genindex.sh*) caused the management plane CPU usage to remain high for a longer period of time than expected.High MP CPUThe script searches log directories intensivelyConfigure Max Days for the Log Types to reduce retention days to reduce amount of logs to index.8.1.17, 9.0.11, 9.1.6,10.0.2
PAN-154181Panorama
8.1.16
 
Fixed an issue where, on Panorama, context switching to the web interface of a managed firewall running PAN-OS 8.1.16 did not work.Context switch is unableA bug fix prevented context switch from workingNone8.1.17
PAN-1511979.1.3
10.0.0
Fixed an issue where a process (*authd*) restarted when an administrator authenticated to the firewall with an Active Directory (AD) account. This issue occurred when LDAP was configured with FQDN, used DHCP instead of a static management IP address, and used the management interface to connect to the LDAP server.Authd crashThe boundary case that DHCP assigned mgmt IPUse service route for LDAP9.0.10, 9.1.4, 10.0.1
PAN-1412219.0.0-9.0.9
9.1.0-9.1.2
Fixed an issue where a commit or content update operation with an error was not prevented from executing in the dataplane, which caused corruption in the dataplane policy cache.DP crash- When DP phase1 parse error happens on config commit, the abort signal didn't cleanup properly,thus policy cache is corrupted
Make sure the config does not error out in DP
9.0.10, 9.1.3
PAN-1445988.1.0-8.1.15
9.0.0-9.0.9
9.1.0-9.1.2
Fixed an issue where dataplane free memory was depleted, which affected new GlobalProtect connections to the firewallGP connection failureThe URL data structure is not being freed during the clientless VPN app access.No8.1.16, 9.0.10, 9.1.3
PAN-1501728.1.15,9.0.9,9.1.3Fixed an issue where dataplane processes restarted when attempting to access websites that had the `NotBefore` attribute less than or equal to Unix Epoch Time in the server certificate with forward proxy enabled.DP restart when parsing certificateThe 'NotBefore' value was not initialized properly1) Import the server's issuer CA to the firewall and mark it trusted, OR
2) Disable decryption to those servers with NotBefore <= 1970/1/1 00:00:00 UTC
This is not practical solution
8.1.15-h3, 8.1.16, 9.0.9-h1, 9.0.10, 9.1.3-h1, 9.1.4, 
PAN-1373878.1.0-8.1.14
9.0.0-9.0.8
9.1.0-9.1.2
Fixed an issue where URL filtering used the IP address instead of the hostname, which led to incorrect URL categorization.Issue on Host header handling causes URL filter functionmiss handling when Host header does not come in 1st packetEnable jumbo frame, or use custom-url-category or custom-appid to detect string "/webapp/wcs/stores/".8.1.15, 9.0.9, 9.1.3
PAN-1480688.1.0-8.1.14
9.0.0-9.0.8
9.1.0-9.1.2
Fixed an issue where SSL connections were blocked if you enabled decryption with the option to block sessions that have expired certificates. This issue included servers that sent an expired AddTrust certificate authority (CA) in the certificate chain.SSL decryption fails to some sitefixed SSL cert verification processDisable certificate expiration check.
(if no expiration check is acceptable)
8.1.15, 9.0.9, 9.1.3
PAN-103290PA3200series
8.1.14 only
Fixed an issue where the firewall stopped recording dataplane diagnostic data in dp-monitor.log after a few hours of uptime.DP crashday-one issue crash when handingNo workaround8.1.15
PAN-139587PA5200,PA7000series
8.1.0-8.1.14
9.0.0-9.0.8
9.1.0-9.1.4
Fixed an issue where high and continuous CPU utilization was seen on dataplanes after IPSec Encapsulating Security Payload (ESP) rekeying occurred for multiple tunnels.High CPU/ High packet descriptorESP rekey issueAfter failover, reboot the failing FW8.1.15, 9.0.9 , 9.1.4
PAN-1444798.1.14 onlyFixed an issue where SNMP objects from the HOST-RESOURCES-MIB returned incorrect values when queried.snmp for the specific MIB does not workregression of a snmp fixNo workaround8.1.15
PAN-136701PA7000series
9.0.0-9.0.7
9.1.0-9.1.1
Added the following CLI commands to address an issue where packets for new sessions dropped when handling predict sessions:
- `set session hwpredict disable yes`
- `show session hwpredict status`
packet drop on predict session matchingadded workaround commandto disable predict lookup in FPP-HW and use FPP-SW. This is controlled using a operational command.9.0.8, 9.1.2
PAN-121626

PA3200series
8.1.0-8.1.13
9.0.0-9.0.6
9.1.0-9.1.1

Fixed an intermittent issue where firewalls dropped packets, which caused issues such as traffic latency, slow file transfers, reduced throughput, internal path monitoring failures, and application failures.Traffic issue Issue on  memory timingNo workaround8.1.14,9.0.7,9.1.2
PAN-125534PA5200,PA7000series
8.1.0-8.1.13
9.0.0-9.0.7

 
Fixed an issue where firewalls experienced high packet descriptor (on-chip) usage during uploads to the WildFire Cloud or WF-500 appliance.Excessive WF uploads caused high packet descriptorExcessive WF uploads surpress platform resources.

Limit maximum number of outstanding WF uploads

Configure Device > Setup > WildFire > General Settings > File Size Limits
to specify the following recommended values for WildFire file size limits:

- pe 8 MB
- apk 10 MB
- pdf 500 KB
- ms-office 500 KB
- jar 5 MB
- flash 5 MB
- MacOSX 1 MB
- archive 10 MB
- linux 10 MB
- script 20 KB

8.1.14,9.0.8,9.1.2
PAN-135260PA7000series only
8.1.12 only
Fixed an intermittent issue where the dataplane process (*all_pktproc_X*) on a Network Processing Card (NPC) restarted when processing IPSec tunnel traffic.DP crashCrash during flow lookup
Added a validation code 
No workaround8.1.13,9.0.7,9.1.2
PAN-1368208.1.0-8.1.13Fixed an issue where a high availability (HA) failover occurred after the firewall reported the following error message in the **System** log: `Dataplane down: controlplane exit failure`.DP crash / down
Internal path monitor fails
NFS transfer issue on DP
Tweaking NFS options
No workaround8.1.14,9.0.0
PAN-102096PA7000series
8.1.0-8.1.12
Fixed an issue where first packet processor packet buffer is not allocated with proper alignment, which caused memory corruption.internal path monitor failure , FPP crashPossible memory corruption on FPPNo workaround8.1.13
PAN-133440PA5200,PA7000series
8.1.8-8.1.12
9.0,9.1
Fixed an issue where fragmented traffic caused high dataplane use and firewall performance issues.high CPU/ high packet bufferfragment reassemble issueConsider blocking fragments via zone protection. 8.1.13,9.0.7,9.1.2
PAN-131993Panorama series
8.1.11-8.1.12
9.0,9.1
Fixed an issue where a process (*reportd*) would crash while running a log query.reportd crashdoublefree while trying cleanup when handling a log queryAllow the query to run to completion before closing the Tab/browser8.1.13,9.0.7,9.1.2
PAN-115875LFC(PA7000)
9.0.0-9.0.5
Fixed an issue where a PA-7080b HA pair rebooted when large sized packet traffic impacted the front panel ports of the Log Forwarding Card (LFC).LFC restartLFC front port error handling failure on receiving jumbo framesAvoid connecting the Front Panel ports to networks with jumbo frames9.0.6 and 9.1.0
PAN-1236679.0.0-9.0.5Fixed an issue where the "snmpd" process was crashing when polling for global counters.snmpd crash and OOM(out of memory) in kernelmemory leak of snmpd when accessing global counter OIDsWorkaround to avoid this crash is to avoid polling OIDs in the global counters table.9.0.6 and 9.1.0
PAN-123322PA3200,PA5200,PA7000series
8.1.0-8.1.11
9.0.0-9.0.5
"PA-3200 Series, PA-5200 Series, and PA-7000 Series firewalls running PAN-OS "<8.1.11 | 9.0.5>" only") There is an intermittent issue where a process ("all_pktproc") stops responding due to a Work Query Entry (WQE) corruption that is caused by duplicate child sessions.dataplane crashCrash when handing packet in predict sessionNone8.1.12,9.0.6 and 9.1.0
PAN-128269PA5200series only
8.1.10-8.1.11
9.0.0-9.0.5
"PA-5250, PA-5260, and PA-5280 firewalls with 100GB AOC cables only") When you upgrade the first peer in a high availability (HA) configuration to "[PAN-OS 8.1.9-h4 or a later] / [a PAN-OS 9.0]" release, the High Speed Chassis Interconnect (HSCI) port does not come up due to an FEC mismatch until after you finish upgrading the second peer.HSCI interface downInternal chip configuration affected AOC moduleConsult Techsupport for upgrade procedure, otherwise avoid the releases8.1.12,9.0.6 and 9.1.0
PAN-1244819.0.0-9.0.4Fixed an issue where the dataplane stopped responding when SMTP sessions were used.DP crash/ Internal Path Monitor FailureMIME boundary is mistakenly calculatedapp-override the smtp9.0.5
PAN-126547

8.1.0-8.1.10
9.0.0-9.0.4

Fixed an issue where a process ("configd") stopped responding when an XML API call with "type=config&action=get" triggered during a commit.configd crashNull was not set to a pointer when xml node is freedDo not run xml api to get predefined xpath8.1.11 and 9.0.5
PAN-120662

PA-7000 series only(XM cards are not affected)
8.1.0-8.1.10
9.0.0-9.0.4

["PA-7000 Series firewalls using PA-7000-20G-NPC cards only"] Fixed an intermittent issue where an out-of-memory (OOM) condition caused the dataplane or internal path monitoring to stop responding.DP crash/ Internal Path Monitor Failure
Insufficient memory was allocated to Linux kernel
No workaround8.1.11 and 9.0.4
PAN-119862PA5050 only
8.1.0-8.1.11
Fixed an intermittent issue where an out-of-memory (OOM) condition caused the dataplane or internal path monitoring to stop responding. With this fix, session capacity is reduced by 400,000.DP crash/ Internal Path Monitor FailureOut of memory on DP0No workaround8.1.11
PAN-1156958.0.x
8.1.0-8.1.9
9.0.0-9.0.3
Fixed an intermittent issue where a large number of packets were received before acknowledgments were complete, which depleted descriptor queue entries and resulted in high latency during data transfers even though CPU usage looked normalHigh packet descriptor and packet bufferAs a result, one or a few aggressive TCP sessions can take all descriptor queue entries due to ack packetsclear session causing the issue 8.1.10 and 9.0.4
PAN-1166138.0.x
8.1.0-8.1.8
9.0.0-9.0.3
Fixed an issue on a VM-Series firewall deployed in Microsoft Azure where packets dropped silently due to a kernel errortraffic drop when burst traffica kernel error when processing bust traffic on AzureNo workaround8.1.9 and 9.0.4
PAN-1201948.1.5-8.1.9
9.0.0-9.0.3
("Virtual and M-Series Panorama appliances and Log Collectors only") Fixed an issue where closed Elasticsearch (ES) indices were continuing to receive and re-queue logs, which resulted in high CPU usage.Log ingestion failure and high CPUmonthly index closed unexpectedlyContact Techsupport8.1.10 and 9.0.4
PAN-1184078.1.0-8.1.8
9.0.0-9.0.3
Fixed an issue where an internal path monitoring failure due to a buffer leak caused the firewall to rebootDP restart due to Internal packet path monitoring failuremess-up of buffer poolNo workaround8.1.9 and 9.0.4
PAN-1177208.1.0-8.1.9
9.0.0-9.0.3
("GlobalProtect Clientless VPN environments only") Fixed an issue where a process ("all_pktproc") stopped responding and caused the firewall to restart unexpectedly when processing GlobalProtect Clientless VPN traffic. To leverage this fix, you must first upgrade ("Devices>Dynamic Updates") to GlobalProtect Clientless VPN content release 79 or a later release.DP crashexception when handling clientless VPN packet with large packetchange clientlessVPN to GP(SSLVPN)
or downgrade to 8.1.8 or older
8.1.10 and 9.0.4
PAN-1139718.1.0-8.1.8
9.0.0-9.0.3
("PA-7000 Series firewalls only") Fixed an issue where the High Speed Chasis Interconnect (HSCI) link flapped after you rebooted the firewall.HSCI flapSignal errors on SMC 8.1.9 and 9.0.4
PAN-1117088.1.0-8.1.8
9.0.0-9.0.2
("PA-3200 Series firewalls only") Fixed a rare software issue that caused the dataplane to restart unexpectedly. To leverage this fix, you must run the "debug dataplane set pow no-desched yes" CLI command (increases CPU utilization).DP crashDeschedule issue on CPU used in PA3200No workaround8.1.9 and 9.0.3
PAN-1177298.1.8 onlyFixed an issue where the firewall incorrectly displayed application dependency
warnings ("Policies > Security") after you initiated a commit
Application dependency shows up on commitdue to incomplete fix of PAN-98386No workaround8.1.9
PAN-107005PA3200 series only
8.1.0-8.1.4
9.0.0-9.0.2
Fixed an issue on PA-3200 Series firewalls where packets dropped when a VSS-Monitoring Ethernet trailer was being appended by an external device. L4checksum fails for VSS monitoring trailer and the packet dropsNetwork offload processor drops the packet due to its L4 checkup validationNo workaround. upgrade PANOS8.1.5 and 9.0.3
PAN-1128148.1.6-8.1.7 and 
9.0.0-9.0.1
Fixed an issue where H.323-based calls lost audio because the predicted H.245 session was not converted to Active status, which caused the firewall to drop the H.245 traffic. predict session failurepredict session fails to create when the predict session is created by S2C flow and it's source NATedDo not use Source NAT8.1.8 and 9.0.2
PAN-1030238.0.14-8.1.15
8.1.2-8.1.6
Fixed an intermittent issue where a content install (content) caused a firewall configuration failure and the firewall to stop responding.FQDN objects are resolved as 0.0.0.0. and pushed to DP. that causes traffic issueContent install job involves wrong config mistakenlyCommit force or force another FQDN refresh.8.0.16 ,8.1.7 and 9.0.0
PAN-108241PA-3200 series/ 8.1.0-8.1.5Fixed an issue on a PA-3200 Series firewall where multiple dataplane processes (all_pktproc, flow_mgmt, flow_ctrl, and pktlog_forwarding) stopped responding when overloaded with traffic.DP crashflow ager process double freeEnable software aho/dfa and pscan can greatly reduce likelihood of seeing issue.8.1.6 and 9.0.0
PAN-1095948.0.14, 8.1.5 onlyFixed an issue where the dataplane restarted when an IPsec rekey event occurred and caused a tunnel process (tund) failure when one--but not both--HA peer is running PAN-OS 8.0.14 or PAN-OS 8.1.5.DP restart due to tund crash during version mismatch in HA peers during upgrade processDP restart due to tund crash which is caused by ike rekey in HA pairPrior to upgrading HA peers, temporarily adjust IKE lifetimes to longer than default to ensure that rekey event does not occur during upgrade process. Can also break HA between peers and upgrade individually as standalone.8.0.15, 8.1.6
PAN-108785PA3200 series/ 8.1.0-8.1.5Fixed an intermittent issue on a firewall in an HA active/passive configuration where a ping test stopped responding on Ethernet 1/1, 1/2, and 1/4 due to input errors on the corresponding switch port after an HA failover.eth1/1,2,4 corrupts packet on transmit after HA failoverinterface initialization steps after HA failover called unnecessary instructionsmanually shut/no shut the interfaces8.1.6 and 9.0.0
PAN-1077918.1.4Fixed an issue where after upgrading from PAN-OS 8.1.3 to 8.1.4 the CLI two-factor administrator authentication failed.2FA failssocket handling bug for 2FAnone8.1.5 and 9.0.0
PAN-1073658.1.4Fixed an issue on Panorama M-Series and virtual appliances where after you make a change to a template and attempt to push to a target device, the device does not appear in the Push Scope Selection list ("Commit > Push to Devices > Edit Selections > Device Groups").Cannot specify device in templateException in php codenone8.1.5
PAN-1072718.1.4Fixed an issue on a PA-3200 Series firewall running PAN-OS 8.1.4 in an HA configuration where the HA1-B (backup) port did not come up as expected.HA1B port is unusableadditional fix of PAN-89402use other interface for HA18.1.5
PAN-1002448.0.x,8.1.xFixed an issue where a failed commit or commit validation followed by a non-user-committed event (such as an FQDN refresh, an external dynamic list refresh, or an antivirus update) resulted in an unexpected change to the configuration that caused the firewall to drop traffic.traffic drop due to wrong policy appliedlast-candidatecfg.xml has been changed which should not happen when commit fails.  That config was involved in next FQDN/EDL updatePerforming manual FQDN refresh or commit appears to resolve the issue, until the next occurence.8.0.14,8.1.5
PAN-1006138.0.10-,8.1.2-8.1.4Fixed an issue on a PA-5200 Series firewall in a high availability (HA) active/active configuration with a virtual wire (vwire) subinterface where session setup packets sent to peer firewalls were sent back as HA2/HA3 race conditions, which caused an increase in packet descriptors and traffic to stop responding.traffic can be affected intermittently due to high packet descriptorDue to the race condition on session setup, packets loop in HA2/HA3 that affects Packet descriptorSession setup/owner set for first-packet/first-packet.  Otherwise, use Active/Passive mode8.1.5
PAN-1060168.0.x,8.1.xFixed an issue on PA-800 Series firewalls where a kernel memory spike caused the firewall to restart.unexpected system restartlack of kernel memorynone8.0.14,8.1.5
PAN-1069368.0.x,8.1.xFixed and issue where PA-800 Series firewalls intermittently restarted due to a kernel error.unexpected system restartheavy use of serial driver caused watch dog timeoutnone8.0.14,8.1.5
PAN-104116

8.1.3,8.0.12

Fixed an issue where a hardware packet buffer leak caused firewall performance to degrade.Hardware packet buffers depletionIn rare condition, the hardware packet buffer is not releasednone8.1.4,8.0.13

 

PAN-103921

PA-3200 series/

8.1.0-8.1.3

Fixed an issue on a PA 3200 Series firewall where the dataplane failed due to an internal path monitoring failure.Internal path monitor failureCommunication failure in link between MP and DP none8.1.4 and 9.0.0
PAN-103442

PA-3200 series/

8.1.0-8.1.3

Fixed an intermittent issue on a PA-3200 Series firewall where the forwarding information base (FIB) did not update correctly, which prevented successful forwarding of offloaded traffic.Some offloaded traffic is not forwarded correctly.FIB entry in DP is no update properly due to programming errorDisable session offload8.1.4 and 9.0.0
PAN-98116

PA-3000 series /

8.1.0-8.1.2

Fixed an issue where the PA-3000 series firewalls passed file-descriptors in a dataplane ("pan_comm") process during content (apps and threat) installation as well as FQDNRefresh job execution, which caused the hardware Layer 7 engine to incorrectly identify applications.App-ID(L7 process) stop working

 DP crash

File descriptor leak in pan_comm process in charge of commit in DPnone

8.1.3

 PAN-99212

 8.0.10-8.0.11

, 8.1.0-8.1.2

 Fixed an issue where the firewall incorrectly dropped ARP packets and increased the "flow_arp_throttle" counter. ARP does not work /Traffic stopARP packet throttling feature mistakenly counts number of arp inspected and drops arp packets none 

8.0.12 and  8.1.3

 PAN-98397

 PA-3200 series/

8.1.0-8.1.2

 Fixed an issue on PA-3200 series firewalls where the offload processor did not process route-deletion update messages , which left behind stale route entries and caused sessions to become unresponsive during the session-offload stage. Packet drop due to routing table problem in Offload chipFIB in Offload chip(FE100) has not updated properly after route deletion Disabling session offload 8.1.3
PAN-94912

PA-5200 series/

8.0.0-8.0.9

8.1.0-8.1.1

  Fixed an issue where PA-5200 Series and PA-3200 Series firewalls in an active/active high availability (HA) configuration sent packets in the wrong direction in a virtual wire deployment. 

MAC flapping happen on neighouring switch.

Traffic disruption can happen

 In ha Active-Active vwire case, when device forwards packets through ha3 link. the header info is correctly set in some cases, causing such packets are forwarded back to the HA peer, instead of forwarding locally. In one of the cases (00810651), disabling session offload has resolved the issue. 8.0.10 and 8.1.2
PAN-90890

8.0.0-8.0.9

8.1.0

Fixed an issue where the User-ID process ("useridd") stopped responding when a virtual system connected to more than one User-ID agent with NT LAN Manager (NTLM) enabled.

useridd process crash/

Useridd high file descriptor/ Useridd

instability

memory corruption of connection stateconfigure only one user-id-agent with NTLM enabled in each vsys.

8.0.10 and 8.1.1

PAN-93839

PAN-3000series and PAN-5000series

/

8.0.0-8.0.9

8.1.0

Fixed an issue where administrators failed to log in to the firewall due to an out-of-memory condition that intermittently caused the firewall to continuously restart processes. (PAN-90143 provided an initial memory enhancement in PAN-OS 8.0.9 that reduced the frequency of these out-of-memory events.)

low memory in MP kernel leads system instability such as admin login failure

/ Out of memory in MP

Linux kernels on PANOS 8.x/9.x have the memory leak which being fixed in the main stream linux. Port the patch from the main stream linux kernel.Reboot system

8.0.10 and 8.1.1

PAN-799898.0.0-8.0.8Fixed an issue on firewalls with custom signatures configured where low memory conditions intermittently caused commit or content installation failures with the following error: "Threat database handler failed."commit failuredevsrvr use fork() system call to spawn a child process(tdb_compile) to compile content during commit. When free memory is low, this fork() call can fail, which will fail commit or content installation.reboot system8.1.0, 8.0.9
PAN-90143

PA-5000 series/

8.0.0-8.0.8 and 8.1.0

Fixed an issue where administrators intermittently failed to log in to the firewall because it intermittently restarted processes continuously due to an out-of-memory condition. system stability/System unresponsiveKernel trackable memory is constantly decreasing. Changing the kernel configuration by disabling page mobility could stop the dropping.reboot system8.1.1, 8.0.9
PAN-92268

PA-7000,PA5200,PA3200 Series/

7.1.0-7.1.16 and 8.0.0-8.0.8

Fixed an issue on PA-7000 Series, PA-5200 Series, and PA-3200 Series firewalls where one or more dataplanes did not pass traffic when you ran several operational commands (from any firewall user interface or from the Panorama management server) while committing changes to device or network settings or while installing a content update. 

Traffic dropmiss-programing on Pancomm use wrong bypass queue iddo another commit if this happens.

 8.1.0 ,8.0.9

and 7.1.17,

PAN-92564

 8.0.0-8.0-8, 8.1.0

 Fixed an issue where a small percentage of writable third-party SFP transceivers (not purchased from Palo Alto Networks®) stopped working or experienced other issues after you upgraded the firewall to which the SFPs are connected to a PAN-OS [8.0 | 8.1] release. With this fix, you must not reboot the firewall after you download and install the PAN-OS [8.0 | 8.1] base image until after you download and install the PAN-OS [8.0.9 | 8.1.x] release. For additional details, upgrade considerations, and instructions for upgrading your firewalls, refer to the PAN-OS 8.1 upgrade information.  unsupported SFP stop working SDK had an I2C read error inserted. This caused PanOS 8.0, (and initial 8.1.0) to have this I2C bus driver to have this logical error in the Read functions, that messed up the Controller to Device protocol sequence.Use supported SFP

8.1.1,8.0.9

PAN-89718

PA-7000series

/

8.0.0-8.0.7

and all older Mainlines

Fixed an issue where PA-7000 Series firewalls rebooted continuously because the "brdagent" process stopped responding during bootup due to HSCI interface initializationFirewall reboots

FPP brdagent is tied up initializing the marvell PHYs and can't respond to heartbeats. As a result it gets killed by masterd

Disable HSCI ports or remove HSCI QSFP+ module during reboot

8.1.0,8.0.8

PAN-86882

8.0.0-8.0.7

and all older Mainlines

Fixed an issue where the firewall dataplane stopped responding after you used nested wildcards ("*") with "." or "/" as delimiters in the URLs of a custom URL category ("Objects > Custom Objects > URL Category") or in the "Allow List" of a URL Filtering profile ("Objects > Security Profiles > URL Filtering > <URL-filtering-profile> > Overrides"). With this fix, the firewalls does not allow you to use nested wildcards in such cases. For details, see "NESTED WILDCARD(*) IN URLS MAY SEVERELY AFFECT PERFORMANCE". DP crash and restart due to custom URL lookup

Misconfiguration on custom URL category using nested asterisk causes DP cpu highload

Note: fix is addtional configuration check to prevent

Use fewer number of  asterisk in configuration. see the link in Description for details

8.1.0,8.0.8

PAN-836878.0.0-8.0.6Fixed an issue on Panorama M-Series appliances where the "configd" process stopped responding during a "Commit > Commit and Push" operation where Panorama pushed configuration changes to Collector Groups.

configd crash 

 

During commit, a tables data structures under collector settings is destructed. Do not do Panorama commit and collector group push at same time.8.1.0,8.0.7
PAN-85938

8.0.0-8.0.6

7.1.0-7.1.13

Fixed an issue where PAN-OS removed the IP address-to-username mappings of end users who logged in to a GlobalProtect internal gateway within a second of logging out from it.

user-ip mapping information is not generated properlywhen Global protect Logout/login event happened in the same second, user-id in firewall can't determine the sequence of these events as we use timestamp(second granularity) to distinguish them. 

No Workaround available

8.0.7,7.1.14
PAN-82125

PA-5000series

/

8.0.0-8.0.6

Fixed an issue where the firewall management plane or control plane continuously rebooted after an upgrade to PAN-OS 8.0, and displayed the following error message: "rcu_sched detected stalls on CPUs/tasks".

continous MP/CP restarti2c issue due to SFP module holding the bus and cause i2c controller reset can't be finished.

Use supported SFP 

8.1.0, 8.0.7

PAN-82273

8.0.0-8.0.5,

7.1.6-7.1.13

Fixed an issue where blocking proxy sessions to enforce Decryption policy rules caused packet buffer depletion, which eventually resulted in packet loss.

Hardware buffer leak issue that could affect any type of traffic handled by DPLeaking packet buffer due to RST packets generated as part of policy-enforcement (denied traffic) in combination with no-decrypt rules

 

 

1. in ssl no-decrypt rule, in decryption profile remove actions from "No decrypt"

OR

2. change deny rule in policy to drop

 

8.0.6, 7.1.14

PAN-84545 

PA-800 series

/

8.0.0-8.0.5

Fixed an issue where PA-800 Series firewalls became unresponsive until you rebooted them, and the firewalls generated no logs from when they stopped responding to when they finished rebooting.

 

 

System unresponsive. no CLI/console/ping response

manual restart is required to recover from the issue

PA-800 uses a proprietary MDIO kernel driver. This driver had a bug in it that was causing a deadlock condition to take place.

 

 

No workaround

8.0.6

PAN-82830

PA-5000 series and PA-3000 series

/

8.0.0-8.0.5

Fixed an issue where PA-5000 Series and PA-3000 Series firewalls that were running low on memory briefly became unresponsive, stopped processing traffic, and stopped generating logs.

 

Firewall "hangs", and it cannot be accessed via SSH/GUI. No logs are being written, and there is no mgmt console output.

Larger memory footprint of 8.0 is causing the issue.

Downgrade to 7.1.x (issue only reported on 8.0.x so far)

 

8.0.6

PAN-81100

With low memory platform such as PA-200 and M-100 primarily.  Other platforms can happen the same issue

/

8.0.0-8.0.5

Fixed an issue on the firewall and Panorama management server where a memory leak caused several operations to fail, such as commits, FQDN refreshes, and content updates.

 

Commit failing and/or memory leak with error: fork() failed!

/ Symptoms include failing to commit, GUI unresponsive, HA config sync failing, MP memory leak, daemon crashes, high MP CPU.

In 8.0 we upgraded to 64-bits. Hence virt and res memory usage will go up slightly.

 

On M-100, upgrading to 32GB memory should  greatly reduce occurrences.

For PA-200 or other platforms, no workaround exists short of downgrading to 7.1.x.

 

8.0.6

PAN-78718

PA-7000 series with Panorama

/

8.0.0-8.0.5

7.1.0-7.1.12, 7.0.0-7.0.18

PA7050 logging stops / Logrcvr crashing on PA-7050LPC stopped saving and displaying new logs due to a memory leak after a Panorama management server running a PAN-OS 8.0 or newer

The issue commonly happens on a 7K FW running 7.x release , which is managed by a Panorama running Rome (8.0).

FW fails processing GTP report definitions which causes memory leak. 

 

 

 

From Panorama running 8.0(or newer) CLI config:

set deviceconfig setting management disable-predefined-reports [ gtp-spoofed-end-ip gtp-malicious-wildfire-submissions top-gtp-attackers top-gtp-victims gtp-users-visiting-malicious-url ]

 

8.0.6, 7.1.13, 7.0.19

PAN-82095

All software QoS platform listed in the description

/

8.0.0 to 8.0.5

7.1.0 to 7.1.14

Fixed an issue on PA-3000 Series, PA-800 Series, PA-500, PA-220, PA-200, and VM-Series firewalls where QoS throughput dropped on interfaces configured to use a QoS profile with an "Egress Max" set to 0Mbps or more than 1143 Mbps ("Network > Network Profiles > QoS Profile").

 QoS enforces max bandwidth with lesser traffic than configuredCoding error limitting max to 1Gbps

 

Lower the QoS bandwidth below 1143Mbit/s, downgrade to <=7.1.10 and/or <=8.0.3

 

8.0.6, 7.1.14

PAN-82275

VM-series

/

8.0.0 to 8.0.4
VM sereis: traffic getting dropped Traffic getting dropped due to flow_qos_pkt_timeout QoS packet is not dequeued after 82 days

The QoS timer variable was not reset properly.

Disable QoS config

 

8.0.5

PAN-81590

PA-5200 series

/

8.0.0 to 8.0.4
Internal link instability between DP and CE(Content Engine)Affects 5200 platforms. System can continue to boot even if CE init fails. This causes issues with Layer7 inspection and HA pathmonitor,etc.

controlplane-console-output.log shows following error:

nac0: Memory channels init incomplete
 

It's internal link issue between DP and CE.
imporved link init and recovery mechanism

 

Use software aho and dfa.

> debug dataplane fpga set sw_aho yes
> debug dataplane fpga set sw_dfa yes

 

8.0.5

PAN-81990

PA-5220,PA-5250

/

8.0.4
Multiple DP restarts by all_pktprocDP crash due to small memory pool size in 8.0.4. Seen only on PA-5220 and PA-5250. With same cause, Other symptoms such as GP(GlobalProtect) connections dropping and SSL decryption traffic failing could happen

fixed memory pool size on the affected platform

 

Use other platforms other than PA-5220 or PA-5250. Or downgrade to 8.0.3.

 

8.0.5

PAN-78572

M-series

/

8.0.0 to 8.0.4
Logd high memory on M-seriesTypical symptoms:
-Traffic and threat logs delayed on Panorama for 24 hours.
-Oom kernel crash
-commit failure
-memory allocation failure

Due to indexing of messages in evtmgr queues start building up. This causes the memory buildup in logd and results in indexing process not being able to startup.

no workaround

 

8.0.5

PAN-80445

M-series

/

8.0.0 to 8.0.3
Reportd memory leak on M-seriesReportd memory increases until you run out. Can cause sluggish performance or loss of ability to manage.

Reportd memory leak happens only on M-series in combo mode. 

fixed various memory leak in reportd process

 

Do not use combo mode. Use dedicated log collectors. 

 

8.0.4

PAN-74655

Not platform specific

/

7.0.x, 7.1.x
High DP CPU with high urlcache_lookup processing timeHigh DP utilization and general traffic slowness caused by URL filtering. Urlcache related function process time goes up in "debug dataplane pow performance"

Issue with URL cache when cache gets above 1 million URLs in MP cache and device-server is consuming high CPUs. DP also consumed high CPU to lookup local cache grows big

 

Clear DP and MP cache:

>clear url-cache all

>delete url-database all

 

PAN-DB cloud update has the fix in March/2017

PANOS fix in 6.1.18, 7.0.16 and 7.1.10.

 



Attachments
Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000Cm68CAC&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Attachments
Choose Language