Palo Alto Networks Knowledgebase: Critical Issues Addressed in PAN-OS Releases

Critical Issues Addressed in PAN-OS Releases

111106
Created On 10/14/19 04:30 AM - Last Updated 10/17/19 08:29 AM
MFA PA-3000 Series PA-3200 Series PA-5200 Series Virtual Appliance Content Release 8.1 8.0 7.1 7.0 9.0 PAN-OS Panorama
Symptom
 
 


Environment
 
 


Cause
 
 


Resolution

Last Updated On : Oct 14th , 2019

 

 

This list is limited to critical severity issues as determined by Palo Alto Networks and is provided for informational purposes only.

  • Please doublecheck the information in release notes to see the latest info about fixed versions.
  • Please create a case with your support provider for a detailed investigation if you feel you have encountered one of these issues.
  • Maintenance releases are the primary mechanism to fix issues.
  • A maintenance release is signified by the third digit in the release version number (for example the .2 in PAN-OS 8.0.2 ).

 

Bugs

Affected Platform(if any)

/Affected Version

Description (release note)Impact

Root cause

Workaround

Fixed release

PAN-1156958.0.x
8.1.0-8.1.9
9.0.0-9.0.3
Fixed an intermittent issue where a large number of packets were received before acknowledgments were complete, which depleted descriptor queue entries and resulted in high latency during data transfers even though CPU usage looked normalHigh packet descriptor and packet bufferAs a result, one or a few aggressive TCP sessions can take all descriptor queue entries due to ack packetsclear session causing the issue 8.1.10 and 9.0.4
PAN-1166138.0.x
8.1.0-8.1.8
9.0.0-9.0.3
Fixed an issue on a VM-Series firewall deployed in Microsoft Azure where packets dropped silently due to a kernel errortraffic drop when burst traffica kernel error when processing bust traffic on AzureNo workaround8.1.9 and 9.0.4
PAN-1201948.1.5-8.1.9
9.0.0-9.0.3
("Virtual and M-Series Panorama appliances and Log Collectors only") Fixed an issue where closed Elasticsearch (ES) indices were continuing to receive and re-queue logs, which resulted in high CPU usage.Log ingestion failure and high CPUmonthly index closed unexpectedlyContact Techsupport8.1.10 and 9.0.4
PAN-1184078.1.0-8.1.8
9.0.0-9.0.3
Fixed an issue where an internal path monitoring failure due to a buffer leak caused the firewall to rebootDP restart due to Internal packet path monitoring failuremess-up of buffer poolNo workaround8.1.9 and 9.0.4
PAN-1177208.1.0-8.1.9
9.0.0-9.0.3
("GlobalProtect Clientless VPN environments only") Fixed an issue where a process ("all_pktproc") stopped responding and caused the firewall to restart unexpectedly when processing GlobalProtect Clientless VPN traffic. To leverage this fix, you must first upgrade ("Devices>Dynamic Updates") to GlobalProtect Clientless VPN content release 79 or a later release.DP crashexception when handling clientless VPN packet with large packetchange clientlessVPN to GP(SSLVPN)
or downgrade to 8.1.8 or older
8.1.10 and 9.0.4
PAN-1139718.1.0-8.1.8
9.0.0-9.0.3
("PA-7000 Series firewalls only") Fixed an issue where the High Speed Chasis Interconnect (HSCI) link flapped after you rebooted the firewall.HSCI flapSignal errors on SMC 8.1.9 and 9.0.4
PAN-1117088.1.0-8.1.8
9.0.0-9.0.2
("PA-3200 Series firewalls only") Fixed a rare software issue that caused the dataplane to restart unexpectedly. To leverage this fix, you must run the "debug dataplane set pow no-desched yes" CLI command (increases CPU utilization).DP crashDeschedule issue on CPU used in PA3200No workaround8.1.9 and 9.0.3
PAN-1177298.1.8 onlyFixed an issue where the firewall incorrectly displayed application dependency
warnings ("Policies > Security") after you initiated a commit
Application dependency shows up on commitdue to incomplete fix of PAN-98386No workaround8.1.9
PAN-107005PA3200 series only
8.1.0-8.1.4
9.0.0-9.0.2
Fixed an issue on PA-3200 Series firewalls where packets dropped when a VSS-Monitoring Ethernet trailer was being appended by an external device. L4checksum fails for VSS monitoring trailer and the packet dropsNetwork offload processor drops the packet due to its L4 checkup validationNo workaround. upgrade PANOS8.1.5 and 9.0.3
PAN-1128148.1.6-8.1.7 and 
9.0.0-9.0.1
Fixed an issue where H.323-based calls lost audio because the predicted H.245 session was not converted to Active status, which caused the firewall to drop the H.245 traffic. predict session failurepredict session fails to create when the predict session is created by S2C flow and it's source NATedDo not use Source NAT8.1.8 and 9.0.2
PAN-1030238.0.14-8.1.15
8.1.2-8.1.6
Fixed an intermittent issue where a content install (content) caused a firewall configuration failure and the firewall to stop responding.FQDN objects are resolved as 0.0.0.0. and pushed to DP. that causes traffic issueContent install job involves wrong config mistakenlyCommit force or force another FQDN refresh.8.0.16 ,8.1.7 and 9.0.0
PAN-108241PA-3200 series/ 8.1.0-8.1.5Fixed an issue on a PA-3200 Series firewall where multiple dataplane processes (all_pktproc, flow_mgmt, flow_ctrl, and pktlog_forwarding) stopped responding when overloaded with traffic.DP crashflow ager process double freeEnable software aho/dfa and pscan can greatly reduce likelihood of seeing issue.8.1.6 and 9.0.0
PAN-1095948.0.14, 8.1.5 onlyFixed an issue where the dataplane restarted when an IPsec rekey event occurred and caused a tunnel process (tund) failure when one--but not both--HA peer is running PAN-OS 8.0.14 or PAN-OS 8.1.5.DP restart due to tund crash during version mismatch in HA peers during upgrade processDP restart due to tund crash which is caused by ike rekey in HA pairPrior to upgrading HA peers, temporarily adjust IKE lifetimes to longer than default to ensure that rekey event does not occur during upgrade process. Can also break HA between peers and upgrade individually as standalone.8.0.15, 8.1.6
PAN-108785PA3200 series/ 8.1.0-8.1.5Fixed an intermittent issue on a firewall in an HA active/passive configuration where a ping test stopped responding on Ethernet 1/1, 1/2, and 1/4 due to input errors on the corresponding switch port after an HA failover.eth1/1,2,4 corrupts packet on transmit after HA failoverinterface initialization steps after HA failover called unnecessary instructionsmanually shut/no shut the interfaces8.1.6 and 9.0.0
PAN-1077918.1.4Fixed an issue where after upgrading from PAN-OS 8.1.3 to 8.1.4 the CLI two-factor administrator authentication failed.2FA failssocket handling bug for 2FAnone8.1.5 and 9.0.0
PAN-1073658.1.4Fixed an issue on Panorama M-Series and virtual appliances where after you make a change to a template and attempt to push to a target device, the device does not appear in the Push Scope Selection list ("Commit > Push to Devices > Edit Selections > Device Groups").Cannot specify device in templateException in php codenone8.1.5
PAN-1072718.1.4Fixed an issue on a PA-3200 Series firewall running PAN-OS 8.1.4 in an HA configuration where the HA1-B (backup) port did not come up as expected.HA1B port is unusableadditional fix of PAN-89402use other interface for HA18.1.5
PAN-1002448.0.x,8.1.xFixed an issue where a failed commit or commit validation followed by a non-user-committed event (such as an FQDN refresh, an external dynamic list refresh, or an antivirus update) resulted in an unexpected change to the configuration that caused the firewall to drop traffic.traffic drop due to wrong policy appliedlast-candidatecfg.xml has been changed which should not happen when commit fails.  That config was involved in next FQDN/EDL updatePerforming manual FQDN refresh or commit appears to resolve the issue, until the next occurence.8.0.14,8.1.5
PAN-1006138.0.10-,8.1.2-8.1.4Fixed an issue on a PA-5200 Series firewall in a high availability (HA) active/active configuration with a virtual wire (vwire) subinterface where session setup packets sent to peer firewalls were sent back as HA2/HA3 race conditions, which caused an increase in packet descriptors and traffic to stop responding.traffic can be affected intermittently due to high packet descriptorDue to the race condition on session setup, packets loop in HA2/HA3 that affects Packet descriptorSession setup/owner set for first-packet/first-packet.  Otherwise, use Active/Passive mode8.1.5
PAN-1060168.0.x,8.1.xFixed an issue on PA-800 Series firewalls where a kernel memory spike caused the firewall to restart.unexpected system restartlack of kernel memorynone8.0.14,8.1.5
PAN-1069368.0.x,8.1.xFixed and issue where PA-800 Series firewalls intermittently restarted due to a kernel error.unexpected system restartheavy use of serial driver caused watch dog timeoutnone8.0.14,8.1.5
PAN-104116

8.1.3,8.0.12

Fixed an issue where a hardware packet buffer leak caused firewall performance to degrade.Hardware packet buffers depletionIn rare condition, the hardware packet buffer is not releasednone8.1.4,8.0.13

 

PAN-103921

PA-3200 series/

8.1.0-8.1.3

Fixed an issue on a PA 3200 Series firewall where the dataplane failed due to an internal path monitoring failure.Internal path monitor failureCommunication failure in link between MP and DP none8.1.4 and 9.0.0
PAN-103442

PA-3200 series/

8.1.0-8.1.3

Fixed an intermittent issue on a PA-3200 Series firewall where the forwarding information base (FIB) did not update correctly, which prevented successful forwarding of offloaded traffic.Some offloaded traffic is not forwarded correctly.FIB entry in DP is no update properly due to programming errorDisable session offload8.1.4 and 9.0.0
PAN-98116

PA-3000 series /

8.1.0-8.1.2

Fixed an issue where the PA-3000 series firewalls passed file-descriptors in a dataplane ("pan_comm") process during content (apps and threat) installation as well as FQDNRefresh job execution, which caused the hardware Layer 7 engine to incorrectly identify applications.App-ID(L7 process) stop working

 DP crash

File descriptor leak in pan_comm process in charge of commit in DPnone

8.1.3

 PAN-99212

 8.0.10-8.0.11

, 8.1.0-8.1.2

 Fixed an issue where the firewall incorrectly dropped ARP packets and increased the "flow_arp_throttle" counter. ARP does not work /Traffic stopARP packet throttling feature mistakenly counts number of arp inspected and drops arp packets none 

8.0.12 and  8.1.3

 PAN-98397

 PA-3200 series/

8.1.0-8.1.2

 Fixed an issue on PA-3200 series firewalls where the offload processor did not process route-deletion update messages , which left behind stale route entries and caused sessions to become unresponsive during the session-offload stage. Packet drop due to routing table problem in Offload chipFIB in Offload chip(FE100) has not updated properly after route deletion Disabling session offload 8.1.3
PAN-94912

PA-5200 series/

8.0.0-8.0.9

8.1.0-8.1.1

  Fixed an issue where PA-5200 Series and PA-3200 Series firewalls in an active/active high availability (HA) configuration sent packets in the wrong direction in a virtual wire deployment. 

MAC flapping happen on neighouring switch.

Traffic disruption can happen

 In ha Active-Active vwire case, when device forwards packets through ha3 link. the header info is correctly set in some cases, causing such packets are forwarded back to the HA peer, instead of forwarding locally. In one of the cases (00810651), disabling session offload has resolved the issue. 8.0.10 and 8.1.2
PAN-90890

8.0.0-8.0.9

8.1.0

Fixed an issue where the User-ID process ("useridd") stopped responding when a virtual system connected to more than one User-ID agent with NT LAN Manager (NTLM) enabled.

useridd process crash/

Useridd high file descriptor/ Useridd

instability

memory corruption of connection stateconfigure only one user-id-agent with NTLM enabled in each vsys.

8.0.10 and 8.1.1

PAN-93839

PAN-3000series and PAN-5000series

/

8.0.0-8.0.9

8.1.0

Fixed an issue where administrators failed to log in to the firewall due to an out-of-memory condition that intermittently caused the firewall to continuously restart processes. (PAN-90143 provided an initial memory enhancement in PAN-OS 8.0.9 that reduced the frequency of these out-of-memory events.)

low memory in MP kernel leads system instability such as admin login failure

/ Out of memory in MP

Linux kernels on PANOS 8.x/9.x have the memory leak which being fixed in the main stream linux. Port the patch from the main stream linux kernel.Reboot system

8.0.10 and 8.1.1

PAN-799898.0.0-8.0.8Fixed an issue on firewalls with custom signatures configured where low memory conditions intermittently caused commit or content installation failures with the following error: "Threat database handler failed."commit failuredevsrvr use fork() system call to spawn a child process(tdb_compile) to compile content during commit. When free memory is low, this fork() call can fail, which will fail commit or content installation.reboot system8.1.0, 8.0.9
PAN-90143

PA-5000 series/

8.0.0-8.0.8 and 8.1.0

Fixed an issue where administrators intermittently failed to log in to the firewall because it intermittently restarted processes continuously due to an out-of-memory condition. system stability/System unresponsiveKernel trackable memory is constantly decreasing. Changing the kernel configuration by disabling page mobility could stop the dropping.reboot system8.1.1, 8.0.9
PAN-92268

PA-7000,PA5200,PA3200 Series/

7.1.0-7.1.16 and 8.0.0-8.0.8

Fixed an issue on PA-7000 Series, PA-5200 Series, and PA-3200 Series firewalls where one or more dataplanes did not pass traffic when you ran several operational commands (from any firewall user interface or from the Panorama management server) while committing changes to device or network settings or while installing a content update. 

Traffic dropmiss-programing on Pancomm use wrong bypass queue iddo another commit if this happens.

 8.1.0 ,8.0.9

and 7.1.17,

PAN-92564

 8.0.0-8.0-8, 8.1.0

 Fixed an issue where a small percentage of writable third-party SFP transceivers (not purchased from Palo Alto Networks®) stopped working or experienced other issues after you upgraded the firewall to which the SFPs are connected to a PAN-OS [8.0 | 8.1] release. With this fix, you must not reboot the firewall after you download and install the PAN-OS [8.0 | 8.1] base image until after you download and install the PAN-OS [8.0.9 | 8.1.x] release. For additional details, upgrade considerations, and instructions for upgrading your firewalls, refer to the PAN-OS 8.1 upgrade information.  unsupported SFP stop working SDK had an I2C read error inserted. This caused PanOS 8.0, (and initial 8.1.0) to have this I2C bus driver to have this logical error in the Read functions, that messed up the Controller to Device protocol sequence.Use supported SFP

8.1.1,8.0.9

PAN-89718

PA-7000series

/

8.0.0-8.0.7

and all older Mainlines

Fixed an issue where PA-7000 Series firewalls rebooted continuously because the "brdagent" process stopped responding during bootup due to HSCI interface initializationFirewall reboots

FPP brdagent is tied up initializing the marvell PHYs and can't respond to heartbeats. As a result it gets killed by masterd

Disable HSCI ports or remove HSCI QSFP+ module during reboot

8.1.0,8.0.8

PAN-86882

8.0.0-8.0.7

and all older Mainlines

Fixed an issue where the firewall dataplane stopped responding after you used nested wildcards ("*") with "." or "/" as delimiters in the URLs of a custom URL category ("Objects > Custom Objects > URL Category") or in the "Allow List" of a URL Filtering profile ("Objects > Security Profiles > URL Filtering > <URL-filtering-profile> > Overrides"). With this fix, the firewalls does not allow you to use nested wildcards in such cases. For details, see "NESTED WILDCARD(*) IN URLS MAY SEVERELY AFFECT PERFORMANCE". DP crash and restart due to custom URL lookup

Misconfiguration on custom URL category using nested asterisk causes DP cpu highload

Note: fix is addtional configuration check to prevent

Use fewer number of  asterisk in configuration. see the link in Description for details

8.1.0,8.0.8

PAN-836878.0.0-8.0.6Fixed an issue on Panorama M-Series appliances where the "configd" process stopped responding during a "Commit > Commit and Push" operation where Panorama pushed configuration changes to Collector Groups.

configd crash 

 

During commit, a tables data structures under collector settings is destructed. Do not do Panorama commit and collector group push at same time.8.1.0,8.0.7
PAN-85938

8.0.0-8.0.6

7.1.0-7.1.13

Fixed an issue where PAN-OS removed the IP address-to-username mappings of end users who logged in to a GlobalProtect internal gateway within a second of logging out from it.

user-ip mapping information is not generated properlywhen Global protect Logout/login event happened in the same second, user-id in firewall can't determine the sequence of these events as we use timestamp(second granularity) to distinguish them. 

No Workaround available

8.0.7,7.1.14
PAN-82125

PA-5000series

/

8.0.0-8.0.6

Fixed an issue where the firewall management plane or control plane continuously rebooted after an upgrade to PAN-OS 8.0, and displayed the following error message: "rcu_sched detected stalls on CPUs/tasks".

continous MP/CP restarti2c issue due to SFP module holding the bus and cause i2c controller reset can't be finished.

Use supported SFP 

8.1.0, 8.0.7

PAN-82273

8.0.0-8.0.5,

7.1.6-7.1.13

Fixed an issue where blocking proxy sessions to enforce Decryption policy rules caused packet buffer depletion, which eventually resulted in packet loss.

Hardware buffer leak issue that could affect any type of traffic handled by DPLeaking packet buffer due to RST packets generated as part of policy-enforcement (denied traffic) in combination with no-decrypt rules

 

 

1. in ssl no-decrypt rule, in decryption profile remove actions from "No decrypt"

OR

2. change deny rule in policy to drop

 

8.0.6, 7.1.14

PAN-84545 

PA-800 series

/

8.0.0-8.0.5

Fixed an issue where PA-800 Series firewalls became unresponsive until you rebooted them, and the firewalls generated no logs from when they stopped responding to when they finished rebooting.

 

 

System unresponsive. no CLI/console/ping response

manual restart is required to recover from the issue

PA-800 uses a proprietary MDIO kernel driver. This driver had a bug in it that was causing a deadlock condition to take place.

 

 

No workaround

8.0.6

PAN-82830

PA-5000 series and PA-3000 series

/

8.0.0-8.0.5

Fixed an issue where PA-5000 Series and PA-3000 Series firewalls that were running low on memory briefly became unresponsive, stopped processing traffic, and stopped generating logs.

 

Firewall "hangs", and it cannot be accessed via SSH/GUI. No logs are being written, and there is no mgmt console output.

Larger memory footprint of 8.0 is causing the issue.

Downgrade to 7.1.x (issue only reported on 8.0.x so far)

 

8.0.6

PAN-81100

With low memory platform such as PA-200 and M-100 primarily.  Other platforms can happen the same issue

/

8.0.0-8.0.5

Fixed an issue on the firewall and Panorama management server where a memory leak caused several operations to fail, such as commits, FQDN refreshes, and content updates.

 

Commit failing and/or memory leak with error: fork() failed!

/ Symptoms include failing to commit, GUI unresponsive, HA config sync failing, MP memory leak, daemon crashes, high MP CPU.

In 8.0 we upgraded to 64-bits. Hence virt and res memory usage will go up slightly.

 

On M-100, upgrading to 32GB memory should  greatly reduce occurrences.

For PA-200 or other platforms, no workaround exists short of downgrading to 7.1.x.

 

8.0.6

PAN-78718

PA-7000 series with Panorama

/

8.0.0-8.0.5

7.1.0-7.1.12, 7.0.0-7.0.18

PA7050 logging stops / Logrcvr crashing on PA-7050LPC stopped saving and displaying new logs due to a memory leak after a Panorama management server running a PAN-OS 8.0 or newer

The issue commonly happens on a 7K FW running 7.x release , which is managed by a Panorama running Rome (8.0).

FW fails processing GTP report definitions which causes memory leak. 

 

 

 

From Panorama running 8.0(or newer) CLI config:

set deviceconfig setting management disable-predefined-reports [ gtp-spoofed-end-ip gtp-malicious-wildfire-submissions top-gtp-attackers top-gtp-victims gtp-users-visiting-malicious-url ]

 

8.0.6, 7.1.13, 7.0.19

PAN-82095

All software QoS platform listed in the description

/

8.0.0 to 8.0.5

7.1.0 to 7.1.14

Fixed an issue on PA-3000 Series, PA-800 Series, PA-500, PA-220, PA-200, and VM-Series firewalls where QoS throughput dropped on interfaces configured to use a QoS profile with an "Egress Max" set to 0Mbps or more than 1143 Mbps ("Network > Network Profiles > QoS Profile").

 QoS enforces max bandwidth with lesser traffic than configuredCoding error limitting max to 1Gbps

 

Lower the QoS bandwidth below 1143Mbit/s, downgrade to <=7.1.10 and/or <=8.0.3

 

8.0.6, 7.1.14

PAN-82275

VM-series

/

8.0.0 to 8.0.4
VM sereis: traffic getting dropped Traffic getting dropped due to flow_qos_pkt_timeout QoS packet is not dequeued after 82 days

The QoS timer variable was not reset properly.

Disable QoS config

 

8.0.5

PAN-81590

PA-5200 series

/

8.0.0 to 8.0.4
Internal link instability between DP and CE(Content Engine)Affects 5200 platforms. System can continue to boot even if CE init fails. This causes issues with Layer7 inspection and HA pathmonitor,etc.

controlplane-console-output.log shows following error:

nac0: Memory channels init incomplete
 

It's internal link issue between DP and CE.
imporved link init and recovery mechanism

 

Use software aho and dfa.

> debug dataplane fpga set sw_aho yes
> debug dataplane fpga set sw_dfa yes

 

8.0.5

PAN-81990

PA-5220,PA-5250

/

8.0.4
Multiple DP restarts by all_pktprocDP crash due to small memory pool size in 8.0.4. Seen only on PA-5220 and PA-5250. With same cause, Other symptoms such as GP(GlobalProtect) connections dropping and SSL decryption traffic failing could happen

fixed memory pool size on the affected platform

 

Use other platforms other than PA-5220 or PA-5250. Or downgrade to 8.0.3.

 

8.0.5

PAN-78572

M-series

/

8.0.0 to 8.0.4
Logd high memory on M-seriesTypical symptoms:
-Traffic and threat logs delayed on Panorama for 24 hours.
-Oom kernel crash
-commit failure
-memory allocation failure

Due to indexing of messages in evtmgr queues start building up. This causes the memory buildup in logd and results in indexing process not being able to startup.

no workaround

 

8.0.5

PAN-80445

M-series

/

8.0.0 to 8.0.3
Reportd memory leak on M-seriesReportd memory increases until you run out. Can cause sluggish performance or loss of ability to manage.

Reportd memory leak happens only on M-series in combo mode. 

fixed various memory leak in reportd process

 

Do not use combo mode. Use dedicated log collectors. 

 

8.0.4

PAN-74655

Not platform specific

/

7.0.x, 7.1.x
High DP CPU with high urlcache_lookup processing timeHigh DP utilization and general traffic slowness caused by URL filtering. Urlcache related function process time goes up in "debug dataplane pow performance"

Issue with URL cache when cache gets above 1 million URLs in MP cache and device-server is consuming high CPUs. DP also consumed high CPU to lookup local cache grows big

 

Clear DP and MP cache:

>clear url-cache all

>delete url-database all

 

PAN-DB cloud update has the fix in March/2017

PANOS fix in 6.1.18, 7.0.16 and 7.1.10.

 

 


Additional Information
 
 


Attachments
Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000Cm68CAC&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Attachments
Choose Language