Troubleshooting loss of dataplane logs on the Firewall

• The firewall may be intermittently providing logs.
• The firewall isn't providing the required logs as per the configuration and inspected traffic.
• The Dataplane logs can be any of the following:
  • Netflow 
  • Traffic 
  • Threat
  • URL Filtering  
  • Decryption   
  • Tunnel Inspection

• Palo Alto Networks Firewalls.
• Supported PAN-OS.
• DataPlane Logs.

• When the NGFW processes traffic in the dataplane and if there is a corresponding configuration to log the event, the information is sent to one of the four queues, depending on the nature of the log.
• The four DP log queues are as follows:
  • Threat queue
  • Traffic queue
  • Threat-var (variable length threat log) queue
  • Deep Packet Inspection queue
• The priority of the queue is based on the above chronological orders. 
• The dataplane has a channel that links it to the management plane for log processing (DP to MP).
• This same channel is also used for packet capture and any requests from DP to the cloud (e.g., URL, DNS Security, Wildfire, etc.).
• Due to the possibility of excessive logging, other functions that are more vital to the inspection of traffic may be dropped. To prevent this from happening, a rate-limiting mechanism was implemented to regulate either the logging count rate (logs/sec) or bandwidth usage (KB/sec).
• The command below provides us with the information to visualize the rate-limiting mechanism:

admin@NGFW> show system setting logging

Max. logging rate:            1280       cnt/s
Max. packet logging rate:     2560        KB/s
Traffic log generation rate:  1597       cnt/s  383280    Byte/s
Threat log generation rate:   1          cnt/s  240       Byte/s
Analytics generation rate:    38         cnt/s  47878     Byte/s
Log sent rate:                1765       cnt/s  425384    Byte/s
Threat Log sent rate:         1          cnt/s  248       Byte/s
Analytics sent rate:          34         cnt/s  46656     Byte/s
Log send failure rate:        0          cnt/s  0         Byte/s
Threat Log send failure rate: 0          cnt/s  0         Byte/s
Analytics send failure rate:  0          cnt/s  0         Byte/s
Current traffic log count:    392         
Current threat log count:     2334      
Current analytics log count:  3625      
Random traffic log drop:      off  
Log suppression:              on
default-policy-logging:       off

• The above output indicates that most of the log information is generated from the Traffic log generation rate, which can guide us on where to address the issue.
• When either the max. logging rate or max packet logging rate is breached, not all logs in the queue will be sent to the management plane for logging.
• Logs that are not sent to the management plane due to bandwidth restrictions are never seen at all. 
• When the management plane receives that log, the logrcvr daemon is responsible for writing the info to the file/disk.

1. Threat queue: is seldom used by the firewall, therefore no solution will be implemented here
2. Traffic queue:  in most customer environments, addressing the issue here solves the most number of TAC cases.

   1. Netflow logs are chatty in nature that they can hog the whole queue in extreme cases only netflow log info is generated by the NGFW. When NetFlow is enabled on an interface, a session hitting the interface will generate a session start log, multiple session update logs (depending on how long the session lives), and a session end log. These netflow logs are all considered traffic logs. 

      1. Disable all or decrease the number of interfaces enabled with Netflow. Go to Network > Interfaces > Ethernet x/x > Netflow Profile > None 
   2. Another log that consumes a significant amount of space in the traffic queue is the traffic log, disable the following logs
      1. Session start log. 
      2. intrazone-default log
      3. interzone-default log
      4. Go to Policies > Security > Security policy name > Actions Log Setting
   3. You can use the command > show system setting logging to verify whether the amount of logging is within the maximum parameters, if not repeat the above steps.
3. Threat-var (variable length threat log) queue 
   1. Provided below is the checklist for the logging and packet capture features as well as their location and where to disable them.

3. 2. You can use the command > show system setting logging to verify whether the amount of logging is within the maximum parameters, if not repeat the above steps.
4. Deep Packet Inspection (DPI) queue: This segment of the queue is the least prioritized among all queues. If you reach this stage and continue to exceed the maximum logging rate or maximum packet logging rate, it is advisable to upgrade to a higher-end model platform to meet your logging requirements.