How to troubleshoot and verify log forwarding issues for LPC on PA-7000 series firewall

• Verify Log Processing Card (LPC) is detected by PA-7000 series firewall
• Traffic logs are forwarding correctly by LPC
• Troubleshoot logging issues for LPC 

• PA-7000 series firewall with LPC card
• Syslog

• Check LPC is up and detected by PA-7000 series firewall

7K(active)> show chassis inventory 
Slot       Component            Serial Number   Ports  Revision  Power (w)
           Chassis              010108001644           1.0

1          PA-7000-20GXM-NPC    013701000814    24     1.0       350
2          PA-7000-20GQ-NPC     011701002948    14     1.0       350
3          empty
4          PA-7050-SMC          002101001677    0      2.0       300
5          empty
6          empty
7          PA-7000-LPC          001401001198    0      1.0       300
8          empty

7K(active)> show chassis status
Slot       Component          Card Status        Config Status   Disabled
1          PA-7000-20G-NPC    Up                 Success
2          PA-7000-20G-NPC    Up                 Success
3          PA-7000-20GQ-NPC   Up                 Success
4          empty
5          empty
6          PA-7080-SMC        Up                 Success
7          PA-7000-LPC        Up                 Success

• If LPC does not come up, check system logs for reason

2019/02/23 15:20:38 critical general        general 0  chassis: restarts exhausted, rebooting system
2019/02/23 15:20:38 critical general        general 0  chassis: Exitted 3 times, rebooting to the maintenance partition
2019/02/23 15:20:38 critical general        general 0  LPC slot 7 failed, rebooting the system

• Check which interface is configured as the log-card interface for log fowarding 

7K(active)> show interface all
...
name                id    vsys zone             forwarding               tag    address
------------------- ----- ---- ---------------- ------------------------ ------ ------------------
...
ethernet1/8       523   0                     logfwd                   0      N/A

• Check configured information for log card interface

7050(active)> debug log-card-interface info slot s7
 -------------------------------------------------------------------------------
Name: log-card-interface
MAC address:
  Port MAC address 58:49:3b:d6:f4:4a
 
Ip address: 10.212.106.58
Netmask: 255.255.255.0

Default gateway: 10.212.106.1

• Verify syslog server is able to be reached from the LPC

7K(active)> debug log-card-interface ping slot s7 host <ip of syslog>
64 bytes from 10.212.64.149: icmp_seq=1 ttl=253 time=0.381 ms
64 bytes from 10.212.64.149: icmp_seq=2 ttl=253 time=0.127 ms

• Verify LPC is receiving logs

7K(active)> debug log-card-interface info slot s7 | match packet
packets received                  9122564
packets transmitted               17677564751 

7K(active)> debug log-card-interface info slot s7 | match packet
packets received                  9122565
packets transmitted               17677565619  <<incrementing

• Verify Log is being forward out the Log Forwarding interface. 

7K(active)> debug log-receiver statistics | match syslog
External Forwarding stats:
     Type  Enqueue Count     Send Count     Drop Count    Queue Depth     Send Rate(last 1min)
    syslog    17673493691    17673493691         

7K(active)> debug log-receiver statistics | match syslog
    syslog    17673495661    17673495661      <<incrementing

• Verify the Log Forwarding interface is forwarding traffic. In this example, it is ethernet 1/11

7K(active)> show interface ethernet1/11 | match tx-bytes
tx-bytes                      443080192

7K(active)> show interface ethernet1/11 | match tx-bytes
tx-bytes                      443080256     <<incrementing

• Check global counter  to see if there is any throttle which indicate logging queue is full

log_traffic_loss_queue_full 570139124 0 info log resource Number of traffic logs that are lost due to next queue is full
log_traffic_loss_cnt 2514786839 373 info log resource Number of traffic logs that are lost

• Verify lpmgrd is running.

7K(active)> show system software status | match lp
Slot 7, Role lp
Process  lpmgrd       running  (pid: 1682)

7K(active)> debug software restart process log-receiver

• Check if VLAN is configured correctly on Firewall and remote device.

7K(active)> show interface ethernet1/11.22

--------------------------------------------------------------------------------
Name: ethernet1/11.22, ID: 1354, 802.1q tag: 22
Operation mode: log-card-forward
Interface management profile: N/A
Service configured:
Zone: N/A, virtual system: vsys1
Adjust TCP MSS: no

Zone: N/A, virtual system: N/A

• Additional debug is to take a packet capture (PCAP) on the peer device directly connected to the interface that is configured for log forwarding.