How to Troubleshoot Using Counters via the CLI

How to Troubleshoot Using Counters via the CLI

92711
Created On 09/25/18 19:24 PM - Last Updated 08/05/19 20:11 PM


Resolution

Counters are a very useful set of indicators for the processes, packet flows and sessions on the PA firewall and can be used to troubleshoot various scenarios.

To troubleshoot dropped packets show counter global filter severity drop can be used. Repeating the command multiple times helps narrow down the drops.
 

> show counter global filter severity drop

Global counters:
Elapsed time since last sampling: 34.999 seconds

name                                   value     rate severity  category  aspect    description
--------------------------------------------------------------------------------

flow_rcv_err                              98        0 drop      flow      parse     Packets dropped: flow stage receive error
flow_rcv_dot1q_tag_err                     1        0 drop      flow      parse     Packets dropped: 802.1q tag not configured
flow_no_interface                        263        0 drop      flow      parse     Packets dropped: invalid interface
flow_ipv6_disabled                     30622        0 drop      flow      parse     Packets dropped: IPv6 disabled on interface
flow_policy_nat_land                    6732        0 drop      flow      session   Session setup: source NAT IP allocation result in LAND attack
flow_fwd_l3_mcast_drop                  2756        0 drop      flow      forward   Packets dropped: no route for IP multicast
flow_fwd_l3_ttl_zero                       4        0 drop      flow      forward   Packets dropped: IP TTL reaches zero
flow_fwd_l3_noroute                        5        0 drop      flow      forward   Packets dropped: no route
flow_fwd_l3_noarp                          1        0 drop      flow      forward   Packets dropped: no ARP
flow_action_reset                          1        0 drop      flow      pktproc   TCP clients reset via responding RST
flow_arp_rcv_err                         162        0 drop      flow      arp       ARP receive error
flow_host_decap_err                      412        0 drop      flow      mgmt      Packets dropped: encapsulation error to control plane
flow_host_service_deny                153865        0 drop      flow      mgmt      Device management session denied
flow_host_service_unknown               2762        0 drop      flow      mgmt      Session discarded: unknown application to control plane
flow_tunnel_encap_err                     33        0 drop      flow      tunnel    Packet dropped: tunnel encapsulation error
appid_lookup_invalid_flow                  1        0 drop      appid     pktproc   Packets dropped: invalid session state
proxy_offload_check_err                 1030        0 drop      proxy     pktproc   The number offload proxy setup check failed because of not SYN or no certificate
url_request_pkt_drop                     204        0 drop      url       pktproc   The number of packets get dropped because of waiting for url category request

--------------------------------------------------------------------------------
Total counters shown: 18
--------------------------------------------------------------------------------
Using the above command with delta option  allows viewing packets dropped since the last time the command was issued.
> show counter global filter delta yes severity drop

Global counters:
Elapsed time since last sampling: 55.446 seconds
name                                   value     rate severity  category  aspect    description
--------------------------------------------------------------------------------

flow_ipv6_disabled                         3        0 drop      flow      parse     Packets dropped: IPv6 disabled on interface
flow_fwd_l3_mcast_drop                     2        0 drop      flow      forward   Packets dropped: no route for IP multicast
flow_host_service_deny                    26        0 drop      flow      mgmt      Device management session denied
flow_host_service_unknown                  2        0 drop      flow      mgmt      Session discarded: unknown application to control plane

--------------------------------------------------------------------------------
Total counters shown: 4
--------------------------------------------------------------------------------
Apart from the severity drop, there are various other severities that this command can be used for based on the scenario. A few examples are: error, informational and warning.

To troubleshoot Management Server Statistics, use show counter management-server
The counters can be used to view management server statistics (number of logs written to trigger counters assigned to each management server process)
This command is useful when suspecting a hardware issue that would require RMA replacement.
 
> show counter management-server
Log action not taken            :          0
Logs dropped because not logging:          0
User information from AD read   :          2
Certificates information read   :          0

License information fetched from update server:          0
Sighash refcount                :          1
Tunnelhash refcount             :          1
URLcat refcount                 :          1
ip2loc refcount                 :          1

To view management interface statistics use show counter interface management command. This is used to assist in troubleshooting connectivity.
> show counter interface management
Interface: Management Interface
-------------------------------------------------------------------------------
Logical interface counters:
-------------------------------------------------------------------------------

bytes received                    505700037
bytes transmitted                 295080711
packets received                  772181
packets transmitted               874087
receive errors                    0
transmit errors                   0
receive packets dropped           0
transmit packets dropped          0
multicast packets received        0

-------------------------------------------------------------------------------

The same counter can be used to check data plane interface statistics as well. Use the command show counter interface <interface id>. Example below.
> show counter interface tunnel.51
Interface: tunnel.51
--------------------------------------------------------------------------------
Logical interface counters read from CPU:
--------------------------------------------------------------------------------
bytes received                           0
bytes transmitted                        0
packets received                         0
packets transmitted                      0
receive errors                           0
packets dropped                          0
packets dropped by flow state check      0
forwarding errors                        0
no route                                 0
arp not found                            0
neighbor not found                       0
neighbor info pending                    0
mac not found                            0
packets routed to different zone         0
land attacks                             0
ping-of-death attacks                    0
teardrop attacks                         0
ip spoof attacks                         0
mac spoof attacks                        0
ICMP fragment                            0
layer2 encapsulated packets              0
layer2 decapsulated packets              0

--------------------------------------------------------------------------------
Layer two troubleshooting can be dealt with in term of the irregularities in the ARP entries received by using the arp aspect of the global counter with the command show counter global filter aspect arp
> show counter global filter aspect arp

Global counters:
Elapsed time since last sampling: 8.330 seconds
name                                   value     rate severity  category  aspect    description
--------------------------------------------------------------------------------
flow_arp_pkt_rcv                       42685        0 info      flow      arp       ARP packets received
flow_arp_pkt_xmt                        1875        0 info      flow      arp       ARP packets transmitted
flow_arp_pkt_replied                    6995        0 info      flow      arp       ARP requests replied
flow_arp_pkt_learned                      17        0 info      flow      arp       ARP entry learned
flow_arp_rcv_gratuitous                  494        0 info      flow      arp       Gratuitous ARP packets received
flow_arp_rcv_err                         162        0 drop      flow      arp       ARP receive error
flow_arp_resolve_xmt                    1843        0 info      flow      arp       ARP resolution packets transmitted

--------------------------------------------------------------------------------

Total counters shown: 7
Various other counters are helpful when troubleshooting, here are a few examples
> show counter global name

  aho_alloc_lookup_failed              warn      failed to alloc regex lookup
  aho_fpga                             info      The total requests to FPGA for AHO
  aho_fpga_invalid_wqe                 error     when getting result from fpga, wqe index was not valid
  aho_fpga_ret_error                   error     Dropped results from FPGA caused by unexecpted type
  aho_fpga_ret_invalid_fid             error     Dropped results from FPGA caused by invalid flow id
  aho_fpga_ret_length_error            error     Dropped results from FPGA caused by short length
  aho_fpga_ret_multi_bufs              info      Aho fpga result with multiple buffers
  aho_fpga_ret_offset_error            error     Dropped results from FPGA caused by invalid offset
  aho_fpga_ret_wrong_size              error     Dropped results from FPGA caused by wrong packet size
  aho_fpga_state_verify_failed         info      when getting result from fpga, session's state was changed
  aho_fpga_unmatched_type              error     when getting result from fpga, type in session was not matched
  aho_fpga_unmatched_wqe               warn      when getting result from fpga, wqe was not matched in session
  aho_match_overflow                   info      number of aho matches overflow
  aho_sw                               info      The total usage of software for AHO
  aho_sw_fpga_fail                     warn      Usage of software AHO caused by failure for sending fpga request
  aho_sw_fpga_full                     info      Usage of software AHO caused by fpga requests threshold
  aho_sw_fpga_unavailable              warn      Usage of software AHO caused by fpga unavailable
  aho_too_many_matches                 info      too many signature matches within one packet
  aho_too_many_mid_res                 info      too many signature middle results within one packet
  appid_dfa_invalid_result             error     The invalid dfa result for appid
  appid_exceed_pkt_limit               warn      App. identification failed caused by limitation of total queued packe
  appid_exceed_queue_limit             warn      App. identification failed caused by limitation of session queued pac
  appid_exceed_queue_limit_post        warn      App. identification failed caused by limitation of session queued pac
  appid_fini_with_wqe_2_fpga           info      session ends with wqe in fpga
  appid_flow_state_fail                info      The session's state was changed
  appid_ident_by_cache                 info      Application identified by cache
  appid_ident_by_dport                 info      Application identified by L4 dport
  appid_ident_by_dport_first           info      Application identified by L4 dport first
  appid_ident_by_heuristics            info      Application identified by heuristics
  appid_ident_by_icmp                  info      Application identified by icmp type
  appid_ident_by_ip                    info      Application identified by ip protocol
  appid_ident_by_sport                 info      Application identified by L4 sport
  appid_ident_by_sport_first           info      Application identified by L4 sport first
  appid_ident_by_supernode             info      Application identified by supernode
  appid_lookup_invalid_flow            drop      Packets dropped: invalid session state
  appid_match_overflow                 info      The dfa matches overflow
  appid_no_policy                      error     App. identification failed because of no policy
  appid_override                       info      Application identified by override rule
  appid_proc                           info      The number of packets processed by Application identification
  appid_reset_sess_tcp_reass           error     reset sess failed at tcp reassembly
  appid_result_id_changed              info      The session's appid status was changed
  appid_result_no_policy               info      The session's policy was changed during appid proc
  appid_skip_terminal                  info      The dfa result is terminal
  appid_ssl_no_cert_no_reset           info      ssl sessions with unknown server certificate but no previous reset
  appid_stop_by_ager                   info      Application identification terminated by session ager
  appid_stop_by_ager_nopkts            info      Ager can't stop appid because no packets were queued
  appid_unknown_by_stop                info      The number of unknown applications because of being stopped


 


Attachments
Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClXOCA0&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Attachments
Choose Language