How to Troubleshoot Using Counters via the CLI
225216
Created On 09/25/18 19:24 PM - Last Modified 08/05/19 20:11 PM
Resolution
Counters are a very useful set of indicators for the processes, packet flows and sessions on the PA firewall and can be used to troubleshoot various scenarios.
To troubleshoot dropped packets show counter global filter severity drop can be used. Repeating the command multiple times helps narrow down the drops.
> show counter global filter severity drop
Global counters:
Elapsed time since last sampling: 34.999 seconds
name value rate severity category aspect description
--------------------------------------------------------------------------------
flow_rcv_err 98 0 drop flow parse Packets dropped: flow stage receive error
flow_rcv_dot1q_tag_err 1 0 drop flow parse Packets dropped: 802.1q tag not configured
flow_no_interface 263 0 drop flow parse Packets dropped: invalid interface
flow_ipv6_disabled 30622 0 drop flow parse Packets dropped: IPv6 disabled on interface
flow_policy_nat_land 6732 0 drop flow session Session setup: source NAT IP allocation result in LAND attack
flow_fwd_l3_mcast_drop 2756 0 drop flow forward Packets dropped: no route for IP multicast
flow_fwd_l3_ttl_zero 4 0 drop flow forward Packets dropped: IP TTL reaches zero
flow_fwd_l3_noroute 5 0 drop flow forward Packets dropped: no route
flow_fwd_l3_noarp 1 0 drop flow forward Packets dropped: no ARP
flow_action_reset 1 0 drop flow pktproc TCP clients reset via responding RST
flow_arp_rcv_err 162 0 drop flow arp ARP receive error
flow_host_decap_err 412 0 drop flow mgmt Packets dropped: encapsulation error to control plane
flow_host_service_deny 153865 0 drop flow mgmt Device management session denied
flow_host_service_unknown 2762 0 drop flow mgmt Session discarded: unknown application to control plane
flow_tunnel_encap_err 33 0 drop flow tunnel Packet dropped: tunnel encapsulation error
appid_lookup_invalid_flow 1 0 drop appid pktproc Packets dropped: invalid session state
proxy_offload_check_err 1030 0 drop proxy pktproc The number offload proxy setup check failed because of not SYN or no certificate
url_request_pkt_drop 204 0 drop url pktproc The number of packets get dropped because of waiting for url category request
--------------------------------------------------------------------------------
Total counters shown: 18
--------------------------------------------------------------------------------
Using the above command with delta option allows viewing packets dropped since the last time the command was issued.
> show counter global filter delta yes severity drop
Global counters:
Elapsed time since last sampling: 55.446 seconds
name value rate severity category aspect description
--------------------------------------------------------------------------------
flow_ipv6_disabled 3 0 drop flow parse Packets dropped: IPv6 disabled on interface
flow_fwd_l3_mcast_drop 2 0 drop flow forward Packets dropped: no route for IP multicast
flow_host_service_deny 26 0 drop flow mgmt Device management session denied
flow_host_service_unknown 2 0 drop flow mgmt Session discarded: unknown application to control plane
--------------------------------------------------------------------------------
Total counters shown: 4
--------------------------------------------------------------------------------
Apart from the severity drop, there are various other severities that this command can be used for based on the scenario. A few examples are: error, informational and warning.To troubleshoot Management Server Statistics, use show counter management-server.
The counters can be used to view management server statistics (number of logs written to trigger counters assigned to each management server process)
This command is useful when suspecting a hardware issue that would require RMA replacement.
> show counter management-server
Log action not taken : 0
Logs dropped because not logging: 0
User information from AD read : 2
Certificates information read : 0
License information fetched from update server: 0
Sighash refcount : 1
Tunnelhash refcount : 1
URLcat refcount : 1
ip2loc refcount : 1
To view management interface statistics use show counter interface management command. This is used to assist in troubleshooting connectivity.
> show counter interface management
Interface: Management Interface
-------------------------------------------------------------------------------
Logical interface counters:
-------------------------------------------------------------------------------
bytes received 505700037
bytes transmitted 295080711
packets received 772181
packets transmitted 874087
receive errors 0
transmit errors 0
receive packets dropped 0
transmit packets dropped 0
multicast packets received 0
-------------------------------------------------------------------------------
The same counter can be used to check data plane interface statistics as well. Use the command show counter interface <interface id>. Example below.
> show counter interface tunnel.51
Interface: tunnel.51
--------------------------------------------------------------------------------
Logical interface counters read from CPU:
--------------------------------------------------------------------------------
bytes received 0
bytes transmitted 0
packets received 0
packets transmitted 0
receive errors 0
packets dropped 0
packets dropped by flow state check 0
forwarding errors 0
no route 0
arp not found 0
neighbor not found 0
neighbor info pending 0
mac not found 0
packets routed to different zone 0
land attacks 0
ping-of-death attacks 0
teardrop attacks 0
ip spoof attacks 0
mac spoof attacks 0
ICMP fragment 0
layer2 encapsulated packets 0
layer2 decapsulated packets 0
--------------------------------------------------------------------------------
Layer two troubleshooting can be dealt with in term of the irregularities in the ARP entries received by using the arp aspect of the global counter with the command show counter global filter aspect arp
> show counter global filter aspect arp
Global counters:
Elapsed time since last sampling: 8.330 seconds
name value rate severity category aspect description
--------------------------------------------------------------------------------
flow_arp_pkt_rcv 42685 0 info flow arp ARP packets received
flow_arp_pkt_xmt 1875 0 info flow arp ARP packets transmitted
flow_arp_pkt_replied 6995 0 info flow arp ARP requests replied
flow_arp_pkt_learned 17 0 info flow arp ARP entry learned
flow_arp_rcv_gratuitous 494 0 info flow arp Gratuitous ARP packets received
flow_arp_rcv_err 162 0 drop flow arp ARP receive error
flow_arp_resolve_xmt 1843 0 info flow arp ARP resolution packets transmitted
--------------------------------------------------------------------------------
Total counters shown: 7
Various other counters are helpful when troubleshooting, here are a few examples
> show counter global name
aho_alloc_lookup_failed warn failed to alloc regex lookup
aho_fpga info The total requests to FPGA for AHO
aho_fpga_invalid_wqe error when getting result from fpga, wqe index was not valid
aho_fpga_ret_error error Dropped results from FPGA caused by unexecpted type
aho_fpga_ret_invalid_fid error Dropped results from FPGA caused by invalid flow id
aho_fpga_ret_length_error error Dropped results from FPGA caused by short length
aho_fpga_ret_multi_bufs info Aho fpga result with multiple buffers
aho_fpga_ret_offset_error error Dropped results from FPGA caused by invalid offset
aho_fpga_ret_wrong_size error Dropped results from FPGA caused by wrong packet size
aho_fpga_state_verify_failed info when getting result from fpga, session's state was changed
aho_fpga_unmatched_type error when getting result from fpga, type in session was not matched
aho_fpga_unmatched_wqe warn when getting result from fpga, wqe was not matched in session
aho_match_overflow info number of aho matches overflow
aho_sw info The total usage of software for AHO
aho_sw_fpga_fail warn Usage of software AHO caused by failure for sending fpga request
aho_sw_fpga_full info Usage of software AHO caused by fpga requests threshold
aho_sw_fpga_unavailable warn Usage of software AHO caused by fpga unavailable
aho_too_many_matches info too many signature matches within one packet
aho_too_many_mid_res info too many signature middle results within one packet
appid_dfa_invalid_result error The invalid dfa result for appid
appid_exceed_pkt_limit warn App. identification failed caused by limitation of total queued packe
appid_exceed_queue_limit warn App. identification failed caused by limitation of session queued pac
appid_exceed_queue_limit_post warn App. identification failed caused by limitation of session queued pac
appid_fini_with_wqe_2_fpga info session ends with wqe in fpga
appid_flow_state_fail info The session's state was changed
appid_ident_by_cache info Application identified by cache
appid_ident_by_dport info Application identified by L4 dport
appid_ident_by_dport_first info Application identified by L4 dport first
appid_ident_by_heuristics info Application identified by heuristics
appid_ident_by_icmp info Application identified by icmp type
appid_ident_by_ip info Application identified by ip protocol
appid_ident_by_sport info Application identified by L4 sport
appid_ident_by_sport_first info Application identified by L4 sport first
appid_ident_by_supernode info Application identified by supernode
appid_lookup_invalid_flow drop Packets dropped: invalid session state
appid_match_overflow info The dfa matches overflow
appid_no_policy error App. identification failed because of no policy
appid_override info Application identified by override rule
appid_proc info The number of packets processed by Application identification
appid_reset_sess_tcp_reass error reset sess failed at tcp reassembly
appid_result_id_changed info The session's appid status was changed
appid_result_no_policy info The session's policy was changed during appid proc
appid_skip_terminal info The dfa result is terminal
appid_ssl_no_cert_no_reset info ssl sessions with unknown server certificate but no previous reset
appid_stop_by_ager info Application identification terminated by session ager
appid_stop_by_ager_nopkts info Ager can't stop appid because no packets were queued
appid_unknown_by_stop info The number of unknown applications because of being stopped