Getting Started: Packet Capture

Getting Started: Packet Capture

1054461
Created On 09/25/18 19:02 PM - Last Modified 01/18/24 15:51 PM


Symptom


A typical situation an administrator encounters minutes after finishing configuring a security policy is a question about why some obscure application is acting funny ever since that new network thingamajig was installed. This is where troubleshooting begins.

In the past installments, we checked out several aspects of how to configure your firewall and set it up from scratch.

One of the more advanced tools at the disposal of an admin is the ability to perform packet captures and look at global counters.

Note: Some of the details discussed in the article will cause performance impact. If you are unsure at any step, please work with the Palo Alto Networks TAC team to capture the packets during a maintenance window.



Environment


  • Any PAN-OS.
  • Palo Alto Networks Firewall.


Resolution


The first place to go is the Packet Capture menu on the GUI, where you can manage filters, add capture stages, and easily download captures.

Navigate to packet capture by going to the Monitor tab and then clicking on Packet Capture. Make sure the Virtual System is set to all

Within the Packet Capture menu settings, you can perform the following options: Configure Filtering, Configure Capturing, Clear all Settings, and extract the Capture Files

Before we get started, there are a few things you should know:

  • Four packet filters can be added with a variety of attributes.
  • Packet captures are session-based, so a single filter is capable of capturing both client2server and server2client.
  • Packets are captured on the dataplane vs on the interface (this explains the next bullet).
  • Pre-Parse Match is a feature that can capture all files before they are processed by the engines running on the dataplane, which can help troubleshoot issues where an engine may not be properly accepting an inbound packet. This option should be used only if instructed by the support and on a low volume time of day as it will capture everything.
  • When filtering is enabled, new sessions are marked for filtering and can be captured, but existing sessions are not being filtered and may need to be restarted to be able to capture them.
  • Offloaded sessions can't be captured so offloading may need to be disabled temporarily. An offloaded session will display 'layer7 processing: completed' in the show session details. Note that this may cause performance impact due to the matching packets being handled by the CPU. Run this during a maintenance window and take the help of Support if required.

It's a lot to remember, but it will all make sense after you've tried your hand at a couple of packet captures of various protocols, I promise!

 

  1. Let's add a couple of packet filters:

Example of a user editing the Packet Capture filters.

 

Filters 1 and 3 are my actual filters: I want to check connections from my client at IP 192.168.0.34 making HTTP connections on port 80 TCP to 198.51.100.97 and SSH connections on port 22 TCP to 198.51.100.1

 

Filters 2 and 4 are my 'backup' filters: I mentioned earlier that packet capture is session aware, but just in case something happens to the returning packet upstream that causes it to fail to match my NAT rule (maybe an upstream device mangles the source port or does something odd to the sequence number), I usually set a returning rule to catch any stray packets that would get dropped due to their not matching.

Note: Ensure the filter is as precise as possible. Never run filters matching the entire subnets such as 192.168.0.0/16 or 0.0.0.0/0 as this may cause performance impact and outage.

 

If we now switch the Filtering button to ON, the filters will be applied to any new sessions that match the criteria:

Under Configure Filtering, the Filtering is now switched to the ON position

 

  1. A simple way to check if the filter is working is to check if global counters are increasing if a new session is initiated.

From the CLI, I execute this command:

 

> show counter global filter delta yes packet-filter yes

Global counters:
Elapsed time since last sampling: 2.647 seconds

--------------------------------------------------------------------------------
Total counters shown: 0
--------------------------------------------------------------------------------
 

And no counters are displayed.

  • 'delta yes' indicates I want to view counters that have incremented since the last time I executed this command.
  • 'packet-filter yes' indicates I want to see only global counters that match my filters.

If I now refresh my browser and execute the command again:

 

> show counter global filter packet-filter yes delta yes

Global counters:
Elapsed time since last sampling: 2.630 seconds

name value rate severity category aspect description
--------------------------------------------------------------------------------
pkt_sent                     22 8 info packet pktproc Packets transmitted
pkt_outstanding              22 8 info packet pktproc Outstanding packet to be transmitted
session_allocated            4  1 info session resource Sessions allocated
session_installed            4  1 info session resource Sessions installed
appid_proc                   2  0 info appid pktproc The number of packets processed by Application identification
appid_use_dfa_1              2  0 info appid pktproc The number of packets using the second DFA table
nat_dynamic_port_xlat        4  1 info nat resource The total number of dynamic_ip_port NAT translate called
dfa_sw                       8  3 info dfa pktproc The total number of dfa match using software
ctd_sml_opcode_set_file_type 1  0 info ctd pktproc sml opcode set file type
aho_sw                       5  1 info aho pktproc The total usage of software for AHO
ctd_pkt_slowpath             4  1 info ctd pktproc Packets processed by slowpath
--------------------------------------------------------------------------------
Total counters shown: 11
--------------------------------------------------------------------------------
 

You'll see that there are a bunch of counters. I've added a link to some more information about global counters at the bottom of this article, and we'll be covering them more in a future episode. But for now, as long as there are counters, we should be good to go.

 

  1. Next you're going to configure the stages—there are 4:
  • drop stage is where packets get discarded. The reasons may vary and, for this part, the global counters may help identify if the drop was due to a policy deny, a detected threat, or something else.
  • receive stage captures the packets as they ingress the firewall before they go into the firewall engine. When NAT is configured, these packets will be pre-NAT.
  • transmit stage captures packets how they egress out of the firewall engine. If NAT is configured, these will be post-NAT.
  • firewall stage captures packets in the firewall stage.

 

For every stage, you can assign a name for the output file and set a maximum packet or byte count:

Packet Capture Stage with File name set to drop.pcap

 

Configure Capturing, 3 files are generated (receive, transmit, drop)

 

  1. When all the desired stages are set, you can switch the capture button to ON, or you can use the CLI, clear the existing sessions which match the filters specified. This is to make sure no session has been active since before the filters were enabled. Then use the capture on command to start the capture as displayed below.
> show session all        => Note down the session number matching the configured filters.
> clear session id <id#>  => This is to clear any existing session that matches the filters configured.

Example:
 > show session all
-------------------------------------------------------------------------------- 
ID          Application    State   Type Flag  Src[Sport]/Zone/Proto (translated IP[Port]) 
Vsys                                          Dst[Dport]/Zone (translated IP[Port]) 
-------------------------------------------------------------------------------- 
27240        web-browsing   ACTIVE  FLOW  NS   192.168.0.34[61003]/trust/6  (198.51.100.230[31047]) 
vsys1                                          198.51.100.97[80]/ISP1  (198.51.100.97[80]) 
....(output omitted)....

> clear session id 27240
session 27240 cleared

> debug dataplane packet-diag set capture on 

Packet capture is enabled
 

 

You can now launch the sessions you'd like to capture. To verify if the session has started, use the show session command:

 

> show session all

--------------------------------------------------------------------------------
ID          Application    State   Type Flag  Src[Sport]/Zone/Proto (translated IP[Port])
Vsys                                          Dst[Dport]/Zone (translated IP[Port])
--------------------------------------------------------------------------------
32637        web-browsing   ACTIVE  FLOW  NS   192.168.0.34[61903]/trust/6  (198.51.100.230[31547])
vsys1                                          198.51.100.97[80]/ISP1  (198.51.100.97[80])
32635        ssh            ACTIVE  FLOW  NS   192.168.0.34[61901]/trust/6  (198.51.100.230[52812])
vsys1                                          198.51.100.1[22]/ISP1  (198.51.100.1[22])
 

When you're done, the capture can be turned off by toggling the button back to the OFF position or using the debug command:

 

> debug dataplane packet-diag set capture off 
Packet capture is disabled 

> debug dataplane packet-diag clear filter-marked-session all
Unmark All sessions in packet debug
  Note: Do not forget to turn off the packet capture after debugging. Failing to do so may tax the CPU and can cause performance problems.

There will now be captured files available for download which you can analyze with Wireshark:

Configure Capturing, 3 files are generated (receive.pcap and transmit.pcap)

 

After you download the packet captures, you may need to merge the transmit and receive files together:

  • The receive stage will have the client's private IP to the web server public IP, and the returning packet from the web server public IP  to the firewall external IP (receive stage is pre-NAT).
  • The transmit stage will have the firewall external IP (client NAT) to the server public IP, and the returning public IP to the client private IP.

As that may be a little confusing when you're trying to follow the TCP stream, you'll want to take note of this important difference between those two files.

Receive.pcap packet capture opened in WireShark denoting the Source and Destination IP addresses.

Transmit.pcap packet capture opened in WireShark denoting the Source and Destination IP addresses.

 

These two stages will ensure you are able to verify if NAT is being applied properly. You can also see if there is any difference between the packets sent out and received from both the client and server perspectives.

 

The merged result should look something like this and allows you to compare packet-per-packet what is being sent out and what is being received.

The receive.pcap and transmit.pcap are megered together within the WireShark view

 

A couple of useful articles to help you better understand how packets flow through the system:

What is the Significance of Global Counters?

Packet Flow Sequence in PAN-OS

 

 

Management plane

 

To Troubleshoot connectivity issues with the management plane, the built-in tcpdump command can be used to capture useful information:

admin@myNGFW> tcpdump filter "port 53"
Press Ctrl-C to stop capturing

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
^C
16 packets captured
32 packets received by filter
0 packets dropped by kernel
The resulting output is stored in a mgmt.pcap file on the management plane: 

admin@myNGFW> view-pcap mgmt-pcap mgmt.pcap
15:24:07.512889 IP 10.0.0.1.36606 > 10.0.0.98.domain:  49243+ PTR? 0.0.0.10.in-addr.arpa. (41)
15:24:07.513106 IP 10.0.0.98.domain > 10.0.0.1.36606:  49243 NXDomain 0/1/0 (100)
15:24:17.196515 IP 10.0.0.1.45455 > 10.0.0.98.domain:  3126+ PTR? 0.0.0.10.in-addr.arpa. (41)
15:24:17.196762 IP 10.0.0.98.domain > 10.0.0.1.45455:  3126 NXDomain 0/1/0 (100)
15:24:31.126186 IP 10.0.0.1.57230 > 10.0.0.98.domain:  2978+ A? panos.wildfire.paloaltonetworks.com. (53)
15:24:31.126196 IP 10.0.0.1.57230 > 10.0.0.98.domain:  55173+ AAAA? panos.wildfire.paloaltonetworks.com. (53)
15:24:31.161436 IP 10.0.0.98.domain > 10.0.0.1.57230:  2978 4/0/0[|domain]
15:24:31.194586 IP 10.0.0.98.domain > 10.0.0.1.57230:  55173 2/0/0[|domain]
15:25:56.746679 IP 10.0.0.1.46048 > 10.0.0.98.domain:  43532+ A? wildfire.paloaltonetworks.com. (47)
15:25:56.746689 IP 10.0.0.1.46048 > 10.0.0.98.domain:  16653+ AAAA? wildfire.paloaltonetworks.com. (47)
15:25:56.794940 IP 10.0.0.98.domain > 10.0.0.1.46048:  43532 1/0/0 (63)
15:25:56.795553 IP 10.0.0.98.domain > 10.0.0.1.46048:  16653 0/1/0 (131)
15:25:59.038459 IP 10.0.0.1.51804 > 10.0.0.98.domain:  23806+ A? panos.wildfire.paloaltonetworks.com. (53)
15:25:59.038471 IP 10.0.0.1.51804 > 10.0.0.98.domain:  31471+ AAAA? panos.wildfire.paloaltonetworks.com. (53)
15:25:59.071431 IP 10.0.0.98.domain > 10.0.0.1.51804:  23806 4/0/0[|domain]
15:25:59.113552 IP 10.0.0.98.domain > 10.0.0.1.51804:  31471 2/0/0[|domain]
 

Please check out this article for additional information How To Packet Capture (tcpdump) On Management Interface

 


 



Additional Information


1. To control the packet capture file size, a single file is limited to 200mb and a second file is automatically created once the size is exceeded, both files will then act as a ring buffer where the primary pcap file is used to write active capture data and the *.pcap.1 file is used as a buffer. Once the primary pcap reaches it's capacity again, the buffer file is discarded, the primary file is renamed to *.pcap.1 and a fresh primary file is started. More information here: What is The Limitation of the Packet Capture File Size on PAN-OS?

2. For small captures, it could be handy to capture everything into a single file, so it is possible to have every stage capture to the same filename. Please be aware of the abovementioned limitation in size, and the decreased visibility for each stage. The recommended method is to always have an individual file per stage.

Since PAN-OS 8.1.0, filters can be added for source and network subnets this is available only via the CLI and NOT WebGUI:

admin@firewall> debug dataplane packet-diag set filter match 
+ destination           Destination IP address
+ destination-netmask   Destination netmask              <<<< new option for network
+ destination-port      Destination port
+ ingress-interface     Ingress traffic interface name
+ ipv6-only             IPv6 packet only
+ lacp                  LACP packet
+ non-ip                Non-IP packet
+ protocol              IP protocol value
+ source                Source IP address
+ source-netmask        Source netmask                     <<<< new option for network
+ source-port           Source port
  |                     Pipe through a command
  <Enter>               Finish input
 
admin@firewall> debug dataplane packet-diag set filter match source 192.168.1.0 source-netmask 24

admin@firewall> debug dataplane packet-diag set filter match destination 192.168.2.0 destination-netmask 24

admin@firewall> debug dataplane packet-diag show setting 

DP dp0:

--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
  Enabled:                   no
  Match pre-parsed packet:   no            
  Index 1: 192.168.1.0/24[0]->0.0.0.0/0[0], proto 0 <<<<<<<<< network is captured /24
           ingress-interface any, egress-interface any, exclude non-IP
  Index 2: 0.0.0.0/0[0]->192.168.2.0/24[0], proto 0 <<<<<<<<< network is captured /24
           ingress-interface any, egress-interface any, exclude non-IP
--------------------------------------------------------------------------------
 

Note: Exercise caution when enabling network-wide captures, there could be a lot of packets resulting in high dataplane CPU consumption and possible traffic impact.



Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClTJCA0&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Choose Language