Palo Alto Networks Knowledgebase: Getting Started: Flow Basic

Getting Started: Flow Basic

23504
Created On 02/07/19 23:51 PM - Last Updated 02/07/19 23:52 PM
Resolution

Let me fix that for you: Flow Basic—

 

In the previous episode, we leveraged debug filters to allow the Palo Alto Networks firewall to collect packet captures  we could use for troubleshooting. But sometimes, you may need to look deeper into what's going on inside the firewall. 

 

Flow basic is the equivalent of a packet capture on every stage inside the firewall process, from receiving the packet to making security decisions, applying NAT, App-ID and so on, which makes it a very powerful tool. 

 

Wield this power with due care as the process can be CPU intensive if your filters are set up broadly or lots of traffic needs to be captured. 

 

Before you get started, make sure the dataplane is not overloaded:

 

 

> show running resource-monitor minute 

Resource monitoring sampling data (per minute):

CPU load (%) during last 60 minutes:
core    0       1       2       3       4       5   
     avg max avg max avg max avg max avg max avg max
       0   0   0   1   0   0   0   0   0   0   0   0
       0   0   0   1   0   0   0   0   0   0   0   0
       0   0   0   0   0   0   0   0   0   0   0   0
       0   0   0   0   0   0   0   0   0   0   0   0

 

The first thing we need to do is set up filters. Last time I showed you how to do this from the GUI—this time, let's take a look at the CLI:

 

First we're going to verify that nothing's been configured yet that could interfere with our new settings:

 

> debug dataplane packet-diag show setting 

--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
  Enabled:                   no
  Match pre-parsed packet:   no            
--------------------------------------------------------------------------------
Logging
  Enabled:                   no
  Log-throttle:              no
  Sync-log-by-ticks:         yes            
  Features:
  Counters:
--------------------------------------------------------------------------------
Packet capture
  Enabled:                   no
  Snaplen:                   0            
--------------------------------------------------------------------------------

 

If anything's still configured, we can clear out all filters and previous flow basic logs using these commands:

 

> debug dataplane packet-diag clear all
> debug dataplane packet-diag clear log log

 

We can now go ahead and create and enable the filters, making sure pre-parse is disabled. A second filter from the server to the NAT IP on the external interface of the firewall will help capture returning packets before they are NAT'ed in the 'ingress stage.' More about that below:

 

> debug dataplane packet-diag set filter match source 192.168.0.34 destination 198.51.100.97 destination-port 80 protocol 6 non-ip exclude 
> debug dataplane packet-diag set filter match source 198.51.100.97 destination 198.51.100.230 source-port 80 protocol 6 non-ip exclude 
> debug dataplane packet-diag set filter on

> debug dataplane packet-diag show setting -------------------------------------------------------------------------------- Packet diagnosis setting: -------------------------------------------------------------------------------- Packet filter Enabled: yes Match pre-parsed packet: no Index 1: 192.168.0.34[0]->198.51.100.97[80], proto 6 ingress-interface any, egress-interface any, exclude non-IP Index 2: 198.51.100.97[80]->198.51.100.230[0], proto 6 ingress-interface any, egress-interface any, exclude non-IP -------------------------------------------------------------------------------- Logging Enabled: no Log-throttle: no Sync-log-by-ticks: yes Features: Counters: -------------------------------------------------------------------------------- Packet capture Enabled: no Snaplen: 0 --------------------------------------------------------------------------------

 

When you're ready to enable logging, you'll see there are several features you can enable. Each one sets a capturing process on a specific engine or daemon that can help drill down even further. 'appid' can help troubleshoot why a certain app may not be getting identified in a flow and 'ctd' can help troubleshoot vulnerability signatures, and so on:

 

> debug dataplane packet-diag set log feature 
 > all        all 
 > appid      appid 
 > cfg        cfg 
 > ctd        ctd 
 > flow       flow 
 > misc       misc 
 > module     module 
 > pow        pow 
 > proxy      proxy 
 > ssl        ssl 
 > tcp        tcp 
 > tunnel     tunnel 
 > url_trie   url_trie 
 > zip        zip 

 

For now, we'll start with the 'flow' feature, which relates to all the base-level operations like inspecting TCP handshake, building sessions on the firewall, and performing NAT. In each feature, you can enable yet more subsections for even greater detail, but we'll stick to the basic setting for now.

 

> debug dataplane packet-diag set log feature flow 
  ager      ager 
  all       all 
  arp       arp 
  basic     basic 
  ha        ha 
  log       log 
  nd        nd 
  np        np 
  receive   receive 
  track     track 

> debug dataplane packet-diag set log feature flow basic 
> debug dataplane packet-diag show setting 

--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
  Enabled:                   yes
  Match pre-parsed packet:   no            
  Index 1: 192.168.0.34[0]->198.51.100.97[80], proto 6
           ingress-interface any, egress-interface any, exclude non-IP
  Index 2: 198.51.100.97[80]->198.51.100.230[0], proto 6
           ingress-interface any, egress-interface any, exclude non-IP
--------------------------------------------------------------------------------
Logging
  Enabled:                   no
  Log-throttle:              no
  Sync-log-by-ticks:         yes            
  Features:
    flow    : basic 
  Counters:
--------------------------------------------------------------------------------
Packet capture
  Enabled:                   no
  Snaplen:                   0            
--------------------------------------------------------------------------------

 

When you're ready to initiate traffic make sure any existing sessions have been terminated, then disable session offloading to ensure all packets are captured even if the session would normally be offloaded into hardware and finally go ahead and enable the logging feature.

 

> show session all filter source 192.168.0.34 destination 198.51.100.97

No Active Sessions

If there are still active sessions you can clear them by using the clear session command:

> clear session all filter source 192.168.0.34 destination 198.51.100.97

 

> set session offload no

> debug dataplane packet-diag set log on

 

You can now go ahead and start the session you want to capture, wait for it to gracefully end, then disable logging:

 

> show session all filter source 192.168.0.34 destination 198.51.100.97

--------------------------------------------------------------------------------
ID          Application    State   Type Flag  Src[Sport]/Zone/Proto (translated IP[Port])
Vsys                                          Dst[Dport]/Zone (translated IP[Port])
--------------------------------------------------------------------------------
51187        web-browsing   ACTIVE  FLOW  NS   192.168.0.34[64969]/trust/6  (198.51.100.230[42882])
vsys1                                          198.51.100.97[80]/ISP1  (198.51.100.97[80])
> show session id 51187

Session           51187

        c2s flow:
                source:      192.168.0.34 [trust]
                dst:         198.51.100.97
                proto:       6
                sport:       64969           dport:      80
                state:       INIT            type:       FLOW
                src user:    pantac\tpiens
                dst user:    unknown

        s2c flow:
                source:      198.51.100.97 [ISP1]
                dst:         198.51.100.230
                proto:       6
                sport:       80              dport:      42882
                state:       INIT            type:       FLOW
                src user:    unknown
                dst user:    pantac\tpiens

        start time                           : Wed Feb 10 13:43:27 2016
        timeout                              : 15 sec
        total byte count(c2s)                : 644
        total byte count(s2c)                : 308
        layer7 packet count(c2s)             : 5
        layer7 packet count(s2c)             : 2
        vsys                                 : vsys1
        application                          : web-browsing  
        rule                                 : web-out
        session to be logged at end          : False
        session in session ager              : False
        session updated by HA peer           : False
        address/port translation             : source
        nat-rule                             : outbound-nat(vsys1)
        layer7 processing                    : enabled
        URL filtering enabled                : False
        session via syn-cookies              : False
        session terminated on host           : False
        session traverses tunnel             : False
        captive portal session               : False
        ingress interface                    : ethernet1/3
        egress interface                     : ethernet1/1
        session QoS rule                     : N/A (class 4)
        tracker stage firewall               : TCP RST - client
        end-reason                           : tcp-rst-from-client
> show session all filter source 192.168.0.34 destination 198.51.100.97

No Active Sessions
> debug dataplane packet-diag set log off

Packet log is disabled

> set session offload yes

 

Each dataplane CPU will generate its own flow log, so depending on the amount of traffic, the type and amount of sessions, there may be several files located on the dataplane. Each CPU that participated in the capture will have a pan_task_X.log entry:

 

> less dp-log 
bfd.log             brdagent.log        dp-monitor.log      dp-monitor.log.1    
dp-monitor.log.2    dp-monitor.log.3    dp-monitor.log.4    masterd.log         
masterd_apps.log    masterd_detail.log  mprelay.log         pan_comm_0.log      
pan_dha.log         pan_task_1.log      pan_task_10.log     pan_task_11.log     
pan_task_2.log      pan_task_3.log      pan_task_4.log      pan_task_5.log      
pan_task_6.log      pan_task_7.log      pan_task_8.log      pan_task_9.log      
panio.log           panio.log.old       pdtrc.log           supervisor.log      
sysdagent.log       

 

A nifty little tool is provided to aggregate these files into a single file:

 

> debug dataplane packet-diag aggregate-logs 

packet-diag.log is aggregated

 

The final output file is then stored on the management plane as pan_packet_diag.log:

 

> less dp-log pan_packet_diag.log for platforms with separate/multiple dataplanes
> less mp-log pan_packet_diag.log for vm and platforms with integrated dataplane

Let's take a look at the stages a packet goes through as it is seen in flow basic:

 

The packet is received on the ingress interface and checked to see if it matches an existing session. If not, it is sent to 'slowpath' for session creation.

 

== 2016-02-10 14:53:09.977 -0800 ==
Packet received at ingress stage
Packet info: len 66 port 18 interface 18 vsys 1
  wqe index 208735 packet 0x0x80000000b4d600c6
Packet decoded dump:
L2:     00:0c:29:1e:9c:8c->b4:0c:25:ed:37:12, type 0x0800
IP:     192.168.0.34->198.51.100.97, protocol 6
        version 4, ihl 5, tos 0x00, len 52,
        id 93, frag_off 0x4000, ttl 128, checksum 3848
TCP:    sport 64984, dport 80, seq 2270663711, ack 0,
        reserved 0, offset 8, window 8192, checksum 18967,
        flags 0x0002 ( SYN), urgent data 0
TCP option:
00000000: 02 04 05 b4 01 03 03 02  01 01 04 02                ........ ....
Flow lookup, key word0 0xfdd8005000010600 word1 0
Session setup: vsys 1
No active flow found, enqueue to create session
           

 

 

Next, slowpath receives the packet. In slowpath, the packet is checked for source and destination zone based on routes or PBF entries. The packet's also checked to see if security rules exist that allow this session, based on the 5 tuples (source zone, source IP subnet, destination zone, destination IP subnet, destination port), and if NAT needs to be applied. If everything checks out, a session is created.

 

== 2016-02-10 14:53:09.978 -0800 ==
Packet received at slowpath stage
Packet info: len 66 port 18 interface 18 vsys 1
  wqe index 208735 packet 0x0x80000000b4d600c6
Packet decoded dump:
L2:     00:0c:29:1e:9c:8c->b4:0c:25:ed:37:12, type 0x0800
IP:     192.168.0.34->198.51.100.97, protocol 6
        version 4, ihl 5, tos 0x00, len 52,
        id 93, frag_off 0x4000, ttl 128, checksum 3848
TCP:    sport 64984, dport 80, seq 2270663711, ack 0,
        reserved 0, offset 8, window 8192, checksum 18967,
        flags 0x0002 ( SYN), urgent data 0
TCP option:
00000000: 02 04 05 b4 01 03 03 02  01 01 04 02                ........ ....
Session setup: vsys 1
PBF lookup (vsys 1) with application web-browsing
Session setup: ingress interface ethernet1/3 egress interface ethernet1/1 (zone 5)
NAT policy lookup, matched rule index 0
Policy lookup, matched rule index 0, 
Allocated new session 51187.
Packet matched vsys 1 NAT rule 'outbound-nat' (index 1),
source translation 192.168.0.34/64984 => 198.51.100.230/52924
Created session, enqueue to install

 

The packet is forwarded to fastpath, NAT translation is applied, and the translated packet is sent out of the egress interface to the next hop.

== 2016-02-10 14:53:09.978 -0800 ==
Packet received at fastpath stage
Packet info: len 66 port 18 interface 18 vsys 1
  wqe index 208735 packet 0x0x80000000b4d600c6
Packet decoded dump:
L2:     00:0c:29:1e:9c:8c->b4:0c:25:ed:37:12, type 0x0800
IP:     192.168.0.34->198.51.100.97, protocol 6
        version 4, ihl 5, tos 0x00, len 52,
        id 93, frag_off 0x4000, ttl 128, checksum 3848
TCP:    sport 64984, dport 80, seq 2270663711, ack 0,
        reserved 0, offset 8, window 8192, checksum 18967,
        flags 0x0002 ( SYN), urgent data 0
TCP option:
00000000: 02 04 05 b4 01 03 03 02  01 01 04 02                ........ ....
Flow fastpath, session 51187
2016-02-10 14:53:09.978 -0800  pan_flow_process_fastpath(src/pan_flow_proc.c:1414): SESSION-DSCP: set se
ssion DSCP: 0x00
NAT session, run address/port translation
Syn Cookie: pan_reass(Init statete): c2s:0 c2s:nxtseq 2270663712 c2s:startseq 2270663712 c2s:win 0 c2s:s
t 3 c2s:newsyn 0 :: s2c:nxtseq 0 s2c:startseq 0 s2c:win 8192 s2c:st 0 s2c:newsyn 0 ack 0 nosyn 0 plen 0
Forwarding lookup, ingress interface 18
L3 mode, virtual-router 1
Route lookup in virtual-router 1, IP 198.51.100.97
Route found, interface ethernet1/1, zone 5
Resolve ARP for IP 198.51.100.97 on interface ethernet1/1
ARP entry found on interface 16
Transmit packet on port 16


 

The returning SYN/ACK packet is received at the ingress stage and matched to the existing session, it is then forwarded to the fastpath stage. Reverse NAT is applied and the packet is sent out of the internal interface back to the client:

 

== 2016-02-10 14:53:09.978 -0800 ==
Packet received at ingress stage
Packet info: len 66 port 16 interface 16 vsys 1
  wqe index 206169 packet 0x0x80000000b72c28c6
Packet decoded dump:
L2:     00:0c:29:de:25:fa->b4:0c:25:ed:37:10, type 0x0800
IP:     198.51.100.97->198.51.100.230, protocol 6
        version 4, ihl 5, tos 0x00, len 52,
        id 10919, frag_off 0x4000, ttl 128, checksum 31342
TCP:    sport 80, dport 52924, seq 3191786586, ack 2270663712,
        reserved 0, offset 8, window 8192, checksum 33843,
        flags 0x0012 ( SYN ACK), urgent data 0
TCP option:
00000000: 02 04 05 b4 01 03 03 08  01 01 04 02                ........ ....
Flow lookup, key word0 0x50cebc00050600 word1 0
Flow 102375 found, state 2, HA 0
Active flow, enqueue to fastpath process


== 2016-02-10 14:53:09.979 -0800 ==
Packet received at fastpath stage
Packet info: len 66 port 16 interface 16 vsys 1
  wqe index 206169 packet 0x0x80000000b72c28c6
Packet decoded dump:
L2:     00:0c:29:de:25:fa->b4:0c:25:ed:37:10, type 0x0800
IP:     198.51.100.97->198.51.100.230, protocol 6
        version 4, ihl 5, tos 0x00, len 52,
        id 10919, frag_off 0x4000, ttl 128, checksum 31342
TCP:    sport 80, dport 52924, seq 3191786586, ack 2270663712,
        reserved 0, offset 8, window 8192, checksum 33843,
        flags 0x0012 ( SYN ACK), urgent data 0
TCP option:
00000000: 02 04 05 b4 01 03 03 08  01 01 04 02                ........ ....
Flow fastpath, session 51187
NAT session, run address/port translation
Syn Cookie: pan_reass(Init statete): c2s:1 c2s:nxtseq 2270663712 c2s:startseq 2270663712 c2s:win 8192 c2
s:st 3 c2s:newsyn 0 :: s2c:nxtseq 3191786587 s2c:startseq 3191786587 s2c:win 8192 s2c:st 3 s2c:newsyn 0 
ack 2270663712 nosyn 0 plen 0
Forwarding lookup, ingress interface 16
L3 mode, virtual-router 1
Route lookup in virtual-router 1, IP 192.168.0.34
Route found, interface ethernet1/3, zone 1
Resolve ARP for IP 192.168.0.34 on interface ethernet1/3
ARP entry found on interface 18
Transmit packet on port 18



 

The final ACK to complete the handshake is received, which triggers the session to be registered and the flow to be created in the fastpath stage. No further route or NAT lookups will need to be performed by the firewall.

== 2016-02-10 14:53:09.979 -0800 ==
Packet received at ingress stage
Packet info: len 60 port 18 interface 18 vsys 1
  wqe index 193163 packet 0x0x80000000b49c60c6
Packet decoded dump:
L2:     00:0c:29:1e:9c:8c->b4:0c:25:ed:37:12, type 0x0800
IP:     192.168.0.34->198.51.100.97, protocol 6
        version 4, ihl 5, tos 0x00, len 40,
        id 94, frag_off 0x4000, ttl 128, checksum 3859
TCP:    sport 64984, dport 80, seq 2270663712, ack 3191786587,
        reserved 0, offset 5, window 16425, checksum 57360,
        flags 0x0010 ( ACK), urgent data 0
TCP option:
Flow lookup, key word0 0xfdd8005000010600 word1 0
Flow 102374 found, state 2, HA 0
Active flow, enqueue to fastpath process


== 2016-02-10 14:53:09.979 -0800 ==
Packet received at fastpath stage
Packet info: len 60 port 18 interface 18 vsys 1
  wqe index 193163 packet 0x0x80000000b49c60c6
Packet decoded dump:
L2:     00:0c:29:1e:9c:8c->b4:0c:25:ed:37:12, type 0x0800
IP:     192.168.0.34->198.51.100.97, protocol 6
        version 4, ihl 5, tos 0x00, len 40,
        id 94, frag_off 0x4000, ttl 128, checksum 3859
TCP:    sport 64984, dport 80, seq 2270663712, ack 3191786587,
        reserved 0, offset 5, window 16425, checksum 57360,
        flags 0x0010 ( ACK), urgent data 0
TCP option:
Flow fastpath, session 51187
NAT session, run address/port translation
Forwarding lookup, ingress interface 18
L3 mode, virtual-router 1
Route lookup in virtual-router 1, IP 198.51.100.97
Route found, interface ethernet1/1, zone 5
Resolve ARP for IP 198.51.100.97 on interface ethernet1/1
ARP entry found on interface 16
Transmit packet on port 16


 

Client and server are now free to start communicating:

== 2016-02-10 14:53:09.980 -0800 ==
Packet received at ingress stage
Packet info: len 398 port 18 interface 18 vsys 1
  wqe index 202239 packet 0x0x80000000b341e8c6
Packet decoded dump:
L2:     00:0c:29:1e:9c:8c->b4:0c:25:ed:37:12, type 0x0800
IP:     192.168.0.34->198.51.100.97, protocol 6
        version 4, ihl 5, tos 0x00, len 384,
        id 95, frag_off 0x4000, ttl 128, checksum 3514
TCP:    sport 64984, dport 80, seq 2270663712, ack 3191786587,
        reserved 0, offset 5, window 16425, checksum 24524,
        flags 0x0018 ( ACK PSH), urgent data 0
TCP option:
Flow lookup, key word0 0xfdd8005000010600 word1 0
Flow 102374 found, state 2, HA 0
Active flow, enqueue to fastpath process


== 2016-02-10 14:53:09.980 -0800 ==
Packet received at fastpath stage
Packet info: len 398 port 18 interface 18 vsys 1
  wqe index 202239 packet 0x0x80000000b341e8c6
Packet decoded dump:
L2:     00:0c:29:1e:9c:8c->b4:0c:25:ed:37:12, type 0x0800
IP:     192.168.0.34->198.51.100.97, protocol 6
        version 4, ihl 5, tos 0x00, len 384,
        id 95, frag_off 0x4000, ttl 128, checksum 3514
TCP:    sport 64984, dport 80, seq 2270663712, ack 3191786587,
        reserved 0, offset 5, window 16425, checksum 24524,
        flags 0x0018 ( ACK PSH), urgent data 0
TCP option:
Flow fastpath, session 51187
NAT session, run address/port translation
session 51187 packet sequeunce old 0 new 1

Forwarding lookup, ingress interface 18
L3 mode, virtual-router 1
Route lookup in virtual-router 1, IP 198.51.100.97
Route found, interface ethernet1/1, zone 5
Resolve ARP for IP 198.51.100.97 on interface ethernet1/1
ARP entry found on interface 16
Transmit packet on port 16


== 2016-02-10 14:53:09.981 -0800 ==
Packet received at ingress stage
Packet info: len 242 port 16 interface 16 vsys 1
  wqe index 211483 packet 0x0x80000000b58550c6
Packet decoded dump:
L2:     00:0c:29:de:25:fa->b4:0c:25:ed:37:10, type 0x0800
IP:     198.51.100.97->198.51.100.230, protocol 6
        version 4, ihl 5, tos 0x00, len 228,
        id 10920, frag_off 0x4000, ttl 128, checksum 31165
TCP:    sport 80, dport 52924, seq 3191786587, ack 2270664056,
        reserved 0, offset 5, window 256, checksum 13539,
        flags 0x0018 ( ACK PSH), urgent data 0
TCP option:
Flow lookup, key word0 0x50cebc00050600 word1 0
Flow 102375 found, state 2, HA 0
Active flow, enqueue to fastpath process


== 2016-02-10 14:53:09.981 -0800 ==
Packet received at fastpath stage
Packet info: len 242 port 16 interface 16 vsys 1
  wqe index 211483 packet 0x0x80000000b58550c6
Packet decoded dump:
L2:     00:0c:29:de:25:fa->b4:0c:25:ed:37:10, type 0x0800
IP:     198.51.100.97->198.51.100.230, protocol 6
        version 4, ihl 5, tos 0x00, len 228,
        id 10920, frag_off 0x4000, ttl 128, checksum 31165
TCP:    sport 80, dport 52924, seq 3191786587, ack 2270664056,
        reserved 0, offset 5, window 256, checksum 13539,
        flags 0x0018 ( ACK PSH), urgent data 0
TCP option:
Flow fastpath, session 51187
NAT session, run address/port translation
session 51187 packet sequeunce old 1 new 2

Forwarding lookup, ingress interface 16
L3 mode, virtual-router 1
Route lookup in virtual-router 1, IP 192.168.0.34
Route found, interface ethernet1/3, zone 1
Resolve ARP for IP 192.168.0.34 on interface ethernet1/3
ARP entry found on interface 18
Transmit packet on port 18


 

You can now use flow basic to follow the packets through the Palo Alto Networks firewall, to better understand all the stages a packet goes through. Also take a look at this article that explains all this in greater detail: Packet Flow Sequence in PANOS

 

When you feel comfortable, feel free to add additional features like 'appid' for more detail, but do keep an eye on the dataplane resource-monitor to make sure the dataplane is not getting taxed.

 

I hope you found this article interesting. If you haven't already done so, please check out the other articles in the Getting Started series.

 

Feel free to leave a comment below.

 

Regards,

Tom



Attachments
Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClS9CAK&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Attachments
Choose Language