How to Verify and Troubleshoot Netflow
Objective
This document will show you how to verify and troubleshoot Netflow on the Palo Alto Networks Firewall
Environment
- Firewall:
- Untrust interface: 100.2.3.4
- Trust interface: 192.168.133.1
- DMZ interface: 172.16.1.1
- Netflow Server (w/ Netflow Analysis/Collector software installed):
- 172.16.1.10
- Client PC:
- 192.168.133.10
Procedure
Table of Contents
1. Netflow Overview
2. How to configure Netflow
3. Verify Netflow configuration via Firewall Web UI
4. Verify Netflow configuration via Firewall CLI
5. Check reachability to your Netflow Server
6. Verify counters for Netflow statistics are incrementing
7. Verify counters for Netflow-related global counters are incrementing
8. Verify firewall is not exceeding Max Logging Rate count/sec
9. Check firewall system resources such as CPU, memory, buffers, etc.
10. Verify Netflow packets are leaving the firewall using a packet capture on the Firewall
11. Verify Netflow packets are arriving to the Netflow Server using a packet capture on the Netflow Server
Netflow Overview
If you have configured Netflow on your firewall, whenever traffic flows through any data interface on the firewall with a Netflow Profile configured, the firewall will create a Netflow record and send that information to your Netflow server
There are four main types of Netflow events/records sent from the firewall to a Netflow server:
Flow created - sent to Netflow server whenever a new traffic flow comes into the firewall (i.e. when a traffic flow/session is created in the firewall)
Flow update - sent periodically to Netflow server every X minutes as more and more packets ingress and egress the firewall for that traffic flow
Flow deleted - sent to Netflow server whenever an existing traffic flow closes (FIN, RST), times out/expires, or is deleted/cleared on the firewall
Flow denied - sent to Netflow server whenever a traffic flow gets denied by firewall policy
Netflow packets are exported by the Firewall to a Netflow Server. They contain information about the traffic ingressing/egressing your firewall such as:
Source IP
Source Port
Destination IP
Destination Port
Traffic Direction (Ingress vs. Egress)
Traffic Action: Session created, Session updated, Session deleted, Session denied
Traffic Protocol (TCP, UDP, ICMP, etc.)
etc.
How to configure Netflow
1. Go to Device > Server Profiles > Netflow > click Create >> give it a Name, IP Address, and Port >> click Ok
2. Go to Network >> Interfaces >> select the Interface you want to enable Netflow on >> click Netflow Profile dropdown to select the Netflow Server Profile created in Step 1 above >> click Ok
3. Go to Device >> Setup >> Services tab >> click Service Route Configuration >> click Source Interface >> select the interface out which your Netflow Server lives i.e. the interface out which Netflow packets will be sent to go to your Netflow Server >> select a Source IP Address >> click Ok
Note: You cannot use the Management (MGT) interface to send NetFlow records from the PA-7000 series, PA-5200 series, and PA-3200 series firewalls, you must configure a Service Route using Step 3 above - for all other firewall models, Step 3 above is optional)
4. Commit
Walkthrough w/ Screenshots: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClJzCA
Verify Netflow configuration via Firewall Web UI
1. Go to Device > Server Profiles > Netflow > click to Edit >> verify it has a Name, correct IP Address, and correct Port (usually 2055)
2. Go to Network >> Interfaces >> select the Interface over which traffic is flowing >> verify it has the correct Netflow Profile selected in the drop-down
3. Go to Device >> Setup >> Services tab >> click Service Route Configuration >> click Netflow >> click Source Interface >> verify the correct Source Interface is selected (i.e. the interface out which Netflow packets should be sent to reach your Netflow Server)
4. Go to Device >> Setup >> Services tab >> click Service Route Configuration >> click Netflow >> click Source Interface >> verify the correct Source IP Address is selected (i.e. the interface IP Address of the interface out which Netflow packets should be sent to go to your Netflow Server)
Note: If using a PA-7000 series, PA-5200 series, or PA-3200 series firewall, you must configure a Service Route out a data interface (not the Management (MGT) interface - MGT interface can not be used for Netflow on these models)
5. Verify that you Commited the configuration to your Firewall (and Pushed if Panorama is in use)
Verify Netflow configuration via Firewall CLI
> set cli config-output-format set > configure # show | match netflow set deviceconfig system route service netflow source address 172.16.1.1 set deviceconfig system route service netflow source interface ethernet1/3 set network interface ethernet ethernet1/3 layer3 netflow-profile pantaclab_netflow_server_profile set network interface ethernet ethernet1/6 layer3 netflow-profile pantaclab_netflow_server_profile set vsys vsys1 server-profile pantaclab_netflow_server_profile server lab_netflow_server host 172.16.1.10 set vsys vsys1 server-profile pantaclab_netflow_server_profile server lab_netflow_server port 2055 set vsys vsys1 server-profile pantaclab_netflow_server_profile template-refresh-rate minutes 30 set vsys vsys1 server-profile pantaclab_netflow_server_profile template-refresh-rate packets 20 set vsys vsys1 server-profile pantaclab_netflow_server_profile active-timeout 5
Check reachability to your Netflow Server
a. If using management interface as the Source Interface for Netflow
> ping host <netflow_server_ip_address>
b. If using a data interface (ethernet1/X) as the Source Interface for Netflow
> ping source <firewall_source_interface_ip_address> host <netflow_server_ip_address>
Also, ensure UDP port 2055 is open between the Firewall and the Netflow Server
Verify counters for Netflow statistics are incrementing
> debug log-receiver netflow statistics > debug dataplane netflow statistics (for PA-7000/PA-5200/PA-3200 series) Netflow Statistics -------------------------------------------------------------------------------- Template Config Last Refreshes Pkts Sent (errors) -------------------------------------------------------------------------------- Server Profile : pantaclab_netflow_server_profile 30 min/20 pkts 28 min/8 pkts 3223 (0) Total Netflow packets exported : 3223 (errors 0)
To clear the counters above, run this command:
> debug log-receiver netflow clear
Verify counters for Netflow-related global counters are incrementing
> show counter global | match netflow name value rate severity category aspect description -------------------------------------------------------------------------------------------------------------------------------------------------------------- log_netflow_cnt 2 0 info log system Number of netflow records (For PA-7000, PA-5200, and PA-3200 series firewalls only): flow_netflow_event_created 58 1 info flow netflow NetFlow flowsets added for flow_created event flow_netflow_event_deleted 43 0 info flow netflow NetFlow flowsets added for flow_deleted event flow_netflow_event_update 1 0 info flow netflow NetFlow flowsets added for flow_update event flow_netflow_event_denied 213 1 info flow netflow NetFlow flowsets added for flow_denied event flow_netflow_wqe_success 1 0 info flow netflow WQE created for NetFlow flow_netflow_data_send_success 1 0 info flow netflow NetFlow data packet sent flow_netflow_data_flush_success 1 0 info flow netflow NetFlow data packet flushed Note: If a traffic flow is 'hardware offloaded' you may see the following counter increment when the offload processor informs the DP that the DP needs to create + send out a Netflow event/record): > show counter global | match fpga_flow flow_fpga_flow_update 3 0 info flow offload fpga flow update transaction flow_fpga_flow_delete 3 0 info flow offload fpga flow delete transactions flow_fpga_rcv_stats
To clear the counters above, run this command:
> clear counter global
Verify firewall is not exceeding Max Logging Rate count/sec
Check the maximum logging rate your firewall can handle:
> show system setting logging Max. logging rate: 50000 cnt/sec
Check the counters below to see if any counters are exceeding their maximum logging rate (per second) above:
> debug log-receiver statistics Log incoming rate: 3/sec Log written rate: 3/sec
Check firewall system resources such as CPU, memory, buffers, etc.
Go to Monitor >> System Logs >> look for/make sure you do not see any logs about high CPU, high memory, high buffer utilization, etc. using filters
Verify Netflow packets are leaving the firewall using a packet capture
Warning: The below steps (packet capture) may/will impact the performance/stability of the firewall and traffic if not performed correctly/carefully - proceed with caution
a. If using management interface as the Source Interface for Netflow
> tcpdump snaplen 0 filter “port 2055” <ctrl+C after 5-10 seconds> > view-pcap mgmt-pcap mgmt.pcap 17:15:06.895193 IP 172.16.1.1.39843 > 172.16.1.10.2055: UDP, length 1369 17:15:10.224250 IP 172.16.1.1.39843 > 172.16.1.10.2055: UDP, length 1356 17:15:13.021765 IP 172.16.1.1.39843 > 172.16.1.10.2055: UDP, length 1392
If you would like to export these pcaps off the firewall to your SCP or TFTP Server for viewing in Wireshark:
> scp export mgmt-pcap from mgmt.pcap to <username@host:path> > tftp export mgmt-pcap from mgmt.pcap to <host>
b. If using a data interface (ethernet1/X) as the Source Interface for Netflow
Start packet capture
Go to Monitor >> Packet Capture >> click Manage Filters >> click Add >> type your Netflow server IP Address in the Destination column >> click Ok
Under Stage >> click Add >> add 'transmit' stage >> click Ok
Toggle Filters to On
Toggle Packet Capture to On
Stop packet capture
Toggle Packet Capture to Off
Toggle Filters to Off
Click Refresh (top-right)
Click Download to download your 'transmit' packet capture to your PC
Open the packet capture in Wireshark - filter by 'udp.port==2055' and verify the below
Verify the Netflow packets are leaving the firewall with the correct Source IP Address (interface configured in Device >> Setup >> Services tab >> Service Route Configuration)
Verify the Netflow packets are leaving the firewall out the correct interface by looking at the Src MAC Address (interface configured in Device >> Setup >> Services tab >> Service Route Configuration)
Verify the Netflow packets are leaving the firewall with the correct Destination IP Address (configured in Device >> Netflow Server Profile >> click Edit)
Verify the Netflow packets are leaving the firewall with the correct Destination Port (configured in Device >> Netflow Server Profile >> click Edit)
Example:
Tip: If you see 'No Template Found' in your Wireshark packet captures of Netflow packets / can't view the Flowsets - don't worry - just do the packet capture again but for a longer time - this happens because Wireshark must capture a Netflow Template along with the Netflow Flowsets in order to be able to display the Netflow Flowsets to you.
Verify Netflow packets are arriving to the Netflow Server using a packet capture
Take a packet capture on your Netflow Server using the packet capture software/utility of your choice (Wireshark, tcpdump, etc.) - ( filter by 'udp.port==2055' )
Make sure the packets that left the firewall (in the previous step) are showing up in this Netflow Server packet capture
Additional Information
Examples of Flow Created, Flow Update, Flow Deleted, and Flow Denied events created by the PAN Firewall:
Flow Created
Flow Update
Flow Deleted
Flow Denied
Tip: If you see 'No Template Found' in your Wireshark packet captures of Netflow packets, then do the packet capture again but for a longer time - this is because Wireshark must capture a Netflow Template along with the Netflow Flowsets in order to be able to display the Netflow Flowsets
What is a Netflow Template?
Netflow Template - this is a packet sent from the firewall to the Netflow server occasionally per configuration (Netflow servers need this template packet to interpret/understand the format/fields present in the Flow Created/Updated/Deleted/Denied packets above)
Netflow Server interface numbering convention for PAN firewalls:
https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/monitoring/firewall-interface-identifiers-in-snmp-managers-and-netflow-collectors.html
How to configure Netflow on Palo Alto Networks Firewall:
https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/monitoring/netflow-monitoring/configure-netflow-exports.html
Palo Alto Networks Firewall Netflow Templates:
https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/monitoring/netflow-monitoring/netflow-templates.html
Using Netflow Profiles with subinterfaces:
https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClzyCAC