Running performance test for VM-Series firewall - Tips & Tricks
31925
Created On 07/12/20 00:49 AM - Last Modified 07/20/20 23:44 PM
Symptom
- Low firewall performance throughput when running iperf command from a host passing through a device.
- Usually throughput is not satisfactory for single stream iperf commands:
# iperf3 -c 172.17.50.126 -t60 -i1 [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-60.00 sec 2.15 GBytes 307 Mbits/sec 1758 sender [ 4] 0.00-60.00 sec 2.15 GBytes 307 Mbits/sec receiver
- Throughput is only 307 Mbps, although on VM-700 IPSEC Throughput supported is up to 6 Gbps.
- Slight improvement on running multi-thread iperf command:
# iperf3 -c 172.17.50.126 -t60 -i1 -P4 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 19.6 MBytes 164 Mbits/sec 11 120 KBytes [ 6] 0.00-1.00 sec 23.5 MBytes 197 Mbits/sec 10 144 KBytes [ 8] 0.00-1.00 sec 20.4 MBytes 171 Mbits/sec 30 102 KBytes [ 10] 0.00-1.00 sec 13.9 MBytes 117 Mbits/sec 34 64.0 KBytes [SUM] 0.00-1.00 sec 77.4 MBytes 649 Mbits/sec 85 # iperf3 -c 172.17.50.126 -t60 -i20 -P16 [SUM] 0.00-60.00 sec 6.01 GBytes 861 Mbits/sec 24601 sender [SUM] 0.00-60.00 sec 6.01 GBytes 860 Mbits/sec receiver
Environment
- Platform: VM-Series on Microsoft Azure, AWS, GCP, VMware etc...
- PAN-OS / Plugin Version: Any
- Deployment: Existing
Cause
- Running single session iperf commands to measure performance throughput of firewall is not a recommended solution.
- Iperf with single session can't scale much as packets cannot be distributed to across all DP cores and queues by NIC.
To validate this behavior, check PAN-OS CLI output for command:
> debug dataplane pow status > show running resource-monitor
- Another limitation on high-performance throughput test is network limitation. Unless you client and server are connected with the 10Gb network, chances are the network will be the bottle net and you cannot reach the full handling potential of your server.
Resolution
- Performance team recommends to run Iperf with mult-threads.
- Scale up more sessions.
- Try using wrk for performance test and other tools that can generate more number of connections so that the sessions gets distributed across multiple cores.
- wrk is a modern HTTP benchmarking tool capable of generating significant load when run on a single multi-core CPU. It combines a multi-threaded design with scalable event notification systems such as epoll and kqueue.
- Install wrk tool on either Linux or MAC host and generate multi-thread, multi-connection HTTP traffiic. Refer documents below: