GlobalProtect Client Return Traffic Is Failing After the GP Gateway Was Deleted and Then Readded OR When Routing table has multiple routes for GP IP Pool

After GlobalProtect gateway configuration is deleted, committed and then readded with exact same IP Pool configuration, return traffic fails

Or when a firewall is restarted, the return GP traffic from the firewall gets dropped or egressed via an unexpected non-tunnel interface
 

• PAN-OS
• GlobalProtect

Firewall implicitly adds the necessary static routes for the configured GlobalProtect IP Pools. Therefore, there is no need for the firewall admin to add static routes manually for the GP IP Pools

If there are multiple routes for the GP IP Pool & one of them is via a non-GP-tunnel interface, there is a high possibility that the route will become ACTIVE in the FIB table after a reboot or after modifying the GP IP Pool

Example Scenario: 

This issue is likely caused by a static route erroneously configured matching the same IP Pool Network but with a different egress interface

After the gateway was deleted and committed, the associated GlobalProtect IP pool route was removed and in its place a previous manually configured static route was installed in the forwarding table (FIB)
After the gateway configuration was restored with the same IP Pool, during the commit, the firewall will no longer install this route in its forwarding table as there is already an existing route matching the same prefix, admin distance and metric
Note: The firewall will use the first configured route. Any other matching routes (matching prefix, AD and metric) will not be installed

For example, the output below seen from command: > show routing route, has 2 routing routes for prefix 172.16.1.0/24
Both prefixes are matching including their metric and administrative distance (AD = 0 for both)
In this example, the route installed (The route showing "A" for Active) is using tunnel.1 interface correctly

VIRTUAL ROUTER: default (id 1)
  ==========
destination                       nexthop                       metric flags      age   interface          next-AS    
0.0.0.0/0                         10.46.42.1                    10     A S              ethernet1/3 
172.16.1.0/24                     10.10.10.10                   10       S E            ethernet1/2                   
172.16.1.0/24                     172.16.1.0                    10     A S E            tunnel.1   <<<< This Route is installed in FIB table

 
However after GlobalProtect gateway was deleted, committed and then be readded,, you can see from output below that the manually configured static route becomes Active and is installed in the FIB table.

VIRTUAL ROUTER: default (id 1)
  ==========
destination                       nexthop                       metric flags      age   interface          next-AS    
0.0.0.0/0                         10.46.42.1                    10     A S              ethernet1/3
172.16.1.0/24                     10.10.10.10                   10     A S E            ethernet1/2  <<<< This Route is installed in FIB table                   
172.16.1.0/24                     172.16.1.0                    10       S E            tunnel.1    

To resolve the routing conflict for the GP IP Pools, delete the manual static routes via non-GP-tunnel interfaces and make sure the routing table has only the static route via GP's tunnel interface

For this example, you can delete static route for 172.16.1.0/24 via non-tunnel interface and make sure there is only one route via tunnel.1 interface that is marked Active

NOTE:
This issue can also arise in the following manner and order:
1. GlobalProtect gateway configuration's IP Pool is modified where it no longer matches the manually configured static route. This would cause the manually configured static route to be installed in the FIB table
2. Then, the same gateway IP Pool is reconfigured to match the erroneously configured static route. This route from GlobalProtect IP Pool will not be installed since the firewall already has the manually configured static route installed in the FIB table.
 

NOTE:
This routing conflict cannot be corrected by restarting routed, committing force, or even rebooting until the stale routes are removed from the configuration.

The reason why some customers might have configured a static route for their GP IP pool is to use it in their redistribution profile for BGP and/or OSPF. However, note that redistribution of this IP Pool route can be accomplished without having to configure a static route. The client just needs to add the tunnel interface in the redistribution profile