OSPF routes are not reflecting in RIB after HA A/P failover and neighborship stuck in exstart state with Cisco nexus VPC cluster

• OSPF adjacency established between Palo Alto Networks Firewalls in Active/Passive HA setup and Cisco Nexus switches configured with L3 HSRP.
• OSPF routes advertised from neighboring Nexus switches are properly showing on Active FW-A.

FW-A(active)> show routing route
VIRTUAL ROUTER: DR-VR (id 8)
  ==========
destination                                 nexthop                                 metric flags      age   interface          next-AS    
0.0.0.0/0                                   192.168.139.42                          11     A O1       1809717 ethernet1/18.705              
10.19.12.0/24                               192.168.139.42                          20     A O2       1809717 ethernet1/18.705              
10.19.13.0/24                               192.168.139.42                          20     A O2       1809717 ethernet1/18.705              
10.19.14.0/24                               192.168.139.42                          20     A O2       1809717 ethernet1/18.705              
10.19.15.0/25                               192.168.139.42                          58     A Oo       1809717 ethernet1/18.705              
10.19.16.0/25                               192.168.139.42                          54     A Oi       1809717 ethernet1/18.705      

• Running <show routing protocol ospf neighbor> on FW-A (Active node), will show DR 192.168.139.42 in Full state, but BDR 192.168.139.43 in "exchange start"  

  virtual router:                DR-VR
  neighbor address:              192.168.139.42
  local address binding:         0.0.0.0
  type:                          dynamic
  status:                        full <<<<<<<<<<<<<<<<
  neighbor router ID:            192.168.129.245
  area id:                       0.0.0.0
  neighbor priority:             1
  lifetime remain:               30
  messages pending:              0
  LSA request pending:           0
  options:                       0x42: O E 
  hello suppressed:              no
  restart helper status:         not helping
  restart helper time remaining: 0
  restart helper exit reason:    none
  ==========
  virtual router:                DR-VR
  neighbor address:              192.168.139.43
  local address binding:         0.0.0.0
  type:                          dynamic
  status:                        exchange start <<<<<<<<<<<<<<<<
  neighbor router ID:            192.168.129.246
  area id:                       0.0.0.0
lines 1-62
  neighbor priority:             1
  lifetime remain:               39
  messages pending:              0
  LSA request pending:           0
  options:                       0x00
  hello suppressed:              no
  restart helper status:         not helping
  restart helper time remaining: 0
  restart helper exit reason:    none

• If a failover takes place and FW-B becomes Active, the OSPF routes are no longer showing in the routing table, even though the routes are showing in <show routing protocol ospf lsdb> output.
• Running <show routing protocol ospf neighbor> on FW-B (Active after failover), will show DR 192.168.139.42 stuck in "exchange start" , but BDR 192.168.139.43 in Full state.

  virtual router:                DR-VR
  neighbor address:              192.168.139.42
  local address binding:         0.0.0.0
  type:                          dynamic
  status:                        exchange start <<<<<<<<<<<<<<<<
  neighbor router ID:            192.168.129.245
  area id:                       0.0.0.0
  neighbor priority:             1
  lifetime remain:               37
  messages pending:              0
  LSA request pending:           0
  options:                       0x00
  hello suppressed:              no
  restart helper status:         not helping
  restart helper time remaining: 0
  restart helper exit reason:    none
  ==========
  virtual router:                DR-VR
  neighbor address:              192.168.139.43
  local address binding:         0.0.0.0
  type:                          dynamic
lines 1-59
  status:                        full <<<<<<<<<<<<<<<<
  neighbor router ID:            192.168.129.246
  area id:                       0.0.0.0
  neighbor priority:             1
  lifetime remain:               32
  messages pending:              0
  LSA request pending:           0
  options:                       0x42: O E 
  hello suppressed:              no
  restart helper status:         not helping
  restart helper time remaining: 0
  restart helper exit reason:    none

• In the packet capture taken on the FW-B, the time-to-leave exceeded ICMP packets are seen sent from BDR to the FW.

• Examining the details of the TTL Exceeded packet in the ICMP portion will tell us that OSPF packet sent from the firewall to the DR router has been lost in transit.

• This also provides details about the packet id lost in transit, which can be traced in the same PCAP

• By filtering the PCAP with that packet ID we can confirm that OSPF DBD packet sent from the firewall to DR router is the one that has been lost in transit.

PAN-OS: All

• OSPF uses a TTL value of 1 for its packets
• In this vPC setup, the Port Channel Hashing Algorithm on Cisco L2 switch causes the OSPF packets from FW-B destined to DR Nexus to be sent via the BDR Nexus.
• Since these packets traverse the VPC peer link, their TTL decrements to 0, causing them to be dropped and preventing OSPF adjacency formation. 
• ICMP TTL Exceeded messages are generated by BDR and sent to the FW.

By enabling 'layer3 peer-router' command on Nexus switches, the vPC peer link can be used for Layer 3 routing, allowing OSPF packets to traverse without TTL decrement.

# config t
(config) # vpc domain <domain-ID>
(config-vpc-domain) # layer3 peer-router
(config-vpc-domain) # end