SDWAN PAN Direct Access - tunnel not coming up after onboarding of HA pair

• When onboarding  Firewall HA pair to Prisma Access, the tunnel fails to come up.
• No commit error is seen on Prisma Access or firewalls.
• The tunnels to Prisma Access are seen down.
• On the firewall, ikemgr.log (less mp-log ikemgr.log)  "authentication failed" errors are seen.

admin@NGFW> less mp-log ikemgr.log
+0800  [PNTF]: {    4:     }: ====> IKEv2 CHILD SA NEGOTIATION STARTED AS INITIATOR, non-rekey; gateway gw_0102_prisma-compute-node_0101 <====
                                                      ====> Initiated SA: x.x.x.x[500]-y.y.y.y[500] message id:0x00000000 parent SN:5968 <====
+0800  [PWRN]: {    3:     }: z.z.z.z[500] - y.y.y.y[500]:0x563f3c8bfc30 received notify type AUTHENTICATION_FAILED
+0800  [INFO]: {    3:     }: z.z.z.z[500] - y.y.y.y[500]:(nil) closing IKEv2 SA gw_0101_prisma-compute-node_0101:5967, code 12
+0800  [PNTF]: {    3:     }: ====> IKEv2 IKE SA NEGOTIATION FAILED AS INITIATOR, non-rekey; gateway gw_0101_prisma-compute-node_0101 <====
                                                      ====> Failed SA: z.z.z.z[500]-y.y.y.y[500] SPI:8201d0b411d1c4df:3c025cb758b62f04 SN 5967 <====
+0800  [PWRN]: {    4:     }: x.x.x.x[500] - y.y.y.y[500]:0x563f3c62f8a0 received notify type AUTHENTICATION_FAILED
+0800  [INFO]: {    4:     }: x.x.x.x[500] - y.y.y.y[500]:(nil) closing IKEv2 SA gw_0102_prisma-compute-node_0101:5968, code 12
+0800  [PNTF]: {    4:     }: ====> IKEv2 IKE SA NEGOTIATION FAILED AS INITIATOR, non-rekey; gateway gw_0102_prisma-compute-node_0101 <====
                                                      ====> Failed SA: x.x.x.x[500]-y.y.y.y[500] SPI:3df51aae451610a6:3238ecf9f4598c3d SN 5968 <====

• Panorama
• sd_wan plugin - 2.2.7+ / 3.0.7-h3+ / 3.1.3+ / 3.2.0+
• cloud_services plugin
• SDWAN PAN Direct Access (Prisma Access as Hub)
• Branch devices in High Availability

• An improvement introduced in the sdwan plugin from version 2.2.7+ / 3.0.7-h3+ / 3.1.3+ / 3.2.0+ is making the autogenerated configuration based on the LOWER Serial Number device in a HA pair.
• In case, the active device onboarded to Prisma Access is not the LOWER Serial Number device, the configuration between the Prisma Access and the firewalls will not match.
• In the diagram below, if Branch-B2 with S/N 22222 is the active device onboarded to Prisma Access, the tunnel will not come up.

1. Delete the current onboarding
2. Make the "Lower Serial Number" device active by performing a failover.
3. Confirm the status on Panorama > Managed Devices > Summary.
4. Configure the Prisma Access onboarding on that device.
5. Commit.
6. Push to Remote Network
7. Confirm Prisma Access commit is complete.
8. Push to the HA pair.

SDWAN Plugin Features Notes

[..] by using the lower serial number among the HA devices for generating the SD-WAN configurations that remove tunnel flaps.

This improvement also introduces the following SD-WAN configuration changes:
- the IKE key ID is formed with the lower serial number between the HA devices.
- the SD-WAN generated configurations, such as route table entry in virtual router, tunnel name, IKE gateway name, BGP import rule name, routing profile, BGP peer, and BGP filtering profile will be reset.
- Tunnel names and corresponding IP address would change as the tunnel names are created from a lower serial number among the two HA devices.

As the firewalls are the initiator, the logging is limited.