VOIP call with application h.225 dropping after 10-15 minutes

• Voip calls with application h.225 and destination port 1720 drop after 15 minutes 
• Firewall does not show any drops from packet capture 
• Global counters using Filters from packet capture shows the following counter : "ctd predict queue got enque due to predict waiting fpp"

Global counters: Elapsed time since last sampling: 10.810 seconds name value rate severity category aspect description --------

flow_fpga_rcv_fastpath 1 0 info flow offload fpga packets for fastpath received 

ctd_predict_queue_enque 1 0 info ctd pktproc ctd predict queue got enque due to predict waiting fpp 

ctd_predict_queue_len 1 0 info ctd pktproc ctd predict queue length

• session shows first end-reason : unknown 

Session          506534

        c2s flow:
                source:      10.10.10.10 [Voice]
                dst:         1.1.1.1
                proto:       6
                sport:       11014           dport:      1720
                state:       ACTIVE          type:       FLOW
                src user:    unknown
                dst user:    unknown

        s2c flow:
                source:      1.1.1.1 [External]
                dst:         2.2.2.2
                proto:       6
                sport:       1720            dport:      26953
                state:       ACTIVE          type:       FLOW
                src user:    unknown
                dst user:    unknown

        start time                           : Tue Mar  3 17:05:00 2020
        timeout                              : 3600 sec
        time to live                         : 3600 sec 
        total byte count(c2s)                : 1839
        total byte count(s2c)                : 2068
        layer7 packet count(c2s)             : 13
        layer7 packet count(s2c)             : 17
        vsys                                 : vsys1
        application                          : h.225  
        rule                                 : Voip
        service timeout override(index)      : False
        session to be logged at end          : True
        session in session ager              : True
        session updated by HA peer           : False
        address/port translation             : source
        nat-rule                             : Internet (vsys1)
        layer7 processing                    : enabled
        ctd version                          : 5
        URL filtering enabled                : True
        URL category                         : any
        session via syn-cookies              : False
        session terminated on host           : False
        session terminate tunnel             : False
        captive portal session               : False
        ingress interface                    : ethernet1/14.1
        egress interface                     : ethernet1/1
        session QoS rule                     : N/A (class 4)
        end-reason                           : unknown

• Then when session fails end-reason is: "tcp-rst-from-client"

Session          506534

c2s flow:
                source:      10.10.10.10 [Voice]
                dst:         1.1.1.1
                proto:       6
                sport:       11014           dport:      1720
                state:       ACTIVE          type:       FLOW
                src user:    unknown
                dst user:    unknown

        s2c flow:
                source:      1.1.1.1 [External]
                dst:         2.2.2.2
                proto:       6
                sport:       1720            dport:      26953
                state:       ACTIVE          type:       FLOW
                src user:    unknown
                dst user:    unknown

        start time                           : Tue Mar  3 17:05:00 2020
        timeout                              : 15 sec
        time to live                         : 14 sec 
        total byte count(c2s)                : 1959
        total byte count(s2c)                : 2365
        layer7 packet count(c2s)             : 15
        layer7 packet count(s2c)             : 20
        vsys                                 : vsys1
        application                          : h.225  
        rule                                 : Voip
        service timeout override(index)      : False
        session to be logged at end          : True
        session in session ager              : True
        session updated by HA peer           : False
        address/port translation             : source
        nat-rule                             : Internet(vsys1)
        layer7 processing                    : enabled
        ctd version                          : 5
        URL filtering enabled                : True
        URL category                         : any
        session via syn-cookies              : False
        session terminated on host           : False
       session traverses tunnel             : False
        session terminate tunnel             : False
       captive portal session               : False
        ingress interface                    : ethernet1/14.1
        egress interface                     : ethernet1/1
      session QoS rule                     : N/A (class 4)
       tracker stage firewall               : TCP RST - client
       end-reason                           : tcp-rst-from-client

• Debug from flow basic capture shows message "queue the work waiting for predict install to complete"

In the sessions with destination port 1720, the following message is seeing  "queue the work waiting for predict install to complete".
A Deny message is also shown in Flow Basic "CP-DENY TCP non data packet getting through "


Packet received at fastpath stage, tag 506534, type ATOMIC
Packet info: len 60 port 77 interface 142 vsys 1
wqe index 552402 packet 0x0x80000003956e08e4, HA: 0, IC: 0
Packet decoded dump:
L2: yy:yy:yy:yy:yy:yy->xx:xx:xx:xx:xx:xx, VLAN 20 (0x8100 0x6014), type 0x0800
IP: 10.10.10.10->1.1.1.1, protocol 6
version 4, ihl 5, tos 0xb8, len 40,
id 27568, frag_off 0x4000, ttl 64, checksum 20256(0x4f20)
TCP: sport 11014, dport 1720, seq 3489772155, ack 3912119931,
reserved 0, offset 5, window 123, checksum 23596,
flags 0x10 ( ACK), urgent data 0, l4 data len 0
TCP option:
Flow fastpath, session 506534 (set work 0x800000038f420800 exclude_video 0 from sp 0x8000000132c16880 exclude_video 0)
HSCI link selection: tag 0
NAT session, run address/port translation
CP-DENY TCP non data packet getting through
queue the work waiting for predict install to complete

• PANOS 8.1.x
• PANOS 9.0.5
• PAN-3260

• Parent sessions are dropped when installing duplicate predict session.
• Firewall is silencing blocking destination port 1790, and application h.225 with destination port 1720 will depend on destination port 1790.  
• Since application h.225 does not hear from destination port 1790. Firewall drops the session with message: "queue the work waiting for predict install to complete"

1. Issue ID PAN-122311 -  Fixed an issue where parent sessions were dropped when you installed duplicate predict session. Fix is in version 9.0.6
2. A workaround without upgrading Firewall is: Application Override