Firewall fails to install dynamic updates pushed by air-gapped panorama with error: HTTP response code said error.

• Internet connected panorama can push the dynamic updates to SCP Server
• Air-gapped panorama can receive updates from SCP Server.
• While pushing these Dynamic Updates from Air-gapped panorama to the firewalls it fails.
• Error message similar to "Failed to upload image. Device msg: 'Failed to download panupv3-all-wildfire-704046-707406. Download error: HTTP response code said error.." is displayed
•  Customer has to manually push the updates on the firewall.
• Air-Gapped Panorama shows below system logs:

high general general 0 0007EV42603 Deployment job upload wildfire to PA-VM failed. Device msg:'Failed to download panupv3-all-wildfire-704039-707399. Download error: HTTP response code said error.'

Note: Air-gapped networks are the ones where panorama management server, managed firewalls, Log Collectors, and WildFire appliances are not connected to the internet.

• Air Gapped Panorama
• PAN-OS 9.1 and above
• Dynamic Updates
• SCP Server

• From the dlsrvr.log (less mp-log dlsrvr.log)on air-gapped panorama,  firewall is trying to download the dynamic update file from panorama its not getting the access to the folder where the dynamic update images are stored:

admin@Panorama> less mp-log dlsrvr.log
xxxxxx [error] 9477#0: *105 open() "/opt/pancfg/mgmt/deploy/wildfire-images/panupv3-all-wildfire-709038-712398" failed (13: Permission denied), client: ::ffff:10.111.111.112, server: d2.dlsrvr.pan, request: "POST /dl.get HTTP/1.1", upstream: "http://127.0.0.1:28250/unauth/php/dlsrvr.php", host: "d2.dlsrvr.pan:28443"

•  Here SCP server is assigning the relevant file permissions to the air-gapped panorama and from the above logs it seems that the dynamic updates file does not have relevant read/write permissions and hence firewall is unable to download the dynamic update files into its database.

Note:  IN PAN-OS 11.x version and above, replace  dlsrvr.log as dlsrvr_error.log 

Installing Content Updates on Panorama Without An Internet Connection