Firewall Data planes fail to connect Enhanced Application Logging ingest endpoint for Device Security security

• Logs are not forwarded to Logging service and Device Security portal shows the firewall as Disconnected
• The EAL ingest endpoint is available on the output of request logging-service-forwarding customerinfo show: 

  Ingest endpoint: <redacted>.in2-lc-prod-de.gpcloudservice.com
  EAL ingest endpoint: <redacted>.fei-lc-prod-de.gpcloudservice.com
  Query endpoint: <redacted>.api2-lc-prod-de.gpcloudservice.com:444
  Customer ID: 12345678
  Region : de

• The output of show Device Security eal all CLI command on the firewall shows:
  1. Management plane is able to connect: 

     Management Plane:
     DPI Cloud server: xyz.fei-lc-prod-de.gpcloudservice.com:443
     Cloud connection: connected

  2. The connection from all Data planes fail: 

     Dataplane s1dp0:
     DPI Cloud server: xyz.fei-lc-prod-de.gpcloudservice.com:443
     Cloud connection: failed

  3. The Error details for failed connection from Data plane slots is: 

     Error code: dial tcp: lookup xyz.fei-lc-prod-de.gpcloudservice.com on 127.132.1.1:53: server misbehaving
     

• The icd_dp.log file (less dp0-log icd_dp.log) shows: 

  {"level":"error","time":"Nov 24 09:48:31.827","message":"[DPI]Failed to connect server [tcp]xyz.fei-lc-prod-de.gpcloudservice.com:443,  ->  err fail to parseTlsCert, err fail to load client cert[/opt/pancfg/mgmt/lcaas/ssl/lcaas.pem], err o pen /opt/pancfg/mgmt/lcaas/ssl/lcaas.pem: no such file or directory"}
  {"level":"error","time":"Nov 24 09:48:31.857","message":"DPI deliver 0[ack: false] take out 3684 stale eal logs. remain 14483(30000000000, force true, totalByte 3273409)"}
  {"level":"error","time":"Nov 24 09:48:31.882","message":"[DPI]grpc tcp connection failed: dial tcp: lookup xyz.fei-lc-prod-de.gpcloudservice.com on 127.132.1.1:53: server misbehaving"}
  {"level":"warn","log":"grpc","time":"Nov 24 09:48:31.882","message":"grpc: addrConn.createTransport failed to connect to {xyz.fei-lc-prod-de.gpcloudservice.com:443  <nil> 0 <nil>}. Err :connection error: desc = \"transport: error while dia ling: dial tcp: lookup xyz.fei-lc-prod-de.gpcloudservice.com on 127.132.1.1:53: server misbehaving\". Reconnecting..."}
  {"level":"error","time":"Nov 24 09:48:31.883","message":"[DPI]Failed to connect server [tcp]xyz.fei-lc-prod-de.gpcloudservice.com:443,  ->  err dial tcp: lookup xyz.fei-lc-prod-de.gpcloudservice.com on 127. 132.1.1:53: server misbehaving"}

• The ping test from firewall towards xyz.fei-lc-prod-de.gpcloudservice.com FQDN/URL is successful

• Any Palo Alto Networks firewall
• PanOS version 10.1 or higher 
• Device Security Security subscription is activated for the firewall
• Logging service license is available on the firewall
• Device certificate is installed on the firewall
• Private DNS server addresses are configured under Device > Setup > Services > Settings

The EAL ingest endpoint is resolved by the Data planes of firewall using their own mechanism which mimics the following behavior:

• If the Primary DNS server configured on the firewall does not contain the A record against a FQDN, it will not be retried to be resolved using Secondary DNS server.
• Hence the DNS resolution failure will bar the Data planes of firewall from successfully connecting.
• Since the Management plane uses a different logic for DNS resolution, it can reach out to Secondary DNS server in such circumstances and if the server is able to respond with relevant A record, connection will be established using Management plane while the Data planes remain Disconnected.

1. Ensure when configuring private DNS servers on firewalls with Device Security security subscription activated andBoth (Primary and Secondary) DNS servers are able to resolve the EAL ingest endpoint.
2. This endpoint can be obtained from the output of request logging-service-forwarding customerinfo show CLI command.