gRPC connection to iot.services-edge.paloaltonetworks is broken, error: context deadline exceeded
34051
Created On 05/18/21 15:51 PM - Last Modified 06/10/25 22:59 PM
Symptom
gRPC connection to iot.services-edge.paloaltonetworks.com cannot be established and times out.
High Severity system logs are also generated indicating connection issues.
2021/04/26 09:49:58 high iot grpc-co 0 gRPC connection to iot.services-edge.paloaltonetworks.com:443 is broken, error: context deadline exceeded time: 2021-04-26 09:49:58
IoT statistics can give us historical information about the connection status. Also, from the same output we can verify device certificate status. Alternatively we can use cli command "show device-certificate status"
admin@PA-VM-100-1> show iot icd statistics all Summary of ICD gRPC client [tcp//iot.services-edge.paloaltonetworks.com:443]: number of connection reset: 0 number of connection failed: 12480 number of connection established: 0 number of connection attempts: 12481 number of connection released: 12479 number of connection selected: 0 number of selections failed: 287959 number of bytes sent: 0 number of bytes received: 0 Last gRPC connection Attempt: 2021-04-29 17:12:12 +0200 CEST Last successful gRPC connection: 1970-01-01 01:00:00 +0100 CET Summary of gRPC connections [configured source IP: ]: ICD device cert status: Installed Validity: Notbefore: 2021-04-23 12:26:29 +0000 UTC Notafter: 2021-07-22 12:26:29 +0000 UTC EnforcerURL: enforcer.iot.services-edge.paloaltonetworks.com:443 max gRPC connections: 1, max alive time: 900, max bytes sent: 0 [0]gRPC conn[192.168.10.32:56326 -> 35.223.164.209:443], state false, selected 0, sent 0, received 0, close @1970-01-01 01:00:00 +0100 CET, backup true grpc stats: wire 0, app: 0, num: 0 Error code: context deadline exceeded Unknown IP Query LRU statistics: number of entries : 0 number of expired entries : 0 number of queries to cloud : 0 number of queries ignored : 8822081 number of queries answered : 0 Verdict LRU statistics: number of verdicts : 0 number of verdicts ignored : 0 number of verdicts pushed out : 0 GETALL duration : nil Summary of connections to dataplane[slots: 1, dps: 1]: s1dp1: address 127.1.1.2, online true, conn state true Current iot bookmark: not received yet Last verdict request: 1970-01-01 01:00:00 +0100 CET Summary of ICD Redis [unix:/opt/pancfg/cache/iotd/redis_iotd.sock]: number of verdicts enqueued: 0 number of verdicts discarded: 0 number of unknown verdict types: 0 number of unknown verdicts failed: 0 number of verdicts dropped: 0 number of failed logging updates: 0 number of verdicts persisted: 0 number of verdicts unchanged: 0 number of bad content errors: 0 number of Redis conn attempts: 1 number of successful conn: 1 Last good Redis connection: 2021-04-26 09:07:56 +0200 CEST Last failed Redis connection: Never Last verdict writing time: Never Summary of ICD device verdict to Iotd daemon: number of verdicts enqueued: 0 number of verdicts discarded: 0 number of verdicts sent: 0 number of verdicts unsent: 0 Last verdict sent to Iotd: 1970-01-01 01:00:00 +0100 CET Summary of ICD device verdict to DP: number of verdicts enqueued: 0 number of verdicts discarded: 0 number of verdicts sent: 0 number of verdicts unsent: 0 Last verdict sent to DP: 1970-01-01 01:00:00 +0100 CET Current Time: 2021-04-29 17:12:20.526017905 +0200 CEST m=+288265.567775167
If we check icd.log file (>less mp-log icd.log) we can see handshake failure:
{"level":"info","time":"2021-04-26T09:49:36.917084697+02:00","message":"Count of OCSP urls: 1"}
{"level":"info","time":"2021-04-26T09:49:36.928677307+02:00","message":"Got OCSP response verify OK from http://r3.o.lencr.org"}
{"level":"info","time":"2021-04-26T09:49:36.928705544+02:00","message":"Check OCSP CN=iot.services-edge.paloaltonetworks.com took 22.890651ms"}
{"level":"info","time":"2021-04-26T09:49:36.928744605+02:00","message":"Certificate is valid: CN=iot.services-edge.paloaltonetworks.com"}
{"level":"info","time":"2021-04-26T09:49:36.929032089+02:00","message":"Cert expiration check. Subject: CN=R3,O=Let's Encrypt,C=US NotAfter: 2021-09-29 19:21:40 +0000 UTC NotBefore: 2020-10-07 19:21:40 +0000 UTC"}
{"level":"error","time":"2021-04-26T09:49:36.940562214+02:00","message":"CRL HTTP Get failed: status code = Ƿ"}
{"level":"info","time":"2021-04-26T09:49:36.940602182+02:00","message":"downloadCRLWithURL crlURL=http://crl.identrust.com/DSTROOTCAX3CRL.crl took 11.537066ms"}
{"level":"info","time":"2021-04-26T09:49:36.940619216+02:00","message":"Fetch CRL from http://crl.identrust.com/DSTROOTCAX3CRL.crl successfully before timeout."}
{"level":"info","time":"2021-04-26T09:49:36.940629875+02:00","message":"Check CRL crlURL=http://crl.identrust.com/DSTROOTCAX3CRL.crl took 11.58108ms"}
{"level":"error","time":"2021-04-26T09:49:36.940638759+02:00","message":"Downloading CRL from http://crl.identrust.com/DSTROOTCAX3CRL.crl"}
{"level":"warn","log":"grpc","time":"2021-04-26T09:49:36.940721237+02:00","message":"grpc: addrConn.createTransport failed to connect to {iot.services-edge.paloaltonetworks.com:443 0 <nil>}. Err :connection error: desc = \"transport: authentication handshake failed: Downloading CRL from http://crl.identrust.com/DSTROOTCAX3CRL.crl\". Reconnecting..."}
{"level":"info","time":"2021-04-26T09:49:37.587058192+02:00","message":"Cert expiration check. Subject: CN=iot.services-edge.paloaltonetworks.com NotAfter: 2021-06-27 05:53:01 +0000 UTC NotBefore: 2021-03-29 05:53:01 +0000 UTC"}
Environment
- Palo Alto Firewall
- Supported PAN-OS
- Device Security
Cause
The above logs are indication of connection failures to crl http://crl.identrust.com/
Resolution
To resolve this issue make sure that there is no device on the path blocking connections to http://crl.identrust.com/ or http://r3.o.lencr.org.
Additional Information
Connection to iot.services-edge.paloaltonetworks may also fail If the firewall cannot connect to OCSP server address http://r3.o.lencr.org.
From icd.log we can see:
{"level":"info","time":"2021-04-02T14:14:15.612020435+11:00","message":"Count of OCSP urls: 1"}
{"level":"info","time":"2021-04-02T14:14:15.659498115+11:00","message":"Check OCSP CN=iot.services-edge.paloaltonetworks.com took 115.017831ms"}
{"level":"error","time":"2021-04-02T14:14:15.659665712+11:00","message":"Send OCSP request failed: http://r3.o.lencr.org error: OCSP HTTP response error: status code:503"}
{"level":"warn","log":"grpc","time":"2021-04-02T14:14:15.66013365+11:00","message":"grpc: addrConn.createTransport failed to connect to {iot.services-edge.paloaltonetworks.com:443 0 <nil>}. Err :connection error: desc = \"transport: authentication handshake failed: Send OCSP request failed: http://r3.o.lencr.org error: OCSP HTTP response error: status code:503\". Reconnecting..."}
{"level":"info","time":"2021-04-02T14:14:15.934510834+11:00","message":"Cert expiration check. Subject: CN=iot.services-edge.paloaltonetworks.com NotAfter: 2021-04-11 22:51:38 +0000 UTC NotBefore: 2021-01-11 22:51:38 +0000 UTC"}