iot.services-edge.paloaltonetworks への gRPC 接続が切断されました。エラー: コンテキストの期限を超えました

34049

Created On 05/18/21 15:51 PM - Last Modified 06/10/25 22:59 PM

Symptom

gRPC 接続先iot.services-edge.paloaltonetworks.com確立できずタイムアウトになります。

接続の問題を示す高重大度のシステムログも生成されます。

2021/04/26 09:49:58 high     iot            grpc-co 0  gRPC connection to iot.services-edge.paloaltonetworks.com:443 is broken, error: context deadline exceeded time: 2021-04-26 09:49:58

IoT 統計により、接続ステータスに関する履歴情報が得られます。また、同じ出力からデバイス証明書のステータスを確認できます。あるいは、cli コマンドを使用することもできます。デバイス証明書のステータスを表示」

admin@PA-VM-100-1> show iot icd statistics all

Summary of ICD gRPC client [tcp//iot.services-edge.paloaltonetworks.com:443]:
number of connection reset:       0
number of connection failed:      12480
number of connection established: 0
number of connection attempts:    12481 
number of connection released:    12479 
number of connection selected:    0
number of selections failed:      287959
number of bytes sent:             0
number of bytes received:         0
Last gRPC connection Attempt:     2021-04-29 17:12:12 +0200 CEST
Last successful gRPC connection:  1970-01-01 01:00:00 +0100 CET 


Summary of gRPC connections [configured source IP: ]:
ICD device cert status: Installed
        Validity: 
                Notbefore: 2021-04-23 12:26:29 +0000 UTC 
                Notafter: 2021-07-22 12:26:29 +0000 UTC
EnforcerURL: enforcer.iot.services-edge.paloaltonetworks.com:443

max gRPC connections: 1, max alive time: 900, max bytes sent: 0
[0]gRPC conn[192.168.10.32:56326 -> 35.223.164.209:443], state false, selected 0, sent 0, received 0, close @1970-01-01 01:00:00 +0100 CET, backup true
grpc stats: wire 0, app: 0, num: 0
Error code: context deadline exceeded


Unknown IP Query LRU statistics:
number of entries          : 0
number of expired entries  : 0
number of queries to cloud : 0
number of queries ignored  : 8822081
number of queries answered : 0


Verdict LRU statistics:
number of verdicts            : 0
number of verdicts ignored    : 0
number of verdicts pushed out : 0
GETALL duration               : nil


Summary of connections to dataplane[slots: 1, dps: 1]:
s1dp1: address 127.1.1.2, online true, conn state true


Current iot bookmark: not received yet
Last verdict request:             1970-01-01 01:00:00 +0100 CET


Summary of ICD Redis  [unix:/opt/pancfg/cache/iotd/redis_iotd.sock]:
number of verdicts enqueued:          0
number of verdicts discarded:         0
number of unknown verdict types:      0
number of unknown verdicts failed:    0
number of verdicts dropped:           0
number of failed logging updates:     0
number of verdicts persisted:         0
number of verdicts unchanged:         0
number of bad content errors:  0
number of Redis conn attempts:        1
number of successful conn:            1
Last good Redis connection:        2021-04-26 09:07:56 +0200 CEST
Last failed Redis connection:      Never
Last verdict writing time:         Never


Summary of ICD device verdict to Iotd daemon:
number of verdicts enqueued:      0
number of verdicts discarded:     0
number of verdicts sent:          0
number of verdicts unsent:        0
Last verdict sent to Iotd:        1970-01-01 01:00:00 +0100 CET


Summary of ICD device verdict to DP:
number of verdicts enqueued:      0
number of verdicts discarded:     0
number of verdicts sent:          0
number of verdicts unsent:        0
Last verdict sent to DP:          1970-01-01 01:00:00 +0100 CET

Current Time:  2021-04-29 17:12:20.526017905 +0200 CEST m=+288265.567775167

確認してみるとicd.logファイル(>less mp-log icd.log ) ハンドシェイクの失敗がわかります。

{"level":"info","time":"2021-04-26T09:49:36.917084697+02:00","message":"Count of OCSP urls: 1"}
{"level":"info","time":"2021-04-26T09:49:36.928677307+02:00","message":"Got OCSP response verify OK from http://r3.o.lencr.org"}
{"level":"info","time":"2021-04-26T09:49:36.928705544+02:00","message":"Check OCSP CN=iot.services-edge.paloaltonetworks.com took 22.890651ms"}
{"level":"info","time":"2021-04-26T09:49:36.928744605+02:00","message":"Certificate is valid: CN=iot.services-edge.paloaltonetworks.com"}
{"level":"info","time":"2021-04-26T09:49:36.929032089+02:00","message":"Cert expiration check. Subject: CN=R3,O=Let's Encrypt,C=US NotAfter: 2021-09-29 19:21:40 +0000 UTC NotBefore: 2020-10-07 19:21:40 +0000 UTC"}
{"level":"error","time":"2021-04-26T09:49:36.940562214+02:00","message":"CRL HTTP Get failed: status code = Ƿ"}
{"level":"info","time":"2021-04-26T09:49:36.940602182+02:00","message":"downloadCRLWithURL crlURL=http://crl.identrust.com/DSTROOTCAX3CRL.crl took 11.537066ms"}
{"level":"info","time":"2021-04-26T09:49:36.940619216+02:00","message":"Fetch CRL from http://crl.identrust.com/DSTROOTCAX3CRL.crl successfully before timeout."}
{"level":"info","time":"2021-04-26T09:49:36.940629875+02:00","message":"Check CRL crlURL=http://crl.identrust.com/DSTROOTCAX3CRL.crl took 11.58108ms"}
{"level":"error","time":"2021-04-26T09:49:36.940638759+02:00","message":"Downloading CRL from http://crl.identrust.com/DSTROOTCAX3CRL.crl"}
{"level":"warn","log":"grpc","time":"2021-04-26T09:49:36.940721237+02:00","message":"grpc: addrConn.createTransport failed to connect to {iot.services-edge.paloaltonetworks.com:443 0  <nil>}. Err :connection error: desc = \"transport: authentication handshake failed: Downloading CRL from http://crl.identrust.com/DSTROOTCAX3CRL.crl\". Reconnecting..."}
{"level":"info","time":"2021-04-26T09:49:37.587058192+02:00","message":"Cert expiration check. Subject: CN=iot.services-edge.paloaltonetworks.com NotAfter: 2021-06-27 05:53:01 +0000 UTC NotBefore: 2021-03-29 05:53:01 +0000 UTC"}

Environment

パロアルト Firewall
サポートされています PAN-OS
IOT安全

Cause

上記のログは、crl http://crl.identrust.com/ への接続障害を示しています。

Resolution

この問題を解決するには、パス上に http://crl.identrust.com/ への接続をブロックするデバイスがないことを確認してください。 http://r3.o.lencr.org 。

Additional Information

への接続iot.services-edge.paloaltonetworks失敗する可能性もあります。firewallに接続できませんOCSPサーバーアドレスhttp://r3.o.lencr.org。

icd.log から次のことがわかります。