Panorama does not show logs and log collector ES in red status after upgrade to 11.1.6-h1

• Logs are not visible since the upgrade.
• ES status is red with multiple unassigned shards:

> show log-collector-es-cluster health
{
  "cluster_name" : "__pan_cluster__",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 0,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 1796,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 0.0
}

• The unassigned shards show the error "no_valid_shard_copy", in the CLI.

> show log-collector-es-cluster state routing_table
{
  "cluster_name" : "__pan_cluster__",
  "cluster_uuid" : "_3t-JkAYRnKnPibN0Ou4mA",
  "routing_table" : {
    "indices" : {
      "pan_20250201_7h_system_017507002887-0" : {
        "shards" : {
          "9" : [
            {
              "state" : "UNASSIGNED",
              "primary" : true,
              "node" : null,
              "relocating_node" : null,
              "shard" : 9,
              "index" : "pan_20250201_7h_system_017507002887-0",
              "recovery_source" : {
                "type" : "EXISTING_STORE",
                "bootstrap_new_history_uuid" : false
              },
              "unassigned_info" : {
                "reason" : "CLUSTER_RECOVERED",
                "at" : "2025-02-20T11:16:46.251Z",
                "delayed" : false,
                "allocation_status" : "no_valid_shard_copy" <<<<<<<<<<
              }
            }
          ],

• The certificate status shows as expired.

> debug elasticsearch show certs 

ElasticSearch Certificate info
CA Cert
  Subject: 2dfde7b5-XXX-473a-a1bb-YYYYYYYYYY
  Issuer:  CCCCCCCC-XXX-473a-a1bb-YYYYYYYYYY
  Validity
    From:  Aug 17 12:34:16 2022 GMT
    To:    Aug 16 12:34:16 2032 GMT
  Status:  CA
CC Cert
  Subject: AAAAAAA-b52f-BBBB-bca7-CCCCCCCCCCCC
  Issuer:  CCCCCCC-XXXX-473a-a1bb-YYYYYYYYYYYY
  Validity
    From:  Oct 31 04:02:03 2024 GMT
    To:    Jan 29 04:02:03 2025 GMT <<<<< Expired
  Status:  Ok

• Check the log __pan_cluster__.log and confirm that there is a certificate error:

>less var/log/elasticsearch/__pan_cluster__.log
[WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [017507002887]client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/x.y.m.7:9300, remoteAddress=/x.y.q.7:36415, profile=default}
[WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [017507002887]client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/x.y.m.7:9300, remoteAddress=/x.y.q.7:46851, profile=default}
[WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [017507002887]client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/x.y.m.7:9300, remoteAddress=/x.y.q.7:36777, profile=default}
[WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [017507002887]client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/142.88.1.7:9300, remoteAddress=/x.y.q.7:39947, profile=default}

• Log-Collector.
• Panorama.
• PAN-OS 11.1.6-h1 although can affect others version.

The certificates are not correctly updated after the upgrade, causing TLS connection problems between nodes inside the cluster.

1. Renew the certificates to fix the problem, join log-collector CLI, and run the following command:

>debug elasticsearch repair certs  
Wait some minutes and restart ES:
>debug elasticsearch es-restart option all

2. Check whether the ES status changes and the unassigned shard reduces.
3. If the issues persist, root access is needed, please open a Support Case for assistance.