ES stuck in red and Active shards shows "0" for M-600 and software version is 11.x

ES stuck in red and Active shards shows "0" for M-600 and software version is 11.x

1069
Created On 09/13/25 22:49 PM - Last Modified 04/21/26 22:03 PM


Symptom


  • Elastic search on Panorama logger showing red status
  • Monitoring tab not displaying traffic
  • Log collectors retrieving traffic from firewalls, but Panorama nodes are not processing logs correctly
  • Elastic search on the secondary panorama node in red status
  • All primary shards are present in unassigned state 
  • On panorama Elastic search is stuck in red and Cluster health shows 0 Active shards 

> show log-collector-es-cluster
health

  "cluster_name" : "__pan_cluster__",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 0,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 3136,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 0.0

  •  > show log-collector-es-cluster state routing_table shows no_valid_shard_copy
> show log-collector-es-cluster state routing_table 
...
"allocation_status" : "no_valid_shard_copy"
...
  •  "> debug elasticsearch show certs" gives server error message or expired certs. 
  • Issue is not resolved after running ""> debug elasticsearch certs repair" and es restart as well.
  • The es process shows running in "show system software status"

__pan_cluster__.log (less es-log __pan_cluster__.log ): (only in 11.1 and higher)

03:48:45 Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed 
03:48:45 at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369) ~[?:?]
04:01:53 Caused by: java.security.cert.CertificateExpiredException: NotAfter: Wed Jun 11 04:02:02 PDT 2025
04:01:53 at sun.security.x509.CertificateValidity.valid(CertificateValidity.java:277) ~[?:?]
10:51:27 [2025-07-12T10:51:27,061][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [017507001549]client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.46.27.46:9300, remoteAddress=/10.46.27.46:37635, profile=default}

 



Environment


  • Panorama M-600
  • PAN-OS 11.1.6-hx


Cause


Elasticsearch cluster certificate status displayed with a past expiration date, which caused all shards to be unassigned.



Resolution


  1. The issue is resolved under PAN-275032.
  2. Upgrading to the fixed versions will resolve the issue.
  3. The following versions have the fix
    • 11.1.8
    • 11.2.6
    • 11.2.8
    • 11.1.6-h4
    • 11.1.4-h14
    • 11.1.7-h2
    • 11.2.4-h10 


Additional Information


  • There is a workaround that can be tried by TAC from root
  • This workaround may not work in all cases.
  • Contact Support if you are open to try the workaround. 



Actions
  • Print
  • Copy Link

    https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA1Ki000000kAGBKA2&lang=en_US&refURL=http%3A%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail