ES stuck in red and Active shards shows "0" for M-600 and software version is 11.x
1069
Created On 09/13/25 22:49 PM - Last Modified 04/21/26 22:03 PM
Symptom
- Elastic search on Panorama logger showing red status
- Monitoring tab not displaying traffic
- Log collectors retrieving traffic from firewalls, but Panorama nodes are not processing logs correctly
- Elastic search on the secondary panorama node in red status
- All primary shards are present in unassigned state
- On panorama Elastic search is stuck in red and Cluster health shows 0 Active shards
> show log-collector-es-cluster
health
"cluster_name" : "__pan_cluster__",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 0,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 3136,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 0.0
> show log-collector-es-cluster state routing_table shows no_valid_shard_copy
> show log-collector-es-cluster state routing_table
...
"allocation_status" : "no_valid_shard_copy"
...
- "> debug elasticsearch show certs" gives server error message or expired certs.
- Issue is not resolved after running ""> debug elasticsearch certs repair" and es restart as well.
- The es process shows running in "show system software status"
__pan_cluster__.log (less es-log __pan_cluster__.log ): (only in 11.1 and higher)
03:48:45 Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
03:48:45 at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369) ~[?:?]
04:01:53 Caused by: java.security.cert.CertificateExpiredException: NotAfter: Wed Jun 11 04:02:02 PDT 2025
04:01:53 at sun.security.x509.CertificateValidity.valid(CertificateValidity.java:277) ~[?:?]
10:51:27 [2025-07-12T10:51:27,061][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [017507001549]client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.46.27.46:9300, remoteAddress=/10.46.27.46:37635, profile=default}
Environment
- Panorama M-600
- PAN-OS 11.1.6-hx
Cause
Elasticsearch cluster certificate status displayed with a past expiration date, which caused all shards to be unassigned.
Resolution
- The issue is resolved under PAN-275032.
- Upgrading to the fixed versions will resolve the issue.
- The following versions have the fix
- 11.1.8
- 11.2.6
- 11.2.8
- 11.1.6-h4
- 11.1.4-h14
- 11.1.7-h2
- 11.2.4-h10
Additional Information
- There is a workaround that can be tried by TAC from root
- This workaround may not work in all cases.
- Contact Support if you are open to try the workaround.