Health status of an Elasticsearch cluster in a Panorama Log-Collector Group becomes red when one of the nodes is restarted
17140
Created On 08/26/21 06:07 AM - Last Modified 06/19/23 02:43 AM
Question
Why is the health status of an Elasticsearch cluster in a Panorama Log-Collector Group red when one of the nodes is being restarted?
Environment
- Panorama with Log-Collector Group.
Answer
A Panorama in a Log-Collector Group is restarted. Once the node in an Elasticsearch cluster starts, it starts loading the shards. During this time, Elasticsearch cluster health status is red and this can take 30 min to 6 hours before status becomes green. The time taken is proportional to the amount of data on the box.
The Elasticsearch health status can be checked from this CLI command 'show log-collector-es-cluster health'.
Example:
In this scenario with 2 Elasticsearch nodes, one of the node was restarted. The output below is taken from the node that was NOT restarted.
1) Output of the command when one of the node was restarting.
admin@Panorama> show log-collector-es-cluster health
{
"cluster_name" : "__pan_cluster__",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 14,
"active_shards" : 14,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 62,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 18.421052631578945
}
2) When the Elasticsearch node comes up.
admin@Panorama> show log-collector-es-cluster health
{
"cluster_name" : "__pan_cluster__",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 14,
"active_shards" : 14,
"relocating_shards" : 0,
"initializing_shards" : 32,
"unassigned_shards" : 30,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 32,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 1454,
"active_shards_percent_as_number" : 18.421052631578945
}
3) The Elasticsearch cluster health status becomes green once the shards loading is completed.
admin@Panorama> show log-collector-es-cluster health
{
"cluster_name" : "__pan_cluster__",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 74,
"active_shards" : 76,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 1,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}