Log Collector Running PAN OS 11.1 Could Not Join the Cluster with Error "master_not_discovered

Log Collector Running PAN OS 11.1 Could Not Join the Cluster with Error "master_not_discovered_exception"

11188

Created On 05/29/24 21:51 PM - Last Modified 08/28/24 22:21 PM

Collector Group

Log Collector

Elastic Search

11.1

Panorama

Symptom

Log collector is up and connected to Panorama but ES status under Panorama > Managed Collectors > Health Status shows red circle and "not running".

When running show-log-collector-es-cluster health command, it takes a while to return an output or eventually returning error as follow,

{
   "error" : {
       "root_cause" : [
           {
              "type" : "master_not_discovered_exception",
              "reason" : null
           }
       ],
       "type" : "master_not_discovered_exception",
       "reason" : null
   },
   "status" : 503
}

When checking es_restart.log (less mp-log es_restart.log), master_not_discovered_exception error is seen, for example,

ERROR Unable to load template /opt/pancfg/mgmt/saved-configs/elasticsearch/templates/system_<serial_number>. {"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

Netstat output (show netstat numeric yes programs yes | match 9300 command output) on log collector shows SYN_SENT towards other log collector(s) in the same cluster.

> show netstat numeric yes programs yes | match 9300
tcp6       0      1 <local_LC_IP>:44095     <remote_LC_IP>:9300      SYN_SENT    43158/java          
tcp6       0      1 <local_LC_IP>:46397     <remote_LC_IP>:9300      SYN_SENT    41947/java          
tcp6       0      1 <local_LC_IP>:42961     <remote_LC_IP>:9300      SYN_SENT    42785/java

Environment

Panorama in mixed mode or logger mode
2-nodes collector group or more
PAN OS 11.1 or later

Cause

TCP port 9300 - 9302 for inter-LC communication is blocked by intermediate device between log collectors.

Resolution

Allow TCP port 9300 - 9302 for inter-LC (log collector) communication on all intermediate devices
Since PAN OS 11.1, inter-LC communication is no longer using SSH tunnels over TCP port 28.

Additional Information