Logs not visible in Panorama when blocks to be ingested in Log Collector exceed the circuit breaker limit

• Logs of a specific Log Collector do not show under the monitor tab of Panorama.
• Log Collector is connected, in sync and receives logs from the firewalls.
• Elasticsearch health status is green.
• The following message can be seen in vld logs: "Error writing block to ES"
• The following message can be seen in __pan_cluster__.log (less es-log __pan_cluster__.log):
• Note: On M-600 and M-700 Panoramas there are 2 more instances of Elasticsearch, so run the commands for all:
  • less es-log __pan_cluster__.log,
  • less es-1-log __pan_cluster__.log
  • less es-2-log __pan_cluster__.log

[xxxx-10-02T15:45:12,646][DEBUG][o.e.a.a.i.s.TransportIndicesStatsAction] [01750700xxxx]failed to execute [indices:monitor/stats] on node [9fG4kUmsSuS9jAUn
bUxxxx]
org.elasticsearch.transport.RemoteTransportException: [01750700xxxx-2][127.0.0.1:95xx][indices:monitor/stats[n]]
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [<transport_request>] would be [22329173281/20.7gb]
, which is larger than the limit of [22328744345/20.7gb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=425663/415.6kb, accounting=22328747618/2
0.7gb]

• A similar message can be found in reportd.log (less mp-log reportd.log)

        "error":        {
                "root_cause":   [{
                                "type": "circuit_breaking_exception",
                                "reason":       "[parent] Data too large, data for [<transport_request>] would be [22328746211/20.7gb], which is larger tha
n the limit of [22328744345/20.7gb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=534/534b, accounting=22328745677/20.7gb]",
                                "bytes_wanted": 22328746211,
                                "bytes_limit":  22328744345
                        },

• Any Panorama
• PAN-OS 11.1.x version

• Elasticsearch is running out of usable memory and uses the circuit breaker to protect itself from large block ingestion.
• The default limit for the parent circuit breaker is 70%.

As a workaround, the circuit breaker limit can be increased:

1. Increase the memory limit using the command below; in most cases, increasing the value from 70% to 73% resolves the issue.
   

   > debug elasticsearch circuit-breaker limit 73

2. Restart reportd process with the following command:

   > debug software restart process reportd

3. Wait for reportd process to be in running state again:

   > show system software status

4. Check that logs are now visible in Panorama.

If the issue is seen in any other version of PAN-OS, contact Palo Alto Support for assistance.