Unable to see logs on Panorama older than a certain time period.
9854
Created On 01/13/23 10:28 AM - Last Modified 06/24/24 23:41 PM
Symptom
- Unable to see logs on Panorama older than a certain time period.
- "show log-collector-es-cluster health" display "green indicating no health issues.
- "show system search-engine-quota" does not display any issues either.
- "reportd.log"(less mp-log reportd.log), display ES_ERROR_INDEX_NOT_FOUND.
--OUTPUT OMITTED----
* Connection #0 to host 127.0.0.1 left intact
2022-12-23 15:52:50.775 +0100 Error: es_check_response_error(pan_es_curl_wrapper.c:3476): ES(es_check_response_error): error [ES_ERROR_INDEX_NOT_FOUND] returned for request
--OUTPUT OMITTED----
Environment
- Any Panorama
- PAN-OS 10.x
- Log Collector configured (local or dedicated)
Cause
- There are no indices for the logs being queried.
- This can be verified by running command "debug elasticsearch es-state option indices".
- If there is no index for the time/day being queried, it means ES indexed logs are lost.
Note: Prior to PanOS 10.1, indices can be checked under /tmp/cli/logs/es_stats.txt in tech-support file.
Resolution
- Re-index the raw logs by following method:
- Pre PAN-OS-9.1, contact TAC support to re-index logs.
- PAN-OS 9.1.3 and greater - Use following command to selectively regenerate logs for a certain time period
admin@Panorama> debug logdb migrate-lc start log-type <type> time period start-date <yyyy-mm-dd> end-date <yyyy-mm-dd>
Example:
admin@Panorama>debug logdb migrate-lc start log-type all time period start-date 2020-02-20 end-date 2020-02-21