Log retention discrepancy between traffic and threat logs in Panorama
9846
Created On 09/21/23 02:29 AM - Last Modified 04/19/24 09:20 AM
Symptom
Threat logs in panorama have shorter retention period compared to traffic logs.
Checking the system logdb quota shows that detailed logs have a retention of 241 days.
> show system logdb-quota
Slot:0
Quotas:
detailed: 60.00%, 1201 GB Expiration-period: 0 days
summary: 30.00%, 600 GB Expiration-period: 0 days
infra_audit: 5.00%, 100 GB Expiration-period: 0 days
platform: 0.10%, 2 GB Expiration-period: 0 days
external: 0.10%, 2 GB Expiration-period: 0 days
Disk usage:
detailed: Logs: 846537 MB, Current Retention: 241 days <-----
summary: Logs: 9946 MB, Current Retention: 353 days
infra_audit: Logs: 82 MB, Current Retention: 368 days
platform: Logs: 0 MB, Current Retention: 0 days
external: Logs: 0 MB, Current Retention: 0 days
Checking traffic logs via GUI shows that the oldest logs is roughly 241 days.
However checking the threat logs only show that the oldest log is only a few months.
Note that the detailed logs contains traffic logs, threat logs, hip match logs and more.
We can configure the detailed logs retention period from panorama.
Panorama > Collector Groups > "CollectorGroup Name" > General > Log Storage Total
Also note that we do not have a separate retention period for Threat and/or Traffic logs as they are combined into detailed logs.
Environment
PANOS-10.2.4, Panorama
Cause
When the detailed logs disk space reaches it quota, it will begin to purge logs and it will do this based on the oldest log.
> debug elasticsearch es-state option indices
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open pan_20221105_68_threat_1234567890-0 lsEvwxqIRAeSoFHp_Ej-1w 1 0 33120 0 1.7gb 1.7gb
green open pan_20221105_68_traffic_1234567890-0 u3G51tyeRX29kHRP14_jJQ 1 0 309729522 0 119.6gb 119.6gb
green open pan_20221107_68_traffic_1234567890-0 sd6CVNzlSEK1Kv3EhpD3Ug 1 0 305644914 0 119.9gb 119.9gb
green open pan_20221110_68_traffic_1234567890-0 ZaAVWBlATxmh2QwwUBkFfA 1 0 304893866 0 118.4gb 118.4gb
green open pan_20221115_68_traffic_1234567890-0 dOLI2h3xTfuqj72YaCw6Bg 1 0 297756328 0 120.1gb 120.1gb
Based from the sample indices above, the oldest logs are on 2022-11-05:
pan_20221105_68_threat_1234567890-0
pan_20221105_68_traffic_1234567890-0
on Jul 26, 2023 the disk quota of the detailed logs was exceeded which prompted the system to remove old indices.
es_purge.log
2023-07-26 17:57:41,066
Disk usage for detailed is 8804723431805, limit is 10002361413058
Disk usage for detailed is 8804723431805, exceeds limit
Found oldest index pan_20221105_68_threat_1234567890-0
deleting index pan_20221105_68_threat_1234567890-0
Found oldest index pan_20221105_68_traffic_1234567890-0
deleting index pan_20221105_68_traffic_1234567890-0
We can see that the threat index pan_20221105_68_threat_1234567890-0 was removed in this case as it was the oldest index. This index contained threat logs from Nov 5, 2022 to Jul 25, 2023.
Resolution
This behavior is expected as retention days is set for aggregated detailed logs. Traffic logs will still have the 241 days retention since it contains the most logs.