Panorama GUI locked out and unable to login with error " Timed out while getting config lock. Please try again"

• Customer might see "Timed out while getting config lock. Please try again" while trying to access panorama GUI 
• configd logs might be filled with below errors 
  

  Error:  pan_cfg_engine_take_rwlock(pan_cfg_engine.c:937): CONFIG_LOCK: read lock TIMEDOUT for cmd: get Error:  pan_cfg_engine_take_rwlock(pan_cfg_engine.c:937): CONFIG_LOCK: read lock TIMEDOUT for cmd: get Error:  pan_cfg_engine_take_rwlock(pan_cfg_engine.c:937): CONFIG_LOCK: read lock TIMEDOUT for cmd: uuid_from_dg_hier Error:  pan_cfg_engine_take_rwlock(pan_cfg_engine.c:937): CONFIG_LOCK: read lock TIMEDOUT for cmd: get Error:  pan_cfg_engine_take_rwlock(pan_cfg_engine.c:937): Error:  pan_cfg_engine_take_rwlock(pan_cfg_engine.c:937): CONFIG_LOCK: write lock TIMEDOUT for cmd: complete CONFIG_LOCK: read lock TIMEDOUT for cmd: get Error:  pan_cfg_engine_take_rwlock(pan_cfg_engine.c:937): CONFIG_LOCK: read lock TIMEDOUT for cmd: get Error:  pan_cfg_engine_take_rwlock(pan_cfg_engine.c:937): CONFIG_LOCK: read lock TIMEDOUT for cmd: deep_search_handler Error:  pan_cfg_engine_take_rwlock(pan_cfg_engine.c:937): CONFIG_LOCK: read lock TIMEDOUT for cmd: uuid_from_dg_hier Error:  pan_cfg_engine_take_rwlock(pan_cfg_engine.c:937): CONFIG_LOCK: read lock TIMEDOUT for cmd: get Error:  pan_cfg_engine_take_rwlock(pan_cfg_engine.c:937): Error:  pan_cfg_engine_take_rwlock(pan_cfg_engine.c:937): CONFIG_LOCK: read lock TIMEDOUT for cmd: get CONFIG_LOCK: read lock TIMEDOUT for cmd: get

   
• max-audit-versions has been set >500 in customer config

• Panorama
• PAN-OS

• The system was configured with a non-default and high value for max-audit-versions:

<max-audit-versions>100000</max-audit-versions>

• This high value prevents the replaydb purge process from functioning as intended.

• As a result, audit entries accumulate over time, increasing load on replay database queries and updates
• Even after restarting configd, the system was only able to purge up to version 31328:

2024-11-11 15:49:38.257 +0000 Removed all the entries with status = 1, rev < 31328 in replaydb.
2024-11-11 15:49:38.257 +0000 Purged entries with version < 31328

• This shows the purge process could not keep up due to the high threshold set by max-audit-versions.
• As a result each replaydb query/update is taking more than a sec. 

2024-11-11T08:38:31.766+0000 I COMMAND  [conn85446] query panreplay.pan_replaydb command: { find: "pan_replaydb", filter: { status: 0 }, sort: { tid: 1 } } planSummary: IXSCAN { tid: 1 } ntoreturn:0 ntoskip:0 keysExamined:20331 docsExamined:20331 cursorExhausted:1 numYields:186 nreturned:11 reslen:63839 locks:{ Global: { acquireCount: { r: 374 } }, Database: { acquireCount: { r: 187 } }, Collection: { acquireCount: { r: 187 } } } 1279ms

• This high latency on these queries contributes directly to MongoDB lock contention, impacting GUI performance

1. Reduce max-audit-versions Setting. The default value is 100. If the customer requires extended audit history, consider setting it to a maximum of 500. This ensures periodic purging occurs and database size remains manageable.
2. Purge ReplayDB Entries Manually using the command "request clean-replay entries all" . After reducing the audit version count and committing the changes and push to devices 
3. Reduce config size if its too high. Review and optimize configuration objects, address groups, rulebases, and other elements to bring the config size under recommended thresholds.