GUI Access Fails with 'Error 504: Gateway Timeout'. HTTP error logs display "Server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting"

• GUI Access to the Palo Alto firewalls fail with 'Error 504: Gateway Timeout'.
• mgmt_httpd_error.log (less mp-log mgmt_httpd_error.log) reports "Server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting" 

  00:15:30.637839 error [7206 7206] prefork.c(806): AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
  00:43:38.814651 error [21292 21292] sapi_apache2.c(356): script '/var/appweb/htdocs/esp/restapi.esp' not found or unable to stat

• Authentication failures for user accounts (e.g., svc-orion) seen repeatedly in authd.log: 

  debug: pan_auth_response_process(pan_auth_state_engine.c:4814): Auth FAILED for user "svc-orion" thru <"Panorama", "shared">: remote server 10.10.10.83 of server profile "RADIUS" is down, or in retry interval, or request timed out (elapsed time 31 secs, max allowed 180 secs)

• Palo Alto Firewalls
• Supported PAN-OS
• RADIUS authentication
• API Management for PAN-OS

• RADIUS server configured in the authentication profile is unavailable/failed.
• API scripts from multiple IP addresses continued to send frequent (every second) login/authentication requests using accounts configured for RADIUS authentication.
• Each failed authentication attempt triggered a timeout (default ~30 seconds), causing the authd process to hang waiting for a reply.
• These timeouts led to many concurrent, hanging HTTPD processes, eventually reaching the MaxRequestWorkers limit and causing the web interface to time out with error 504.

1. Disable the offending API scripts from sending requests to the firewall using RADIUS-authenticated users.
2. Restart the web-backend process to clear the stuck or hanging httpd worker processes: 

> debug software restart process web-backend

3. Tune monitoring/automation scripts on the identified source IPs to remain below 5 requests per second.
4. Check whether the API clients are configured to use HTTP Keep-Alive (persistent connections)? This would help reduce the process overhead caused by multiple short-lived connections.

Note:

• Palo Alto suggests to limit the number of concurrent API calls to five
• What is the limit for API calls to PAN-OS and Panorama?

• Multiple httpd processes are spawned under web_backend and seen in the mp-monitor.log. 

  12:31:10 processes web_backend        7206  0  8     477652   46648        S          46648
  12:31:10 processes httpd(ppid:7206)   605   0  18    939224   50852        S          50852
  12:31:10 processes httpd(ppid:7206)   1468  0  18    939092   50860        S          50860
  12:31:10 processes httpd(ppid:7206)   1478  0  18    939092   50924        S          50924
  12:31:10 processes httpd(ppid:7206)   1821  0  18    939092   50868        S          50868
  12:31:10 processes httpd(ppid:7206)   1830  0  18    939092   50848        S          50848

• No issue with the CLI access. Temporary workaround for GUI access is to restart the web-server/backend. Although it may not work in all scenarios.

  > debug software restart process web-backend
  > debug software restart process web-server