GUI Access Fails with 'Error 504: Gateway Timeout'. HTTP error logs display "Server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting"

• GUI Access to the Palo Alto firewalls fail with 'Error 504: Gateway Timeout'.
• mgmt_httpd_error.log (less mp-log mgmt_httpd_error.log) reports "Server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting" 

  00:15:30.637839 error [7206 7206] prefork.c(806): AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
  00:43:38.814651 error [21292 21292] sapi_apache2.c(356): script '/var/appweb/htdocs/esp/restapi.esp' not found or unable to stat

• Authentication failures for user accounts (e.g., svc-orion) seen repeatedly in authd.log: 

  debug: pan_auth_response_process(pan_auth_state_engine.c:4814): Auth FAILED for user "svc-orion" thru <"Panorama", "shared">: remote server 10.10.10.83 of server profile "RADIUS" is down, or in retry interval, or request timed out (elapsed time 31 secs, max allowed 180 secs)

• Palo Alto Firewalls
• Supported PAN-OS version
• RADIUS authentication

• RADIUS server configured in the authentication profile is unavailable/failed.
• API scripts from multiple IP addresses continued to send frequent (every second) login/authentication requests using accounts configured for RADIUS authentication.
• Each failed authentication attempt triggered a timeout (default ~30 seconds), causing the authd process to hang waiting for a reply.
• These timeouts led to many concurrent, hanging HTTPD processes, eventually reaching the MaxRequestWorkers limit and causing the web interface to time out with error 504.

1. Disable the offending API scripts from sending requests to the firewall using RADIUS-authenticated users.

2. Restart the web-backend process to clear the stuck or hanging httpd worker processes: 

> debug software restart process web-backend

• Multiple httpd processes were spawned under web_backend in mp-monitor.log. 

  12:31:10 processes web_backend        7206  0  8     477652   46648        S          46648
  12:31:10 processes httpd(ppid:7206)   605   0  18    939224   50852        S          50852
  12:31:10 processes httpd(ppid:7206)   1468  0  18    939092   50860        S          50860
  12:31:10 processes httpd(ppid:7206)   1478  0  18    939092   50924        S          50924
  12:31:10 processes httpd(ppid:7206)   1821  0  18    939092   50868        S          50868
  12:31:10 processes httpd(ppid:7206)   1830  0  18    939092   50848        S          50848

The customer should not have any issue with CLI access. Temporary workaround for GUI access (Might not work under all scenarios):

> debug software restart process web-backend
> debug software restart process web-server