Summary: End users performing searches on the GSA report receiving error 503 "Service Unavailable"
Cause: The GSA can only handle a certain number of concurrent requests. If the limit of the concurrent requests is reached and no further requests can be queued, the GSA will return the error 503 "Service Unavailable."
Troubleshooting Steps: First ensure that this is a load-related problem. To do so, you need to understand the search load during the time the users report the error.
- If the problem is still ongoing, take a TCP dump and monitor open connections by using Content Sources > Diagnostics > Real-time Diagnostics in the section Run packet capture. You will be able to see the successful and failing connections. One thing to watch for is whether your network is filtering the TCP packets to close the connection. In that case, the search appliance will keep too many connections open and pile these up. The second thing to watch for is the number of concurrent requests to the search appliance.
- If the problem is not happening anymore, you can export the search logs from Reports > Search Logs page in the Admin Console. You can export logs for the last 90 days. A good tool to analyze the logs is searchstats.py from the gsa-admin-toolkit. Searchstats.py allows you to specify the interval you are interested in. It will help you to understand the QPS the search appliance was serving during the time of the problem.
Fix: Depending on the root cause of the problem, you will either need to:
- Add more capacity (i.e. a second appliance and load balance)
- Distribute the incoming load better (i.e. via implementing a reverse proxy with queuing capability)
- Ensure your network properly closes all opened TCP connections
- Determine whether you can optimize your search queries (use less complex search parameters, speed up secure search via implementing ACLs, etc.)