Search returns error "503: Service Unavailable"
Summary: End users performing searches on the GSA report receiving error 503 "Service Unavailable"
Cause: The GSA can only handle a certain number of concurrent requests. If the limit of the concurrent requests is reached and no further requests can be queued, the GSA will return the error 503 "Service Unavailable."
Troubleshooting Steps: The first thing to do is to ensure that it is a load-related problem. To do so, you need to understand the search load during the time the users report the error.
- If the problem is still ongoing, take a TCP dump and monitor open connections by using Content Sources > Diagnostics > Real-time Diagnostics (see 7.0 to 7.2 Admin Console mapping for prior page location) in the section Run packet capture. This way you will be able to see the successful and failing connections. One thing to watch out for is whether your network is filtering the TCP packets to close the connection. In that case, the search appliance will keep too many connections open and pile these up. The second thing is to watch out for is the number of concurrent requests to the search appliance.
- If the problem is not happening anymore, you can export the search logs from Reports > Search Logs page in Admin Console. You can export logs for the last 90 days. A good tool to analyze the logs is searchstats.py from the gsa-admin-toolkit. Searchstats.py allows you to specify the interval you are interested in. It will help you to understand the QPS the search appliance was serving during the time of the problem.
Fix: Depending on the root cause of the problem, you will either need to:
- add more capacity (i.e. a second appliance and load balance)
- distribute the incoming load better (i.e. via implementing a reverse proxy with queuing capability)
- ensure your network properly closes all opened TCP connections
- look whether you can optimize your search queries (use less complex search parameters, speed up secure search via implementing ACLs, etc.)