This is the situation we are trying to deal with, high load from AI bot crawlers (left).
When that CPU utilization is under 20% the servers earn 'credit'. That credit can build to a maximum, and is used when the load is over 20%.

The high load causes the slow loading of pages, which may exceed a time setting to cause the 'Gateway' error message. But in the current extreme case, rather than coping with a short peak load, it is persistent. So all of the credit is eventually used, with the very likely display of the Gateway error, until you are lucky enough to get a page delivered.

The effect of taking the servers off-line for maintenance yesterday can be seen with the very low dip. Both in terms of the low load allowing the credit to build, and of throwing off the AI bots. Until they woke up again.

Our amateur skills are not sufficient for solving this problem and we are seeking outside help.