High latency/drops between Cisco switches in two locations. How to troubleshoot?

Cisco switches puts ICMP at the bottom of the priority list. We get the same results if we ping a busy 3750-X.

You need to look at the system utilization on the switches, as I suspect they are so busy that they are doing software processing of packets. Are you running any kind of layer 3 services on these?

There is a quite serious bug in IOS 12.2.53:

CSCth24278 (Catalyst 2960-S switches)

The CPU utilization on the switch remains high (50 to 60 percent) when the switch is not being accessed by a telnet or a console session. When you telnet or console into the switch, the CPU utilization goes down.

There is no workaround.

Upgrade to 12.2.58-SE1 or later to fix this situation.