Apache - extremely slow initial handshake (SSL enabled)

I have a problem with Apache server. The initial SSL handshake takes about 5-7 seconds which is horrible. After that the response time is in miliseconds - but the messages must be sent right after the first one or the handshake will be done again after a few seconds of inactivity. Is there any way to speed it up with Apache configuration?

This is with some Linux distro, right?

Is this server mostly idle? Perhaps Apache is shoveling randomness from /dev/random. That can be sometimes very slow, since /dev/random requires "real" randomness; stuff like moving the mouse, using the keyboard and network traffic gather the entropy. If the kernel entropy pool is nearly empty, processes requiring randomness tend to stall for a (long) while.

Should this be the situation, cat /proc/sys/kernel/random/entropy_avail should be near zero during SSL handshake.

You may switch the randomness source to /dev/urandom by using SSLRandomSeedApache directive, but be warned, this is theoretically not as secure as /dev/random.

Other solution is to use daemon like rngd (available from rng-tools package in many Linux distros) which fills up entropy pool.

Besides what Janne Pikkarainen said, check also if the server is not doing a DNS reverse lookup for SSL requests. Run a tcpdump -pi eth0 port 53.