enabling xdebug remote debug makes apache server very slow

In my case this was caused by having

xdebug.remote_autostart = 1

set in php.ini. That causes xdebug to try to connect to remote debugger on every request. I had some PHP handled styles, auto_preppend_file and other PHP files in the request and for each of them it waited approximately 2secs, which added up to sth. like 15 seconds or so. Setting

xdebug.remote_autostart = 0

solved the problem completely. xdebug connects only when debug cookie is present. Please note you need to remove the debug cookie/param if you are not in debug session for this fix to work.

Here is my config that I use to setup xdebug.

Experienced also low performance with XDebug (loading Captcha in 6 seconds instead of milliseconds) Remarks on this page got me on the way to identify the cause.

Turned off the profiler and loading time was divided by 3. Still slow, but better already.

xdebug.profiler_enable = 0

Just as further reference... in case anyone has the same/similar problem ... (60 sec. timeout)

First double check xdebug.remote_autostart is disabled to avoid the auto connection.
As @LazyOne pointed out, and @Tomáš Fejfar explains as well.

xdebug.remote_autostart
Type: boolean, Default value: 0
Normally you need to use a specific HTTP GET/POST variable to start remote debugging (see Remote Debugging). When this setting is set to 1, Xdebug will always attempt to start a remote debugging session and try to connect to a client, even if the GET/POST/COOKIE variable was not present.

With this, I recover my normal speed when the debug cookie wasn't present...
But!... I still get very slow response (60 sec. timeout) from the server when the cookie was manually activated.

So, after reading the Xdebug Documentation and checking my config,
I find out that I had enable xdebug.remote_connect_back

xdebug.remote_connect_back
Type: boolean, Default value: 0, Introduced in Xdebug > 2.1
If enabled, the xdebug.remote_host setting is ignored and Xdebug will try to connect to the client that made the HTTP request. It checks the $_SERVER['REMOTE_ADDR'] variable to find out which IP address to use. Please note that there is no filter available, and anybody who can connect to the webserver will then be able to start a debugging session, even if their address does not match xdebug.remote_host.

So all calls to the server was trying to be debugged, making the server very slow and also insecure.

Disable this option and and verified that I had well-defined xdebug.remote_host pointing to my machine, I got an acceptable response ~1sec. and only when the cookie is enabled.

So in brief, my configuration file end up like this:

zend_extension             = "/absolute/path/to/your/xdebug-extension.so"
xdebug.remote_enable       = 1
xdebug.remote_autostart    = 0
xdebug.remote_connect_back = 0
xdebug.remote_host         = "192.168.1.2"
xdebug.remote_port         = 9000
xdebug.remote_handler      = "dbgp"
xdebug.remote_mode         = req
xdebug.remote_log          = "/tmp/xdebug.log"

Note: I made this changes in etc/php5/conf.d/xdebug.ini file and not in php.ini

Edit:
As @Riimu & @jdunk point it out ^{thanks to both}, you may want to set also:
^{* see comments for details}

xdebug.remote_cookie_expire_time = 0
// or
xdebug.remote_cookie_expire_time = -9999

Also, if you do want to leave xdebug.remote_autostart = 1 enabled all the time, in your Phpstorm settings try increasing the max simultaneous sections. This should reduce hanging and blocks, but will still result in a performance impact based on my experience.