How to set the limits for Jacobian Integration

It's not going to work. Your $r$ limits are way more complicated. You should get $$8 \int_0^{\pi/4}\int_0^{\sec\theta} re^{-r^2}\,dr\,d\theta.$$ Good luck!!