Elastix(Asterisk) Slow to repsond on incoming PSTN line

I setup a PBX with asterisk and Elastix as a web GUI. I have all incoming calls set to ring to a ring group of our office personnel in a ringall format. The only issue is the time for Asterisk to pickup and pass the phone call to the VOIP phones to ring from our PSTN lines. It takes about 5 rings to start ringing our VOIP phones in the office and it goes to our carrier's voicemail before we get a change to even pickup the phone. I'm not sure why Asterisk is taking so long to pass on the call. We are using Siemens Gigaset DE380 IP R phones with a Digium TD410 card.

Could it be asterisk is expecting caller-id but not receiving it? Just a guess as most caller-id on PSTN lines should happen somewhere around the first ring.