Asterisk Intrusion Prevention

You are correct that " that's the way you get an actual phone to connect with the Asterisk server". Phones register to the server on the same port that's used to register with the provider.

If you are only using SIP to talk to the provider, then simply set-up a firewall to only allow SIP from the provider.

Fail2Ban is one solution but gives adversaries 5 chances when they should be getting none.

You can use the iptables firewall to block ALL inbound connections except SIP from the IP that your trunk provider uses.

iptables is shipped with e.g. AsteriskNow but can also be downloaded and installed separately.

I'm not an iptables expert so won't try to give you the full details about how to configure it in case I get it wrong but there is a wealth of information only a google click away and as your rules will be quite simple, it should not be too complicated to set up