Why is there less latency on loopback than on a carp interface?

A constant 100ms delay looks weird. It looks like packets get buffered and not immediately delivered. Or maybe some of them are dropped and retransmitted. Can you run tcpdump on this interface to show the problem ? I don't know how the IP stack works on FreeBSD, nor how CARP is implemented, but would it be possible for instance that the slave regularly advertises its MAC address with gratuitous ARPs and that the master alternatively sends packets to each side ?

Could you also run tcpdump on the real interface to ensure that nothing gets emitted out ?

Is it possible that the system refrains from caching a CARP device's ARP entry, thus causing an ARP request to be emitted for each packet of a session, that the CARP daemon would have to answer ?

Most of those are some stupid ideas, but it's in order to help you search in the right direction.

Just for clarity, you only changed how it was being accessed, from the 127 address, to the local IP; correct?

If that's the case and it made a difference, something ain't right. Check your routing table with netstat -rn and see what the local IPs are routed to, it should be routed to the lo0 interface (just like 127).

Your netstat -rn output should be vaguely similar to this:

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            1.2.3.1            UGS       131  2655014   nge1
1.2.3.0/23         link#2             U           0       88   nge1
1.2.3.4            link#2             UHS         0    34848    lo0
127.0.0.1          link#5             UH          0    64678    lo0
192.168.0.0/26     link#1             U           2 41703537   nge0
192.168.0.1        link#1             UHS         0    70088    lo0