Is there an equivalent utility to Linux's tracepath for OS X?

You can use the standard ping that the Mac OSX comes with to find pmtu.

ping -D -s <size> <host> shoud do the trick. Please note, the size is just the ICMP datagram payload size. To see the IP MTU, you need to add 8 bytes of the ICMP header and 20 bytes of the IP header (so for example, to send 1500 Bytes long IP packet, you need to specify size as 1472). Please note, -D is important here, as it sets the Dont-Fragment flag on.

You can also use ping -D -g <minsize> -G <maxsize> -h <sizeincr> <host> to run a ping sweep. This pings your host with packets starting from minsize up to maxsize in sizeincr increments. Again, same rule applies to your sizes - these are just for ICMP payload (so you need to add 28 Bytes to the size specified here for the full IP MTU size).

What you're looking for is an ICMP unreachable message back with a "Fragmentation needed but Don't Fragment flag on" sub-type back. The IP address of the sender of the above ICMP message is the host or router which has its MTU set to lower than the size of your packet, on the link which it would use to forward your packet further towards the destination.

Please note as well, that sometimes (badly misconfigured) firewalls or traffic filters will block all ICMP messages and in such case, you'll just see no response. To test which host dropped the packet, you could run traceroute to your destination and then repeatedly ping every hop on the way with '-D' and a size or a sweep to discover the local MTU of the link that hop is connected to. It will be the link between the last host you can ping and the first host you can't that will have a smaller MTU.

Again - this is all subject to routers handling ICMP, which they might not.

The PMTUD (Path MTU Discovery) protocol works in the same fashion. It uses TCP packets with DF flag on. The TCP sessions endpoints agree on their MSS (Maximum Segment Size, which is MTU less IP and TCP headers, 40B) - they choose the smaller of the MSS values advertised by each host in the TCP handshake. Then they send all traffic with DF flags on, and wait for the ICMP unreachable or a TCP ACK. The latter means the PMTU is sufficient, the former mean they need to drop MSS.

Hope that helps!

traceroute is available on the Mac. Traceroute is actually a more featured version of tracepath.

man traceroute will give you the details but traceroute <host> does the same thing as tracepath <host>.