How can I make sure all my Mac's TCP traffic goes through a SOCKS5 proxy?

Why isn't system preferences -> network -> (Select a network on the left side of the window and choose Advanced in the bottom right) -> Proxies (tab at the top) working for you?

If you can set yourself an SSH server, then the free sshuttle can tunnel all TCP traffic through the connection, doing all the firewall work for you.

To forward all TCP traffic and DNS requests to a remote SSH server, the command is simple enough :

sshuttle --dns -vr ssh_server 0/0

Besides TCP and DNS, sshuttle does not forward other requests such as UDP, ICMP, ping etc.

For more information and examples see the article Using Sshuttle in Daily Work.

While setting the system wide proxy settings is a good start, you may also want to look into using iptables to ensure that all traffic is going through the proxy. Some applications do not use system wide configuration settings (Firefox among them), and thus it's imperative that you tailor your rules not to allow direct connections and only to route traffic through the proxy.

EDIT: While I personally use iptables rules to manage potential "leakage" from my VPN, I was actually originally mistaken to think iptables could work with a socks proxy directly. You'll need something like tun2socks in order to make a virtual tunnel interface (such as vpn's use).

Following that, you can set up an iptables script similar to the following:

#!/bin/bash
if [[ $EUID -ne 0 ]]; then
    echo "This script must be run as root" 1>&2
    exit 1
fi

# name of primary network interface (before tunnel)
PRIMARY=eth0

# gateway ip address (before tunnel - adsl router ip address)
# automatically determine the ip from the default route
GATEWAY=`route -n | grep $PRIMARY | egrep "^0\.0\.0\.0" | tr -s " " | cut -d" " -f2`

# provided by tun2socks: interface name
TUNNEL=tun0

# If you'd like, putting the tun2socks command here is a good idea.  It may or may not be necessary to do so, but either way is more convenient than running the two commands separately.

# iptables rules - important!

LOCAL_NET=192.168.0.0/16
#LOCAL_NET=$GATEWAY

# Flush all previous filter rules, you might not want to include this line if you already have other rules setup
iptables -t filter --flush

iptables -t filter -X MYVPN
iptables -t filter -N MYVPN

# Add local routes to routing table
route add -net 192.168.1.0 netmask 255.255.255.0 dev eth0
route add -host 23.21.163.237 dev eth0 gw 192.168.1.1

# Add ssh routes to routing table
ip route add table 128 to 192.168.1.0/24 dev eth0
ip route add table 128 default via 192.168.1.1

# Exceptions for local traffic & vpn server
iptables -t filter -A MYVPN -o lo -j RETURN
iptables -t filter -A MYVPN -o ${TUNNEL} -j RETURN
iptables -t filter -A MYVPN --dst 127.0.0.1 -j RETURN
iptables -t filter -A MYVPN --dst $LOCAL_NET -j RETURN
iptables -t filter -A MYVPN --dst ${SERVER} -j RETURN
iptables -t filter -A MYVPN --dst ${VPN_SERVER} -j RETURN

# Add extra local nets here as necessary

iptables -t filter -A MYVPN -j DROP

# MYVPN traffic leaving this host:
iptables -t filter -A OUTPUT -p tcp --syn -j MYVPN
iptables -t filter -A OUTPUT -p icmp -j MYVPN
iptables -t filter -A OUTPUT -p udp -j MYVPN

Naturally you'll want to make this script reflects your particular network (ie, if you're using something like a 192.168.0.0/24 subnet, adjust accordingly). Also, it is very tightly based on a script I use with a VPN, hence, all the mentions MYVPN or VPN -- while you're not using a VPN, tun2socks effectively behaves as if you are, so everything should work the same.

And special thanks to this answer over at Unix.SE for steering me in the right direction to answer this one.

EDIT again: So, looks like OS X actually would be doing this with ipfw rather than iptables (sorry, I'm mostly a linux person, and thought OS X had iptables available). There are equivalencies such that the script can be adapted, some of which are pointed out here. man ipfw should set you straight on the syntax. I'll leave the original iptables script up as a template so you can see what is going on conceptually. WaterRoof appears like it may help make using ipfw a little more user friendly; other front ends may be available as well.