Routing traffic for specific port range
I have a Ubiquiti Dream Machine (UDM) which is part of a project that is replacing a network topology with a different one. This involves both Internet traffic and VoIP (Asterisk). During the phased introduction of the new topology, I have the following requirement due to limitations we have when configuring the Asterisk server:
On the UDM I need to re-route all UDP traffic pointed at 10.0.0.1, in ports 50000-55000, to 10.0.10.1 on interface br8
I doubt this can be achieved from the UI, but I have SSH access to the UDM where I can touch up the iptables and ip route configurations.
On the UDM, the routing table is like this:
# ip route
10.0.0.0/24 dev br3 proto kernel scope link src 10.0.0.1
10.0.1.0/24 dev br5 proto kernel scope link src 10.0.1.1
10.0.2.0/24 dev br6 proto kernel scope link src 10.0.2.1
10.0.3.0/24 dev br4 proto kernel scope link src 10.0.3.1
10.0.10.0/24 dev br8 proto kernel scope link src 10.0.10.1
10.1.1.0/24 dev br0 proto kernel scope link src 10.1.1.1
10.2.2.0/24 dev br2 proto kernel scope link src 10.2.2.1
192.168.1.0/24 dev eth4 proto kernel scope link src 192.168.1.86
An important note on topology: the 10.0.10.0/24 network is the one bridging VoIP traffic between the old and the new topologies, The old VoIP server there has the following routing table:
# ip route
192.168.1.248/29 dev eth1 proto kernel scope link src 192.168.1.250
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.1
10.0.1.0/24 dev eth3 proto kernel scope link src 10.0.1.1
10.0.10.0/24 dev eth2 proto kernel scope link src 10.0.10.10
10.0.0.0/8 via 10.0.10.1 dev eth2
default via 192.168.1.254 dev eth1
Some networks appear conflicted, but they're not really talking to each other. The only thing talking between these two servers, is VLAN br8 on the UDM which connects to eth2 on the old server, for VoIP traffic only.
The routing is working as I intend it to. VoIP is going through the server at 10.0.10.10, visible from both the old and the new world. When I place a VoIP call, the SIP negotiation happens correctly, meaning that each phone is able to see the server and talk both ways.
My only problem is that when the RTP voice traffic starts flowing, one of the phones sometimes (depending on which party initiates the call) sends it to the old VoIP server address (was 10.0.0.1) instead of to the new one, 10.0.10.10. This happens despite being re-configured to use the new address, and despite having just done the SIP negotiation with the new address! It's an asterisk configuration problem that sends the wrong address inside the SIP invite, fooling the phone. The Asterisk server thinks it's all the same because his own identity is both 10.0.0.1 and 10.0.10.10, but that is true only in the old world, not in the new...
So during the one-way audio calls, I see this kind of traffic on the UDM:
# tcpdump -q -n -c 20 -i any host 10.0.10.10 or host 10.0.2.1833 or host 10.0.10.47
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
12:20:55.545461 IP 10.0.10.47.5032 > 10.0.0.1.50042: UDP, length 172
12:20:55.545461 IP 10.0.10.47.5032 > 10.0.0.1.50042: UDP, length 172
12:20:55.545461 IP 10.0.10.47.5032 > 10.0.0.1.50042: UDP, length 172
12:20:55.565464 IP 10.0.10.47.5032 > 10.0.0.1.50042: UDP, length 172
12:20:55.565464 IP 10.0.10.47.5032 > 10.0.0.1.50042: UDP, length 172
12:20:55.565464 IP 10.0.10.47.5032 > 10.0.0.1.50042: UDP, length 172
12:20:55.570260 IP 10.0.2.183.50030 > 10.0.10.10.54736: UDP, length 172
12:20:55.570260 IP 10.0.2.183.50030 > 10.0.10.10.54736: UDP, length 172
12:20:55.570313 IP 10.0.2.183.50030 > 10.0.10.10.54736: UDP, length 172
12:20:55.570317 IP 10.0.2.183.50030 > 10.0.10.10.54736: UDP, length 172
12:20:55.570764 IP 10.0.2.183.50030 > 10.0.10.10.54736: UDP, length 172
12:20:55.570764 IP 10.0.2.183.50030 > 10.0.10.10.54736: UDP, length 172
12:20:55.570807 IP 10.0.2.183.50030 > 10.0.10.10.54736: UDP, length 172
12:20:55.570810 IP 10.0.2.183.50030 > 10.0.10.10.54736: UDP, length 172
12:20:55.571077 IP 10.0.2.183.50030 > 10.0.10.10.54736: UDP, length 172
12:20:55.571077 IP 10.0.2.183.50030 > 10.0.10.10.54736: UDP, length 172
12:20:55.571109 IP 10.0.2.183.50030 > 10.0.10.10.54736: UDP, length 172
12:20:55.571111 IP 10.0.2.183.50030 > 10.0.10.10.54736: UDP, length 172
12:20:55.585465 IP 10.0.10.47.5032 > 10.0.0.1.50042: UDP, length 172
12:20:55.585465 IP 10.0.10.47.5032 > 10.0.0.1.50042: UDP, length 172
The traffic going to 10.0.10.10 is ok, the one going to 10.0.0.1 is not. I'd like to redirect it but I need your help.
(I know a solution could, or even should, be attempted by fixing Asterisk server config somehow, but there are limitations there, and I am charged only with the re-routing solution. Asking about the re-routing is the sole purpose of my question here).
The current iptables configuration on the UDM is this:
# iptables-save
# Generated by iptables-save v1.6.1 on Thu May 13 12:28:34 2021
*nat
:PREROUTING ACCEPT [149794:35652606]
:INPUT ACCEPT [59043:3818189]
:OUTPUT ACCEPT [205240:13204474]
:POSTROUTING ACCEPT [179113:10974505]
:UBIOS_INPUT_JUMP - [0:0]
:UBIOS_OUTPUT_JUMP - [0:0]
:UBIOS_POSTROUTING_JUMP - [0:0]
:UBIOS_POSTROUTING_USER_HOOK - [0:0]
:UBIOS_PREROUTING_JUMP - [0:0]
-A PREROUTING -j UBIOS_PREROUTING_JUMP
-A INPUT -j UBIOS_INPUT_JUMP
-A OUTPUT -j UBIOS_OUTPUT_JUMP
-A POSTROUTING -j UBIOS_POSTROUTING_JUMP
-A UBIOS_POSTROUTING_JUMP -j UBIOS_POSTROUTING_USER_HOOK
-A UBIOS_POSTROUTING_USER_HOOK -o eth4 -m comment --comment 00000001095216660481 -j MASQUERADE
COMMIT
# Completed on Thu May 13 12:28:34 2021
# Generated by iptables-save v1.6.1 on Thu May 13 12:28:34 2021
*mangle
:PREROUTING ACCEPT [21051971:19450922056]
:INPUT ACCEPT [11815594:10852127372]
:FORWARD ACCEPT [9222781:8597163389]
:OUTPUT ACCEPT [11762182:10802741148]
:POSTROUTING ACCEPT [21125153:19406413016]
COMMIT
# Completed on Thu May 13 12:28:34 2021
# Generated by iptables-save v1.6.1 on Thu May 13 12:28:34 2021
*filter
:INPUT ACCEPT [11703464:10819984868]
:FORWARD ACCEPT [9222781:8597163389]
:OUTPUT ACCEPT [11762124:10802734194]
:UBIOS_FORWARD_IN_USER - [0:0]
:UBIOS_FORWARD_JUMP - [0:0]
:UBIOS_FORWARD_OUT_USER - [0:0]
:UBIOS_FORWARD_USER_HOOK - [0:0]
:UBIOS_INPUT_JUMP - [0:0]
:UBIOS_INPUT_USER_HOOK - [0:0]
:UBIOS_IN_GEOIP - [0:0]
:UBIOS_LAN_IN_USER - [0:0]
:UBIOS_LAN_LOCAL_USER - [0:0]
:UBIOS_LAN_OUT_USER - [0:0]
:UBIOS_OUTPUT_JUMP - [0:0]
:UBIOS_OUTPUT_USER_HOOK - [0:0]
:UBIOS_OUT_GEOIP - [0:0]
:UBIOS_WAN_IN_USER - [0:0]
:UBIOS_WAN_LOCAL_USER - [0:0]
:UBIOS_WAN_OUT_USER - [0:0]
-A INPUT -j UBIOS_INPUT_JUMP
-A FORWARD -j UBIOS_FORWARD_JUMP
-A OUTPUT -j UBIOS_OUTPUT_JUMP
-A UBIOS_FORWARD_IN_USER -i eth4 -m comment --comment 00000001095216663481 -j UBIOS_WAN_IN_USER
-A UBIOS_FORWARD_IN_USER -i br0 -m comment --comment 00000001095216663482 -j UBIOS_LAN_IN_USER
-A UBIOS_FORWARD_IN_USER -i br2 -m comment --comment 00000001095216663483 -j UBIOS_LAN_IN_USER
-A UBIOS_FORWARD_IN_USER -i br3 -m comment --comment 00000001095216663484 -j UBIOS_LAN_IN_USER
-A UBIOS_FORWARD_IN_USER -i br4 -m comment --comment 00000001095216663485 -j UBIOS_LAN_IN_USER
-A UBIOS_FORWARD_IN_USER -i br5 -m comment --comment 00000001095216663486 -j UBIOS_LAN_IN_USER
-A UBIOS_FORWARD_IN_USER -i br6 -m comment --comment 00000001095216663487 -j UBIOS_LAN_IN_USER
-A UBIOS_FORWARD_IN_USER -i br8 -m comment --comment 00000001095216663488 -j UBIOS_LAN_IN_USER
-A UBIOS_FORWARD_JUMP -j UBIOS_FORWARD_USER_HOOK
-A UBIOS_FORWARD_OUT_USER -o eth4 -m comment --comment 00000001095216663481 -j UBIOS_WAN_OUT_USER
-A UBIOS_FORWARD_OUT_USER -o br0 -m comment --comment 00000001095216663482 -j UBIOS_LAN_OUT_USER
-A UBIOS_FORWARD_OUT_USER -o br2 -m comment --comment 00000001095216663483 -j UBIOS_LAN_OUT_USER
-A UBIOS_FORWARD_OUT_USER -o br3 -m comment --comment 00000001095216663484 -j UBIOS_LAN_OUT_USER
-A UBIOS_FORWARD_OUT_USER -o br4 -m comment --comment 00000001095216663485 -j UBIOS_LAN_OUT_USER
-A UBIOS_FORWARD_OUT_USER -o br5 -m comment --comment 00000001095216663486 -j UBIOS_LAN_OUT_USER
-A UBIOS_FORWARD_OUT_USER -o br6 -m comment --comment 00000001095216663487 -j UBIOS_LAN_OUT_USER
-A UBIOS_FORWARD_OUT_USER -o br8 -m comment --comment 00000001095216663488 -j UBIOS_LAN_OUT_USER
-A UBIOS_FORWARD_USER_HOOK -m comment --comment 00000001095216663481 -j UBIOS_FORWARD_IN_USER
-A UBIOS_FORWARD_USER_HOOK -m comment --comment 00000001095216663482 -j UBIOS_FORWARD_OUT_USER
-A UBIOS_INPUT_JUMP -j UBIOS_INPUT_USER_HOOK
-A UBIOS_INPUT_USER_HOOK -i eth4 -m comment --comment 00000001095216663481 -j UBIOS_WAN_LOCAL_USER
-A UBIOS_INPUT_USER_HOOK -i br0 -m comment --comment 00000001095216663482 -j UBIOS_LAN_LOCAL_USER
-A UBIOS_INPUT_USER_HOOK -i br2 -m comment --comment 00000001095216663483 -j UBIOS_LAN_LOCAL_USER
-A UBIOS_INPUT_USER_HOOK -i br3 -m comment --comment 00000001095216663484 -j UBIOS_LAN_LOCAL_USER
-A UBIOS_INPUT_USER_HOOK -i br4 -m comment --comment 00000001095216663485 -j UBIOS_LAN_LOCAL_USER
-A UBIOS_INPUT_USER_HOOK -i br5 -m comment --comment 00000001095216663486 -j UBIOS_LAN_LOCAL_USER
-A UBIOS_INPUT_USER_HOOK -i br6 -m comment --comment 00000001095216663487 -j UBIOS_LAN_LOCAL_USER
-A UBIOS_INPUT_USER_HOOK -i br8 -m comment --comment 00000001095216663488 -j UBIOS_LAN_LOCAL_USER
-A UBIOS_LAN_IN_USER -d 10.0.10.10/32 -j LOG
-A UBIOS_LAN_IN_USER -d 10.0.10.10/32 -m comment --comment 00000001095216662480 -j RETURN
-A UBIOS_LAN_IN_USER -s 10.0.10.10/32 -j LOG
-A UBIOS_LAN_IN_USER -s 10.0.10.10/32 -m comment --comment 00000001095216662481 -j RETURN
-A UBIOS_LAN_IN_USER -s 10.0.2.0/24 -m comment --comment 00000001095216666481 -j RETURN
-A UBIOS_LAN_IN_USER -s 10.0.3.0/24 -m comment --comment 00000001095216666482 -j RETURN
-A UBIOS_LAN_IN_USER -s 10.0.1.0/24 -m comment --comment 00000001095216666483 -j RETURN
-A UBIOS_LAN_IN_USER -s 10.2.2.0/24 -m comment --comment 00000001095216666484 -j RETURN
-A UBIOS_LAN_IN_USER -s 10.0.10.0/24 -m comment --comment 00000001095216666485 -j RETURN
-A UBIOS_LAN_IN_USER -s 10.1.1.0/24 -m comment --comment 00000001095216666486 -j RETURN
-A UBIOS_LAN_IN_USER -s 10.0.0.0/24 -m comment --comment 00000001095216666487 -j RETURN
-A UBIOS_LAN_IN_USER -j LOG
-A UBIOS_LAN_IN_USER -m comment --comment 00000001097364144127 -j RETURN
-A UBIOS_LAN_LOCAL_USER -j LOG
-A UBIOS_LAN_LOCAL_USER -m comment --comment 00000001097364144127 -j RETURN
-A UBIOS_LAN_OUT_USER -d 10.0.2.0/24 -m comment --comment 00000001095216666481 -j RETURN
-A UBIOS_LAN_OUT_USER -d 10.0.3.0/24 -m comment --comment 00000001095216666482 -j RETURN
-A UBIOS_LAN_OUT_USER -d 10.0.1.0/24 -m comment --comment 00000001095216666483 -j RETURN
-A UBIOS_LAN_OUT_USER -d 10.2.2.0/24 -m comment --comment 00000001095216666484 -j RETURN
-A UBIOS_LAN_OUT_USER -d 10.0.10.0/24 -m comment --comment 00000001095216666485 -j RETURN
-A UBIOS_LAN_OUT_USER -d 10.1.1.0/24 -m comment --comment 00000001095216666486 -j RETURN
-A UBIOS_LAN_OUT_USER -d 10.0.0.0/24 -m comment --comment 00000001095216666487 -j RETURN
-A UBIOS_LAN_OUT_USER -j LOG
-A UBIOS_LAN_OUT_USER -m comment --comment 00000001097364144127 -j RETURN
-A UBIOS_WAN_IN_USER -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment 00000001095216663481 -j RETURN
-A UBIOS_WAN_IN_USER -m conntrack --ctstate INVALID -m comment --comment 00000001095216663482 -j DROP
-A UBIOS_WAN_IN_USER -m comment --comment 00000001097364144127 -j DROP
-A UBIOS_WAN_LOCAL_USER -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment 00000001095216663481 -j RETURN
-A UBIOS_WAN_LOCAL_USER -m conntrack --ctstate INVALID -m comment --comment 00000001095216663482 -j DROP
-A UBIOS_WAN_LOCAL_USER -m comment --comment 00000001097364144127 -j DROP
-A UBIOS_WAN_OUT_USER -m comment --comment 00000001097364144127 -j RETURN
COMMIT
I read online about the possibility to do what I intend by:
- marking the packets to be routed in iptables (using mangle)
- using a second routing table for those packets (adding to /etc/iproute2/rt_tables)
- adjusting that table to send stuff where it needs to go
While I understand the general idea, I am no networks specialist, I am new to these concepts and I get lost in the details. I am not sure how to mark, and how much of the main routing table needs to be repeated in the second table. I would appreciate some specific help with these commands and configurations. Thanks in advance!
So, how can I re-route all UDP traffic on the UDM pointed at 10.0.0.1, in ports 50000-55000, to 10.0.10.1 on interface br8?
Solution 1:
This is a task for 'policy based routing' Policy based routing allows you to configure complex routing scenarios. For example, you can route packets based on various criteria, such as the source address, packet metadata, and including protocol.
I am not sure if it can be done on the UDM - but here is a link for the USG:
https://help.ui.com/hc/en-us/articles/360005460813-UniFi-USG-Advanced-Policy-Based-Routing-
Solution 2:
I got it working, finally, so I'll be answering my own question. The solution described as sequence of terminal commands is not permanent, it won't survive reboots or upgrades, but I wanted to focus pedagogically on the routing solution (that's the title of my question) and leave the permanence problems for a script at the end (more briefly explained, since there are other places on the Internet helping to solve that problem).
Should you be doing the same as me and using this solution? Probably not. There are "caveats", see some of the comments on the question. I will try to explain how to do the routing trick, but I am not necessarily recommending it for other situations...
Create a new routing table:
echo "74 myroutes" >> /etc/iproute2/rt_tables
Check with
cat /etc/iproute2/rt_tables
Add default route to new routing table:
ip route add default via 10.0.10.10 dev br8 table myroutes
Check with
ip route list table myroutes
Note that I can keep this quite simple because only a handful of very specific packets are going through here (only to specific host, through specific ports). So the stuff that will use this table can all go in the same direction, since it's already thoroughly been "pre-selected" by iptables marks, see below.
Mark the selected packets in iptables
iptables -t mangle -I PREROUTING -p udp -m udp --dport 50000:55000 -d 10.0.0.1 -j MARK --set-mark 0x74
Check your criteria by adding also a logging line:
iptables -t mangle -I PREROUTING -p udp -m udp --dport 50000:55000 -d 10.0.0.1 -j LOG --log-prefix "I just marked a packet: "
(Remember to remove that line afterwards. To delete iptables mangle rules I didiptables -L -t mangle -n -v --line-numbers and then used the rule number to do something like
iptables -t mangle -D PREROUTING 2)
Make sure the intended packets are getting correctly marked before proceeding!
Use ip rule to tell the kernel to use your new routing table for those marked packets
So I'm starting with these rules:
# ip rule list
0: from all lookup local
32000: from all lookup main
32500: from 192.168.1.86 lookup 201
32766: from all lookup 201
32767: from all lookup default
I need to insert my rule before that first one, which makes it a bit more complicated than just adding a new rule (thanks to @A.B for the help on this in Making an `ip rule` higher priority than `local`).
A word of WARNING, don't lock yourself out of the server by messing the local rule, which will break all your networking into the UDM. And networking is the only way into the UDM :-)... so the order of the following commands matters, the delete can only be done after adding the same local rule at a new "preference" position:
ip rule add pref 10 lookup local
ip rule add pref 5 fwmark 0x74 lookup myroutes
ip rule delete pref 0
So let's check what we have now:
# ip rule list
5: from all fwmark 0x74 lookup myroutes
10: from all lookup local
32000: from all lookup main
32500: from 192.168.1.86 lookup 201
32766: from all lookup 201
32767: from all lookup default
Check by contrasting the results of these two commands:
# ip route get 10.0.0.1
local 10.0.0.1 dev lo src 10.0.0.1
cache <local>
# ip route get 10.0.0.1 mark 0x74
10.0.0.1 via 10.0.10.10 dev br8 src 10.0.10.1 mark 0x74
cache
That's it! Everything's working now and the RTP traffic flows to the correct place again, despite the Asterisk server's buggy configuration.
Making changes permanent across reboots and upgrades
Very briefly (not the main point of this question) here's how to make these changes permanent on a Ubiquiti UDM. You must first install udm-utilities and save the script below with a name like
/mnt/data/on_boot.d/10-add-myroutes.sh
and make it executable with a chmod 700. It will run as root.
The script below executes the above commands at boot time. Where needed, there is some trickery to ensure that it can be run more than once without creating duplicate entries.
#!/bin/sh
# fail-safe mechanism in case our boot script is causing broken networking
# and we need to reboot and rescue by killing script before it runs:
sleep 180
grep -qxF '74 myroutes' /etc/iproute2/rt_tables || echo "74 myroutes" >> /etc/iproute2/rt_tables
ip route add default via 10.0.10.10 dev br8 table myroutes
iptables -t mangle -C PREROUTING -p udp -m udp --dport 50000:55000 -d 10.0.0.1 -j MARK --set-mark 0x74 || iptables -t mangle -I PREROUTING -p udp -m udp --dport 50000:55000 -d 10.0.0.1 -j MARK
ip rule list | grep $'10:\tfrom all lookup local' || ip rule add pref 10 lookup local
ip rule list | grep $'5:\tfrom all fwmark 0x74 lookup myroutes' || ip rule add pref 5 fwmark 0x74 lookup myroutes
# the following logic is a bit different: for safety, we want to ensure the delete only happens when the other "local" line is there
ip rule list | grep $'10:\tfrom all lookup local' && ip rule delete pref 0
If making changes, be very careful around that part with ip rule delete pref 0 which is where you can lose access to your device by not having a local rule in place. If it happens, reboot, quickly log back in while the script is on the 3 minute sleep at the beginning, and kill it (in self-defense!).