Limiting interface bandwidth with tc under Linux
I have a linux router which has a 10GBe interface on the outside and bonded Gigabit ethernet interfaces on the inside.
We have currently budget for 2GBit/s. If we exceed that rate by more than 5% average for a month then we'll be charged for the whole 10Gbit/s capacity. Quite a step up in dollar terms.
So, I want to limit this to 2GBit/s on 10GBe interface.
TBF filter might be ideal, but this comment is of concern.
On all platforms except for Alpha, it is able to shape up to 1mbit/s of normal traffic with ideal minimal burstiness, sending out data exactly at the configured rates.
Should I be using TBF or some other filter to apply this rate to the interface and how would I do it. I don't understand the example given here: Traffic Control HOWTO
In particular "Example 9. Creating a 256kbit/s TBF"
tc qdisc add dev eth0 handle 1:0 root dsmark indices 1 default_index 0
tc qdisc add dev eth0 handle 2:0 parent 1:0 tbf burst 20480 limit 20480 mtu 1514 rate 32000bps
How is the 256K bit/s rate calculated? In this example, 32000bps = 32k bytes per second. Since tc uses bps = bytes per second. I guess burst and limit come into play but how would you go about choosing sensible numbers to reach the desired rate?
This is not a mistake. I tested this and it gave a rate close to 256K but not exactly that.
EDIT
After doing a lot of reading and testing, I've come to the conclusion that TBF is inappropriate because of the bandwidth involved. Whatever settings I tried I couldn't get TBF to provide bandwidth > ~50Mbit/s. According to lartc.org/lartc.pdf, the RED method is better for shaping bandwidth > 100Mbit/s so I will attempt to use that.
However, choosing the a value for min (i.e. Average queue size at which marking becomes a possibility). The example given is this:
You should set the min by calculating that highest acceptable base queueing latency you wish, and multiply it by your bandwidth. For instance, on my 64kbit/s ISDN link, I might want a base queueing latency of 200ms so I set min to 1600 bytes.
how would you choose the highest acceptable base queuing latency? The example is for 64kbit/s.
What would be acceptable for 2Gbit/s?
Solution 1:
-
You should chose acceptable queueing latency basing on type of traffic.
- For example, for voice queuing more that 200ms is already a problem.
- While having 500ms buffer for ftp/torrent traffic is not a big problem at all.
Queuing latency/strategy is a question of traffic type not interface speed. For example VOIP, perhaps, should not be queued at all. Unfortunately the tc RED documentation is not very clear, you would better read some RED info on Juniper/Cisco site and apply that knowledge to tc.
Solution 2:
How is the 256K bit/s rate calculated? In this example, 32,000bps = [32,000] bytes per second.
Yes, the math there is correct. If you're seeing a number close to 256k, it's probably slightly below. Where are you measuring that number from? If it's your browser's downlaod or something similar, they don't count the overhead of packet headers, but tc counts everything.
Solution 3:
In my experience, qdisc TBF is easily able to limit bandwidth to 1 Gbps so I would guess it will also scale to 2 Gbps. However, you'll probably need a real CPU for the job instead of some low-end edge router. Something like 4 GHz i3 will surely be enough.
Try something like
tc qdisc add dev "$DEV" root handle 1: \
tbf rate "$UPLINK_RATE" burst "$UPLINK_BURST" latency "$TBF_LATENCY"
where
DEV="$(ip route | grep "^default " | grep -Po "(?<=dev )[^ ]+")"
UPLINK_RATE="2000Mbit"
UPLINK_BURST="6500"
TBF_LATENCY="14ms"
Note that to use low latency TBF you may need to be running PREEMPT kernel (e.g. Ubuntu linux-lowlatency-hwe-* package) or the system may fail handling all those packages.
See also: https://networkengineering.stackexchange.com/a/54404/36597