Linux traffic shaping using TC

My internet connection goes like this:

Internet <-128kbps link-> Cisco Router (Public IP) <-LAN-> Linux router/server (Public IP) <-LAN-> Regular PCs (Public IPs)

The Cisco router:

• the first Public IP allocated to my institution (/29)
• is programmed to send all packets through the Linux router

The Linux router

• The second Public IP allocated to my institution
• is programmed to forward packets between the regular PCs and the Cisco router
• act as a server too (mail, web, etc.)

The regular PCs (4 of them):

• The rest of the Public IPs
• use the Linux router as the gateway

I enabled the iptables packet logging on the Linux router and sometimes and I find out that:

• Some packets are big, bigger than 20KB. Is that normal? (yes, it is normal. These are not packets, these are IP datagrams as Some Guy kindly explained)
• Too many times the transmitted data (out to the internet) was bigger than 16KB. For example in a particular second 10572 bytes went in (no problem), 63521 bytes went out (to the Cisco router). It would take 4 seconds at least to send that 64KB through the 128kbps link. Meanwhile the Linux router is sending more data to the Cisco router, clogging its buffers. Not good.

Now, how can I configure the Linux router to shape traffic in a way that:

1. Keep transmission speed to the max when the traffic is between these regular PCs and the Linux server.
2. Slow down traffic to the outside world to avoid clogging the “out” line, using all (or almost all) the bandwidth available (128 kbps). No more “>16KB out seconds” on the trace.
3. Guarantee 24kbps out to each regular PC, 24 kbps to the Linux server at any time. (8bkps left for overhead if necessary). IOW, 5 (pseudo) “bands”, 24kbps each one.
4. In case there is any PC not using its full band, fairly share the idle bandwidth among the rest of the transmitting PCs
5. Give priority to certain packets (DNS lookups, control packets), take priority away from others (torrent!!!), INSIDE each band, and not affecting other bands.

I have already marked (using IP tables --set-xmark option) each outgoing packet for each PC:

1. Linux router to the outside world, high prio
2. Linux router to the outside world, normal prio
3. Linux router to the outside world, low prio
4. First regular PC to the outside world, high prio

... And so on.

Each incoming packet is also marked using this scheme starting from 16.

I’m sorry for this long question, but I have given up to set this up using the tc command, there is too little documentation about traffic shaping and I don’t know where to go next.

Solution 1:

Assuming eth0 is a 100mbit Ethernet connection to the Cisco Router, it should be something like this (Isn’t it?):

tc qdisc add dev eth0 root handle 1: htb default 2
# 100 mbps
tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit
# To LAN traffic
tc class add dev eth0 parent 1:1 classid 1:2 htb rate 99000kbit ceil 100mbit
# IN traffic
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 120kbit
# OUT traffic
tc class add dev eth0 parent 1:1 classid 1:4 htb rate 120kbit

# IN “bands” (one for each PC)
tc class add dev eth0 parent 1:3 classid 1:10 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:3 classid 1:11 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:3 classid 1:12 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:3 classid 1:13 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:3 classid 1:14 htb rate 24kbit ceil 120kbit

# OUT “bands” (one for each PC)
tc class add dev eth0 parent 1:4 classid 1:15 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:4 classid 1:16 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:4 classid 1:17 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:4 classid 1:18 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:4 classid 1:19 htb rate 24kbit ceil 120kbit

Would get me something like this:

+-----------------------------------------------------------+
|                      100 mbits (1:1)                      |
+---------+------------------------+------------------------+
| 99mbits |   120 kbits In (1:3)   |  120 kbits Out(1:4)    |
+  (1:2)  +----+----+----+----+----+----+----+----+----+----+
+---------+ PC1| PC2| PC3| PC4| PC5| PC1| PC2| PC3| PC4| PC5|
          |1:10|1:11|1:12|1:13|1:14|1:15|1:16|1:17|1:18|1:19|
          +----+----+----+----+----+----+----+----+----+----+

And for each band:

# PC1, IN
tc qdisc add dev eth0 parent 1:10 handle 20: prio
tc qdisc add dev eth0 parent 20:1 handle 22: sfq perturb 10
tc qdisc add dev eth0 parent 20:2 handle 23: sfq perturb 10
tc qdisc add dev eth0 parent 20:3 handle 24: sfq perturb 10

# PC1, OUT
tc qdisc add dev eth0 parent 1:15 handle 21: prio
tc qdisc add dev eth0 parent 21:1 handle 25: sfq perturb 10
tc qdisc add dev eth0 parent 21:2 handle 26: sfq perturb 10
tc qdisc add dev eth0 parent 21:3 handle 27: sfq perturb 10

+--------------------++--------------------+
|       PC1 IN       ||      PC1 OUT       |
+--------------------++--------------------+
|     PRIO (20:0)    ||     PRIO (21:0)    |
|      |      |      ||      |      |      |
| Prio | Prio | Prio || Prio | Prio | Prio |
|   1  |   2  |   3  ||   1  |   2  |   3  |
|(20:1)|(20:2)|(20:3)||(21:1)|(21:2)|(21:3)|
+------+------+------++------+------+------+
|  SFQ |  SFQ |  SFQ ||  SFQ |  SFQ |  SFQ |
|(22:0)|(23:0)|(24:0)||(25:0)|(26:0)|(27:0)|
+------+------+------++------+------+------+

And so on.

The rules should be like this

# PC1, OUT
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 1 fw flowid 21:1
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 2 fw flowid 21:2
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 3 fw flowid 21:3

# PC1, IN
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 16 fw flowid 20:1
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 17 fw flowid 20:2
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 18 fw flowid 20:3

and so on.

Any suggestion, comments, etc? (I have no experience in the field)

Solution 2:

take a look here - it's quite simple tutorial how to use htb qos mechanism. in essence you mark packets on iptables and then assign to different ques depending on mark.

alternatively you might look at hfsc, it might actually work better with such slow internet connection, although i never used it.

regarding packet size - i cannot imagine 20kB large packet. even jumbo-frames in ethernet are shorter [ 9kB ].

edit: as Some Guy explains - LEN is length of defragmented IP packet, not a size of frame on the wire.