Sharing Bandwidth and Prioritizing Realtime Traffic via HTB, Which Scenario Works Better?

I would like to add some kind of traffic management to our Internet line. After reading a lot of documentation, I think HFSC is too complicated for me (I don't understand all the curves stuff, I'm afraid I will never get it right), CBQ is not recommend, and basically HTB is the way to go for most people.

Our internal network has three "segments" and I'd like to share bandwidth more or less equally between those (at least in the beginning). Further I must prioritize traffic according to at least three kinds of traffic (realtime traffic, standard traffic, and bulk traffic). The bandwidth sharing is not as important as the fact that realtime traffic should always be treated as premium traffic whenever possible, but of course no other traffic class may starve either.

The question is, what makes more sense and also guarantees better realtime throughput:

1. Creating one class per segment, each having the same rate (priority doesn't matter for classes that are no leaves according to HTB developer) and each of these classes has three sub-classes (leaves) for the 3 priority levels (with different priorities and different rates).

2. Having one class per priority level on top, each having a different rate (again priority won't matter) and each having 3 sub-classes, one per segment, whereas all 3 in the realtime class have highest prio, lowest prio in the bulk class, and so on.

I'll try to make this more clear with the following ASCII art image:

Case 1:

root --+--> Segment A
       |       +--> High Prio
       |       +--> Normal Prio
       |       +--> Low Prio
       |
       +--> Segment B
       |       +--> High Prio
       |       +--> Normal Prio
       |       +--> Low Prio
       |
       +--> Segment C
               +--> High Prio
               +--> Normal Prio
               +--> Low Prio

Case 2:

root --+--> High Prio
       |        +--> Segment A
       |        +--> Segment B
       |        +--> Segment C
       |
       +--> Normal Prio
       |        +--> Segment A
       |        +--> Segment B
       |        +--> Segment C
       |
       +--> Low Prio
                +--> Segment A
                +--> Segment B
                +--> Segment C

Case 1 Seems like the way most people would do it, but unless I don't read the HTB implementation details correctly, Case 2 may offer better prioritizing.

The HTB manual says, that if a class has hit its rate, it may borrow from its parent and when borrowing, classes with higher priority always get bandwidth offered first. However, it also says that classes having bandwidth available on a lower tree-level are always preferred to those on a higher tree level, regardless of priority.

Let's assume the following situation: Segment C is not sending any traffic. Segment A is only sending realtime traffic, as fast as it can (enough to saturate the link alone) and Segment B is only sending bulk traffic, as fast as it can (again, enough to saturate the full link alone). What will happen?

Case 1:
Segment A->High Prio and Segment B->Low Prio both have packets to send, since A->High Prio has the higher priority, it will always be scheduled first, till it hits its rate. Now it tries to borrow from Segment A, but since Segment A is on a higher level and Segment B->Low Prio has not yet hit its rate, this class is now served first, till it also hits the rate and wants to borrow from Segment B. Once both have hit their rates, both are on the same level again and now Segment A->High Prio is going to win again, until it hits the rate of Segment A. Now it tries to borrow from root (which has plenty of traffic spare, as Segment C is not using any of its guaranteed traffic), but again, it has to wait for Segment B->Low Prio to also reach the root level. Once that happens, priority is taken into account again and this time Segment A->High Prio will get all the bandwidth left over from Segment C.

Case 2:
High Prio->Segment A and Low Prio->Segment B both have packets to send, again High Prio->Segment A is going to win as it has the higher priority. Once it hits its rate, it tries to borrow from High Prio, which has bandwidth spare, but being on a higher level, it has to wait for Low Prio->Segment B again to also hit its rate. Once both have hit their rate and both have to borrow, High Prio->Segment A will win again until it hits the rate of the High Prio class. Once that happens, it tries to borrow from root, which has again plenty of bandwidth left (all bandwidth of Normal Prio is unused at the moment), but it has to wait again until Low Prio->Segment B hits the rate limit of the Low Prio class and also tries to borrow from root. Finally both classes try to borrow from root, priority is taken into account, and High Prio->Segment A gets all bandwidth root has left over.

Both cases seem sub-optimal, as either way realtime traffic sometimes has to wait for bulk traffic, even though there is plenty of bandwidth left it could borrow. However, in case 2 it seems like the realtime traffic has to wait less than in case 1, since it only has to wait till the bulk traffic rate is hit, which is most likely less than the rate of a whole segment (and in case 1 that is the rate it has to wait for). Or am I totally wrong here?

I thought about even simpler setups, using a priority qdisc. But priority queues have the big problem that they cause starvation if they are not somehow limited. Starvation is not acceptable. Of course one can put a TBF (Token Bucket Filter) into each priority class to limit the rate and thus avoid starvation, but when doing so, a single priority class cannot saturate the link on its own any longer, even if all other priority classes are empty, the TBF will prevent that from happening. And this is also sub-optimal, since why wouldn't a class get 100% of the line's bandwidth if no other class needs any of it at the moment?

Any comments or ideas regarding this setup? It seems so hard to do using standard tc qdiscs. As a programmer it was such an easy task if I could simply write my own scheduler (which I'm not allowed to do).

Solution 1:

If I understand htb correctly, then the rate is "guaranteed". This means you have ideas about the rate of the "realtime" traffic. Only if this rate is exceeded, it will borrow. If several classes want to borrow, the prio should kick in. The guaranteed rates should add up the physical limit. Otherwise it's too much of a hassle.

IMHO, Case A will never really work since you need to have priority or rate-limiting at the root level. The priorities/rates in the different segment don't know anything of each other and will be handled equally.

The thing you probably want is: Put the "rate" for low and normal prio to 0 or close to it and add "ceiling" for the rest of the bandwidth, for high prio you guarantee a rate of 100% of the physical.