What is the best solution for traffic control in a large system (ca. 2000 users)?

following situation: We're a group of students administering the internet connection for the local resident halls, with a total of about 2000 end users.

We have a traffic point system, every MB down- or upload costs points, new points are added by the hour. At the moment, we block a user's internet access when he has spent all his points (by placing him into a REJECT policy in iptables on our Debian gateway router).

We would like only to limit a user's bandwidth. What is the best way to do this?

The simple answer would be to set a rate-limit on the user's switch port (mostly Cisco Catalyst 3550s). However, this is undesirable, as traffic inside our own network and to the university network should remain unlimited. Is there a way to limit bandwidth only for packets with a certain destination or source IP range (so both egress and ingress) in Cisco IOS? I could not find anything.

The other way would be to control the traffic on our gateway router. Several solutions come to my mind:

• tc or tcng - seems like both have a rather arcane syntax and neither offer good features for doing per-IP traffic control. A dedicated QDisc for so many people would probably slow down the router quite a lot. Furthermore, documentation on both is pretty outdated.

• shorewall - seems to have a rather neat syntax for configurations, however, I'm unsure whether it can handle this amount of traffic and users and whether it's suitable for per-IP traffic limiting

• pfSense - looks like an OS intended for purposes like ours. However, it would require us to compeletely reinstall our gateway router. We don't have other BSD systems and pfSense would need to have very good traffic accounting capabilities (we're using fprobe-ulog and ulog-acctd there at the moment), too.

What is your experience? Which solution suits our needs and can be most easily maintained? Do you have other ideas?

If you need any additional information about our system, please don't hesitate to ask.

Thanks in advance.

EDIT: I have implemented the system with iptables and tc.

Every user has a /28-subnet, a VPN IP (both from 10.0.0.0/8) and an external IP, all are steered through one iptables chain. This chain has only one rule, a simple RETURN.

Every five minutes, a Python script reads out the byte counters of these rules. It resets the counters and updates the user's traffic point account in our PostgreSQL database.

If a user's point balance decreases below a certain threshold, two tc classes are created for this user (one for the incoming, one for the outgoing interface on our gateway router), the IPs are entered into tc filters belonging to these classes. The classes are speed-limited by an HTB.

Compared to the previous system with fprobe-ulog and ulog-acctd this is much faster as the byte counting is done by iptables.

Network speed has improved considerably for our users.

Solution 1:

I'm not sure how interested you are in reconfiguring your entire setup (i.e., replace Debian), or how feasible it would be to place something like this behind your gateway, but FreeBSD has a feature in ipfw known as dummynet. Since it sounds like you don't feel like getting a dedicated hardware traffic shaper, this may be an option for you. We currently use it to choke SMTP traffic inbound and outbound through one of our proxy gateways to keep an aging NFS backend system from being overwhelmed and subsequently becoming unresponsive.

With some scripting and intelligent configuration of your rulesets, it would be feasible to be able to traffic control thousands of individual IP addresses.