iptables management tools for large scale environment
Solution 1:
If you're perhaps wanting to make a move from a rule-driven approach to a "describe the final state required" way of doing things, have a look at fwbuilder.
Pros:
- multiple firewalls supported - your core + host-based rules - from 1 set of objects
- SQL-esque "tell me what you want" rather than "tell me how to do it" approach (NB I'm not saying there's any SQL in there! Just that it's descriptive Vs procedural :-)
- It's a GUI, kinda like commercial hardware f/w vendors' interfaces, so it's possible to push some tasks down the employee/skills stack
- supports most "wierd' usage I've tried
- can generate rules for a variety of f/w implementations - BSD/cicso/iptables/etc
- seperates front-end from rule-compiler, which makes me hopeful that speed is a concern to the authors. NB I've nothing near the scale to which you're alluding
- File format isn't binary
- does IPv6
- Creates an iptables-save stylee config for atomic and quick loading
Cons:
- It's a GUI
- Moving your existing ruleset is unlikely to be pain-free
- Whilst GPL and in Debian, Windows+OSX clients are 30-day eval, as no-one's cross-compiled a Free version yet for those OS; hence the commercial arm of the devs has a monopoly on those binaries
- File format is technically XML; NB don't let this put you off: take a look at the tools that they provide (you can use the gui binary to manipulate it via the CLI for example), the CLI XML tools that already exist, and remember that - at your scale - some semblance of meta-data + structure ain't a /bad/ thing! It diffs quite nicely across edits, IIRC.
Link : http://www.fwbuilder.org
Solution 2:
write your own. seriously - at this scale it is reasonable.
use ipset and/or plenty of iptable tables / subtables. whenever possible reload only some subtables / some sets of ipset - this will speed up reconfiguration.
probably you already do it, but still it's worth mentioning - use nested tables to decrease load on the router and average number of lookups needed for packets setting up new connections. obviously -A FORWARD -m state --state ESTABLISHED,RELATED is your topmost rule.
Solution 3:
holy balls (keepin' the theme alive!) man...12,000 core rules?
I'm assuming you've considered all the easy options like simply dropping the sets into CVS? Puppet or CFengine?
Honestly, from the broad overview you've given, I'd strongly suggest re-evaluating your network design. I'm probably a little too simplistic, but I simply cannot fathom a design that would necessitate 12k iptables rules. This really sounds like something that would benefit more from an SLB type solution than a better way to manage the firewall rules.
On a side note, how does one add a comment versus adding an "answer"?