Why should I firewall servers?

PLEASE NOTE: I'm not interested in making this into a flame war! I understand that many people have strongly-held beliefs about this subject, in no small part because they've put a lot of effort into their firewalling solutions, and also because they've been indoctrinated to believe in their necessity.

However, I'm looking for answers from people who are experts in security. I believe that this is an important question, and the answer will benefit more than just myself and the company I work for. I've been running our server network for several years without a compromise, without any firewalling at all. None of the security compromises that we have had could have been prevented with a firewall.

I guess I've been working here too long, because when I say "servers", I always mean "services offered to the public", not "secret internal billing databases". As such, any rules we would have in any firewalls would have to allow access to the whole Internet. Also, our public-access servers are all in a dedicated datacenter separate from our office.

Someone else asked a similar question, and my answer was voted into negative numbers. This leads me to believe that either the people voting it down didn't really understand my answer, or I don't understand security enough to be doing what I'm currently doing.

This is my approach to server security:

1. Follow my operating system's security guidelines before connecting my server to the Internet.

2. Use TCP wrappers to restrict access to SSH (and other management services) to a small number of IP addresses.

3. Monitor the state of this server with Munin. And fix the egregious security problems inherent to Munin-node in its default configuration.

4. Nmap my new server (also before connecting my server to the Internet). If I were to firewall this server, this should be the exact set of ports incoming connections should be restricted to.

5. Install the server in the server room and give it a public IP address.

6. Keep the system secure by using my operating system's security updates feature.

My philosophy (and the basis of the question) is that strong host-based security removes the necessity of a firewall. The overall security philosophy says that strong host-based security is still required even if you have a firewall (see security guidelines). The reason for this is that a firewall that forwards public services to a server enables an attacker just as much as no firewall at all. It is the service itself that is vulnerable, and since offering that service to the entire Internet is a requirement of its operation, restricting access to it is not the point.

If there are ports available on the server that do not need to be accessed by the whole Internet, then that software needed to be shut down in step 1, and was verified by step 4. Should an attacker successfully break into the server through vulnerable software and open a port themselves, the attacker can (and do) just as easily defeat any firewall by making an outbound connection on a random port instead. The point of security isn't to defend yourself after a successful attack - that's already proven to be impossible - it's to keep the attackers out in the first place.

It's been suggested that there are other security considerations besides open ports - but to me that just sounds like defending one's faith. Any operating system/TCP stack vulnerabilities should be equally vulnerable whether or not a firewall exists - based on the fact that ports are being forwarded directly to that operating system/TCP stack. Likewise, running your firewall on the server itself as opposed to having it on the router (or worse, in both places) seems to be adding unnecessary layers of complexity. I understand the philosophy "security comes in layers", but there comes a point where it's like building a roof by stacking X number of layers of plywood on top of each other and then drilling a hole through all of them. Another layer of plywood isn't going to stop the leaks through that hole you're making on purpose.

To be honest, the only way I see a firewall being any use for servers is if it has dynamic rules preventing all connections to all servers from known attackers - like the RBLs for spam (which coincidentally, is pretty much what our mail server does). Unfortunately, I can't find any firewalls that do that. The next best thing is an IDS server, but that assumes that the attacker doesn't attack your real servers first, and that attackers bother to probe your entire network before attacking. Besides, these have been known to produce large numbers of false positives.

Solution 1:

Advantages of firewall:

1. You can filter outbound traffic.
2. Layer 7 firewalls (IPS) can protect against known application vulnerabilities.
3. You can block a certain IP address range and/or port centrally rather than trying to ensure that there is no service listening on that port on each individual machine or denying access using TCP Wrappers.
4. Firewalls can help if you have to deal with less security aware users/administrators as they would provide second line of defence. Without them one has to be absolutely sure that hosts are secure, which requires good security understanding from all administrators.
5. Firewall logs would provide central logs and help in detecting vertical scans. Firewall logs can help in determining whether some user/client is trying to connect to same port of all your servers periodically. To do this without a firewall one would have to combine logs from various servers/hosts to get a centralized view.
6. Firewalls also come with anti-spam / anti-virus modules which also add to protection.
7. OS independent security. Based on host OS, different techniques / methods are required to make the host secure. For example, TCP Wrappers may not be available on Windows machines.

Above all this if you do not have firewall and system is compromised then how would you detect it? Trying to run some command 'ps', 'netstat', etc. on local system can't be trusted as those binaries can be replaced. 'nmap' from a remote system is not guaranteed protection as an attacker can ensure that root-kit accepts connections only from selected source IP address(es) at selected times.

Hardware firewalls help in such scenarios as it is extremely difficult to change firewall OS/files as compared to host OS/files.

Disadvantages of firewall:

1. People feel that firewall will take care of security and do not update systems regularly and stop unwanted services.
2. They cost. Sometimes yearly license fee needs to be paid. Especially if the firewall has anti-virus and anti-spam modules.
3. Additional single point of failure. If all traffic passes through a firewall and the firewall fails then network would stop. We can have redundant firewalls, but then previous point on cost gets further amplified.
4. Stateful tracking provides no value on public-facing systems that accept all incoming connections.
5. Stateful firewalls are a massive bottleneck during a DDoS attack and are often the first thing to fail, because they attempt to hold state and inspect all incoming connections.
6. Firewalls cannot see inside encrypted traffic. Since all traffic should be encrypted end-to-end, most firewalls add little value in front of public servers. Some next-generation firewalls can be given private keys to terminate TLS and see inside the traffic, however this increases the firewall's susceptibility to DDoS even more, and breaks the end-to-end security model of TLS.
7. Operating systems and applications are patched against vulnerabilities much more quickly than firewalls. Firewall vendors often sit on known issues for years without patching, and patching a firewall cluster typically requires downtime for many services and outbound connections.
8. Firewalls are far from perfect, and many are notoriously buggy. Firewalls are just software running on some form of operating system, perhaps with an extra ASIC or FPGA in addition to a (usually slow) CPU. Firewalls have bugs, but they seem to provide few tools to address them. Therefore firewalls add complexity and an additional source of hard-to-diagnose errors to an application stack.

Solution 2:

TCP Wrappers could be arguably called a host-based firewall implementation; you're filtering network traffic.

For the point on an attacker making outbound connections on an arbitrary port, a firewall would provide a means of controlling outgoing traffic as well; a properly configured firewall manages ingress and egress in a way which is appropriate to the risk related to the system.

On the point about how any TCP vulnerability isn't mitigated by a firewall, you're nt familiar with how firewalls work. Cisco has a whole bunch of rules available for download which identify packets constructed in a way that would cause particular operating systems problems. If you grab Snort and start running it with the right rule set, you will also get alerted on this kind of thing. And of course, Linux iptables can filter out malicious packets.

Basically, a firewall is proactive protection. The farther you get away from being proactive, the most likely that you'll find yourself in a situation where you're reacting to a problem rather than preventing the problem. Concentrating your protection at the border, as with a dedicated firewall, makes things easier to manage because you have a central choke point rather than duplicating rules everywhere.

But no single thing is necessarily a final solution. A good security solution generally is multi-layer, where you have a firewall at the border, TCP wrappers at the device, and probably some rules on internal routers as well. You should usually protect the network from the Internet, and protect the nodes from each other. This multi-layer approach isn't like drilling a hole through multiple sheets of plywood, it's more like putting up a pair of doors so an intruder has two locks to break instead of just one; this is called a man trap in physical security, and most every building has one for a reason. :)