Does "TARPIT" have any known vulnerabilities or downsides?

TARPIT can be used to waste an attacker's resources, thus slowing down his attacks and lowering his ability to attack other hosts... looks like a good idea.

It is provided as a Netfilter addon and can be used just like any other IPTables target.

Are there known downsides or vulnerabilities in this approach of dealing with (D)DoS ?

I used to think it was a good idea. But now I know it's a very bad idea unfortunately.

Have you ever run an http benchmark app like apache bench. On a single machine you can set it to create hundreds of connections per second to a target server. Get a few of those clients running and connecting to your server with tarpitting enabled and I think you will see a problem.

Think about how creating thousands of connections per second to your server will impact the server if each connection is trapped in a tarpit.

Your server will quickly consume all it's available resources (or file handles) so that no more connections are allowed. This is worse than just closing the connection. It'd be better to drop the offender for a while than try to tie up his resources. Which is what scripts like fail2ban achieve.

Also, you never want your normal users to be stuck in the tarpit. At least for interactive sessions. How do you decide who's allowed and who's not upfront? For some protocols you can't. For example, HTTP traffic. You have to assume a client is OK until you get activity from them that tells you otherwise. Then you can decide to treat it as bad and next time it'll get caught in the tarpit. Which would seem to be ok except many of these attacks can come from dynamic adsl users etc who just happen to have gotten the latest worm virus.

Given that many attacks happen on PC's by worm virus' without the user even knowing and running off dynamic addresses, you can end up building up an easily outdated tarpit blacklist. Are you starting to see some problems?

Summary

Running a tarpit on a general purpose server does come with risks. If you know what the risks are you can mitigate them, depending on your level of comfort.

You have to ensure you don't accidentally DOS your server w/ tarpit traffic
You have to ensure you fill up your state tables w/ tarpitted connection info
You have to ensure you don't flood your logs with tarpit connection information
You need to ensure a long blacklist doesn't affect performance
You need to ensure that your blacklist automatically expires hosts
You need the ability to whitelist hosts (either permanently or with time-limit)

Thankfully this is all possible, and quite easy using regular iptables and ipset.

Limiting TARPIT resource usage

You can use iptables to limit how many hosts you TARPIT without using too many system resources. See example below. This includes network bandwidth, system memory, state stable entries, and other system resources. Get too many tarpitted connections, start ignoring them. If you organize your rule set in the correct order, none of the tarpitted connections end up on your state tables. Also make sure you don't log, unless you're doing realtime stats with something like a custom ulog -- Direct iptables tarpit logs can quickly fill up a disk.

In my experience my current hosts can easily hold 200+ hosts in a tarpit, with little noticeable effect on memory usage, traffic usage, or cpu usage. Likely I could push this further, but so far on average I'm only trapping around 130 hosts at any given moment.

The reason why I implemented the limits, was as stated in another suggestion, because my first tarpit host became flooded. This was a trivial workaround. I've had no issues since.

Using ipset for efficient blacklists

ipset is an great little tool that allows you to create groups of objects to use in iptables rules. Not only that, but since it can hold the objects in a hash table, the larger the ipset the faster it is compared to the equivalent linear set of iptables rules.

In addition to that, the lists can include counters (packets/bytes), timeout, and exclusion on a per object basis.

You can add/remove from ipset with most programs that automatically block, such as fail2ban, ossec, and more. Because you can set a default timeout, you can ensure that entries are expired regardless what program set the entry.

Example

Here's an example based on what I use on servers I manage that mitigates the risks listed above:

Caveat

### Note:  This does not account for all possible traffic you might need or want
###        This is only an example of mitigating common risks of using a tarpit
###        Use at your own risk

Configuring ipset

# Create the ipset blacklist
# options:
# - create         : create a new ipset
# - blacklist      : Name of the ipset we'll reference in the iptables rule
# - hash:net       : Make blacklist type hash to hold network (ip/mask) data
# - family inet    : This is for IPv4 data (inet6 for IPv6)
# - counters       : We want packet/byte stats counted for each entry
# - comment        : So we can add a comment with each entry (handy)
# - timeout 604800 : Set a default timeout of 604800 seconds (1 week) for each new entry
# - nomatch        : Allow us to enter exclusion entries (never match an entry)
ipset create blacklist hash:net family inet counters comment timeout 604800 nomatch

# Create an entry to never blacklist a trusted network
# options:
# - timeout 0      : entry never expires
# - nomatch        : Tells IPset to never match this entry (whitelist, in our usage)
ipset add blacklist 123.45.67.0/24 comment "Trusted subnet, causes breakage if blocked" timeout 0 nomatch

# Example to blacklist hosts
# no netmask implies /32 (or /128 for ipv6)
ipset add blacklist 34.56.78.90 comment "SSH Bruteforce"
ipset add blacklist 23.45.67.89 comment "SQL Injection" timeout 12345
# etc...

Configuring iptables

# Flush the input table
iptables -F INPUT

# Drop the custom flow TAR
iptables -X TAR

# Set default INPUT policy to DROP
iptables -P INPUT DROP

# Create the chain TAR
iptables -N TAR

# Send all blacklisted hosts to the tarpit
iptables -A INPUT -m set --match-set blacklist src -j TAR

# Allow established connections
# Note: after the blacklist so newly blacklisted threats are immediately ignored
#       Yes, that will cause the server to hold the state open until it times out.
#       Would you rather have a malicious host continuing its attack?
iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# Allow the services we want
# Note:  These create new state table entries/use up memory
# Note:  We do not account for synflood prevention in this example
iptables -A INPUT -p tcp -m multiport --dports 22,80,443 -m tcp --syn -m conntrack --ctstate NEW -j ACCEPT
iptables -A INPUT -p icmp -m icmp --icmp-type 8 -m conntrack --ctstate NEW -j ACCEPT

# Send everything else to tarpit chain
iptables -A INPUT -j TAR

# This is the tarpit chain

# Tarpit up to 10 unique connections a second, any more, and pass to next rule
# Note: 10/s limit is arbitrary, adjust to your preference (experiment)
iptables -A TAR -p tcp -m limit --limit 10/sec -j TARPIT --tarpit 

# Drop everything else that makes it this far
# You can also set to REJECT which will immediately tell all connections to buzz off
iptables -A TAR -j DROP

Looking at trapped hosts realtime

$ sudo tcpdump -npi eth0 'src YOUR.HOST.IP and (tcp[14] = 0 && tcp[15] = 0)'