How to Block 100,000+ Individual IP addresses
Introduction
How do you Block large number of IP address from your web application/server. Obviously that can easily be done in PHP or any programming language
$ipList = []; // array list or from database
if (in_array(getIP(), $ipList)) {
// Log IP & Access information
header("https://www.google.com.ng/search?q=fool"); // redirect
exit(); // exit
}
Or Using htaccess
order allow,deny
deny from 123.45.6.7
deny from 012.34.5.
# .... the list continues
allow from all
The issues
- Am trying to block a whole
100k plus individual IPsnotsubnets - Am trying to avoid user getting to PHP before blocking such IP
- 100000+ is over 1.5MB and that is a lot if information to be loading in
htaccessall the time - Database of IP still growing ... and they would be nee to dynamically add more values
- To set bans in
iptablesfor 100000+ is just ridiculous (Might Be wrong)
Stupid Idea
order allow,deny
deny from database <-------- Not sure if this is possible
allow from all
Question
- Is it possible for
htaccessto get the list from database (Redis,Crunchbase,Mongo, MySQL or even Sqlite) ... any - Is there a visible solution to manage such kind of issue in production
- I know the best solution is
Block the IPs at the firewall levelis there any way to pragmatically add/remove IP to the firewall
Finally
My approach might be totally wrong ... all I want is a visible solution since spammers and botnets are on the rise ...
Please this has nothing to do with DOS attack its a simple ... get lost response
Update
- Firewall : Cisco PIX 515UR
Something that you can try is keeping a list of the IP addresses you want to block in a text file or convert it to a dbm hash file, then use mod_rewrite's RewriteMap. You'd have to set this up in your server/vhost config. You cannot initialize a map in an htaccess file.
RewriteEngine On
RewriteMap deny_ips txt:/path/to/deny_ips.txt
RewriteCond ${deny_ips:%{REMOTE_ADDR}|0} !=0
RewriteRule ^ - [L,F]
The /path/to/deny_ips.txt file would look something like this:
12.34.56.78 1
11.22.33.44 1
etc.
Essentially, an IP that you want to deny and a space then a "1". Any IP in this text file will cause the server to return a 403 Forbidden. To speed things up a bit you can use the httxt2dbm to generate a dbm hash and then you'd define the mapping as so:
RewriteMap deny_ips dbm:/path/to/deny_ips.dbm
I'm not sure what the performance hit is for using mod_rewrite like this with a lot of IPs, but a quick benchmark test on apache 2.2 running on a 3Ghz i686 under linux, the difference between 5 IPs in the list versus 102418 is negligible. According to ab's output, they're nearly identical.
Addressing specific questions:
Is it possible for htaccess to get the list from database (Redis,Crunchbase,Mongo, MySQL or even Sqlite) ... any
Using a rewrite map, you can use the "prg" map type to run an external program for a mapping type. You can then write a perl, php, etc. script to talk to a database in order to look up an IP address. Also note that caveats listed under "Caution". You'd then use this map like you would any other map (RewriteCond ${deny_ips:%{REMOTE_ADDR}|0} !=0). This would essentially create a bottleneck for all requests. Not the best solution for talking to a database.
In apache 2.4 though, there is a dbd/fastdbd map type, which allows you to create queries through mod_dbd. This is a much better option and the mod_dbd module manages connections to the database, pools connections, etc. So the map definition would look something like:
RewriteMap deny_ips "fastdbd:SELECT active FROM deny_ips WHERE source = %s"
Assuming you have a table "deny_ips" with 2 columns "source" (the IP address) and "active" (1 for active, 0 for inactive).
Is there a visible solution to manage such kind of issue in production
If you are storing all of the blocked IPs in the database, it's a matter of managing the contents of your database table. If you are using the dbm map type, I know at least perl has a DBI for managing dbm files, so you can use that to add/remove IP entries from the deny list. I've never used it before so I can't really say much about it. Managing a flat text file is going to be a lot trickier, especially if you plan on removing entries, and not just append to it. Outside of using a database and apache 2.4's mod_dbd, I don't think any of these solutions are out of the box or production ready. It's going to require custom work.
I know the best solution is Block the IPs at the firewall level is there any way to pragmatically add/remove IP to the firewall
For IPtables, there is a perl interface that's marked as Beta, but I've never used it before. There's libiptc but according to netfilter's faq:
Is there an C/C++ API for adding/removing rules?
The answer unfortunately is: No.
Now you might think 'but what about libiptc?'. As has been pointed out numerous times on the mailinglist(s), libiptc was NEVER meant to be used as a public interface. We don't guarantee a stable interface, and it is planned to remove it in the next incarnation of linux packet filtering. libiptc is way too low-layer to be used reasonably anyway.
We are well aware that there is a fundamental lack for such an API, and we are working on improving that situation. Until then, it is recommended to either use system() or open a pipe into stdin of iptables-restore. The latter will give you a way better performance.
So I don't know how viable a libiptc solution is if there's no API stability.
ANOTHER PERSPECTIVE
Hello. You can check if an address is blocked or not, via accessing two bytes in two data chunks each 8KB long. Yes, I am serious... Please be patient because it takes a little bit long to explain it.
THE THEORY
An IP address is an address, actually a 4 byte number.
The question is, what if we make it to address bit positions?.
The answer: Well ok, we will have
2^32 = 4 Giga Bits
of addressing space and that will take
4Gb/8 = 512 Mega Bytes
of allocation. Ouch! But do not worry, we are not going to block everything in the ipverse and 512MB is an exaggeration.
This can open us a path to the solution.
The Lilliputian Case
Think of a Lilliputian world which there exists only ip addresses from 0 to 65535. So addresses are like 0.1 or 42.42 up to 255.255.
Now King of this world wants to block several L-IP (lilliput ip) addresses.
First he builds a virtual 2D bit map which is 256 * 256 bits long that takes up :
64 K Bits = 8 K Bytes.
He decides to block that nasty "revolution" site which he hates because he is the king, the address is 56.28 for instance.
Address = (56 * 256) + 28 = 14364.(bit position in whole map)
Byte in map = floor(14364 / 8) = 1795.
Bit position= 14364 % 8 = 4.(modulus)
He opens the map file, accesses 1795th byte and sets the bit 4 (by an | 16), then writes it back to mark the site as blocked.
When his script sees the 56.28, it does the same calculation and looks at the bit, and if it is set, blocks the address.
Now what is the moral of the story? Well we can use this lilliputian structure.
THE PRACTICE
The Real World Case
We can apply the Lilliputian case to real world with a "use it when you need" approach since allocating a 512MB file is not a good choice.
Think of a database table named BLOCKS with entries like that:
IpHead(key): unsigned 16 bit integer,
Map : 8KB BLOB(fixed size),
EntryCount : unsigned 16 bit integer.
And another table with just one entry with the structure below named BASE
Map : 8KB BLOB(fixed size).
Now lets say you have an incoming address 56.28.10.2
Script accesses BASE table and gets the Map.
It looks up the higher order IP numbers 56.28:
Address = (56 * 256) + 28 = 14364.(bit position in whole map)
Byte in map = floor(14364 / 8) = 1795.
Bit position= 14364 % 8 = 4.(modulus)
Looks at byte 1795 bit 4 in the Map.
If bit is not set no further operation is needed meaning there is no blocked ip address in range 56.28.0.0 - 56.28.255.255 .
If bit is set then the script accesses the BLOCKS table.
The higher order IP numbers were 56.28 which gives 14364 so the script queries the BLOCKS table with index IpHead = 14364. Fetches the record. The record should exist since it is marked at BASE.
Script does the calculation for lower order IP address
Address = (10 * 256) + 2 = 2562.(bit position in whole map)
Byte in map = floor(2562 / 8) = 320.
Bit position= 2562 % 8 = 2.(modulus)
Then it checks if address is blocked by looking at bit 2 of byte 320 of field Map.
Job done!
Q1: Why do we use BASE at all, we could directly query BLOCKS with 14364.
A1: Yes we could but BASE map lookup will be faster then BTREE search of any database server.
Q2: What is the EntryCount field in BLOCKS table for?
A2: It is the count of ip addresses blocked in the map field at the same record. So if we unblock ip's and EntryCount reaches 0 that BLOCKS record becomes unnecessary. It can be erased and the corresponding bit on BASE map will be unset.
IMHO this approach will be lightning fast. Also for the blob allocation is 8K per record. Since db servers keep blobs in seperate files, file systems with 4K, 8K or multiples of 4K paging will react fast.
In case blocked addresses are too dispersed
Well that is a concern, which will make the database BLOCKS table to grow unnecessarily.
But for such cases the alternative is to use a 256*256*256 bit cube which is 16777216 bits long, equaling to 2097152 bytes = 2MB.
For our previous example Higher Ip resolving is :
(56 * 65536)+(28 * 256)+10
So BASE will become a 2MB file instead of a db table record, which will be opened (fopen etc.) and bit will be addressed via seeking (like fseek, never read whole file contents, unnecessary) then access the BLOCKS table with structure below:
IpHead(key): unsigned 32 bit integer, (only 24 bit is used)
Map : 32 unsigned 8 bit integers(char maybe),(256 bit fixed)
EntryCount : unsigned 8 bit integer.
Here is the php example code for block checking of bitplane-bitplane (8K 8K) version:
Side Note: This script can be optimized further via elimination of several calls etc.. But written like this for keeping it easy to understand.
<?
define('BLOCK_ON_ERROR', true); // WARNING if true errors block everyone
$shost = 'hosturl';
$suser = 'username';
$spass = 'password';
$sdbip = 'database';
$slink = null;
$slink = mysqli_connect($shost, $suser, $spass, $sdbip);
if (! $slink) {
$blocked = BLOCK_ON_ERROR;
} else {
$blocked = isBlocked();
mysqli_close($slink); // clean, tidy...
}
if ($blocked) {
// do what ever you want when blocked
} else {
// do what ever you want when not blocked
}
exit(0);
function getUserIp() {
$st = array(
'HTTP_CLIENT_IP',
'REMOTE_ADDR',
'HTTP_X_FORWARDED_FOR'
);
foreach ( $st as $v )
if (! empty($_SERVER[$v]))
return ($_SERVER[$v]);
return ("");
}
function ipToArray($ip) {
$ip = explode('.', $ip);
foreach ( $ip as $k => $v )
$ip[$k] = intval($v);
return ($ip);
}
function calculateBitPos($IpH, $IpL) {
$BitAdr = ($IpH * 256) + $IpL;
$BytAdr = floor($BitAdr / 8);
$BitOfs = $BitAdr % 8;
$BitMask = 1;
$BitMask = $BitMask << $BitOfs;
return (array(
'bytePos' => $BytAdr,
'bitMask' => $BitMask
));
}
function getBaseMap($link) {
$q = 'SELECT * FROM BASE WHERE id = 0';
$r = mysqli_query($link, $q);
if (! $r)
return (null);
$m = mysqli_fetch_assoc($r);
mysqli_free_result($r);
return ($m['map']);
}
function getBlocksMap($link, $IpHead) {
$q = "SELECT * FROM BLOCKS WHERE IpHead = $IpHead";
$r = mysqli_query($link, $q);
if (! $r)
return (null);
$m = mysqli_fetch_assoc($r);
mysqli_free_result($r);
return ($m['map']);
}
function isBlocked() {
global $slink;
$ip = getUserIp();
if($ip == "")
return (BLOCK_ON_ERROR);
$ip = ipToArray($ip);
// here you can embed preliminary checks like ip[0] = 10 exit(0)
// for unblocking or blocking address range 10 or 192 or 127 etc....
// Look at base table base record.
// map is a php string, which in fact is a good byte array
$map = getBaseMap($slink);
if (! $map)
return (BLOCK_ON_ERROR);
$p = calculateBitPos($ip[0], $ip[1]);
$c = ord($map[$p['bytePos']]);
if (($c & $p['bitMask']) == 0)
return (false); // No address blocked
// Look at blocks table related record
$map = getBlocksMap($slink, $p[0]);
if (! $map)
return (BLOCK_ON_ERROR);
$p = calculateBitPos($ip[2], $ip[3]);
$c = ord($map[$p['bytePos']]);
return (($c & $p['bitMask']) != 0);
}
?>
I hope this helps.
If you have questions on the details, I will be happy to answer.
Block the traffic before it reaches the www server using iptables and ipset.
Catch the blacklisted IP traffic in the filter table of the INPUT chain assuming your web server is on the same machine. If you are blocking IPs on a router you will want the FORWARD chain.
First create the ipset:
ipset create ip_blacklist hash:ip
IPs can be added via:
ipset add ip_blacklist xxx.xxx.xxx.xxx
Add the ipset match rule to your iptables (DROP all packets match to ipset):
iptables --table filter --insert INPUT --match set --match-set ip_blacklist src -j DROP
This will stop the blacklisted traffic before the www server.
Edit: I had a chance to look up the default maximum size and it is 65536 so you will need to adjust this to support 100000+ entries:
ipset create ip_blacklist hash:ip maxelem 120000
You can also tweak the hash size:
ipset create ip_blacklist hash:ip maxelem 120000 hashsize 16384 (Must be a power of 2)
My experience is ipset lookup has negligible effect on my system (~45000 entries). There are a number of test cases on the net. Memory for the set is a limiting factor.
You need to do this with an external firewall, not in PHP. I recommend pfSense or PF. I have used it before and it is very easy to use, very intuitive, and extremely powerful. It is the choice of the best sys-admins. I run it on FreeBSD, but it works great on OpenBSD as well. I am a Linux guy so it pains me to say this, but don't try to run it on Linux. BSD is easy, and you can figure it out quickly.
An awesome feature for pfSense is the ability to configure using scripts and restricting the configuration access to a single network interface (so that only things on the LAN can configure it). It also has a couple of ID10T level features to keep you from cutting off your own access accidentally.
You should also be aware that many spammers can switch IPs quickly using things like Tor. To fix this you should include in your block list the addresses that are known tor exit nodes (this list is available from various places).
If you want a way to add / remove via code take a look at denyhosts. You could either maintain the IP list via code or patch the source to read from whatever location you want.