Integrating AbuseIPDB into pfBlockerNG

I have been monitoring IP’s that slip through my current feeds in pfBlockerNG on my pfSense run appliance. When I encountered one that hit my proxy or attempted to hit a port I would often lookup that IP on the “Google”. The search results were almost always first presented from https://www.abuseipdb.com/. This site’s database is fed by community reports and is constantly being updated. Naturally I thought to myself “That would be a great feed!” Alas, after much searching I have found that there wasn’t any current integration with the AbuseIPDB; largely because it wasn’t designed to be a feed as much as a method to lookup or report the abuse of IP’s. AbuseIPDB does integrate with fail2ban however pfSense doesn’t have a package for that either. Since pfBlockerNG does support custom lists and the AbuseIPDB provides a way to get the most recent abusers I decided that I would write a simple little perl script that would collect the most recent reported abusers and consolidate this into a custom feed/list that would be scraped up by pfBlockerNG upon each refresh.

For this approach to work you should have (and understand how to use) the following packages running on your pfSense node:
1. pfBlockerNG-devel
2. cron

AbuseIPDB apikey
You also must create an account at https://www.abuseipdb.com/ and get an apikey to programmatically request the latest list of abuser’s IP’s. Tip: if you have and register your own domain you have higher limits on the number of requests per day.

The idea is as follows:
In cron run a script that reads the existing feed file and puts all IP’s in a perl hash. Next the script will call (via curl) the abuseipdb api to get the latest list. Because we add these to the same hash duplicates are auto magically ignored. Once the 2 lists are combined simply write out the new hash to the original feed file and pfBlockerNG will do the rest.

Here is the implementation:
Create a directory to store the script and create script below

mkdir /usr/local/share/script

Next create the feed file that will store the list of blacklisted IP’s 1 per line. This will prevent the script from erroring out on the first run. (or you could omit this step and ignore the warning on the first run).

touch  /var/db/pfblockerng/Abuse_IPDB.txt 

Then create the script:

vi /usr/local/share/script/sync_aipdb.pl

#!/usr/local/bin/perl
use strict;

# full path of file that pfblocker will consume
my $blacklist = '/var/db/pfblockerng/AbuseIPDB.txt';
my $url = 'https://api.abuseipdb.com/api/v2/blacklist';
my $apikey = '<<YOUR_API_KEY_HERE>>';
my $cmd = "/usr/local/bin/curl -s -G $url -d confidenceMinimum=100 -H \"Key: $apikey\" -H \"Accept: text/plain\"";


# %IP is an associative array (aka hash) used to compare and add any new entries
my %IP;

# open current blacklist if present will warn and move on if not present
open(IF,"<$blacklist") or warn "No IP list located at '$blacklist'.\n$!";
while(<IF>){
  chomp;
  $IP{"$_"} = 1;
}
close(IF);

# Print something to STDOUT so you know its working :)
print localtime . "\tBlacklist Count: " . (keys %IP);

# get latest 10k ips from AbuseIPDB and add new via IP Hash
open (HTTP,"$cmd|") or die "Cannot connect to AbuseIPDB.com!\n$!";
while (<HTTP>){
  chomp;
  $IP{"$_"} = 1;
}
close (HTTP);


# Print something to STDOUT so you know merge is working :)
print "  Merged Count: " . (keys %IP) . "\n";

# Overwrite blacklist file with updated combined list
open(OF,">$blacklist") or die "Cannot write blacklist located at '$blacklist'.\n$!";
foreach my $ip (keys %IP){
  print OF "$ip\n";
}
close (OF);

Remember to make the script executable

chmod 755 /usr/local/share/script/sync_aipdb.pl

Now its true there are many ways to do the above task and some additional error checking wouldn’t hurt . But this is just the basics to get you started. One to do is to write (and test) a quick regex to be sure only ip4 IP’s are being included.

Once the script is in place have the script run in cron (no more that 4 times a day if using the AbuseIPDB free version as a normal user. Notice that my pfBlockerNG feeds are updated 4 times a day at 6 hour intervals (2nd to bottom entry). So I schedule my script to run just a few minutes before that. (bottom crontab entry).

The first run you will get about 10,000 IP addresses, after that it will the output file will grow by the number of new unique IP’s that the database returns.

The below cronlog will verify that your IP Update script is running.

cat /var/log/pfblockerng/sync_aipdb.log
Fri Nov 12 23:55:00 2021 Blacklist Count: 14761 Merged Count: 16922
Sat Nov 13 05:55:00 2021 Blacklist Count: 16922 Merged Count: 18430
Sat Nov 13 11:55:00 2021 Blacklist Count: 18430 Merged Count: 19768

Lastly you will need to add the local feed to pfBlockerNG so when the feeds get updated your new custom list is also added. Instead of passing an http address I simply enter the location of the text file.

And a view of the pfBlockerNG log will show the new IP’s added.

grep Abuse_IPDB /var/log/pfblockerng/pfblockerng.log
[ Abuse_IPDB_v4 ] Downloading update .. completed ..
5950 /var/db/pfblockerng/deny/Abuse_IPDB_v4.txt
Nov 12 13:36 Abuse_IPDB_v4
[ Abuse_IPDB_v4 ] [ 11/12/21 18:00:31 ]
[ Abuse_IPDB_v4 ] Downloading update [ 11/12/21 18:00:39 ] .. completed ..
6114 /var/db/pfblockerng/deny/Abuse_IPDB_v4.txt
Nov 12 18:00 Abuse_IPDB_v4
[ Abuse_IPDB_v4 ] [ 11/13/21 00:00:18 ]
[ Abuse_IPDB_v4 ] Downloading update [ 11/13/21 00:00:24 ] .. completed ..
6179 /var/db/pfblockerng/deny/Abuse_IPDB_v4.txt
Nov 13 00:00 Abuse_IPDB_v4
[ Abuse_IPDB_v4 ] [ 11/13/21 06:00:24 ]
[ Abuse_IPDB_v4 ] Downloading update [ 11/13/21 06:00:31 ] .. completed ..
6240 /var/db/pfblockerng/deny/Abuse_IPDB_v4.txt
Nov 13 06:00 Abuse_IPDB_v4
[ Abuse_IPDB_v4 ] [ 11/13/21 12:00:28 ]
[ Abuse_IPDB_v4 ] Downloading update [ 11/13/21 12:00:35 ] .. completed ..
6358 /var/db/pfblockerng/deny/Abuse_IPDB_v4.txt
Nov 13 12:00 Abuse_IPDB_v4

The end result for me has been a real increase catching the number of IP’s hitting my website that are likely up to no good: (12 hour snapshot below)

Hope some of you find this useful. Also if you can take advantage of fail2ban you should. In my case my next step/script will be CONTRIBUTE to the database by reporting any additional IP’s that leak through my firewall using the reporting portion of the API.