Parse Apache logs for traces of server abuse

End of a month is a perfect time to look at a Web site’s logs – many servers are setup to archive and file away the logs at the end of the month and some servers don’t even keep more than the current month worth of logs. So, it’s November 28th and I’d like to take a look at who’s been abusing this site’s resources. Apache logs for an entire month provide a wealth of information about that!

Every Web site has certain URLs that should not be receiving as many visits as its home page. However, with proliferation of fraudulent forum registrations I routinely find that the registration page of this site’s forum receives approximately 10 times the amount of traffic the homepage does. Most of these visits are frauds, it’s quite apparent considering the amount of registrations I see per month that complete the required email verification.

So, I’ve decided to deny the service at the firewall level to those IPs that produce the largest amounts of fraudulent forum registrations. My main assumption here is that it’s conceivable that a large network would show 50-100 registrations a month (and if your site is much more popular, you’ll have to adjust that upwards) but a small regional network with less than a 1000 addresses simply cannot have this many unless most of them are malicious registration attempts.

Awk to the rescue! Well, OK, not just awk but sort and uniq as well but awk does all the heavy lifting here.

Run this code in the ~/access-logs/ directory on your server. This assumes the standard cPanel practice of creating this symbolic links to the logs otherwise located at /usr/local/apache/domlogs/username . If you host your server on a different platform, adjust the locations accordingly. Also, the code assumes that the log file is named as the domain, i.e. the log file for Web the site at is called ~/access-logs/ – another standard cPanel practice. Adjust your locations accordingly if you are hosted on a different platform.

awk -F\" '($2 ~ "^GET /forums/register.php")' ~/access-logs/*com | awk '{print $1}' | sort | uniq -c | sort -nr > ~/reg_attempts_IPs.txt

In this case, requests for /forums/register.php is what I’m mostly interested in. You can put any URL you like in there, just as long as you can be sure that there cannot be many legitimate requests for that URL.

In my case the command created a list of 2000+ IPs that has requested /forums/register.php which was sorted in the descending order. I decided to block only the worst offenders this time but this IP deny list will get fine-tuned using the results of a few following months and I think that I will end up denying any network that produce 100+ registration attempts in a month.

Here is the list of the worst offenders for November 2011. The first number is the amount of attempts, the second is the IP itself


Note that some of them (example highlighted) are on the same network – the simple script does not come up with subnet calculation by itself. Additionally, I wanted to look at the DNS information before denying the entire subnet – some of these IPs belong to ISPs that recycle IP addresses and they own vast ranges – in millions – and so it did not make much sense to bad those.

But I was able to compile a short list of those that definitely warrant denial at the firewall level. This month the dubious distinction belongs to:

  • – Voxility SRL
  • – T-2 Access Network
  • – Lamboley Gameserver
  • – HostingLtd Web Server Network Previously Altaire Limited
  • – Redstation Limited Mail Infrastructure (WTF?)
  • – intergenia AG
  • – IP-SPACE-FOR-DEDICATED-SERVERS (meaning – proxy)
  • – SteepHost-DC-UA

This list is far from being complete, I will be definitely adding to it in the next months.

Here are the iptables rules for your IP banning convenience:
iptables -A INPUT -s -j DROP
iptables -A INPUT -s -j DROP
iptables -A INPUT -s -j DROP
iptables -A INPUT -s -j DROP
iptables -A INPUT -s -j DROP
iptables -A INPUT -s -j DROP
iptables -A INPUT -s -j DROP
iptables -A INPUT -s -j DROP
iptables -A INPUT -s -j DROP
service iptables save

Run these rules as root and watch the useless load on your server go down.

Happy banning!