logotype

How to block IP addresses from accessing webpages in Ubuntu

 

I have recently had a scenario when one of my websites was using an older version of a CMS that wasn't as secure as it could have been and as such took down one of my Ubuntu servers by overloading it with Apache and MySQL requests. When I examined the web logs I found that most of the dodgy traffic was coming from a few specific IP addresses. I used UFW (Uncomplicated Firewall - a front end to iptables) to give the server a break.


It is worth noting that the rules are in order with the first matching being the rule that is used. We are going to add all www traffic but block specific IP addresses, so we are going to do the blocking first, then at the very end enable www then enable the firewall.

 

Steps to Reproduce:

Install UFW:

sudo apt-get install ufw

Check the Status:

sudo ufw status numbered

Make sure any changes are for IPv4 and IPV6

sudo vi /etc/default/ufw
IPV6=yes

Now make sure you don't cut yourself off

sudo ufw allow ssh
sudo ufw allow ftp
sudo ufw allow www

If you need to open up ports:

sudo ufw allow 2000:2100/tcp

And now to the business end. Look at your Apache log files to find the offenders:

tail -f /var/log/apache2/access.log

Example output:

130.185.139.213 - - [14/Jul/2014:14:41:28 +0000] "GET / HTTP/1.0" 200 6684 "-" "-"
128.199.159.98 - - [14/Jul/2014:14:41:36 +0000] "GET / HTTP/1.1" 200 4763 "http://www.yourwebsite.com/" "Mozilla/5.0 (iPad; CPU OS 6_1_3 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10B329 Safari/8536.25"
178.73.202.206 - - [14/Jul/2014:14:41:44 +0000] "GET /?q=content/adam_lanza_look_warrants\xffasperger testing adults\xffexternal nofollow HTTP/1.1" 404 2961 "http://www.yourwebsite.com/" "Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3"
178.73.202.206 - - [14/Jul/2014:14:41:45 +0000] "GET /?q=node/add HTTP/1.1" 403 1627 "http://www.yourwebsite.com/" "Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3"
178.73.202.206 - - [14/Jul/2014:14:41:45 +0000] "GET /?q=user/register HTTP/1.1" 403 3177 "http://www.yourwebsite.com/?q=node/add" "Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3"
162.243.114.187 - - [14/Jul/2014:14:42:01 +0000] "GET / HTTP/1.1" 200 4748 "http://www.yourwebsite.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36"
87.149.73.204 - - [14/Jul/2014:14:42:25 +0000] "GET /content/pay_attention_family_survival_system_review HTTP/1.1" 404 516 "http://www.yourwebsite.com/" "Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3"
91.217.10.134 - - [14/Jul/2014:14:42:29 +0000] "GET /content/akasse HTTP/1.1" 404 490 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) )"

You will quickly see which IP addresses are hitting your servers the hardest, and which ones are not valid traffic.

(Remember not to block googlebot unless it is someone pretending to be googlebot)

To quickly add the IP addresses into UFW use the following command, replacing the IP addresses you wish to block:

ufw deny from 128.199.159.98 ; ufw deny from 178.73.202.206 ; ufw deny from 87.149.73.204  ; ufw deny from 94.23.59.173 ; ufw deny from 66.35.75.11

Once you are happy then allow all others to access www traffic

ufw allow www

If in future you want to remove a specific rule (like www when you are adding more IP addresses:

ufw status numbered
ufw delete 10 //or which ever number the 80/tcp rule is


If you are unsure if the IP address is a googlebot or not - or you don't know if it is legit you can do a forward and reverse look-up on its IP address. Unfortunately Google doesn't release a list of their bots IP addresses as they are constantly changing. So for example you might see the following line in your Apache access.log:

66.249.79.69 - - [14/Jul/2014:13:58:23 +0000] "GET /?q=shop&page=58 HTTP/1.1" 200 5654 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Run the following:

host 66.249.79.69
69.79.249.66.in-addr.arpa domain name pointer crawl-66-249-79-69.googlebot.com.

then a reverse to confirm:

host crawl-66-249-79-69.googlebot.com
crawl-66-249-79-69.googlebot.com has address 66.249.79.69

If the IP address doesn't match both ways - block it

 Now to enable UFW

sudo ufw enable

 

Copyright 2017 OReillyIT. All rights reserved.