googlebot IP block check script

[eUKMatthew]

Sometimes googlebot IP’s get blocked in CSF / APF firewall and bots are not able to visit your websites not can google list your pages in search result.

Google never makes their IP addresses list open as the IP addresses keep on changing. The only way to find out googlebot IP is by checking Reverse DNS.

following is a shell script to keep an eye on blocked googlebot IP’s.

     #!/bin/bash

    awk ‘{print $1}’ /etc/csf/csf.deny | grep -v ‘#’ > /tmp/deny_list.txt

    for i in `cat /tmp/deny_list.txt`
    do
    host $i | grep googlebot

    if [ $? -eq 0 ]
    then
    echo $i >> /tmp/googlebot-ip-list.txt
    fi

    done

    if [ -f /tmp/googlebot-ip-list.txt ]
    then

    echo “Check the reverse DNS before unblocking above IP” >> /tmp/googlebot-ip-list.txt
    mail -s “Googlebot IP is blocked on `hostname`” your_email@address < /tmp/googlebot-ip-list.txt

    fi

    rm -f /tmp/googlebot-ip-list.txt
    rm -f /tmp/deny_list.txt

create a file in /etc/cron.hourly/ to check googlebot IP every hour or in /etc/cron.daily/ to check on daily basis as per your requirement.

chmod 755 / 750 to execute it.

for apf firewall replace the code

awk ‘{print $1}’ /etc/csf/csf.deny | grep -v ‘#’ > /tmp/deny_list.txt

with

awk ‘{print $1}’ /etc/apf/deny_hosts.rules | grep -v ‘#’ > /tmp/deny_list.txt

Thats it

Matthew
UK Web Hosting - Cheap Windows Linux cPanel Shared Reseller Web Hosting in UK

[Brian]

I am presuming something like this is only needed for Private / VPS / Virtual Private than Cpanel shared hosting ?

[Daniel]

Originally Posted by Brian

I am presuming something like this is only needed for Private / VPS / Virtual Private than Cpanel shared hosting ?

Indeed, This will most likely already be executed on shared servers

[eUKhost.com]

This will work for only VPS, Semi-dedicated or Dedicated Server customers. Shared hosting customers need to rely on our scripts which we have on our servers. We keep on clearing our denied IPs list to allow bots to crawl your websites.

You can check awstats or webalizer statistics from your cpanel control panel as you will manage to figure out when and at what time a specific search engine bot managed to visit your website. Our website gets crawled every few as we have high volume of posts on these forums daily. any new thread from these forums gets listed in google in 10 - 20 minutes.

[WelshTom]

I've always been a bit concerned of the Google Bot ending up the firewall due to mod_security violations or something like that, great resource, thanks for that.