[eUKMatthew]

Sometimes googlebot IP’s get blocked in CSF / APF firewall and bots are not able to visit your websites not can google list your pages in search result.

Google never makes their IP addresses list open as the IP addresses keep on changing. The only way to find out googlebot IP is by checking Reverse DNS.

following is a shell script to keep an eye on blocked googlebot IP’s.

     #!/bin/bash

    awk ‘{print $1}’ /etc/csf/csf.deny | grep -v ‘#’ > /tmp/deny_list.txt

    for i in `cat /tmp/deny_list.txt`
    do
    host $i | grep googlebot

    if [ $? -eq 0 ]
    then
    echo $i >> /tmp/googlebot-ip-list.txt
    fi

    done

    if [ -f /tmp/googlebot-ip-list.txt ]
    then

    echo “Check the reverse DNS before unblocking above IP” >> /tmp/googlebot-ip-list.txt
    mail -s “Googlebot IP is blocked on `hostname`” your_email@address < /tmp/googlebot-ip-list.txt

    fi

    rm -f /tmp/googlebot-ip-list.txt
    rm -f /tmp/deny_list.txt

create a file in /etc/cron.hourly/ to check googlebot IP every hour or in /etc/cron.daily/ to check on daily basis as per your requirement.

chmod 755 / 750 to execute it.

for apf firewall replace the code

awk ‘{print $1}’ /etc/csf/csf.deny | grep -v ‘#’ > /tmp/deny_list.txt

with

awk ‘{print $1}’ /etc/apf/deny_hosts.rules | grep -v ‘#’ > /tmp/deny_list.txt

Thats it

Matthew
UK Web Hosting - Cheap Windows Linux cPanel Shared Reseller Web Hosting in UK