Sometimes
googlebot IP’s get blocked in CSF / APF firewall and bots are not able to visit your websites not can google list your pages in search result.
Google never makes their IP addresses list open as the IP addresses keep on changing. The only
way to find out googlebot IP is by checking Reverse DNS.
following is a shell script to keep an eye on blocked googlebot IP’s.
HTML Code:
#!/bin/bash
awk ‘{print $1}’ /etc/csf/csf.deny | grep -v ‘#’ > /tmp/deny_list.txt
for i in `cat /tmp/deny_list.txt`
do
host $i | grep googlebot
if [ $? -eq 0 ]
then
echo $i >> /tmp/googlebot-ip-list.txt
fi
done
if [ -f /tmp/googlebot-ip-list.txt ]
then
echo “Check the reverse DNS before unblocking above IP” >> /tmp/googlebot-ip-list.txt
mail -s “Googlebot IP is blocked on `hostname`” your_email@address < /tmp/googlebot-ip-list.txt
fi
rm -f /tmp/googlebot-ip-list.txt
rm -f /tmp/deny_list.txt
create a file in /etc/cron.hourly/ to check googlebot IP every hour or in /etc/cron.daily/ to check on daily basis as per your requirement.
chmod 755 / 750 to execute it.
for apf firewall replace the code
HTML Code:
awk ‘{print $1}’ /etc/csf/csf.deny | grep -v ‘#’ > /tmp/deny_list.txt
with
HTML Code:
awk ‘{print $1}’ /etc/apf/deny_hosts.rules | grep -v ‘#’ > /tmp/deny_list.txt
Thats it
Matthew
UK Web Hosting - Cheap Windows Linux cPanel Shared Reseller Web Hosting in UK
googlebot IP block check script
Join our Affiliate Program for FREE and earn 20% commission on each referral. 
Linear Mode
