UK WEB HOSTING FORUM FOR DISCUSSION ON WEB HOSTING SERVICE AND SUPPORT
LINUX HOSTING WINDOWS HOSTING PACKAGES SHOPPING CART OSCOMMERCE ZEN CART AGORA
ECOMMERCE HOSTING ASP MSSQL FRONTPAGE HOSTING PHP MYSQL HOSTING DISCUSSION FORUM
CPANEL RESELLER HOSTING DEDICATED SERVER VPS HOSTING PLESK VIRTUOZZO
Quick Search
Your forum announcement here!

  UK Web Hosting | Dedicated Server Windows and Linux VPS Forum > Services > Search Engine Optimization

Reply
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 18-10-2008, 21:09
System Administrator
 
Join Date: Aug 2007
Posts: 56
Default googlebot IP block check script

Sometimes googlebot IP’s get blocked in CSF / APF firewall and bots are not able to visit your websites not can google list your pages in search result.

Google never makes their IP addresses list open as the IP addresses keep on changing. The only way to find out googlebot IP is by checking Reverse DNS.

following is a shell script to keep an eye on blocked googlebot IP’s.

HTML Code:
     #!/bin/bash

    awk ‘{print $1}’ /etc/csf/csf.deny | grep -v ‘#’ > /tmp/deny_list.txt

    for i in `cat /tmp/deny_list.txt`
    do
    host $i | grep googlebot

    if [ $? -eq 0 ]
    then
    echo $i >> /tmp/googlebot-ip-list.txt
    fi

    done

    if [ -f /tmp/googlebot-ip-list.txt ]
    then

    echo “Check the reverse DNS before unblocking above IP” >> /tmp/googlebot-ip-list.txt
    mail -s “Googlebot IP is blocked on `hostname`” your_email@address < /tmp/googlebot-ip-list.txt

    fi

    rm -f /tmp/googlebot-ip-list.txt
    rm -f /tmp/deny_list.txt
create a file in /etc/cron.hourly/ to check googlebot IP every hour or in /etc/cron.daily/ to check on daily basis as per your requirement.

chmod 755 / 750 to execute it.

for apf firewall replace the code

HTML Code:
awk ‘{print $1}’ /etc/csf/csf.deny | grep -v ‘#’ > /tmp/deny_list.txt
with

HTML Code:
awk ‘{print $1}’ /etc/apf/deny_hosts.rules | grep -v ‘#’ > /tmp/deny_list.txt
Thats it

Matthew
UK Web Hosting - Cheap Windows Linux cPanel Shared Reseller Web Hosting in UK
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #2 (permalink)  
Old 19-10-2008, 07:30
Brian's Avatar
Premium Member
 
Join Date: Nov 2005
Location: New Mexico
Posts: 849
Default

I am presuming something like this is only needed for Private / VPS / Virtual Private than Cpanel shared hosting ?
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #3 (permalink)  
Old 19-10-2008, 09:57
Daniel's Avatar
Got root?
 
Join Date: Aug 2008
Location: England, UK
Posts: 175
Send a message via MSN to Daniel Send a message via Skype™ to Daniel
Default

Quote:
Originally Posted by Brian View Post
I am presuming something like this is only needed for Private / VPS / Virtual Private than Cpanel shared hosting ?
Indeed, This will most likely already be executed on shared servers
__________________
Dan Miller
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #4 (permalink)  
Old 19-10-2008, 15:54
eUKhost.com's Avatar
Chief Marketing Officer
 
Join Date: Sep 2005
Posts: 4,997
Send a message via AIM to eUKhost.com Send a message via MSN to eUKhost.com Send a message via Skype™ to eUKhost.com
Default

This will work for only VPS, Semi-dedicated or Dedicated Server customers. Shared hosting customers need to rely on our scripts which we have on our servers. We keep on clearing our denied IPs list to allow bots to crawl your websites.

You can check awstats or webalizer statistics from your cpanel control panel as you will manage to figure out when and at what time a specific search engine bot managed to visit your website. Our website gets crawled every few as we have high volume of posts on these forums daily. any new thread from these forums gets listed in google in 10 - 20 minutes.
__________________
UK Web Hosting || Business Hosting || eUKhost Knowledgebase
Toll Free : 0808 262 0255 || MSN : mark @ eukhost.com || AIM : eukmark
A bunch of Sheep led by a Lion is better than a bunch of Lions led by a Sheep.
__________________________________________________

Great Opportunity :: Join our Affiliate Program for FREE and earn 20% commission on each referral.
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #5 (permalink)  
Old 20-10-2008, 14:24
WelshTom's Avatar
Moderator
 
Join Date: May 2007
Location: Newport, Wales
Posts: 935
Send a message via AIM to WelshTom Send a message via MSN to WelshTom Send a message via Yahoo to WelshTom
Default

I've always been a bit concerned of the Google Bot ending up the firewall due to mod_security violations or something like that, great resource, thanks for that.
__________________
Thomas Williams
Jst Hosting
Freephone: 0800 242 5302
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
Reply



Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is On
Trackbacks are On
Pingbacks are On
Refbacks are On


All times are GMT. The time now is 03:03.

 

Powered by vBulletin® Version 3.7.2
Copyright ©2000 - 2009, Jelsoft Enterprises Ltd.
LinkBacks Enabled by Web Hosting 3.1.0
Copyright © 2001-2008, eUKhost.com. All rights reserved.

 
Site Map

VPS Hosting
VPS Hosting plans

Dedicated Server Hosting
Dedicated Server plans

Business Web Hosting
100% uptime Hosting

Cpanel Hosting
cPanel Shared Hosting

Reseller Hosting
Reseller Web Hosting

Windows Hosting
Windows Shared Hosting

Windows VPS

Windows VPS Hosting

Semi Dedicated Servers
Semi-Dedicated Hosting

Dedicated Server Mirroring
Dedicated Server Mirroring

Webhosting Knowledgebase
Frequently asked Questions

Web Hosting Blog
eUKhost Blog

Web Hosting Support
Support Helpdesk

UK Data Center
eUKhost Datacenter

Web Hosting Forum
eUKhost Forum

Support Tutorials
Online Flash Tutorials

Offsite Back-up Plans
Remote Backup Service

Customer Testimonials
eUK Customer Testimonials


knowledgebase articles

eUKhost.com Services

Pre-Sales Questions
Pre-sales FAQ's

Domain Names
Domain registration FAQ's

cPanel Hosting
cPanel Hosting FAQ's

Windows Web Hosting
Plesk Control Panel

Reseller Hosting
Reseller Hosting FAQ's

VPS Hosting
Virtual Private Server

Semi-Dedicated Servers
Semi-Dedicated FAQ's

Dedicated Servers
Dedicated Server Hosting


popular blog categories


Web Hosting
Website Hosting articles

UK Web Hosting
UK Hosting articles

Dedicated Server Hosting
Dedicated Server guidelines

VPS Hosting
VPS hosting articles

cPanel Hosting
cPanel Hosting articles

Linux Operating System
Linux Operating techniques

Windows Web Hosting
Windows plesk articles

Web Hosting
Web Hosting Service