This will work for only VPS, Semi-dedicated or Dedicated Server customers. Shared hosting customers need to rely on our scripts which we have on our servers. We keep on clearing our denied IPs list to allow bots to crawl your websites.
You can check awstats or webalizer statistics from your cpanel control panel as you will manage to figure out when and at what time a specific search engine bot managed to visit your website. Our website gets crawled every few as we have high volume of posts on these forums daily. any new thread from these forums gets listed in google in 10 - 20 minutes.
