View Single Post
  #34 (permalink)  
Old 29-07-2007, 18:29
twiseman twiseman is offline
Member
 
Join Date: May 2007
Posts: 41
Default

Depends on the bot in question.

Some like Google and Yahoo you can actually limit how often they visit your website hosting. I know Google can be done through the webmaster tools, not sure on Yahoo but I am sure it was mentioned in the blog link above.

You can also limit where the bot can access using robots.txt if you don't want it to index a particular portion of your website hosting. Or you can use robots.txt to create a sitemap if you find it's not index particular parts of your website hosting. I have used the latter method and it has improved the pages indexed on the search engines, especially dynamic content from my CMS.

Google robots.txt there are a loads of website hostings regarding it's use and if you have a google account they give you ways to test your file live.

Tim
Reply With Quote