Ways you could control Search Bots

  1. robots.txt
  2. Header Meta Tag
  3. Using X-Robots-Tag
  4. Change HTTP Status code ( Eg:  403 – forbidden ..etc )

Optimise Google Crawl Efficiency

X-Robots-Tag HTTP Header

Robots.txt

Robots Meta Tag

Specify a Search Bot

Few Example Search Bots

Robots Meta Tag Values and Search Engine Support

Robots value Google Yahoo! MSN / Live Ask
index Yes Yes Yes Yes
noindex Yes Yes Yes Yes
follow Yes Not Sure Not Sure Yes
nofollow Yes Yes Yes Yes
none Yes Not Sure Not Sure Yes
noarchive Yes Yes Yes Yes
nosnippet Yes No No No
noodp Yes Yes Yes No
noydir No Yes No No

 

Differerence between Robots.txt and X-Robots-Tag Header and Robots Meta Tag

  • Robots.txt is Crawler Directive ( Sets Direction for the crawler )
  • X-Robots-Tag is Indexer Directive ( Set on a per page and/or per element basis )
  • X-Robots-Tag and Meta Robots Tag are similar but X-Robots-Tag has more flexibility compared to Robots Meta Tag

Priority

  • Robots.txt – Powerful
  • Robots Meta Tag – More Powerful
  • X-Robots-Tag HTTP Header- Most Powerful

Example Situation

  • Robots.txt – Basic Site level and Page Level
  • Robots Meta Tag – HTML Page Level
  • X-Robots-Tag HTTP Header – Includes Various File types also ( Eg: PDF, image ..etc )

Block a certain Data Type