WordPress Robots.txt Sample

WordPress Robots.txt

Adding Sitemaps to WordPress Robots.txt

Explanation

Allowing all Bots

  • Allowing any Bots to Crawl

Not Allowing any Bots

  • Not Allowing any Bots to Crawl

Block a Folder

Block a File

 Block a page and/or a directory named private

Block All Sub Folders starting with private

Block URL’s end with

Block URL’s which includes Question Mark (?)

Block a File Type

Block all Paginated pages which don’t have “?” at the end

  • http://www.example.com/blog/? ( Allow )
  • http://www.example.com/blog/?page=2 ( Not Allow )

Helps us to Block Paginated pages from Crawling

Using Hash

Bots / User Agents

Top 10 Bots

Robot
bingbot
Googlebot
Googlebot Mobile
AhrefsBot
Baidu
MJ12bot
proximic
A6
ADmantX
msnbot/2.0b

Individual Crawl request for each Bots


Leave a Reply

Your email address will not be published. Required fields are marked *