WordPress Robots.txt Sample

WordPress Robots.txt

Adding Sitemaps to WordPress Robots.txt

Explanation
Allowing all Bots

  • Allowing any Bots to Crawl

Not Allowing any Bots

  • Not Allowing any Bots to Crawl

Block a Folder

Block a File

 Block a page and/or a directory named private

Block All Sub Folders starting with private

Block URL’s end with

Block URL’s which includes Question Mark (?)

Block a File Type

Block all Paginated pages which don’t have “?”  » Read more about: WordPress Robots.txt Sample  »