WordPress Robots.txt Sample

WordPress Robots.txt

Adding Sitemaps to WordPress Robots.txt

Allowing all Bots

  • Allowing any Bots to Crawl

Not Allowing any Bots

  • Not Allowing any Bots to Crawl

Block a Folder

Block a File

 Block a page and/or a directory named private

Block All Sub Folders starting with private

Block URL’s end with

Block URL’s which includes Question Mark (?)

Block a File Type

Block all Paginated pages which don’t have “?”  » Read more about: WordPress Robots.txt Sample  »

SEO Cannibalisation

SEO Cannibalisation is a process of Consolidating many similar pages which dilutes SEO Value, in to a single page.

Similar Pages

  • Many Pages with Exactly Similar Page Titles
  • Many Pages with content targeting the same keyword

Reason for SEO Cannibalisation

Have too many similar pages?

 » Read more about: SEO Cannibalisation  »

SEO Canonicalisation

Canonicalisation in terms of SEO

Show Google which one is the Preferred URL or Main URL

Canonicalisation Code Implementation

Similar Pages in the eyes of Google

All below pages are the same:


  • http://www.example.com/
  • http://example.com/

Apache Web Server

  • http://example.com/index.html
  • http://www.example.com/index.html
  • http://example.com/index.php
  • http://www.example.com/index.php

Microsoft Internet Information Services (IIS)

  • http://www.example.com/default.asp (or .aspx )
  • http://example.com/default.asp (or .aspx)

Use Redirection to the Preferred URL

This has to be considered if we have exactly same page with multiple URL’s as mentioned above.

 » Read more about: SEO Canonicalisation  »