Robots are very important nowadays to handle bots from crawling your website or pages, Hence it is important to add robots.txt file in your website’s root directory.
One of the main reason of using robots.txt is hiding your private pages from search results like for .e.g. your-website.com/wp-admin
Generally, If you don’t use robots.txt file, bots (spiders) will crawl your entire webpages, some are not necessary or it can also be duplicate contents.
I would better suggest you to use this file to guide bots to crawl your websites/blogs properly.
There is no need to create robots.txt nowadays because while installing WordPress, It automatically creates a virtual robots file with default coding. (Example given below)
How should I verify If I have default/virtual robots.txt file ?
Just you need to verify is by typing this url in your browser:
Still there are many threads on WordPress support forums related to editing this virtual robots.txt, In my opinion editing this virtual robots file is difficult.
This is what I have faced with my blog (BlogDum); I tried every possible ways to solve but failed to edit this virtual file, Hence after doing a good research.
I found out two ways to edit robots.txt file one with plugin and other without plugin, both worked for me like a charm.
Method 1 – Without plugin.
1. Create a text file
2. Add your desired code in this file or you can check mine robots.txt
3. Save file as “robots.txt” – Don’t misspell
4. Upload this file in your root directory (Same place where you add Google webmasters verification code).
5. Now check www.yourdomainname.com/robots.txt
6. If you can see the file and codes then you have successfully uploaded it.
Note: Check if there is already a robots.txt file in your root folder, if yes edit and update it.
Method 2 – Using Plugin
Go to your Dashboard > Plugins > Add New
Type: WP Robots Txt
Or Use this link to download zip file directly: http://wordpress.org/extend/plugins/wp-robots-txt/
Install and activate it.
Go to Dashboard > Settings > Reading