SEM Tutorials

When I start my career in SEO the first challenging task I faced was how to prevent google or other Search Engine spiders from indexing certain folders or pages in a website. You know it is very simple one but think about a newbie who is hearing this stuff at first time in his life from the client. The major problem was there was no senior guy available at that time to ask help and the Internet connection in my laptop was not working as the wireless adapter is damaged. So I couldn't even google it for an answer. Then I started to think how can I prevent Search Engine spiders from a particular page? Then the idea of robots.txt came to my mind. I know that using this we can write rules for the web spiders. So immediately I answered them " It is very simple by writing disallow for a folder or file in robots.txt for Search Engine spiders". They accepted that and they thought I know this stuff very much and leave the question. 

After the client call I started to write diallow code for google spider and then the next doubt came. What is the difference between disallow: and disallow:/

So here I am writing the basic robots.txt optimization tips for newbies who may have some doubts in writing the proper rules in robots.txt.

How to allow spiders to index all pages in my website

It is very simple to write rules for allowing spiders to index all pages and folders in your website. The rule is

User-agent: *
Disallow:

or

User-agent: *
allow:/

How to prevent Search Engine spiders from indexing a website

The rule is

User-agent: *
Disallow:/


How to prevent Search Engine spiders from indexing particular page

The rule to prevent google and other search engine spiders from indexing certain page in our website and allow to index rest of our website is:

User-agent: *
Disallow:/ Folder


The folder can change with the name of the file we need to exclude.

Related Articles

1. How to re-index a manually removed websites in google index list

2. How to remove cached pages from google index 

Home