The Correct Way To Use The robot.txt File

After optimizing your web site most webmasters don't consider using the robot.txt file. This is a really important file for your website. It let the spiders and crawlers know what they can and can't index. This helps in keeping them out of folders you don't want index like the admin or stats folder.

Here's a list of factors Which You Can incorporate in a robot.txt file and there significance:

User-agent: In this field you can define a particular robot to describe access policy even a "*" for many robots clarified in example.


Disallow: In the area you define folders and the files to not add from the crawl.

The # will be to represent comments

Here are some examples of a robot.txt file

User-agent: *


The above would allow all spiders index all of the content but in some cases the search engines might ignore the robots and index the content anyway.

Here another

User-agent: *

Disallow: /cgi-bin/

The above would block all spiders from indexing the cgi-bin directory.

User-agent: googlebot


User-agent: *


Disallow: /cgi-bin/

Disallow: /admin/

Disallow: /stats/

From the above mentioned example googlebot can index everything while the other spiders can't index admin.php, cgi-bin, admin, and stats directory. See which you are able to block single files such as admin.php. Just check up on Alkanyx if you need more details on php script with login.

Leave a Reply

Your email address will not be published. Required fields are marked *