How to configure robots.txt to allow everything?
My robots.txt
in Google Webmaster Tools shows the following values:
User-agent: *
Allow: /
What does it mean? I don't have enough knowledge about it, so looking for your help. I want to allow all robots to crawl my website, is this the right configuration?
Solution 1:
That file will allow all crawlers access
User-agent: *
Allow: /
This basically allows all user agents (the *) to all parts of the site (the /).
Solution 2:
If you want to allow every bot to crawl everything, this is the best way to specify it in your robots.txt:
User-agent: *
Disallow:
Note that the Disallow
field has an empty value, which means according to the specification:
Any empty value, indicates that all URLs can be retrieved.
Your way (with Allow: /
instead of Disallow:
) works, too, but Allow
is not part of the original robots.txt specification, so it’s not supported by all bots (many popular ones support it, though, like the Googlebot). That said, unrecognized fields have to be ignored, and for bots that don’t recognize Allow
, the result would be the same in this case anyway: if nothing is forbidden to be crawled (with Disallow
), everything is allowed to be crawled.
However, formally (per the original spec) it’s an invalid record, because at least one Disallow
field is required:
At least one Disallow field needs to be present in a record.
Solution 3:
I understand that this is fairly old question and has some pretty good answers. But, here is my two cents for the sake of completeness.
As per the official documentation, there are four ways, you can allow complete access for robots to access your site.
Clean:
Specify a global matcher with a disallow segment as mentioned by @unor. So your /robots.txt
looks like this.
User-agent: *
Disallow:
The hack:
Create a /robots.txt
file with no content in it. Which will default to allow all for all type of Bots
.
I don't care way:
Do not create a /robots.txt
altogether. Which should yield the exact same results as the above two.
The ugly:
From the robots documentation for meta tags, You can use the following meta tag on all your pages on your site to let the Bots
know that these pages are not supposed to be indexed.
<META NAME="ROBOTS" CONTENT="NOINDEX">
In order for this to be applied to your entire site, You will have to add this meta tag for all of your pages. And this tag should strictly be placed under your HEAD
tag of the page. More about this meta tag here.
Solution 4:
It means you allow every (*
) user-agent/crawler to access the root (/
) of your site. You're okay.