Robots.txt Generator
Robots.txt Generator Tool
Allowed User Agents:There are over 200 user agents (spiders) that can read through your webpages when indexing. Since new spiders get added/removed daily, we have limited the choices to the major search engines (which make their User-Agent names public). The default setting is to apply your preferences to All Agents. Excluded URLS:
You may impose restrictions on which webpages to disallow indexing. By default, most users will want to allow all directories except their /cgi-bin directory which commonly holds scripts. To enable all webpages, check the "Enable All Webpages" checkbox. Otherwise, enter each webpage or directory path in the exclusion box, one per line (all paths must end with a "/"). If you checked off Enable All Webpages, this box will not be read. Example: "http://www.sample.com/cgi-bin/" (Excludes /cgi-bin/ directory)
Example: "http://www.sample.com/images/" (Excludes /images/ directory)
Example: "http://www.sample.com/hello.html" (Excludes /hello.html webpage)
| Robots.txt Generated Results |
|
|