Believe it or not, there might be times when you DON’T want the search engines to find certain pages or folders of your site.
For example, if you have folders on your web site that you want to be available only to your customers, you can insert a robots.txt file into that particular folder — so that when the search engines spider your entire web site, they will skip this particular folder.
To create a robots.txt file, simply use your web authoring program (or even Notepad) to create a text file. This text file must have a file extension of .txt in order to work properly. And then, in the body of the text file, insert the following:
User-Agent: *
Disallow: /
Then, simply upload your robots.txt file into the folder that you want protected from the search engines.
To shield an individual file from the search engines, insert the following code inside the HEAD tag:
<meta name=”robots” content=”noindex, nofollow”>
Robots.txt files or coding allows you to post pages to your web site that you only want people certain people to view (without the need for password protection). Also, you won’t have to worry about any of the content on these pages competing with those all-important keyword-rich pages that you have throughout the rest of your web site.
For more information regarding search engine optimization, see SEO.
