Creating a Robots.txt fileGet Web Design Tips and Tricks on mps-web-design.com. Creating a Robots.txt file topic will increase your understanding on Web Design Tips and Tricks. We at mps-web-design.com only provide news, articles, information in Web Design Tips and Tricks. Web Design Tips and Tricks at mps-web-design.com provides the most up to date news and articles. If you have questions please do not hesitate to contact us.
These pages, although optimized for different search engines, often turn out to be pretty similar to each other. The search engines now have the ability to detect when a site has created such similar looking pages and are penalizing or even banning such sites. In order to prevent your site from being penalized for spamming, you need to prevent the search engine spiders from indexing pages which are not meant for it, i.e. you need to prevent AltaVista from indexing pages meant for Google and vice-versa. The best way to do that is to use a robots.txt file. You should create a robots.txt file using a text editor like Windows Notepad. Don't use your word processor to create such a file. Here is the basic syntax of the robots.txt file: User-Agent: [Spider Name] Disallow: [File Name] For instance, to tell AltaVista's spider, Scooter, not to spider the file named myfile1.html residing in the root directory of the server, you would write User-Agent: Scooter Disallow: /myfile1.html To tell Google's spider, called Googlebot, not to spider the files myfile2.html and myfile3.html, you would write User-Agent: Googlebot Disallow: /myfile2.html Disallow: /myfile3.html You can, of course, put multiple User-Agent statements in the same robots.txt file. Hence, to tell AltaVista not to spider the file named myfile1.html, and to tell Google not to spider the files myfile2.html and myfile3.html, you would write User-Agent: Scooter Disallow: /myfile1.html User-Agent: Googlebot Disallow: /myfile2.html Disallow: /myfile3.html If you want to prevent all robots from spidering the file named myfile4.html, you can use the * wildcard character in the User-Agent line, i.e. you would write User-Agent: * Disallow: /myfile4.html However, you cannot use the wildcard character in the Disallow line. Once you have created the robots.txt file, you should upload it to the root directory of your domain. Uploading it to any sub-directory won't work - the robots.txt file needs to be in the root directory. I won't discuss the syntax and structure of the robots.txt file any further - you can get the complete specifications from here. Now we come to how the robots.txt file can be used to prevent your site from being penalized for spamming in case you are creating different pages for different search engines. What you need to do is to prevent each search engine from spidering pages which are not meant for it. For simplicity, let's assume that you are targeting only two keywords: 'tourism in Australia' and 'travel to Australia'. Also, let's assume that you are targeting only three of the major search engines: AltaVista, HotBot and Google. Now, suppose you have followed the following convention for naming the files: Each page is named by separating the individual words of the keyword for which the page is being optimized by hyphens. To this is added the first two letters of the name of the search engine for which the page is being optimized. Hence, the files for AltaVista are tourism-in-australia-al.html travel-to-australia-al.html The files for HotBot are tourism-in-australia-ho.html travel-to-australia-ho.html The files for Google are tourism-in-australia-go.html travel-to-australia-go.html As I noted earlier, AltaVista's spider is called Scooter and Google's spider is called Googlebot. A list of spiders for the major search engines can be found here. Now, we know that HotBot uses Inktomi and from this list, we find that Inktomi's spider is called Slurp. Using this knowledge, here's what the robots.txt file should contain: User-Agent: Scooter Disallow: /tourism-in-australia-ho.html Disallow: /travel-to-australia-ho.html Disallow: /tourism-in-australia-go.html Disallow: /travel-to-australia-go.html User-Agent: Slurp Disallow: /tourism-in-australia-al.html Disallow: /travel-to-australia-al.html Disallow: /tourism-in-australia-go.html Disallow: /travel-to-australia-go.html User-Agent: Googlebot Disallow: /tourism-in-australia-al.html Disallow: /travel-to-australia-al.html Disallow: /tourism-in-australia-ho.html Disallow: /travel-to-australia-ho.html When you put the above lines in the robots.txt file, you instruct each search engine not to spider the files meant for the other search engines. When you have finished creating the robots.txt file, double-check to ensure that you have not made any errors anywhere in it. A small error can have disastrous consequences - a search engine may spider files which are not meant for it, in which case it can penalize your site for spamming, or, it may not spider any files at all, in which case you won't get top rankings in that search engine. An useful tool to check the syntax of your robots.txt file can be found here. While it will help you correct syntactical errors in the robots.txt file, it won't help you correct any logical errors, for which you will still need to go through the robots.txt thoroughly, as mentioned above. Bonsai Gardening Secrets. - Discover over 95 pages of insider secrets to creating stunning bonsai trees. PowerPause -The Fantastic Formula. - The PowerPause is a simple, 3 minute, 3 step formula for creating fantastic success in money, health and relationships. I came across the article from Nicholas Carr's post Should the Net forget? This is an interesting consequence that seems to be getting pushed on SEO, rather then perhaps looking at it from the aspect of accountable reporting, no? Nicholas states that:
The result is that:
In SummarySo, in the past as the print info (newspaper issues) simply disappeared or, more recently, as they hid the content behind paywalls and poor SEO, newspapers didn't have to worry about the consequences of What do you think the answer should be? Nicholas Carr asks Should the Net forget? I'm not so sure, and I don't think that the answer is that simple. There's a learning curve to moving print onto the web, and this case encompasses one facet of what needs to be conisdered, but it would be great if some form of integrity from those doing the reporting kept these kinds of things from happening. Article Index: | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 |
More Articles:1. How to Transfer Your Web Site to Another Host Without Losing It You want to change web hosts because you have found a lessexpensive solution for your web site. You are not happy with theservices your host is providing. The response time for technicalsupport is too slow. You make the decision to change web hosts. Now you are faced with the daunting task of switching web hostswithout losing the files and beautiful design of your web site.How do you achieve this? Here are the steps you need to take: 1. Make a complete copy of your web site.Use your favorite FTP… 2. Project Discovery: The First Step to Building Strong Web Communication You're in a hurry to get your Web site going. You've got tight deadlines. But don't let your haste get in the way of your success. A company's Web site should embody the company, reflect its values and strengthen its identity. When you pick your Web design team, make sure they are going to learn your business inside and out. Good Web design firms know to start a project with as much information as possible. They gather it during the Discovery phase. Discovery encompasses a full-scale examinatio… 3. TEN Answers That Turn Your Visitors Into Your Customers By Radhika Venkata You know... Before purchasing your product a person comes to your order page minimum of 7 times? Nobody buys a product or service in their first visit. People search, compare, estimate, test... They look for affordable and good product that fulfil their needs.Before buying they get so many questions in their mind.If you answer these questions, sure they will become your potential customers.****1.What did I get from your product?****List out the benefits on your order page. If you don't have a … 4. How To Supercharge Your Website By Jason A. Martin I am often asked “Jason, I get people to my website, but nothing happens. What am I doing wrong?” Is this you? Did you build a website only to have people come and leave? If so, you can still save your website. By following some tried and true rules, your website can become supercharged and be all that you envisioned.First, we need to dispel one myth. The Internet amazes almost no one anymore. When I started in the early 1990’s it was a whole different story. Today, the Internet is a major f… |
||||
