Regarding robots.txt, most people may think that it can be written or not. If you write it wrong, you will be in trouble, so you might as well not write it at all. In fact, in search engine optimization, robots.txt has several wonderful uses. Now let’s take a look at the wonderful uses of robots.txt one by one.
1. When creating a new site, guide the spider to crawl the list page. When a new website is first built, write the website's column address in your robots.txt for crawling. Examples are as follows
# http://www.xxxxx.cn/ Website optimized robots.txt
User-agent: *
Disallow: /
Disallow: /plus/advancedsearch.php
Disallow: /plus/car.php
Disallow: /plus/carbuyaction.php
Disallow: /plus/shops_buyaction.php
Disallow: /plus/erraddsave.php
Disallow: /plus/posttocar.php
Disallow: /plus/disdls.php
Disallow: /plus/feedback_js.php
Disallow: /plus/mytag_js.php
Disallow: /plus/rss.php
Disallow: /plus/search.php
Disallow: /plus/recommend.php
Disallow: /plus/stow.php
Disallow: /plus/count.php
Disallow: /include
Disallow: /templets
The default robots.txt of dedecms is used, which I personally feel is pretty good.
2. Block low-quality pages from being included. For example, one of your web pages is a dynamic page, content.asp, and you can obtain different content through the parameter id. If you need to add a parameter style to display pages with different styles but the same content to different users, then, For a page with the same content with ID 1, there may be several links.
content.asp?id=1
content.asp?id=1&style=red
content.asp?id=1&style=green
For these connections, the content is the same but the connections are different, which may lead to the appearance of duplicate content. Here, you can block the inclusion of duplicate pages through robots.txt, as follows:
User-agent: *
Disallow: content.asp?id=*&style=*
3. No need to submit sitemap to major search engines. Google, Microsoft and Yahoo all already support this extension of robots.txt. So it also saves time to submit sitemap
As for the specific writing method of robots.txt, it is recommended to refer to the guide given by Baidu, which is very detailed http://www.baidu.com/search/robots.html
First published on Webmaster.com, please keep the link http://www.zzyoua.cn/ for reprinting.