The most headache-inducing thing for webmasters is that in addition to program and server security, the content may be considered to be collected. Of course, the network itself is resource sharing now. I will not talk about the right or wrong of collection here, but just talk about personal opinions on the prevention of collection.
1. How to distinguish search crawlers
In the past, I have seen codes on the Internet about using asp code to capture search crawlers, and record the crawler's access records to files through the code. I made changes to this code (the original author is unknown) and added it to the program to prevent collection. Now it is made into a function and called directly on the page.
The code copy is as follows:
FunctionCheckSearchRobots(SearchRobotsAgent)
CheckSearchRobots=False
'Custom search crawler, due to the length, this is just a part of the records of crawlers in the main domestic search engines
DimSearchRobots:SearchRobots="Baidu@Baidu|Google@Google|crawler@alexa|Yahoo@Yahoo|sohu@Sohu|MSN@MSN|QihooBot@Qihoo"
dimArrSearchRobots,ArrSearchRobots2,SearchRobotsTrue,SearchRobotsName,iSearchRobots
SearchRobotsTrue=False
ArrSearchRobots=split(SearchRobots,chr(124))
ForiSearchRobots=0ToUbound(ArrSearchRobots)
ArrSearchRobots2=split(ArrSearchRobots(iSearchRobots),"@")
IfInstr(lcase(SearchRobotsAgent),lcase(ArrSearchRobots2(0)))>0then
SearchRobotsTrue=true:SearchRobotsName=ArrSearchRobots2(1):exitfor
EndIf
Next
IfSearchRobotsTrueandlen(SearchRobotsName)>0then'If it is a crawler, update the crawler information
CheckSearchRobots=True
EndIf
EndFunction
Put the above code in the asp program (see the relevant article for details on how to place it), and then make the following call where you need to call: IfCheckSearchRobots(SearchRobotsAgent)=FalseThen
Response.Write "Not a search engine, can be partially restricted!"' Here you can add a refresh number to determine the specified time. For example, under normal circumstances, the access speed is N times/minute. If this number exceeds (extraordinary access), it can be restricted.