nginx防止爬虫

时间:2015-10-20 15:54 来源:linux.it.net.cn 作者:IT

方法一：修改nginx.conf

server {

listen 80;

#添加如下内容即可防止爬虫

if ($http_user_agent ~* "qihoobot|Baiduspider|Googlebot|Googlebot-Mobile|Googlebot-Image|Mediapartners-Google|Adsbot-Google|Feedfetcher-Google|Yahoo! Slurp|Yahoo! Slurp China|YoudaoBot|Sosospider|Sogou spider|Sogou web spider|MSNBot|ia_archiver|Tomato Bot")
{
return 403;

}

方法2：网站更目录下增加Robots.txt

http://tool.it.net.cn/robots/

(责任编辑：IT)