You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
13 lines
174 B
13 lines
174 B
8 months ago
|
User-agent: Nutch
|
||
|
|
Disallow: /
|
||
|
|
User-agent: baiduspider
|
||
|
|
Disallow: /
|
||
|
|
User-agent: naverbot
|
||
|
|
Disallow: /
|
||
|
|
User-agent: yeti
|
||
|
|
Disallow: /
|
||
|
|
User-agent: *
|
||
|
|
Disallow:
|
||
|
|
Disallow: /cgi-bin/
|
||
|
|
|