robots.txt
googlebot
yahoobot
msnbot