Web Crawlers, Robots.txt, and SEO

By Jason Eichner

There are many different sorts of robots browsing the web, for as many different reasons. Some are harmless, some are extremely important to your business, and some can be harmful. But what are they, and how can you influence them?

Web crawlers, often called robots, are programs that browse your internet, fulfilling a specific purpose. The most famous is likely the Googlebot: your best friend if you care about your Google rankings. Googlebot browses (crawls) your websites to check that it is safe and to gather data to fill your entry in search results. Other less reputable robots will scan your website looking for e-mail addresses. It then appends those e-mails to a spam list. These robots can also find “mailto:” links, so any e-mail address on a website is at risk of getting added to spam lists.

What can you do with these robots? A lot, and very little.