How does a search engine read the whole web?

A search engine reads the web with a crawler: a program that starts from a list of known pages, downloads each one, notices every link on it, and adds those links to a to-do list of pages to visit next. Repeat that millions of times and a few starting points unfold into a map of the web. The pages it copies are handed to the search engine, which indexes them so that people can search them quickly.

The starting list is called the seeds, and the growing to-do list is the crawl frontier. The hard part is that the web is far too big and changes too fast. A crawler can only download a limited number of pages in a given time, so it has to prioritize, and by the time it returns, a page may have changed or vanished. Even the biggest engines index only part of the web: a 2009 study put it at no more than 40 to 70 percent of the indexable web.

Crawlers are also fast, much faster than human visitors, and that can be a problem for the websites they visit. In 1994, Martijn Koster proposed a simple fix after, as the story goes, a badly behaved crawler written by Charles Stross accidentally knocked over Koster's server. The fix was a plain text file called robots.txt that a site places at its root, listing which parts of the site robots may visit.

Well-behaved robots fetch that file before anything else on the site. But it is a request, not a lock. The standard relies on voluntary compliance, and a bad robot can even read the file as a list of interesting places. Pages blocked in robots.txt can still show up in search results if other pages link to them.