Spider (software)

A spider is usually a component of search engine software that resides on the search engine's own servers. Spiders are usually programmed to interpret the RAW code pulled from a URL on the internet. Depending on the search engine or project defines what specifications the spiders are usually built to, however most are built to the main specifications set by WC3

The spiders are usually made to interpret and follow links within the RAW code, each link then has it's RAW code pulled to be interpreted. The Spider accessing each of these links is why it's called "Spidering", since the Spider attempts to weave a Web from the links.

Most websites are spidered and a cache(copy) taken by the search engines servers. Those cache's can then be used to better define the website that was visited by allowing greater analysis of the websites content.

Spiders will attempt to return to URL's at a future date to update the cache with any new data it can retrieve so as to not let the cache go stale.

Other names that are sometimes used are:
Agent
Robot(Bot)