QUESTIONS ABOUT THIS SITE

Undertoad · 01-06-2007, 10:36 AM

Search engines like Yahoo! and Google "spider" the site, which is to say they read and index everything on the front page and then read and index everything that the front page links to, and then everything that every other page links to, and so on.

In any given week Yahoo! and Google read and index the entire site several times. because they have found that doing so doesn't slow down the connection, and because they aren't smart enough not to read old threads. Little things like the cookies, the date stamps, etc. change and this tricks them into believing the whole site has changed.

A "slurp spider" is one process at Yahoo! reading. When there are a hundred of them, they are reading the entire site again.

01-06-2007, 10:36 AM	#11
Undertoad Radical Centrist Join Date: Jan 2001 Location: Cottage of Prussia Posts: 31,423	Search engines like Yahoo! and Google "spider" the site, which is to say they read and index everything on the front page and then read and index everything that the front page links to, and then everything that every other page links to, and so on. In any given week Yahoo! and Google read and index the entire site several times. because they have found that doing so doesn't slow down the connection, and because they aren't smart enough not to read old threads. Little things like the cookies, the date stamps, etc. change and this tricks them into believing the whole site has changed. A "slurp spider" is one process at Yahoo! reading. When there are a hundred of them, they are reading the entire site again.

Currently Active Users Viewing This Thread: 175 (0 members and 175 guests)