From: Kir Kolyshkin (no email)
Date: Sat Dec 28 2002 - 09:08:12 EST
Paul Stewart wrote:
> When we restart indexer (runs 8 hours daily) it always reloads the
> robots.txt files from what appears to be every site when it first starts
> up...
That is the right behaviour, ASPseek never trust "old" robots.txt and
always reloads it when starting a new indexing session. In other words,
"Period" directive doesn't apply to robots.txt.
--
== kir_at_asplinux.ru == 7551596_at_ICQ == 6722750_at_sms.beemail.ru ==
Dream like you'll live forever...Love like you've never been hurt...
Work like you don't need the money...and Dance like nobody is watching!
-- Satchel Paige
|
|
|