From: Searcher (no email)
Date: Tue Oct 29 2002 - 14:06:46 EST
>Mike, I'm reindexing 9890 sites a day (those days that everything
>works allright). The first run took much more, but it seems that yours is
>taking too much time to complete.
Ah, finally, someone who's running a similar task. I can see that the system is
constantly connecting to a good dozen systems or so and pulling down what ever
it's indexing. I know it's alive and I can also see that the drive space is
constantly on the rise so it's not being slow about it that I can tell?
>Now I'm running on a RedHat 7.3, but before I was using FreeBSD on a
>fairly similar configuration and it was many times slower than now.
I'm running on RedHat 7.2 myself, with the dual 350Mhz CPU's and 8GB's of memory
as I mentioned. The system seems to be handing everything ok, the load is not
very high or anything but it's been running non stop for over two weeks now.
>Some times seems that stoping and restarting the indexer is faster than
>simply wait the end of the run.
I think I started the process from the console and left it there. I've been
monitoring the system from an SSH login. My point is, I can't quit the session
unless I hit CTL-C or is there some other way, from another login?
>The disk space depends on the content of the pages, but on my case
>it is using 1GB for the cache and 2GB for the database.
I know it's into the millions of pages because there is constantly a queue of
between 1.2 and 1.5 million pages. It's already into 45GB's of drive space usage
now. I'm indexing 10028 sites, not pages but I notice you also said you are
indexing 9000+ *sites*. I had assumed that the drive usage would be pretty normal
for the number of sites that I'm indexing.
>By the way, I use the following mysql command to report top 10 sites:
>sometimes there's a huge site that shouldn't be indexed, which is
>consuming lots of resources.
I'll check this out.
>Hope some of this may help you.
Invaluable, thanks very much for your help.