Re: [aseek-users] Time frame thoughts

From: Searcher (no email)
Date: Tue Oct 29 2002 - 14:06:46 EST

>Mike, I'm reindexing 9890 sites a day (those days that everything
>works allright). The first run took much more, but it seems that yours is
>taking too much time to complete.

Ah, finally, someone who's running a similar task. I can see that the system is
constantly connecting to a good dozen systems or so and pulling down what ever
it's indexing. I know it's alive and I can also see that the drive space is
constantly on the rise so it's not being slow about it that I can tell?

>Now I'm running on a RedHat 7.3, but before I was using FreeBSD on a
>fairly similar configuration and it was many times slower than now.

I'm running on RedHat 7.2 myself, with the dual 350Mhz CPU's and 8GB's of memory
as I mentioned. The system seems to be handing everything ok, the load is not
very high or anything but it's been running non stop for over two weeks now.

>Some times seems that stoping and restarting the indexer is faster than
>simply wait the end of the run.

I think I started the process from the console and left it there. I've been
monitoring the system from an SSH login. My point is, I can't quit the session
unless I hit CTL-C or is there some other way, from another login?

>The disk space depends on the content of the pages, but on my case
>it is using 1GB for the cache and 2GB for the database.

I know it's into the millions of pages because there is constantly a queue of
between 1.2 and 1.5 million pages. It's already into 45GB's of drive space usage
now. I'm indexing 10028 sites, not pages but I notice you also said you are
indexing 9000+ *sites*. I had assumed that the drive usage would be pretty normal
for the number of sites that I'm indexing.

>By the way, I use the following mysql command to report top 10 sites:
>sometimes there's a huge site that shouldn't be indexed, which is
>consuming lots of resources.

I'll check this out.

>Hope some of this may help you.

Invaluable, thanks very much for your help.


Hosted Email Solutions

Invaluement Anti-Spam DNSBLs

Powered By FreeBSD   Powered By FreeBSD