Re: improving concurrency/performance (fwd)

From: Andrew McNamara (no email)
Date: Tue Nov 08 2005 - 22:48:20 EST

  • Next message: Sebastian Haas: "Data format error: : Mailbox doesn't exist"

    I forwarded John's message to Andrew Morton, linux kernel maintainer, and
    this is his reply (it was cc'ed to the list, but, not being a subscriber,
    I presume it bounced):

    ------- Forwarded Message

    Date: Tue, 08 Nov 2005 15:21:31 -0800
    From: Andrew Morton <>
    To: Andrew McNamara <>
    cc: "John Madden" <>,
             
    Subject: Re: improving concurrency/performance (fwd)

    Andrew McNamara <> wrote:
    >
    > This guy is having a problem with cyrus-imap and ext3 - when multiple
    > processes are attempting to write to the one filesystem (but not the one
    > file), performance drops to next to nothing when only five processes are
    > writing. An strace shows most of the time is being spent in fdatasync
    > and fsync.
    >
    > ...

    Yes, on ext3, an fsync() syncs the entire filesystem. It has to, because
    all the metadata for each file is shared - it's just a string of
    journallable blocks. Similar story with the data, in ordered mode.

    So effectively, fsync()ing five files one time each is performing 25 fsync()s.

    One fix (which makes the application specific to ext3 in ordered-data or
    journalled-data mode) is to perform a single fsync(), with the
    understanding that this has the side-effect of fsyncing all the other
    files. That's an ugly solution and is rather hard to do if the workload
    consists of five separate processes!

    So I'd recommending mounting the filesystem with the `-o data=writeback'
    mode. This way, each fsync(fd) will sync fd's data only. That's much
    better than the default data-ordered mode, wherein a single fsync() will
    sync all the other file's data too.

    In data=writeback mode it is still the case that fsync(fd) will sync the
    other file's metadata, but that's a single linear write to the journal and
    the additional cost should be low.

    Bottom line: please try data=writeback, let me know.

    > ------- Forwarded Message
    >
    > Date: Tue, 08 Nov 2005 09:25:54 -0500
    > From: "John Madden" <>
    > To: Jure =?iso-8859-1?Q?Pe=E8ar?= <>
    > cc:
    > Subject: Re: improving concurrency/performance
    >
    > > As expected, these are from locking operations. 0x8 is file descriptor,
    > > which, if I read lsof output correctly, points to config/socket/imap-0.lock
    > > (what would that be?) and 0x7 is F_SETLKW which reads as "set lock or wait
    > > for it to be released" in the manual page.
    >
    > Yup, that's exactly the sort of thing I was suspecting -- the performance I
    > was seeing just didn't make sense.
    >
    > imap-0.lock is in /var/imap/socket for me. I believe it's one of the lock
    > file s created when cyrus is started, so it wouldn't make any sense for
    > imapd to ever be spinning on it.
    >
    > The delays I was seeing ocurred when multiple imapd's were writing to the
    > spool at the same time. I do see a lot of this though:
    >
    > fcntl(6, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
    >
    > It looks like the lock to open a file in the target mailbox. But again,
    > very l ow actual throughput and still little or no iowait. However, adding
    > a -c to the strace, the top three syscalls are:
    >
    > % time seconds usecs/call calls errors syscall
    > - ------ ----------- ----------- --------- --------- ----------------
    > 52.68 0.514720 1243 414 fdatasync
    > 29.87 0.291830 846 345 fsync
    > 4.19 0.040898 27 1519 fcntl
    >
    > Makes me wonder why the fsync's are taking so long since the disk is
    > performing so well. Anyone know if that's actually typical?
    >
    > --
    > John Madden
    > UNIX Systems Engineer
    > Ivy Tech Community College of Indiana
    >
    >
    > - ----
    > Cyrus Home Page: http://asg.web.cmu.edu/cyrus
    > Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
    > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
    >
    > ------- End of Forwarded Message

    ------- End of Forwarded Message

    ----
    Cyrus Home Page: http://asg.web.cmu.edu/cyrus
    Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
    List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
    

  • Next message: Sebastian Haas: "Data format error: : Mailbox doesn't exist"





    Hosted Email Solutions

    Invaluement Anti-Spam DNSBLs



    Powered By FreeBSD   Powered By FreeBSD