Cyrus IMAP 2.3.9 on Solaris 10 with ZFS and SAN

From: Pascal Gienger (no email)
Date: Thu Sep 20 2007 - 16:09:39 EDT

  • Next message: Vincent Fox: "RE: Cyrus IMAP 2.3.9 on Solaris 10 with ZFS and SAN"

    Just a little note, for those who have perhaps the same problem.

    We saw performance problems after we switched from a Linux installation to
    a Solaris 10 cluster connected to our SAN (using scsi_vhci and 2 Qlogic
    Controllers).

    Problems arose when real load came to the machine, despite having tested it
    with some load simulation scenarios...

    Looking at the attached chart[0], you will notice up to 10 seconds time to
    select a mailbox, far too long.

    The problem was ZFS - ZFS is perfect for rubbish, cheap disks with ugly
    firmwares. So if your storage is "too good" it makes things go bad.

    First: The file prefetch algorithm does not seem to be very good for 20
    million mail files and 300,000 cyrus meta files... ;-)

    So first step was (at 11am in the chart) to disable this prefetching
    routine. [1]

    This cut off 4 seconds from requests, but still 10 second-timeouts were
    seen.

    Next step was to disable the zfs cache flush. As said, ZFS is good for
    rubbish disks, so every 5 seconds it instructs the SATA or SCSI drives to
    flush their internal ram to disk. This is good for cheap disks, but a no-go
    if you have a SAN RAID storage having 2 Gigabytes and more of RAM storage,
    buffered with a battery. In fact, our storage system really flushed all 5
    second its complete RAM cache, you even saw it watching the "blinkenlights".
    Plus, every fsync() call did the same... :(

    There is a trick: You can disable this in ZFS [2]. You see the result at
    4pm (16:00 european notation in the chart). CHILDREN DO NOT TRY THIS AT
    HOME..... Only do this if you don't have any "real" physical disk storages
    attached to your system with zfs pools on them - otherwise you will lose
    data on power outages as the ram cache of your hard disk is not buffered by
    a battery.

    Now the machine runs and handles all mail without noticeable delays.

    [0] http://priscilla.rz.uni-konstanz.de/mailserver/

    [1] in /etc/system: set zfs:zfs_array_rd_sz=0
        on a live system using mdb -kw: zfs_array_rd_sz/Z0x0

    [2] in /etc/system: set zfs:zfs_nocacheflush=1
        on a live system using mdb -kw: zfs_nocacheflush/W0t1 


    ----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
    mailselect.png
  • Next message: Vincent Fox: "RE: Cyrus IMAP 2.3.9 on Solaris 10 with ZFS and SAN"





    Hosted Email Solutions

    Invaluement Anti-Spam DNSBLs



    		Powered By FreeBSD   Powered By FreeBSD