From: Andrew Morgan (no email)
Date: Wed May 02 2007 - 13:23:41 EDT
On Wed, 2 May 2007, Thorsten Büker wrote:
> Dear all,
>
> For quite a few month I ran a backup-scenario set up according to [1]
> including synchronisation of the mailboxes using "rsync -vaR". (In the
> meantime I migrated to the usage of LVM snapshots.)
>
> As restoring a backup became necessary due to filesystem corruption I
> fetched the mailboxes from the r'synched folders as well as the mailbox
> databases and further necessary stuff. The restore went smoothly.
>
> But wondering about the heavily increased amount of partition in use, I
> noticed that my call of rsync lacked of the "--delete" attribute. In
> consequence the restored mailbox directories include not only the messages,
> which should be in the inbox, but all files beeing in the directories during
> the backup cronjobs. (My mailbox sizes ~700 MB instead of ~100 MB.)
>
> With regard to backup performance I'd like to clean up the relevant
> directories by ensuring uniformity of the inbox according to the Cyrus
> mailbox database and the files in the mailbox directory.
>
> In the wiki I found Andrew's collection of scripts [2] and had a try. My
> intention is/was to firstly manually select mailbox directories, which
> contain much too much unneeded files -- mainly mailboxes used via imap. After
> temporarily stopping mail delivery and mailbox access, each chosen mailbox
> might be renamed and its contents copied to a new mailbox of its initial
> name. After starting mail delivery and access again, the mailboxes and
> directories containing wrong fails might be deleted at all.
>
> But, as usual, the idea doesn't work yet. A first try to copy the
> existing mailbox "thorsten" to "thorsten_neu" leads to the following:
>
> ~~~~
>
> mail:/var/spool/cyrus/mail/t/user#
> /usr/local/sbin/cyrus/copy_user_mailbox.pl thorsten thorsten_neu
> Successfully created inbox for thorsten_neu
> Successfully set quota for thorsten_neu to 102400
> "user.thorsten"
> Copying folder user.thorsten to user.thorsten_neu
> "user.thorsten.Archiv"
> Copying folder user.thorsten.Archiv to user.thorsten_neu.Archiv
> "user.thorsten.Drafts"
> Copying folder user.thorsten.Drafts to user.thorsten_neu.Drafts
> "user.thorsten.Mailinglisten"
> Copying folder user.thorsten.Mailinglisten to
> user.thorsten_neu.Mailinglisten
> Error copying messages to user.thorsten_neu.Mailinglisten
> "user.thorsten.Mailinglisten.Announces"
> Not connected at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 104
> Error sending '24 SELECT user.thorsten.Mailinglisten.Announces' to IMAP:
> at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 104
> Copying folder user.thorsten.Mailinglisten.Announces to
> user.thorsten_neu.Mailinglisten.Announces
> Not connected at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 129
> Error sending '25 STATUS user.thorsten.Mailinglisten (MESSAGES)' to
> IMAP: at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 129
> Use of uninitialized value in numeric eq (==) at
> /usr/local/sbin/cyrus/copy_user_mailbox.pl line 129.
>
> [.. various errors at lines 104 and 129 ..]
>
> Not connected at /usr/local/sbin/cyrus/copy_user_mailbox.pl line 152
> Error sending '84 LOGOUT' to IMAP: at
> /usr/local/sbin/cyrus/copy_user_mailbox.pl line 152
>
> ~~~~
>
> Please note, the additional output on the folder name results from
> uncommenting line 92 of the script. The relevant sections look like
> below [6]. Evaluating, why the connection to Cyrus gets lost while looping
> through the folders, a discovered "word too long" in /var/log/mail.log.
>
> Well, in this case (my personal inbox) erasing various mails inside
> "user.thorsten.Mailinglisten" (~2200 mails) helped to make copying work. But
> regarding customer's inboxes this might not be first choice ;-)
>
> In google I found [3], suggesting to recompile imapd. Alas, is there any way
> to modify the script instead of recompiling imapd to permit a larger value?
> Indeed I'd prefer to keep sarge's "2.1.18-1+sarge2".
Ack, someone using my scripts! :)
I bet the problem is that on a large mailbox, $imap->messages will return
a really big list of UIDs (longer than the allowed MAXWORD of 32768). You
may be able to fix this by calling "$imap->Ranges(1);" earlier in the
script. According to the Mail::IMAPClient docs, with that enabled it will
try to generate a condensed list of UIDs when possible.
However, if that still does not work then you may have to add a loop to
the code to operate on "chunks" of UIDs at a time, rather than the whole
list at once. Instead of:
my $uidlist = $imap1->copy($targetf, $imap1->messages);
you'd do something like:
@alluids = $imap1->messages;
foreach $uid (@alluids) {
$uiddone = $imap1->copy($targetf, $uid);
}
There is probably some smarter way to use bigger chunks of uids rather
than 1 at a time.
> Furthermore I tried to rename a mailbox via cyradm in advance, but this leads
> to the following:
>
> ~~~~
>
> mail.myhostname.de> renm user.thorsten user.thorsten_orig
> renamemailbox: Operation is not supported on mailbox
>
> ~~~~
Did you set "allowusermoves: 1" in imapd.conf?
> The amount of results by google is rather poor, honestly spoken one result
> [4]. The presented workaround [5] contains quite a lot of work dealing
> manually with each subfolder. Is there any other way to rename a mailbox?
>
> Or, going even further, is there any more clever attempt than the one above
> to solve the problem? Thanks in advance!
If I understand you correctly, you want to get a list of "valid" UIDs from
the folder listing, and then delete (from the filesystem) anything which
is not in that list of "valid" UIDs?
Or do you intend to just copy the mailbox to a new mailbox, delete the old
mailbox, and copy it back? That sounds pretty reasonable to me. One
problem you might have is the loss of the seen state for each mailbox
because the mailbox unique ID will change with the copy. The unique ID is
stored in the cyrus.header file in each mailbox/folder, so you might be
able to save those and copy them back as you go.
An ugly problem!
Andy
---- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
|
|
|