my cyrus went crazy - "PANIC: DB_RUNRECOVERY: Fatal error" and others

• Previous message: Edward Rudd: "Re: Can't use cyrus deliver with mboxes containing a '@'"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] [ attachment ]

Hello,

my environment:

Gentoo
net-mail/cyrus-imapd 2.2.12
sys-libs/db 4.1.25_p1-r4 1.85-r2

today in the morning I have noticed that postfix cannot deliver mail to
mailboxes. A fast look at /var/log/everything/* showed that Cyrus went
crazy with messages like these:

Mar 19 12:29:33 [lmtpunix] DBERROR db4: environment not yet opened
Mar 19 12:29:33 [lmtpunix] DBERROR: opening /var/imap/deliver.db:
Invalid argument
Mar 19 12:29:33 [lmtpunix] DBERROR: opening /var/imap/deliver.db:
cyrusdb error
Mar 19 12:29:33 [lmtpunix] FATAL: lmtpd: unable to init duplicate
delivery database
Mar 19 12:29:33 [master] service lmtpunix pid 30445 in READY state:
terminated abnormally

unfortunately logs have already rotated so many times, that could not
find information about the beginning of the disaster.
Anyway - /etc/init.d/cyrus restart fortunately helped and until now
everything was fine.

But then... cyrus went down again with logs like this:

Mar 19 22:27:28 [postfix/pipe] 86247530B5: to=<>,
orig_to=<>, relay=cyrus, delay=1, status=sent
(pyza.from.cx)
Mar 19 22:27:29 [postfix/pipe] 86247530B5: to=<>,
orig_to=<>, relay=cyrus, delay=2, status=sent
(pyza.from.cx)
Mar 19 22:27:29 [postfix/qmgr] 86247530B5: removed

(till now it was ok)

Mar 19 22:30:01 [/usr/sbin/cron] (root) CMD (test -x /usr/sbin/run-crons
&& /usr/sbin/run-crons )
Mar 19 22:32:15 [ctl_cyrusdb] checkpointing cyrus databases
Mar 19 22:32:17 [ctl_cyrusdb] DBERROR db4: DB_LOGC->get: log record
checksum mismatch
Mar 19 22:32:17 [ctl_cyrusdb] DBERROR db4: DB_LOGC->get: catastrophic
recovery may be required
Mar 19 22:32:17 [ctl_cyrusdb] DBERROR db4: PANIC: DB_RUNRECOVERY: Fatal
error, run database recovery
Mar 19 22:32:17 [ctl_cyrusdb] DBERROR: critical database situation
Mar 19 22:33:20 [imap] DBERROR db4: fatal region error detected; run
recovery
Mar 19 22:33:20 [imap] DBERROR: dbenv->open '/var/imap/db' failed:
DB_RUNRECOVERY: Fatal error, run database recovery
Mar 19 22:33:20 [imap] DBERROR: init() on berkeley
Mar 19 22:33:20 [imap] sql_select option missing
Mar 19 22:33:20 [imap] auxpropfunc error no mechanism available_
Mar 19 22:33:20 [imap] login: localhost [127.0.0.1] marcin plaintext
User logged in
Mar 19 22:38:21 [imap] DBERROR db4: fatal region error detected; run
recovery

and then:

Mar 19 22:51:05 [imaps] idle for too long, closing connection
                 - Last output repeated twice -
Mar 19 22:52:05 [imaps] DBERROR db4: fatal region error detected; run
recovery
Mar 19 22:52:05 [imaps] DBERROR: error closing: DB_RUNRECOVERY: Fatal
error, run database recovery
Mar 19 22:52:05 [imaps] DBERROR: error closing tlsdb: cyrusdb error
Mar 19 22:52:05 [imaps] DBERROR db4: fatal region error detected; run
recovery
Mar 19 22:52:05 [imaps] DBERROR: error exiting application:
DB_RUNRECOVERY: Fatal error, run database recovery
Mar 19 22:52:13 [imaps] DBERROR db4: fatal region error detected; run
recovery
Mar 19 22:52:13 [imaps] DBERROR: error closing: DB_RUNRECOVERY: Fatal
error, run database recovery
Mar 19 22:52:13 [imaps] DBERROR: error closing tlsdb: cyrusdb error
Mar 19 22:52:13 [imaps] DBERROR db4: fatal region error detected; run
recovery
Mar 19 22:52:13 [imaps] DBERROR: error exiting application:
DB_RUNRECOVERY: Fatal error, run database recovery
(I use imaps for remote mail clients)

I tried to restart cyrus, but got only:

Mar 19 22:59:57 [master] exiting on SIGTERM/SIGINT
Mar 19 22:59:58 [master] setrlimit: Unable to set file descriptors limit
to -1: Operation not permitted
Mar 19 22:59:58 [master] retrying with 1024 (current max)
Mar 19 22:59:58 [master] process started
Mar 19 22:59:58 [ctl_cyrusdb] DBERROR db4: DB_ENV->log_flush: LSN past
current end-of-log
Mar 19 22:59:58 [ctl_cyrusdb] DBERROR db4: /var/imap/deliver.db: unable
to flush page: 0
Mar 19 22:59:58 [ctl_cyrusdb] DBERROR db4: txn_checkpoint: failed to
flush the buffer cache Invalid argument
Mar 19 22:59:58 [ctl_cyrusdb] DBERROR db4: PANIC: Invalid argument
Mar 19 22:59:58 [ctl_cyrusdb] DBERROR: critical database situation
Mar 19 22:59:58 [master] process 3885 exited, status 75_
Mar 19 22:59:59 [master] ready for work
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR db4: fatal region error detected;
run recovery
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR: dbenv->open '/var/imap/db'
failed: DB_RUNRECOVERY: Fatal error, run database recovery
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR: init() on berkeley
Mar 19 22:59:59 [ctl_cyrusdb] checkpointing cyrus databases
Mar 19 22:59:59 [tls_prune] DBERROR db4: fatal region error detected;
run recovery
Mar 19 22:59:59 [tls_prune] DBERROR: dbenv->open '/var/imap/db' failed:
DB_RUNRECOVERY: Fatal error, run database recovery
Mar 19 22:59:59 [tls_prune] DBERROR: init() on berkeley
Mar 19 22:59:59 [tls_prune] DBERROR db4: environment not yet opened
Mar 19 22:59:59 [tls_prune] DBERROR: opening /var/imap/tls_sessions.db:
Invalid argument
Mar 19 22:59:59 [tls_prune] DBERROR: opening /var/imap/tls_sessions.db:
cyrusdb error
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR db4: txn_checkpoint interface
requires an environment configured for the tran
saction subsystem
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR: couldn't checkpoint: Invalid argument
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR: sync /var/imap/db: cyrusdb error
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR db4: DB_ENV->log_archive interface
requires an environment configured for the logging subsystem
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR: error listing log files: Invalid
argument
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR: archive /var/imap/db: cyrusdb error
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR db4: txn_checkpoint interface
requires an environment configured for the tran
saction subsystem
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR: couldn't checkpoint: Invalid argument
Mar 19 22:59:59 [ctl_cyrusdb] DBERROR: sync /var/imap/db: cyrusdb error
[...]

I tried ctl_cyrusdb -r, with no effect :

Mar 19 23:48:16 [ctl_cyrusdb] DBERROR db4: DB_ENV->log_flush: LSN past
current end-of-log
Mar 19 23:48:16 [ctl_cyrusdb] DBERROR db4: /var/imap/deliver.db: unable
to flush page: 0
Mar 19 23:48:16 [ctl_cyrusdb] DBERROR db4: txn_checkpoint: failed to
flush the buffer cache Invalid argument
Mar 19 23:48:16 [ctl_cyrusdb] DBERROR db4: PANIC: Invalid argument
Mar 19 23:48:16 [ctl_cyrusdb] DBERROR: critical database situation

Please help!
_________________
Rgds, Grzes

---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html

• Previous message: Edward Rudd: "Re: Can't use cyrus deliver with mboxes containing a '@'"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] [ attachment ]