From: Phil Howard (no email)
Date: Tue Jul 02 2002 - 05:03:07 EDT
Just to point out first, unlike the current project, the next mail based
project does have more time available to it and doesn't yet have a fixed
schedule. Thus there is time to write some C code and do things like
adding a new map type.
On Mon, Jul 01, 2002 at 09:42:53PM -0400, wrote:
| On Mon, 1 Jul 2002, Phil Howard wrote:
|
| > 3. NOT one giant map with every user at domain dot That will be too big to
| > maintain. A separate map for each domain is best.
|
| Your intuition is wrong here. Searching multiple (possible hundreds) of
| maps for a single key scales poorly. In fact one single map is best.
| Making a sensible front-end tool that provides per domain views is a
| separate problem (can use say "select ... where ..."). A good IMAP server
| comes with virtual user and virtual domain administration tools.
If each map is for a specific domain, why search them all? Why not just
search the one map for that domain, using only the username part as the
key, and stop there?
I want to be able to be able to link maps together so that example.com
and example.net are the same namespace, without there being duplicates
for every user in a big single giant map. But the biggest reason is
load balancing. I want to be able to easily and quickly move a group of
domains from one server to another one.
As a result of this, these maps would have only the user part as the
key, so even if you wanted to search all the maps for some user at domain,
you can't actually carry out a reasonable search. If you searching
for , and there exists an entry for "foo" in the map
for example.org, is that considered a match?
As for the administration tools in an IMAP server, this project won't
be able to use them, as the administration will be done by a separate
project. Manual administration is intended to be very rare, so any
manual oriented administration tools are not applicable.
| > 4. Mail delivered to ${prefix}/${domain}/${user}/ in maildir format
|
| Any VDA will do this, but for some the mailboxes for each user may need to
| be explicitly mapped to the appropriate path.
If the regular expression map can do substitution, then maybe it can
be used to construct the path from the user at domain key. Otherwise a
new map type could easily solve the problem. One that did read a map
file but applied the substitution if the map file didn't have a match
would make sense to me. Then you could still have a mapping for the
rare cases of an unusual relationship, without having to build and
keep a giant identity map.
| > 5. If ${prefix}/${domain}/${user}/ exists, the address is valid for
| > delivery. If it doesn't, then the address is non-existant.
|
| This cannot be done from "smtpd", it needs to be able to run "chrooted"
| and with low privileges. You need a map to validate the users. Surely
| users need a password entry to read their mail, so a path entry for their
| mailbox is not asking too much. Any multi-column database (MySQL, LDAP,
| userdb, ...) will be able to all the necessary data in one row per user.
Obviously it needs to know in some way. Can the specific chroot path be
configured (even if at patch time)? That can be a path where all the maps
are located and it just reads them. In Linux I can also do bind mounting
to put one directory inside another, and that works fine in chroot (I know
because I have a script to do Slackware installs in chroot that uses this
capability to "pre-mount" the target directory into the chroot as "/mnt"
where the Slackware installer installs to, and it works).
| > 6. If ${prefix}/${domain}/${user}/.forward exists, obey it.
|
| This is available with some VDAs (but not the "virtual" delivery
| agent bundled with Postfix).
Is Postfix's VDA considered useful?
| > 7. One single system user owns everything from ${prefix}/ on down.
|
| $virtual_uid_maps = static:12345
| $virtual_gid_maps = static:12345
|
| > To carry out some of these things, the thought I had was to write a new
| > map type handler which does a lookup for an existing directory or file.
| > The "name" for the map will actually be a complex specification that
| > tells the path, what to return if the file object does not exist, what
| > to return if it is a directory, and what to return if it is a file with
| > a special code to indicate that the file should be read and its content
| > be returned. Then Postfix can think it is a map, but it's just a directory.
| >
|
| This will not work because of "chroot" and security issues. Just drive
| both the VDA and Postfix from a shared "userdb", MySQL or LDAP database.
What directory will it be chroot-ed into?
| > Users will be added/deleted/changed by web CGI programs. Rebuilding a
| > whole map is a bad idea in this case.
| >
|
| Berkeley DB maps support incremental insertion, but this is not
| crashproof unless you turn on logging and transactions, doing Postfix
| compatible locking may also be an issue, also Postfix may need to open
| the map with logging and transactions turned on in order to do crash
| recovery, this is not the case with the current Berkeley DB <-> Postfix
| glue. For live incremental updates your best may be LDAP, MySQL or (via a
| patch) PostSQL.
|
| Note on the other hand that even large Berkeley DB maps can be rebuilt
| from scratch very quickly (5s on my machine for ~16MB on disk .db file
| with ~140000 records). If you build a copy of the map, open it, sync it,
| and rename it, you have a light weight crash-proof (atomic) Berkeley DB
| rebuild. If this happens infrequently (less than once every 5 minutes) you
| will be better off with Berkeley DB than with any complex/slow DBMS than
| supports incremental insertion of records.
|
| Does anyone else have good experiences with other dynamic map types that
| support efficient frequent incremental rebuilds and are robust against
| crashes?
|
| [ Michael Tokarev: how quickly does your tinycdb build large maps ~100000
| rows ~16 MB image ].
With reiserfs, it is now more practical to use the filesystem as a database.
While it has journaling, the key factors here is that it uses hashing to
manage the directories, and has a feature called tail-packing for efficient
space utilization with small files. It's not quite as fast as ext2, but it
sure beats ext2 when you have 3.7 million files in a single directory (which
I have done).
With Linux bind mounts I can present the tree of such a filesystem based db
into any directory which is to serve as the chroot jail for some process.
The only risk is that bind mounts inherit the rw vs ro property of the whole
filesystem it is picked from. Making sure the process has no uid rights to
write on those directories or files should deal with it.
I haven't ruled out using a db file rebuilt periodically from other data.
But I probably would go for cdb in that case.
-- ----------------------------------------------------------------- | Phil Howard - KA9WGN | Dallas | http://linuxhomepage.com/ | | | Texas, USA | http://phil.ipal.org/ | ----------------------------------------------------------------- - To unsubscribe, send mail to with content (not subject): unsubscribe postfix-users
|
|
|