Re: ZMailer "hangs up" on message queues...

[Matthias Urlichs <smurf@noris.de>]

At 12:44 -0700 2001-07-24, Michael Loftis wrote:
>We've been submitting messages to the postoffice/router directory int he
>format <NUM>-<NUM>  where num is a non-zero number, one is the PID the
>other a message sequence count.

You cannot spool mails directly to the router directory. That's what 
the public/ directory is there for.

The algorithm to use is really very simple:
- Create file with unique name (pid+timestamp) in public/.
- Write envelope, header, body.
- Close file.
- Stat() file, retrieving inode number.
- Rename file to ../router/INODENUMBER.

The stat() will _not_ cost you anything because it's already in the 
cache, and even if not, you need it for the next step (the rename) 
anyway.

If you write directly to router/NUMBER, you risk (a) that the number 
isn't unique, or (b) that the router will try to process your message 
while you didn't yet write it and it's still empty, i.e. you have a 
race condition. Ouch.


If you really have performance problems, change file systems. 
Seriously. Old file systems like FFS(*BSD) or ext2(Linux) have a 
linear directory, so the processing of mails is an O(n^2) operation 
WRT the number of stalled mails. This may bite you sooner or later -- 
for instance, after somebody has sent you 200 emails to a domain with 
a nonresponsive DNS server.

By contrast, some newer file systems (reiserfs under Linux) handle 
directory contents as a B-tree, and thus only get slowed down 
O(n*log(n)).

The same thing applies to the scheduler's directories.

Another datapoint to ask yourself is, do you need to be 1000% sure 
that every mail gets delivered, no matter when the system crashes? If 
you mount your file system asynchronously (the default on Linux), you 
do not have that guarantee, but directory operations are ten times 
faster...
-- 
Matthias Urlichs
-
To unsubscribe from this list: send the line "unsubscribe zmailer" in
the body of a message to majordomo@nic.funet.fi